ToborVerse Robot Ingestion API Reference

This document explains how to programmatically upload raw robot datasets (episodes) directly from a physical robot to the ToborVerse ingestion backend.

All ingestion uploads use an optimized S3 Multipart Upload flow. Rather than streaming gigabytes of data through the API server, the robot asks the server for direct-to-S3 upload links, uploads the data chunk-by-chunk directly to AWS S3, and then notifies the API server when done.

---

1. Credentials & Authentication

Every robot provisioned on the platform has two credentials: * robot_serial: The robot serial number used to identify the robot. * api_key: A cryptographically random key prefixing with tv_robot_.

These credentials must be sent with every API request using custom HTTP headers:

X-Robot-serial: <your_robot_serial>
X-Robot-API-Key: <your_api_key>

Note: Do not pass the API key or serial number in request bodies. Keep them in the headers to avoid log leakage.

---

2. Ingestion Upload Workflow

The ingestion pipeline consists of 4 distinct steps:

sequenceDiagram
    participant Robot as Robot Client
    participant API as ToborVerse API
    participant S3 as AWS S3 Storage

    Note over Robot: Calculates file size & MD5 ETag
    Robot->>API: 1. POST /api/episodes/start (headers + metadata)
    API-->>Robot: Returns episode_id, upload_id, and presigned URLs

    loop For each chunk
        Robot->>S3: 2. PUT <Presigned URL> (raw bytes)
        S3-->>Robot: Returns HTTP 200 + ETag Header
        Robot->>API: 3. POST /api/episodes/:id/parts (ETag + part_number)
        API-->>Robot: Acknowledges part
    end

    Robot->>API: 4. POST /api/episodes/:id/complete (upload_id)
    API-->>Robot: Finalized & marked "uploaded"

Step 1: Initiate Ingestion Upload

Before transmitting any bytes, notify the server about the file you want to upload. The backend uses the file size to calculate how many chunks are needed, and pre-generates presigned AWS S3 URLs for each chunk.

• Endpoint: POST /api/episodes/start
• Headers:
• Content-Type: application/json
• X-Robot-serial: <robot_serial>
• X-Robot-API-Key: <api_key>

• Body:

  {
    "file_size_bytes": 15695235,
    "zstd_md5": "4a76f97ee817de979de5c156c69dc34f-3",
    "task": "autonomous hallway collection"
  }
  

  Note on zstd_md5: This is the expected ETag of the completed S3 multipart upload. See the JS example code below on how this checksum is calculated.

• Response (201 Created):

  {
    "episode_id": "01a043c5-4ab8-7ed3-b24c-3d1bd5400ae4",
    "upload_id": "mbDKb1FLzAem...",
    "chunk_size_bytes": 5242880,
    "total_parts": 3,
    "parts": [
      {
        "part_number": 1,
        "url": "https://app-toborverse-2026.s3.us-west-1.amazonaws.com/orgs/..."
      },
      {
        "part_number": 2,
        "url": "https://app-toborverse-2026.s3.us-west-1.amazonaws.com/orgs/..."
      },
      {
        "part_number": 3,
        "url": "https://app-toborverse-2026.s3.us-west-1.amazonaws.com/orgs/..."
      }
    ]
  }
  

---

Step 2: Upload Chunks Directly to S3

For each object inside the returned parts array, slice your local file according to the chunk_size_bytes (usually 5MB) and upload it directly.

• Method: PUT
• URL: Use the specific presigned url field from the part object.
• Body: Raw binary slice of the file.
• Response: Extract the ETag header from the response. Ensure you strip out any double quotes around the tag (e.g. convert "abcdef..." to abcdef...).

---

Step 3: Acknowledge Part Upload

After uploading a chunk to S3, notify the ToborVerse API server immediately. This records the chunk's ETag in the database to allow the backend to stitch the file together later.

• Endpoint: POST /api/episodes/:episode_id/parts
• Headers:
• Content-Type: application/json
• X-Robot-serial: <robot_serial>
• X-Robot-API-Key: <api_key>
• Body:

  {
    "upload_id": "mbDKb1FLzAem...",
    "part_number": 1,
    "etag": "0f887635613a0f3409be92e054806b29"
  }
  

• Response (201 Created):

  {
    "part": {
      "id": "54",
      "org_id": "01a03493-6d3b-7993-b252-c7be819e118c",
      "episode_id": "01a043c5-4ab8-7ed3-b24c-3d1bd5400ae4",
      "part_number": 1,
      "etag": "0f887635613a0f3409be92e054806b29",
      "uploaded_at": "2026-08-27T15:10:26.543Z"
  }
  

---

Step 4: Finalize Ingestion Upload

Once all chunks are uploaded and verified by the API, tell the server to complete the ingestion. The backend will ask S3 to assemble the file, verify the final file integrity check, and trigger downstream extraction.

• Endpoint: POST /api/episodes/:episode_id/complete
• Headers:
• Content-Type: application/json
• X-Robot-serial: <robot_serial>
• X-Robot-API-Key: <api_key>
• Body:

  {
    "upload_id": "mbDKb1FLzAem..."
  }
  

• Response (200 OK):

  {
    "episode_id": "01a043c5-4ab8-7ed3-b24c-3d1bd5400ae4",
    "status": "uploaded",
    "mp4_s3_path": "orgs/01a03493-6d3b-7993-b252-c7be819e118c/episodes/01a043c5-4ab8-7ed3-b24c-3d1bd5400ae4/raw.tar.zst"
  }
  

---

3. Reference Javascript Implementation

The following is a clean node.js implementation to demonstrate this flow. Save this script on the robot, make sure you have the source dataset (pick_cube_xr_lerobot_dataset.tar.zst), and run it.

const fs = require('fs');
const { createHash } = require('crypto');

// Configuration
const API_URL = 'http://localhost:5000/api'; // Or your staging endpoint
const ROBOT_SERIAL = '786543875902356723';
const API_KEY = 'tv_robot_ynOrjS1axBxELzpViWevNuxUIP3lj1sTWeyGalch_hI';

const FILE_PATH = './pick_cube_xr_lerobot_dataset.tar.zst';
const CHUNK_SIZE_BYTES = 5242880; // 5MB standard

// Helper to pre-calculate S3 Multipart ETag locally
function calculateS3MultipartETag(filePath, chunkSize) {
  const fileBuffer = fs.readFileSync(filePath);
  const totalParts = Math.ceil(fileBuffer.length / chunkSize);

  if (totalParts <= 1) {
    return createHash('md5').update(fileBuffer).digest('hex');
  }

  const partHashes = [];
  for (let i = 0; i < totalParts; i++) {
    const start = i * chunkSize;
    const end = Math.min(start + chunkSize, fileBuffer.length);
    const chunk = fileBuffer.subarray(start, end);
    partHashes.push(createHash('md5').update(chunk).digest());
  }

  const combinedHash = createHash('md5').update(Buffer.concat(partHashes)).digest('hex');
  return `${combinedHash}-${totalParts}`;
}

async function uploadDataset() {
  if (!fs.existsSync(FILE_PATH)) {
    console.error(`❌ File not found: ${FILE_PATH}`);
    process.exit(1);
  }

  const stat = fs.statSync(FILE_PATH);
  const fileSizeBytes = stat.size;

  console.log(`📂 Processing file: ${FILE_PATH} (${fileSizeBytes} bytes)`);
  const zstdMd5 = calculateS3MultipartETag(FILE_PATH, CHUNK_SIZE_BYTES);
  console.log(`🏷️ Local MD5 ETag calculated: ${zstdMd5}`);

  const authHeaders = {
    'Content-Type': 'application/json',
    'X-Robot-serial': ROBOT_SERIAL,
    'X-Robot-API-Key': API_KEY
  };

  // Step 1: Initiate upload on backend
  console.log('\n[Step 1] Initiating upload request...');
  const startRes = await fetch(`${API_URL}/episodes/start`, {
    method: 'POST',
    headers: authHeaders,
    body: JSON.stringify({
      file_size_bytes: fileSizeBytes,
      zstd_md5: zstdMd5
    })
  });

  if (!startRes.ok) {
    throw new Error(`Failed to initiate upload: ${await startRes.text()}`);
  }

  const { episode_id, upload_id, parts } = await startRes.json();
  console.log(`✅ Upload initiated. Episode ID: ${episode_id}`);

  // Step 2 & 3: Upload each part & notify API
  const fileDescriptor = fs.openSync(FILE_PATH, 'r');

  for (const part of parts) {
    const { part_number, url } = part;
    const start = (part_number - 1) * CHUNK_SIZE_BYTES;
    const end = Math.min(start + CHUNK_SIZE_BYTES, fileSizeBytes);
    const length = end - start;

    const buffer = Buffer.alloc(length);
    fs.readSync(fileDescriptor, buffer, 0, length, start);

    console.log(`\n[Step 2] Uploading part ${part_number}/${parts.length} to S3 (${length} bytes)...`);
    const s3Res = await fetch(url, {
      method: 'PUT',
      body: buffer
    });

    if (!s3Res.ok) {
      throw new Error(`S3 upload failed for part ${part_number}: ${await s3Res.text()}`);
    }

    const etag = s3Res.headers.get('ETag').replace(/"/g, '');
    console.log(`✅ Uploaded part ${part_number} (ETag: ${etag})`);

    console.log(`[Step 3] Notifying API for part ${part_number}...`);
    const notifyRes = await fetch(`${API_URL}/episodes/${episode_id}/parts`, {
      method: 'POST',
      headers: authHeaders,
      body: JSON.stringify({
        upload_id,
        part_number,
        etag
      })
    });

    if (!notifyRes.ok) {
      throw new Error(`Failed to notify API for part ${part_number}: ${await notifyRes.text()}`);
    }
    console.log(`✅ Part ${part_number} acknowledged by backend.`);
  }

  fs.closeSync(fileDescriptor);

  // Step 4: Finalize upload
  console.log('\n[Step 4] Finalizing multipart upload...');
  const completeRes = await fetch(`${API_URL}/episodes/${episode_id}/complete`, {
    method: 'POST',
    headers: authHeaders,
    body: JSON.stringify({ upload_id })
  });

  if (!completeRes.ok) {
    throw new Error(`Failed to complete upload: ${await completeRes.text()}`);
  }

  const finalResult = await completeRes.json();
  console.log('\n🎉 Upload completed successfully!');
  console.log(`Episode S3 path: ${finalResult.mp4_s3_path}`);
}

uploadDataset().catch(err => {
  console.error('\n❌ Ingestion failed:', err.message);
  process.exit(1);
});