# How the engine works

> The capture loop, timestamps, dropped-frame accounting and on-device processing behind every Rayform run.

Source: https://rayform.world/docs/engine
Last updated: 2026-10-19
Status: Rayform is in private preview. The edge engine is in development with preview teams, and commands, formats and APIs described here may change before general availability.

Most capture scripts lose data in ways that never raise an error: a garbage-collection pause stalls a camera callback, a USB bus saturates, a queue grows until frames are dropped, or a timestamp is taken when a message arrives instead of when the sensor exposed it. The model then trains on observations and actions that are a few frames apart. Rayform's capture path is designed to avoid those failure modes and to log the ones it cannot avoid.

## The capture loop

| Stage | What happens |
| --- | --- |
| Buffers | Frame and sample buffers are allocated at start-up, sized from the rig file, to keep memory allocation out of the capture loop |
| Readers | One native reader thread per device, pinned to a core where the platform allows, copies data straight into the arena |
| Timestamps | Each sample keeps its device timestamp (exposure or measurement time) when the hardware provides one, plus the host receive time |
| Accounting | Sequence numbers from each device are checked, so frames the hardware or bus drops are logged with their time |
| Writer | A separate writer drains the arenas to disk in large sequential blocks, so slow storage shows up as back-pressure in the log instead of lost frames |

Throughput depends on your cameras, bus and storage; `rayform capture --bench` measures it on your hardware before you record.

## Time

Every stream is mapped onto one run clock:

1. **Hardware sync** (preferred): cameras triggered from a common signal, or devices sharing PTP. Offsets are measured, not estimated.
2. **Device clocks**: each device's own timestamps are mapped to the host clock, and drift is estimated continuously.
3. **Host time only**: used when a device exposes nothing else; the run is marked accordingly.

`rayform check` reports the method used per stream and the largest residual skew between streams. See [Frames, units and time](https://rayform.world/docs/reference/conventions) for the conventions written to every file.

## Space

Each camera's intrinsics and its pose relative to the robot's base frame come from the [calibration](https://rayform.world/docs/engine/calibration) in use for the run. Depth is reprojected into the base frame on request, and the residual error of the calibration is stored with every run.

## On the device, not in the cloud

Capture, alignment, checks and export all run on your own hardware: the robot's computer, a workstation next to the rig, or an Apple Silicon laptop. Raw streams never leave your network unless you copy them. [Rayform Cloud](https://rayform.world/docs/guides/uploading) is optional and receives only the structured datasets you choose to sync.

## What a run contains

```text
runs/2026-10-19_0912/
  run.json          # rig, task, operator, software versions, clock method per stream
  calibration.json  # intrinsics and extrinsics used for this run
  streams/          # raw camera and joint data, unchanged
  episodes.json     # episode boundaries, discarded attempts, quality scores
  capture.log       # back-pressure, drops and warnings with timestamps
```