Host protocol framing

The on-device protocol task (ports/b2op/app/Protocol.cpp) speaks a compact binary protocol with the web IDE / host tool over USB CDC and BLE NUS.

Wire format

Every frame is:

[0x7E][LEN:varint][SEQ:u8][FLAGS:u8][KIND:u8][PAYLOAD:LEN bytes]

0x7E

Magic byte; also used for re-sync after corruption. Any byte that is not part of an in-progress frame and equals 0x7E starts a new frame.

LEN — LEB128 varint

Payload byte count, little-endian, up to 5 bytes. Bit 7 of each byte is the continuation flag — if set, another byte follows. The 5th byte, if reached, is always the last regardless of bit 7.

Range	Bytes	Format
0–127	1	`[0xxxxxxx]`
128–16 383	2	`[1xxxxxxx][0xxxxxxx]`
16 384–2 097 151	3	`[1xxxxxxx][1xxxxxxx][0xxxxxxx]`
2 097 152–268 435 455	4	`[1xxxxxxx][1xxxxxxx][1xxxxxxx][0xxxxxxx]`
268 435 456–34 359 738 367	5	`[1xxxxxxx][1xxxxxxx][1xxxxxxx][1xxxxxxx][xxxxxxxx]`

SEQ — sequence number

Host → Device: monotonically increasing 0x00–0xFF, wraps.
Device → Host: 0xFF = unsolicited (no matching request); otherwise echoes the host’s SEQ so the host can match the reply to its waiter.

FLAGS

bit 0 NO_ACK — device must not send a response frame. SEQ is still a real counter and is used for in-order delivery when frames arrive out of order over BLE writeValueWithoutResponse.
Bits 1–7: reserved, must be zero.

KIND

One-byte frame type; see the table below.

Per-frame overhead: 5–9 bytes (1 magic + 1–5 varint + 1 SEQ + 1 FLAGS + 1 KIND).

Kind table

Hex	Name	Direction	Payload	Notes
`0x01`	HELLO_REQ	host → device	empty	Request device to resend board info
`0x10`	PING	host → device	empty	Liveness check
`0x11`	MTU_REQ	host → device	empty	Query BLE ATT MTU
`0x12`	RUN	host → device	`path` (UTF-8)	Run a Python file
`0x13`	STOP	host → device	empty	Interrupt running program; `FLAGS=NO_ACK` for fire-and-forget
`0x14`	UPLOAD	host → device	`[path_len:u8][path bytes][file_size:u32 BE]`	File data follows as DATA chunks; see UPLOAD flow
`0x15`	READ	host → device	`path` (UTF-8)	Reply: ACK `[size:u32 BE]`, DATA chunks, OK; see READ flow
`0x16`	LS	host → device	`path` (UTF-8)	Reply: DATA with packed dir entries
`0x17`	MV	host → device	`[src_len:u8][src bytes][dst_len:u8][dst bytes]`	Rename/move; reply: OK or ERR
`0x18`	CP	host → device	`[src_len:u8][src bytes][dst_len:u8][dst bytes]`	Copy; reply: ACK, then OK or ERR; abortable; see CP flow
`0x19`	RM	host → device	`path` (UTF-8)	Delete file; reply: OK or ERR
`0x1A`	MKDIR	host → device	`path` (UTF-8, absolute)	Create directory and parents (`mkdir -p`); reply: OK or ERR; see MKDIR
`0x1B`	ABORT	host → device	`[target_seq:u8]` or empty	Cancel an UPLOAD / READ / CP (empty = all); see ABORT
`0x20`	HELLO	device → host	`[proto_ver:u8][board_name\0][board_ver\0][fw_ver\0]`	SEQ = `0xFF`; sent proactively at boot
`0x21`	OK	device → host	optional message (UTF-8 or structured; see command)	Echoes request SEQ
`0x22`	ERR	device → host	error message (UTF-8)	Echoes request SEQ
`0x23`	ACK	device → host	optional message (UTF-8)	Echoes SEQ; resets host idle timer
`0x24`	PROGRESS	device → host	`[sent:u32 BE][total:u32 BE]`	Echoes SEQ; resets host idle timer
`0x25`	DATA	both	binary payload (command-specific); file chunks are `[offset:u32 BE][bytes]`	Echoes request SEQ; host → device only as UPLOAD chunks
`0x30`	STDOUT	device → host	UTF-8 text	SEQ = `0xFF`; unsolicited
`0x31`	STDERR	device → host	UTF-8 text	SEQ = `0xFF`; unsolicited
`0x32`	PROG_END	device → host	`[ok:u8][message UTF-8]`	SEQ = `0xFF`; see below
`0x33`	PROG_START	device → host	`path` (UTF-8)	SEQ = `0xFF`; see below
`0x34`	EXEC	host → device	Python source (UTF-8)	Execute in REPL context; reply OK or ERR; stdout as STDOUT frames
`0x35`	CMD	host → device	`[CmdId:u8][args…]`	Binary hardware command; reply OK or DATA; see CMD table below

Protocol versioning

The proto_ver byte in every HELLO frame encodes the active protocol version. The host maps this byte to a semver name and refuses to operate if the version is unrecognised (displaying an “update the app” message).

Byte	Semver name
`2`	v1.0.0
`3`	v2.0.0 — UPLOAD data in DATA chunks, READ in DATA chunks, ABORT, CP ACK

Only the device advertises its protocol version; the host never sends a version in HELLO_REQ. The HELLO_REQ payload is always empty.

Structured payloads

HELLO / HELLO_REQ reply (OK payload)

[proto_ver:u8][board_name\0][board_ver\0][fw_ver\0]

proto_ver = 3 for this protocol version. All strings are null-terminated UTF-8. fw_ver is the firmware git tag (from git describe --tags --always --dirty, embedded at build time via version_generated.h). The host parses them to populate boardName, boardVersion, and fwVersion in the UI.

LS DATA payload

Packed directory entries, repeated until LEN bytes are consumed:

[is_dir:u8][name\0]

is_dir: 1 = directory, 0 = file. name is null-terminated UTF-8. Entries end when no bytes remain in the payload. Every entry is returned — dot-files, main.py, boot.py and system folders are not filtered (only the . / .. pseudo-entries are omitted).

PROGRESS payload

[sent:u32 BE][total:u32 BE]

Both fields are big-endian 32-bit unsigned integers.

MTU_REQ OK payload

[mtu:u16 BE]

Big-endian 16-bit unsigned integer, the negotiated ATT MTU.

PROG_START — program execution began

PROG_START (SEQ = 0xFF) is emitted unsolicited whenever a program starts from the menu (i.e. not initiated by a host RUN command). It is also sent on BLE reconnect if a program is already running at that moment (so the host can restore its programRunning state). Payload is the file path (UTF-8).

PROG_START is not sent for host-initiated runs (KIND_RUN); the host already knows the run started because it sent the command and received OK.

PROG_END — every program exit path

PROG_END (SEQ = 0xFF) is emitted unconditionally whenever a running program ends, regardless of whether it was started by the host or from the menu:

Normal return → ok=1
hub.exit() → raises SystemExit → caught by runner → ok=1
2-second centre-button hold → KeyboardInterrupt → caught → ok=1
Uncaught exception → ok=0, message = flattened traceback string

In all cases Protocol::onProgEnd() (called by Runner::run() on mp_task after execution) emits the PROG_END frame.

Streaming transfers

UPLOAD, READ and CP are spread over many protocol-task iterations. Each iteration first parses and dispatches every complete incoming frame, then advances active transfers by one bounded slice (READ: ≤ 2 KiB, and for a request received over USB never more than the free CDC TX fifo space, whether or not the host asserts DTR, and while a BLE central is connected never more than the free NimBLE buffer space, always leaving a reserve so host writes can still be received; CP: ≤ 4 KiB), so the device keeps answering PING / STOP / ABORT / CMD and keeps emitting STDOUT during a transfer. At most one UPLOAD, one READ and one CP can be active at a time; a second request of the same kind gets ERR "UPLOAD busy" / "READ busy" / "CP busy".

File data travels in DATA chunks that carry their own position:

DATA payload = [offset:u32 BE][bytes]

offset is the byte position of the first data byte in the file, so the receiver can place chunks correctly even if they arrive out of order or duplicated. Every chunk frame carries the SEQ of the request that started the transfer.

UPLOAD flow

Host sends one UPLOAD frame whose payload is [path_len:u8][path bytes][file_size:u32 BE].
Device creates parent directories and opens the file. On failure it replies ERR "UPLOAD: open <path>" and the upload ends. Otherwise it sends ACK.
Host sends the file as DATA chunks (same SEQ), [offset:u32 BE][bytes]. A whole chunk frame must fit the device’s 2048-byte frame buffer — use ≤ 1024 data bytes per chunk. The full file is never held in RAM.
Device writes in-order chunks immediately. Chunks that arrive early are held (up to 8) until the gap before them is filled; chunks for data already written are ignored. More than 8 early chunks → ERR "UPLOAD: reorder window full".
Device emits PROGRESS ([sent:u32 BE][total:u32 BE], sent = contiguous bytes written) every 4 KiB; each one resets the host idle timer. Hosts can use it as flow control (keep at most a few KiB in flight past the last PROGRESS).
When all file_size bytes are written the device closes the file and sends a final PROGRESS (sent == total) followed by OK. A zero-length upload completes right after the ACK.

Timeouts

Device idle timeout: 5 s without a chunk → ERR "UPLOAD: timeout".
Host idle timeout: 15 s; reset on ACK and each PROGRESS frame.

Failure

On timeout, write error, bad chunk (payload < 4 bytes), reorder overflow or ABORT the device closes and deletes the partial file. Any chunks still in flight then get ERR "DATA: no active upload"; hosts can ignore those.

READ flow

Host sends READ with the file path.
Device replies ERR ("read: no such file: <path>", "read: open <path>", "bad path", "READ busy") or ACK with payload [size:u32 BE].
Device sends the file as DATA chunks (same SEQ), [offset:u32 BE][≤ 512 bytes]. STDOUT and replies to other requests may arrive between chunks. Hosts place each chunk at its offset.
Device sends OK after the last chunk. If the file read ends early (I/O error) OK still follows; hosts should compare the received length with size.

CP flow

Host sends CP. Malformed payload → ERR; a CP already running → ERR "CP busy".
Device sends ACK, then opens both files (ERR "cp: open <path>" on failure).
The copy runs in the background, ≤ 4 KiB per iteration.
OK when done; ERR "cp: write failed" on a write error (the partial destination is deleted).

ABORT

Payload [target_seq:u8] cancels the UPLOAD, READ or CP started with that SEQ; an empty payload cancels every active transfer.

Each cancelled transfer gets ERR "aborted" on its own SEQ, so the host waiter for it settles. A cancelled UPLOAD or CP deletes its partial file; a cancelled READ stops sending chunks.
The ABORT frame itself then gets OK. When a target_seq matched nothing (already finished, or never started) it gets ERR "ABORT: no such stream" instead. An empty ABORT always gets OK.
Frames already queued before the ABORT was processed (e.g. a READ chunk) may still arrive after it; hosts drop DATA for a SEQ they no longer wait for. Over BLE the device keeps at most ~2.4 KB of stream data queued in the BLE stack, so this tail is short; the ERR "aborted" arrives after it.
If the transfer had already finished sending (its OK is in flight), the ABORT gets ERR "ABORT: no such stream" and the transfer’s OK follows — the abort came too late, it was not ignored.
A READ / CP started while another is running gets ERR "READ busy" / "CP busy" — DATA still arriving is from the old transfer.
Over BLE a host write can still occasionally fail (Web Bluetooth NotSupportedError: GATT operation failed) when the device is briefly out of buffers. Hosts should retry a failed GATT write a few times with a short backoff (e.g. 20 / 50 / 100 ms) before failing the request.
The device cannot tell when a host gives up on a transfer. A host that fails a READ / UPLOAD / CP locally (bad chunk, timeout, user cancel) must send ABORT for it, otherwise the device keeps streaming.

MKDIR

Payload is the absolute directory path, UTF-8, no terminator (e.g. /sd/my_project/lib). Same SEQ rules as RM: no ACK, the device replies OK or ERR directly.

The device walks the path from the root down; each segment that does not exist is created. Empty segments (//, trailing /) are ignored.

OK (empty payload) — the path is a directory, including when it already existed (idempotent).
ERR "bad path" — empty, longer than 255 bytes, relative (no leading /), or contains a .. segment.
ERR "not a directory: <seg>" — a segment exists as a file; <seg> is the absolute path up to that segment.
ERR "<ERRNO>" — mkdir raised OSError; the errno name is passed through (e.g. "ENOSPC").

No proto_ver bump. Firmware without MKDIR replies to kind 0x1A with ERR "unknown kind: 0x1a"; older firmware sends a bare "unknown kind". Hosts should treat any ERR starting with "unknown kind" as “MKDIR unsupported”.

Unknown kinds

Any unrecognised KIND is answered with ERR "unknown kind: 0xNN" (lower-case hex, echoing the request SEQ).

Sequence / response matching

Host counter wraps 0x00 → 0xFF → 0x00.
Device echoes the request SEQ in OK / ERR / ACK / PROGRESS / DATA replies.
Frontend waiters is a Map<seq, Waiter> keyed by SEQ, enabling multiple in-flight requests.
NO_ACK frames (STOP with FLAGS=NO_ACK) use a real SEQ for in-order delivery via the device-side reorder buffer but register no waiter on the host.

Fire-and-forget ordering (NO_ACK)

BLE writeValueWithoutResponse can deliver frames out of order. The device maintains a reorder buffer keyed by SEQ. After each NO_ACK frame arrives the buffer is drained in ascending SEQ order. Entries with a gap larger than WINDOW = 16 are flushed anyway to avoid deadlock.

HELLO lifecycle

Device boots, Protocol::init() runs (enabling stdout framing), sends HELLO (SEQ=``0xFF``, KIND=``0x20``) proactively.
Host connect() pings, then calls requestHello() which sends HELLO_REQ (KIND=``0x01``). The device replies OK with the same [proto_ver][board_name\0][board_ver\0] payload.
If the proactive HELLO arrived before the host subscribed, the explicit HELLO_REQ / OK exchange still populates the board info.

Keepalive ping

The host sends a PING (KIND 0x10) every 10 seconds while connected. The device replies OK. This keeps the BLE connection alive on both the OS radio scheduler and the device NimBLE stack. If a PING times out (2 s) the error is swallowed — the BLE disconnect handler fires independently if the link is truly dead.

Why the C-level stdout hook

In this MicroPython build sys.stdout and sys.stderr are immutable dummy objects — assigning sys.stdout = my_stream does not redirect print(), which goes straight to the C function mp_hal_stdout_tx_strn. Wrapping program output in STDOUT frames therefore has to happen inside that C function, not in Python.

ports/b2op/mphalport.c (a local copy that overrides the submodule file) adds a toggleable framing layer. When enabled, each call to mp_hal_stdout_tx_strn coalesces the payload into a 240-byte buffer and flushes a binary STDOUT frame on newline or buffer-full. The STDOUT frame header is:

[0x7E][LEN:varint][SEQ=0xFF][FLAGS=0x00][KIND=0x30]

Framing is enabled at boot by Protocol::init() and remains on for the lifetime of the device. Protocol replies (OK / ERR / DATA / PROG_END) bypass the framing buffer via mp_hal_proto_write(), which flushes any pending framed data first so reply bytes never interleave inside a STDOUT payload.

Transport fan-out

All writes flow through the shared C helper stdout_tx_strn_impl, which drives USB CDC / USB-serial-JTAG directly. BLE NUS is attached via a C function-pointer callback set by BleUart::init() through mp_hal_set_stdout_c_ble_write(). Every framed flush and every protocol write automatically mirrors to BLE without any Python involvement.

On power-off, BleUart::deinit() detaches that callback and tears down NimBLE under the same recursive mutex that guards every other BleUart call, so the protocol task (advertising timeout) and mp_task (stdout mirror) can never touch a freed NimBLE stack. Never call NimBLEDevice::deinit() directly.

Interactive frames (0x34–0x35)

These host-to-device frames enable interactive Python execution and direct hardware control from the web IDE. Both are processed on separate tasks:

EXEC is handled on mp_task (holds the MicroPython GIL).
CMD is handled on hub_protocol_task (holds GIL via C call; no Python interpreter involvement).

KIND_EXEC (0x34) — REPL execution

Payload: Python source string (UTF-8, no NUL terminator required).

The device executes the source in a persistent REPL context using parse_compile_execute with MP_PARSE_SINGLE_INPUT | EXEC_FLAG_IS_REPL. Globals persist across calls, and expression results are printed to stdout (just like the interactive REPL). Stdout flows as normal KIND_STDOUT (0x30) frames during execution.

Reply:

KIND_OK — execution succeeded (or raised SystemExit).
KIND_ERR — a Python exception occurred; payload is the exception text (UTF-8).

Rejected conditions:

A user program is running → ERR "program running".
A previous EXEC is still pending → ERR "busy".

KIND_CMD (0x35) — Binary hardware command

Payload: [CmdId:u8][args…]

The first byte selects the command; remaining bytes are the arguments as described in the table below. Reply is KIND_OK for fire-and-forget commands, or KIND_DATA for query commands.

Port index encoding

Idx	Port	C++ object
0	A	`Hub::Ports::portA`
1	B	`Hub::Ports::portB`
2	C	`Hub::Ports::portC`
3	D	`Hub::Ports::portD`
4	E	`Hub::Ports::portE`
5	F	`Hub::Ports::portF`
6	LED	`Hub::Ports::LED`
7	Accel	`Hub::Ports::Accel`
8	Gyro	`Hub::Ports::Gyro`

endState values: 0 = FLOAT, 126 = HOLD, 127 = BRAKE.

CMD table

CmdId	Name	Payload layout (after CmdId byte)	Reply
`0x01`	MOTOR_START_POWER	`port:u8 power:i8`	OK
`0x02`	MOTOR_START_SPEED	`port:u8 speed:i8 maxPower:u8`	OK
`0x03`	MOTOR_START_SPEED_TIME	`port:u8 ms:u16BE speed:i8 maxPower:u8 endState:u8`	OK
`0x04`	MOTOR_START_SPEED_DEG	`port:u8 deg:u32BE speed:i8 maxPower:u8 endState:u8`	OK
`0x05`	MOTOR_GOTO_ABS_POS	`port:u8 pos:i32BE speed:u8 maxPower:u8 endState:u8`	OK
`0x06`	MOTOR_PRESET_ENCODER	`port:u8 pos:i32BE`	OK
`0x07`	MOTOR_SET_ACC_TIME	`port:u8 ms:u16BE profile:u8`	OK
`0x08`	MOTOR_SET_DEC_TIME	`port:u8 ms:u16BE profile:u8`	OK
`0x09`	PORT_SET_RGB	`port:u8 r:u8 g:u8 b:u8`	OK
`0x0A`	PORT_SET_RGB_IDX	`port:u8 colorIdx:u8`	OK
`0x0B`	PORT_SET_MODE	`port:u8 mode:u8`	OK
`0x0C`	PORT_WRITE_DATA	`port:u8 mode:u8 [data bytes…]`	OK
`0x10`	PORT_GET_VALUE	`port:u8 mode:u8 dataset:u8`	DATA: one or more `f32LE` values
`0x11`	PORT_GET_DEVICE_TYPE	`port:u8`	DATA: `type:u16BE`
`0x12`	PORT_FORCE_DEVICE_TYPE	`port:u8 type:u16BE` (local ports 0–5 only)	OK
`0x20`	HUB_GET_BATTERY	(none)	DATA: `mv:u16BE pct:u8`
`0x21`	HUB_GET_BUTTONS	(none)	DATA: `mask:u8` (bit 0=center, 1=up, 2=down, 3=left, 4=right)
`0x22`	HUB_GET_ORIENTATION	(none)	DATA: `pitch:f32LE yaw:f32LE roll:f32LE`

Serial monitor

make monitor in ports/b2op/ runs b2op_monitor.py instead of the stock idf.py monitor. It decodes the binary frame protocol so the terminal shows clean text rather than raw framing bytes. All standard IDF monitor keyboard shortcuts (Ctrl+] exit, Ctrl+T menu, Ctrl+T Ctrl+R reset, Ctrl+T Ctrl+F flash, etc.) and features (address decoding from ELF, panic backtrace, timestamps, file logging) are preserved.

Protocol trace

Every non-STDOUT frame the device sends is printed as one trace line with time since monitor start, SEQ and FLAGS:

[proto +  12.345s seq=04 fl=00] ACK size=1834 B -> READ stream opened
[proto +  12.351s seq=04 fl=00] DATA #1 off=0 len=231 -> 231/1834 (12.6%) +6ms
[proto +  12.402s seq=04 fl=00] OK
[proto +  12.402s seq=04 fl=00]   stream finished (OK): 1834/1834 B contiguous, ...

An ACK with a 4-byte payload opens a READ stream for its SEQ; each DATA chunk is checked for gaps, duplicates, out-of-order and past-end offsets, and a chunk whose data looks like a frame header (lost payload) is flagged. OK / ERR prints a summary with byte count, chunk count, duration, throughput and whether the file arrived complete. The monitor only sees what the device sends over USB (which mirrors BLE output), not host requests.

Environment variables:

B2OP_MON_HEX=1 — full hex + ASCII dump of every protocol payload.
B2OP_MON_STDOUT=1 — also trace one line per STDOUT / STDERR frame.
B2OP_MON_QUIET=1 — old terse one-line-per-frame output.

Binary color escapes in STDOUT/STDERR payload

Firmware can embed two-byte color markers directly in print() output to request terminal colour changes. The monitor translates them to ANSI sequences before the text reaches the display pipeline.

0x01 0xNN

NN selects the colour:

`NN`	Colour	ANSI equivalent
`0x00`	reset / normal	`\033[0m`
`0x01`	red (bold)	`\033[1;31m`
`0x02`	green	`\033[0;32m`
`0x03`	yellow	`\033[0;33m`
`0x04`	blue	`\033[0;34m`
`0x05`	cyan	`\033[0;36m`
`0x06`	magenta	`\033[0;35m`

Example — emit red text from C++ (inside mp_hal_stdout_tx_strn path):

// In a MicroPython module or C-side print helper:
static const char RED_ON[]  = "\x01\x01";
static const char RED_OFF[] = "\x01\x00";
mp_hal_stdout_tx_strn(RED_ON,  2);
mp_hal_stdout_tx_strn("error text\n", 11);
mp_hal_stdout_tx_strn(RED_OFF, 2);

Example — emit yellow text from Python (hub helper planned):

import sys
sys.stdout.buffer.write(b'\x01\x03Warning\x01\x00\n')

STDERR kind (0x31) is automatically displayed in red regardless of embedded color markers.

Notes

Output is coalesced into a 240-byte buffer. A typical print(...) emits one STDOUT frame per line.
Writes without a trailing newline (e.g. print("x", end="")) stay buffered until the next newline or buffer-full.
sys.stderr on ESP32 flows through the same mp_hal path, so tracebacks appear as STDOUT frames. A separate STDERR kind would require a distinct C hook.
Framing survives soft-reset because the flag is a plain C static; it is set once by Protocol::init() and never cleared.
proto_ver in HELLO = 2. Older ASCII-framed (V1) firmware is not supported; only V2 (or above) is implemented on both device and host.