Host protocol framing — V2

The on-device program runner (ports/b2op/fs/protocol.py) speaks a compact binary protocol with the web IDE / host tool over USB CDC and BLE NUS.

Wire format

Every frame is:

[0x7E][LEN:varint][SEQ:u8][FLAGS:u8][KIND:u8][PAYLOAD:LEN bytes]

0x7E

Magic byte; also used for re-sync after corruption. Any byte that is not part of an in-progress frame and equals 0x7E starts a new frame.

LEN — LEB128 varint

Payload byte count, little-endian, up to 5 bytes. Bit 7 of each byte is the continuation flag — if set, another byte follows. The 5th byte, if reached, is always the last regardless of bit 7.

Range	Bytes	Format
0–127	1	`[0xxxxxxx]`
128–16 383	2	`[1xxxxxxx][0xxxxxxx]`
16 384–2 097 151	3	`[1xxxxxxx][1xxxxxxx][0xxxxxxx]`
2 097 152–268 435 455	4	`[1xxxxxxx][1xxxxxxx][1xxxxxxx][0xxxxxxx]`
268 435 456–34 359 738 367	5	`[1xxxxxxx][1xxxxxxx][1xxxxxxx][1xxxxxxx][xxxxxxxx]`

SEQ — sequence number

Host → Device: monotonically increasing 0x00–0xFF, wraps.
Device → Host: 0xFF = unsolicited (no matching request); otherwise echoes the host’s SEQ so the host can match the reply to its waiter.

FLAGS

bit 0 NO_ACK — device must not send a response frame. SEQ is still a real counter and is used for in-order delivery when frames arrive out of order over BLE writeValueWithoutResponse.
Bits 1–7: reserved, must be zero.

KIND

One-byte frame type; see the table below.

Per-frame overhead: 5–9 bytes (1 magic + 1–5 varint + 1 SEQ + 1 FLAGS + 1 KIND).

Kind table

Hex	Name	Direction	Payload	Notes
`0x01`	HELLO_REQ	host → device	empty	Request device to resend board info
`0x10`	PING	host → device	empty	Liveness check
`0x11`	MTU_REQ	host → device	empty	Query BLE ATT MTU
`0x12`	RUN	host → device	`path` (UTF-8)	Run a Python file
`0x13`	STOP	host → device	empty	Interrupt running program; `FLAGS=NO_ACK` for fire-and-forget
`0x14`	UPLOAD	host → device	`[path_len:u8][path bytes][file data]`	File len = `LEN − 1 − path_len`
`0x15`	READ	host → device	`path` (UTF-8)	Reply: DATA with raw file bytes
`0x16`	LS	host → device	`path` (UTF-8)	Reply: DATA with packed dir entries
`0x17`	MV	host → device	`[src_len:u8][src bytes][dst_len:u8][dst bytes]`	Rename/move; reply: OK or ERR
`0x18`	CP	host → device	`[src_len:u8][src bytes][dst_len:u8][dst bytes]`	Copy; reply: OK or ERR
`0x19`	RM	host → device	`path` (UTF-8)	Delete file; reply: OK or ERR
`0x20`	HELLO	device → host	`[proto_ver:u8][board_name\0][board_ver\0]`	SEQ = `0xFF`; sent proactively at boot
`0x21`	OK	device → host	optional message (UTF-8 or structured; see command)	Echoes request SEQ
`0x22`	ERR	device → host	error message (UTF-8)	Echoes request SEQ
`0x23`	ACK	device → host	optional message (UTF-8)	Echoes SEQ; resets host idle timer
`0x24`	PROGRESS	device → host	`[sent:u32 BE][total:u32 BE]`	Echoes SEQ; resets host idle timer
`0x25`	DATA	device → host	binary payload (command-specific)	Echoes request SEQ
`0x30`	STDOUT	device → host	UTF-8 text	SEQ = `0xFF`; unsolicited
`0x31`	STDERR	device → host	UTF-8 text	SEQ = `0xFF`; unsolicited
`0x32`	PROG_END	device → host	`[ok:u8][message UTF-8]`	SEQ = `0xFF`; see below

Structured payloads

HELLO / HELLO_REQ reply (OK payload)

[proto_ver:u8][board_name\0][board_ver\0]

proto_ver = 2 for this protocol version. Both strings are null-terminated UTF-8. The host parses them to populate boardName and boardVersion in the UI.

LS DATA payload

Packed directory entries, repeated until LEN bytes are consumed:

[is_dir:u8][name\0]

is_dir: 1 = directory, 0 = file. name is null-terminated UTF-8. Entries end when no bytes remain in the payload.

PROGRESS payload

[sent:u32 BE][total:u32 BE]

Both fields are big-endian 32-bit unsigned integers.

MTU_REQ OK payload

[mtu:u16 BE]

Big-endian 16-bit unsigned integer, the negotiated ATT MTU.

PROG_END — every program exit path

PROG_END (SEQ = 0xFF) is emitted unconditionally whenever a running program ends:

Normal return → ok=1
hub.exit() → raises SystemExit → caught by runner → ok=1
2-second centre-button hold → KeyboardInterrupt → caught → ok=1
Uncaught exception → ok=0, message = flattened traceback string

All exit paths converge in _handle(KIND_RUN, ...) inside protocol.py: after runner.run_program() returns, _prog_end(ok, msg) is called unconditionally before on_run_finished.

UPLOAD flow

Host sends one UPLOAD frame whose payload is [path_len:u8][path bytes][file data].
Device sends ACK immediately (before reading all data), so the host can reset its idle timer.
Device emits PROGRESS frames ([sent:u32 BE][total:u32 BE]) roughly every 2 KiB while writing. Each resets the host idle timer.
Device sends OK when the file is flushed to disk.

Sequence / response matching

Host counter wraps 0x00 → 0xFF → 0x00.
Device echoes the request SEQ in OK / ERR / ACK / PROGRESS / DATA replies.
Frontend waiters is a Map<seq, Waiter> keyed by SEQ, enabling multiple in-flight requests.
NO_ACK frames (STOP with FLAGS=NO_ACK) use a real SEQ for in-order delivery via the device-side reorder buffer but register no waiter on the host.

Fire-and-forget ordering (NO_ACK)

BLE writeValueWithoutResponse can deliver frames out of order. The device maintains a reorder buffer:

_no_ack_buf  = {}   # seq → (kind, flags, payload)
_no_ack_next = 0    # next expected SEQ for in-order dispatch

After each NO_ACK frame arrives the buffer is drained in ascending SEQ order. Entries with a gap larger than WINDOW = 16 are flushed anyway to avoid deadlock.

HELLO lifecycle

Device boots, install_stream_redirect() runs, sends HELLO (SEQ=``0xFF``, KIND=``0x20``) proactively.
Host connect() pings, then calls requestHello() which sends HELLO_REQ (KIND=``0x01``). The device replies OK with the same [proto_ver][board_name\0][board_ver\0] payload.
If the proactive HELLO arrived before the host subscribed, the explicit HELLO_REQ / OK exchange still populates the board info.

Why the C-level stdout hook

In this MicroPython build sys.stdout and sys.stderr are immutable dummy objects — assigning sys.stdout = my_stream does not redirect print(), which goes straight to the C function mp_hal_stdout_tx_strn. Wrapping program output in STDOUT frames therefore has to happen inside that C function, not in Python.

ports/b2op/mphalport.c (a local copy that overrides the submodule file) adds a toggleable framing layer. When enabled, each call to mp_hal_stdout_tx_strn coalesces the payload into a 240-byte buffer and flushes a binary STDOUT frame on newline, buffer-full, or when raw_write() is called. The STDOUT frame header is:

[0x7E][LEN:varint][SEQ=0xFF][FLAGS=0x00][KIND=0x30]

Transport fan-out

Framed and raw writes both flow through the shared C helper stdout_tx_strn_impl, which drives USB CDC / USB-serial-JTAG directly. BLE NUS is attached via a Python callback registered with set_frame_sink(). protocol.py sets a sink that forwards every framed flush and every raw_write() payload to ble_uart.instance().write_tx(). Called under the GIL from mp_hal_stdout_tx_strn code paths that originate in Python.

hub.set_framed_output(enabled: bool) → None[source]

Enable or disable stdout framing at the C level. When True, every call to mp_hal_stdout_tx_strn (print, traceback, REPL echo, sys.stdout.write) is wrapped in a binary STDOUT frame before reaching UART / USB CDC / JTAG.

Idempotent to call. Off by default at boot.

hub.raw_write(buf: bytes) → int[source]: Write bytes to stdout bypassing the framing wrapper. Returns the number of bytes written. Flushes any pending framed data first so raw protocol frames never interleave inside a framed payload. Used by protocol.py to emit its own OK / ERR / DATA / PROG_END frames.

hub.flush_output() → None[source]: Explicitly flush any buffered framed stdout data as a STDOUT frame. Normally not needed — auto-flush fires on newlines, buffer-full, or before any raw_write.

hub.set_frame_sink(cb: Callable[[bytes], None] | None) → None[source]: Register a callback that receives the exact bytes emitted by every framed flush and every raw_write(). Used by protocol.py to mirror the stream to BLE NUS. Pass None to detach. Called under the GIL; exceptions inside cb are swallowed.

Notes

Output is coalesced into a 240-byte buffer. A typical print(...) emits one STDOUT frame per line.
Writes without a trailing newline (e.g. print("x", end="")) stay buffered until the next newline, buffer-full, or an explicit flush_output() call.
sys.stderr on ESP32 flows through the same mp_hal path, so tracebacks appear as STDOUT frames. A separate STDERR kind would require a distinct C hook.
Framing survives soft-reset because the flag lives in a plain C global rather than in MP_STATE_*. main.py re-enables it explicitly after each soft-reset in case that changes.
proto_ver in HELLO = 2. Older ASCII-framed (V1) firmware is not supported; only V2 is implemented on both device and host.