Wire-format parity fixes in reader.py for trailing fields the
companion-radio firmware emits but the SDK decoder was dropping, plus an
over-read guard on DEFAULT_FLOOD_SCOPE. Each fix uses the existing
BATT_AND_STORAGE defensive-read pattern (up-front minimum-length check
plus per-field cumulative-length gates), so older firmware that doesn't
emit the trailing fields keeps decoding without raising. Every fix ships
a legacy-frame and modern-frame unit test pair.
AUTOADD_CONFIG (PacketType 25): companion-v1.14.0 firmware emits a
trailing max_hops byte (firmware commit 00566741); the SDK dropped it.
Adds a defensive `if len(data) >= 3` read. Pre-v1.14.0 frames decode to
`{"config": ...}` with no max_hops key.
LOGIN_SUCCESS (PacketType 0x85): the new-style RESP_SERVER_LOGIN_OK path
emits three trailing fields the SDK was dropping — server_timestamp (4B,
firmware commit 0e90b731), acl_permissions (1B, 7947e8a2), and
fw_ver_level (1B, 418ae08b), all first shipped in companion-v1.10.0.
Adds per-field length gates (>=12, >=13, >=14). Legacy 8-byte "OK"-path
frames decode unchanged.
ACK (PacketType 0x82): firmware emits a trailing 4-byte trip_time
(round-trip latency in ms) since companion-v1.0.0a (firmware commit
d9dc76f1, Jan 2025) — on the wire ~16 months but never surfaced by the
SDK. Adds an `if len(data) >= 9` read. Legacy 5-byte ACK frames decode
unchanged.
DEFAULT_FLOOD_SCOPE (PacketType 28): firmware emits a 48-byte populated
frame or a 1-byte sentinel when no scope is set. The SDK unconditionally
read 31+16 bytes, over-reading 47 bytes past the end of the sentinel
frame and dispatching `{"scope_name": "", "scope_key": ""}`. Adds an
`if len(data) >= 48` guard so the sentinel dispatches `{}`. Consumers
detecting "no scope" via `payload["scope_name"] == ""` should switch to
a key-presence check.
RAW_DATA: the payload-framing fix this branch originally carried landed
upstream first in PR #86 (commit 44b21be), with identical decode logic
(discard the reserved 0xFF byte, then read the remaining variable-length
payload). That redundant change is dropped here; this commit retains only
its regression test (test_raw_data_realistic_frame), which exercises the
upstream fix end-to-end — the upstream change shipped without a test.
CHANGELOG:
- Decode max_hops trailing byte in AUTOADD_CONFIG (companion-v1.14.0+).
- Decode server_timestamp / acl_permissions / fw_ver_level trailing
fields in LOGIN_SUCCESS (companion-v1.10.0+).
- Decode trip_time trailing field in ACK (companion-v1.0.0a+).
- DEFAULT_FLOOD_SCOPE dispatches `{}` on the 1-byte sentinel frame
instead of `{scope_name: "", scope_key: ""}`. Detect "no scope" via
key presence.
Tests: 10 tests in tests/unit/test_protocol_surface_gaps.py (legacy +
modern frame pair per finding, including a RAW_DATA regression test for
the upstream fix); all 10 pass on the rebased upstream base.
Why: companion-radio firmware has been emitting these fields on the wire
for between 2 and 16 months; SDK consumers (integrations, bots,
dashboards) lose access to data that is on the wire today.
Why: companion-radio firmware emits RESP_CODE_CHANNEL_DATA_RECV (27) for
group-channel binary data (PAYLOAD_TYPE_GRP_DATA), shipped in
companion-v1.15.0. The SDK had no PacketType value 27, no EventType, and
no reader handler, so these frames hit the unknown-packet-type
fallthrough and the payload was silently dropped.
This adds PacketType.CHANNEL_DATA_RECV = 27 (the previously-skipped enum
slot), EventType.CHANNEL_DATA_RECV, and a reader handler. The fixed
9-byte header (snr, reserved, channel_idx, path_len with the
sentinel/hash-mode encoding) reuses CHANNEL_MSG_RECV_V3's framing; the
typed tail decodes data_type (uint16 little-endian, widened from uint8
in firmware), data_len, and payload (hex string). An up-front length
gate mirrors the defensive-read pattern used by the other handlers.
data_type is exposed as an int (matching the txt_type convention) and
payload as a hex string (matching RAW_DATA's convention for binary data
of unknown encoding); attributes surface channel_idx and data_type so
subscribers can filter without unpacking the payload.
Tests: tests/unit/test_protocol_surface_gaps.py — 5 new tests (enum
slot present, direct-path frame, route-flood path_len bit-split,
under-minimum frame dropped, widened data_type round-trip). Full unit
suite: 146 passed.
Why: MeshCore.subscribe typed callbacks as Callable[[Event], Coroutine[...]]
(async only), while EventDispatcher.subscribe typed them as
Callable[[Event], Union[None, asyncio.Future]] (sync or async). Type-checkers
flag any sync handler passed through MeshCore.subscribe. Fix: align the
annotation to match EventDispatcher's union type; remove unused Coroutine import.
Refs: Forensics report finding R05
Why: send_binary_req() registered the pending request with the reader AFTER
send() returned. If a BINARY_RESPONSE arrives between send() returning and
registration (reachable for TCP-companion proxies), the reader logs "No tracked
request found" and the caller's wait_for_event times out. Fix: pre-register a
placeholder keyed by object id before send(), then swap it for the real tag
from MSG_SENT. On send() failure, the placeholder is cleaned up.
Refs: Forensics report finding R03
Why: Three minor cleanup fixes. M03 adds an else branch in set_flood_scope so
unsupported scope types raise TypeError instead of UnboundLocalError. M05 removes
the dead `out_path_len >> 6` shift in update_contact (high bits always zero due
to reader masking) and initializes path_hash_mode=0 explicitly. M07 normalizes
three `return None` paths in get_contacts to return Event(EventType.ERROR, ...)
so callers can rely on the return type always being Event.
Refs: Forensics report findings M03, M05, M07
Why: 5 seconds is too short for slow-path mesh operations (path-resolving
messaging, long binary responses, remote auth). Also the root cause of tests
that appeared to "hang" — they were falling through to the 5s timeout because
their mock dispatchers don't wire matching responses. Landed as a separate
commit so reviewers can drop it independently if they push back.
Refs: Forensics report finding F09
Why: req_mma() references undefined variables `start` and `end`, causing a
NameError on every call. The logger.error migration warning confirms the method
is intentionally deprecated in favor of req_mma_sync. Since it is broken as
shipped, removing it cannot break any working caller.
Refs: Forensics report finding F13
Why: Five CommandType enum entries had no user-facing SDK method:
SEND_RAW_DATA (25), HAS_CONNECTION (28), GET_CONTACT_BY_KEY (30),
GET_TUNING_PARAMS (43), FACTORY_RESET (51). Added send_raw_data() in
MessagingCommands, has_connection()/get_tuning()/request_factory_reset()/
confirm_factory_reset() in DeviceCommands, get_contact_by_key() in
ContactCommands. FACTORY_RESET uses a two-step token pattern as a
Python-side safety measure against accidental invocation.
Refs: Forensics report finding N09 (also N03 for get_tuning)
Why: Firmware requires strict len > 10 (MyMesh.cpp:1620). When path is
empty, send_trace() builds exactly 10 bytes (cmd+tag+auth+flags), which
is silently rejected. Appending one zero byte when the packet is <= 10
bytes satisfies the firmware gate without changing the semantic content.
Refs: Forensics report finding N05
Why: Three firmware push/response codes had no SDK handler — frames fell
through to the "Unhandled" debug log. CONTACT_DELETED (0x8F) carries a
32-byte pubkey from onContactOverwrite(); CONTACTS_FULL (0x90) is a
1-byte push from onContactsFull(); TUNING_PARAMS (23) is the 9-byte
response to CMD_GET_TUNING_PARAMS carrying rx_delay and airtime_factor.
All three now dispatch typed events. Short frames are guarded.
Refs: Forensics report findings N01, N02, N03
Why: PacketType was missing CONTACTS_FULL (0x90), emitted by
MyMesh::onContactsFull(). CommandType was missing GET_STATS (56),
used by get_stats_core/radio/packets but referenced only as literal
b"\\x38". Adding both enum entries prepares for handler and wrapper
implementations in subsequent commits.
Refs: Forensics report findings N02, R04
Why: asyncio.get_event_loop() inside an async function emits
DeprecationWarning since Python 3.10 and raises in some contexts on
Python 3.12+. The call in CommandHandlerBase.send() is always inside
a running async context where get_running_loop() is the correct API.
Refs: Forensics report finding F19
Why: On Python 3.9/3.10, asyncio.Queue() and asyncio.Lock() bind to
the running event loop at construction time. If the SDK is instantiated
from a synchronous factory before an event loop exists, both primitives
raise "RuntimeError: ... is bound to a different event loop" on first
use. Fix: EventDispatcher defers Queue creation to start(), with a
guard in dispatch() that raises RuntimeError if called before start().
CommandHandlerBase defers Lock creation via a lazy @property accessor.
Both document the contract change in class docstrings.
Refs: Forensics report finding F08
Why: EventDispatcher._process_events() calls task_done() on the queue
immediately after spawning async callback tasks. await queue.join() in
stop() therefore returns as soon as all items are marked done, even if
their async callbacks are still executing. Any caller that does
"await dispatcher.stop(); cleanup()" could race with still-running
callbacks. Fix: after queue.join(), gather all tracked background tasks
before cancelling the dispatch loop.
Refs: Forensics report finding F07
Why: Python's asyncio holds only weak references to tasks created via
create_task(). Under GC pressure (especially Python < 3.11), unretained
tasks can be silently cancelled mid-execution, and any exceptions are
swallowed as "Task exception was never retrieved." Seven call sites
across TCPConnection, BLEConnection, SerialConnection, and
EventDispatcher used fire-and-forget create_task with no stored
reference. Fix: introduce _background_tasks set and _spawn_background()
helper on each class, following the standard pattern from the asyncio
docs (task.add_done_callback(set.discard)).
Refs: Forensics report finding F05
Why: _receive_count incremented inside data_received(), which fires
once per TCP read — not once per completed MeshCore frame. Under TCP
fragmentation a single frame can arrive in multiple segments, inflating
_receive_count relative to _send_count. The disconnect-detection
heuristic (send_count - receive_count >= 5) then never fires because
the receive side is over-counted. Moving the increment into handle_rx
after a complete frame is assembled makes the counter semantically
correct: one increment per MeshCore frame dispatched to the reader.
Refs: Forensics report finding N04
Why: When BLE pairing failed during connect(), the exception was caught
as a warning and connect() continued normally. This left the transport
in a half-usable state — the BLE link was up but the pairing handshake
never completed, so encrypted characteristics could silently fail.
Now the pairing exception disconnects the client and re-raises, giving
the caller a clean failure. Updated the pre-existing
test_ble_connection_with_pin_failed_pairing test to assert the re-raise
behavior instead of the old swallow-and-continue behavior.
Refs: Forensics report finding F17
Why: handle_disconnect resets self.client to self._user_provided_client.
That client's disconnected_callback was not re-registered, so subsequent
BLE disconnects after a successful reconnect cycle were missed —
ConnectionManager never learned the link dropped again. Now re-registers
via set_disconnected_callback with a hasattr guard and try/except for
bleak version compatibility. The next connect() call would also re-create
the client with the callback, but this closes the gap between disconnect
and reconnect.
Refs: Forensics report finding F16
Why: When handle_rx detects an oversize frame header (>300 bytes), it
resets state (header, inframe, frame_expected_size) and re-calls itself
on any remaining data. But when data is empty after the header, the
method fell through to the frame-assembly code with inframe=b"",
eventually dispatching an empty frame to reader.handle_rx. Currently
absorbed by F06's umbrella try/except, but a guaranteed crash if that
guard moves. Adding a bare return after the reset block prevents the
fallthrough in both TCP and serial transports.
Refs: Forensics report finding M06
Why: After create_serial_connection, connect() awaited
_connected_event.wait() with no timeout. If the serial device opened
but connection_made was never called (driver bug, USB adapter glitch),
connect() hung indefinitely. Now wrapped in asyncio.wait_for with a
configurable timeout (default 10s). asyncio.TimeoutError propagates
to the caller for clean failure handling.
Refs: Forensics report finding F18
Why: TCP was the only transport that fired _disconnect_callback on a dead
transport or send failure. Serial and BLE returned silently, leaving
ConnectionManager unaware the link was down. Additionally, transport.write()
(TCP/serial) and write_gatt_char (BLE) had no try/except — any write-time
exception escaped the send() coroutine unhandled. This commit makes all three
transports symmetric: (a) fire _disconnect_callback when transport is None/dead,
(b) wrap the write call in try/except and fire _disconnect_callback on failure.
Refs: Forensics report findings F04, NEW-A
Why: send_appstart() only expected [SELF_INFO]. If firmware returned
RESP_CODE_ERR (version mismatch, unsupported feature flag), wait_for_event
never matched and the command hung until DEFAULT_TIMEOUT (5s) fired.
Bootstrap is called on every initial connect, so a 5s hang on error was
user-visible. Now expects [SELF_INFO, ERROR] so firmware errors are
returned immediately as Event objects instead of timing out.
Refs: Forensics report finding M02
Why: When send_msg() returned an ERROR event (e.g. firmware rejected
the send), the error-check logged the failure but did not return or
continue. Execution fell through to result.payload["expected_ack"],
which raised KeyError because the ERROR payload is {"reason": "..."}.
The retry loop — the entire purpose of this function — never ran.
Now the ERROR path increments attempt counters and continues the
loop, preserving the retry semantics the function name promises.
Refs: Forensics report findings F21, M01
Why: wait_for_event matches a single EventType; when callers pass
[X, ERROR] to send() or wait_for_events, the return value may be an
error response whose payload is {"reason": "..."} — not the command-
specific keys the caller expects. Without a documented contract and
a convenience helper, every call site independently forgets to check
.type before accessing payload keys, leading to KeyError (F21/M01,
M04) or silent fallthrough. The is_error() helper and docstrings on
send()/wait_for_events() establish the contract that subsequent
commits in this branch rely on.
Refs: Forensics report finding F22
ConnectionManager._attempt_reconnect called self.connection.connect()
directly, bypassing MeshCore.connect() which runs send_appstart().
Firmware requires CMD_APP_START after every transport-level connection
to initialize the session. Without it, the reconnected transport has
no active session — sends go unanswered, tcp_no_response fires after
5 attempts, handle_disconnect re-enters _attempt_reconnect, and the
reconnect storm begins.
Fix: add an optional reconnect_callback parameter to
ConnectionManager.__init__. MeshCore passes self._on_reconnect which
calls send_appstart() after the transport reconnects. The callback
is invoked inside _attempt_reconnect immediately after a successful
connect(), before the CONNECTED event is emitted. Callback failures
are logged as warnings but do not break the reconnect — the transport
is up regardless. Default None keeps the API backwards-compatible
for direct ConnectionManager users.
Refs: Forensics report finding F02
_attempt_reconnect previously tail-recursed via asyncio.create_task,
re-assigning self._reconnect_task from inside the running coroutine.
This orphaned the current task — disconnect() cancelled only the
newest pointer, leaving previous-generation attempts in flight. Those
orphaned tasks could set _is_connected = True after the caller thought
the session was closed.
Fix: replace with a single iterative while loop that holds one
persistent task for the entire reconnect session. The task is created
once in handle_disconnect and torn down only on success, max attempts
exhausted, or disconnect() cancellation.
Refs: Forensics report finding F03
The three transports had inconsistent connect() return contracts:
TCP returned an asyncio.Future (fixed by F01), serial returned
self.port (always truthy), BLE returned self.address or None.
ConnectionManager's success check `if result is not None:` was
tautological for TCP and serial. With F01 fixed, the check is now
meaningful for all three.
Add a comprehensive docstring to ConnectionProtocol documenting the
contract: return truthy on success (included in CONNECTED event
payload), return None for soft failure (retry), or raise for hard
failure (also retry, logged). Also import Awaitable for the F02
reconnect_callback type hint that follows.
Refs: Forensics report finding N11
TCPConnection.connect() returned a resolved asyncio.Future wrapping
self.host instead of the plain string. ConnectionManager put this
Future directly into the CONNECTED event payload, which crashed any
downstream serializer (e.g. HA recorder's sanitize_event_data) that
tried to walk the payload dict. BLE and serial already return plain
strings. Fix: delete the Future creation and return self.host
directly.
Refs: Forensics report finding F01
A BATTERY frame with len(data) < 3 caused dbuf.read(2) to return short
bytes; int.from_bytes(b"", ...) silently yielded 0, propagating a bogus
level=0 to HA sensors. Same silent-zero class as N07 (storage fields).
Option B: early-return with debug log, matching the NEW-C pattern for
STATUS_RESPONSE. No BATTERY event is dispatched for malformed frames.
Not in the original forensics report — discovered during G1 N07 work and
logged in issues_log.md. Resolved here because no later branch touches
this handler.
Files changed:
- src/meshcore/reader.py: add `if len(data) < 3: return` guard before
the level read in the BATTERY branch
- tests/unit/test_reader.py: add test_g1_battery_too_short_for_level —
sends a 1-byte frame (type only), asserts no BATTERY event dispatched
and debug log emitted
Three small cleanup fixes bundled per proposal §4.1 commit order.
N08 — CONTROL_DATA empty payload guard. The handler reads
`payload = dbuf.read()` then immediately dereferences `payload[0]`
without checking length. A zero-length payload (firmware truncation
or garbled frame) raises IndexError. Pre-F06 the IndexError would
escape; post-F06 it would log and skip the dispatch via the umbrella.
Adding an explicit `if len(payload) == 0: return` after the read
short-circuits the empty case before it touches `payload[0]`, with a
debug log noting the empty payload. The `return` exits handle_rx
cleanly without engaging the F06 umbrella's parse-error path, which
is the correct behavior — an empty CONTROL_DATA frame is not a parse
error, it's an unusable frame.
F12 — print(res) leftover debug. The RAW_DATA handler had a stray
`print(res)` polluting stdout. Replaced with `logger.debug(res)` to
match the surrounding `logger.debug("Received raw data")` line.
N10 — magic numbers 16 and 17. Two `elif packet_type_value == 16/17`
branches hardcoded the integer values for CONTACT_MSG_RECV_V3 and
CHANNEL_MSG_RECV_V3, both already declared in packets.py:94-95.
Replaced with `PacketType.CONTACT_MSG_RECV_V3.value` and
`PacketType.CHANNEL_MSG_RECV_V3.value` to eliminate drift risk if
the enum is ever renumbered.
Findings: N08 (Info), F12 (Info), N10 (Info)
File: src/meshcore/reader.py
`uncrypted[4:4]` is the empty slice. `int.from_bytes(b"", "little")`
returns 0, so `txt_type` was always 0 for every decrypted channel
message — silently masking the upper 6 bits of byte 4. The line
immediately above (`attempt = uncrypted[4] & 3`) already proves byte
4 is in range, so widening the slice to `[4:5]` is safe.
This is a one-character fix and changes the observable value of
`txt_type` for all callers. Existing consumers that branched on
`txt_type` were effectively dead code; this restores the intended
behavior.
Finding: R02 (Info)
File: src/meshcore/meshcore_parser.py
The ADVERT branch in MeshcorePacketParser.parsePacketPayload reads the
flags byte with `pk_buf.read(1)[0]`, which IndexErrors on a short
advert payload (the minimum advert is 32 + 4 + 64 + 1 = 101 bytes
before any optional fields). Pre-F06, the IndexError would escape as a
swallowed task exception. With F06's umbrella now in place it would
log and skip the dispatch, but the proposal §4.1 NEW-B asks for a
narrower local guard so a malformed advert doesn't poison the rest of
the parse path.
The optional `lat/lon/feat1/feat2` reads after the flags byte also
silently produce zeros on short reads (`int.from_bytes(b"", ...)`
returns 0), which would propagate bogus zero coordinates upstream.
Wrapping the whole branch limits the blast radius to a single
malformed advert.
Wrap the entire body of the ADVERT elif (from `pk_buf = io.BytesIO(...)`
through the final `log_data["adv_feat2"]` assignment) in
`try/except (IndexError, ValueError)` and log a debug message with the
exception type, message, and `pkt_payload` length on failure. This
matches the defensive pattern the proposal specifies.
Finding: NEW-B (S3)
File: src/meshcore/meshcore_parser.py
Add a 2-byte hard-floor length check at the top of
MeshcorePacketParser.parsePacketPayload. The minimum viable payload is 1
header byte + 1 path_byte for a direct route; anything shorter would
crash on `path_byte = pbuf.read(1)[0]` (IndexError on the empty buffer).
The reader.py LOG_DATA branch only requires `len(data) > 3`, so a 4-byte
LOG_DATA frame produces a 1-byte payload here — that path is reachable.
The caller in reader.py dereferences log_data['route_type'],
['payload_type'], ['path_len'], and ['path'] immediately after the
parse, so an empty log_data would KeyError on the dispatch (caught by
the F06 umbrella, but the dispatch would still be skipped). Populate
sentinel values matching the existing route_typename = "UNK" pattern in
the function before the early return so the caller's downstream lookups
don't KeyError, then return early with a debug log.
Finding: R01 (S2)
File: src/meshcore/meshcore_parser.py
Why: parse_status with offset=8 reads up through data[56:60]
(the rx_airtime field), so a full STATUS_RESPONSE push frame is
60 bytes: 1 type + 1 reserved + 6 pubkey + 52 status fields. The
push handler in handle_rx previously called parse_status with no
length check at all, so a short frame would slice through empty
data and silently produce zeros for every missing field. HA sensor
telemetry would silently report all-zero status — same class as N07.
The BINARY_RESPONSE STATUS path at the bottom of handle_rx already
gates parse_status with `len(response_data) >= 52` on its
offset-stripped buffer; this commit adds the equivalent gate for
the push path: `if len(data) < 60: log + return`. The `return`
short-circuits cleanly out of the umbrella try block without
dispatching a STATUS_RESPONSE event for the bogus parse.
Refs: Forensics report finding NEW-C (S3)
Why: The BATTERY handler previously gated the used_kb / total_kb
reads on `len(data) > 3`, which is wrong. The full
RESP_CODE_BATT_AND_STORAGE frame is 11 bytes (1 type + 2 level +
4 used_kb + 4 total_kb), so a 4-10 byte truncated frame would pass
the guard, and io.BytesIO.read(4) silently returns short bytes
instead of raising. int.from_bytes(b"", ...) returns 0, so HA
sensor telemetry silently reports zero storage on a truncated frame.
Tighten the guard to `len(data) >= 11` so the storage fields are
only parsed when the full frame is present. Inline comment added
to document the expected frame layout.
Note: the unconditional 2-byte `level` read at the top of the
handler has the same class of issue (no guard, silent zero on a
1-byte frame). That is out of scope for finding N07 and has been
logged in issues_log.md as a separate item.
Refs: Forensics report finding N07 (S3)
Why: The ALLOWED_REPEAT_FREQ branch in handle_rx had `except e:` —
syntactically valid Python only if `e` happens to be bound to an
exception class, which it isn't. The first time the inner read loop
actually raised, the except clause itself would raise NameError
("name 'e' is not defined") and propagate out of the handler. The
proposal correctly notes this is unreachable in practice today
because `int.from_bytes(b"", ...)` returns 0 so the loop terminates
cleanly, but it is a latent footgun. Replace with the standard
`except Exception as e:` form and swap the `print(e)` for a proper
`logger.warning(...)` call to match the rest of the file (which uses
the module logger, not stdout).
Refs: Forensics report finding F11 (S3)
