docs: ADRs 0005-0007 for socket protocol 2

ADR 0005: the web UI and Home Assistant talk to the device over one
websocket protocol (ops on loop(), sync with next, a JSON envelope a
binary encoding can reuse, short keys described by a schema).
ADR 0006: two security modes and eleven HTTP routes on port 80.
ADR 0007: the firmware no longer updates itself from GitHub.

ADR 0002 is amended: its single-writer rule stands, but edits now get
one reply and reads no longer go through a ModelSnapshot.
SECURITY_PLAN.md is revised to match, and keeps the token key, login
backoff, CORS and Origin work. PROTOCOL.md starts as an outline, and
GLOSSARY.md gains the socket protocol and security terms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
cjkas 2026-10-22 20:50:33 +02:00
parent 539a49eb77
commit e25747d395
7 changed files with 202 additions and 52 deletions

View file

@ -1,5 +1,5 @@
---
status: accepted
status: accepted, amended by ADR-0005
---
# The loop task is the only writer of the shade model
@ -13,6 +13,8 @@ The shade model (rooms, shades, groups and repeaters) is changed only from `loop
## Consequences
> **Amended by [ADR 0005](0005-one-websocket-protocol.md) (socket protocol 2).** The rule above stands. These consequences are replaced: model edits no longer answer "accepted" and report through `requestResult`, because every socket op gets one reply from `loop()`. Reads no longer go through a ModelSnapshot, because `loop()` formats a sync from the live model. The caps on reads and the UI's 5 s `requestResult` wait go with them.
- async_tcp must still never wait for `loop()`, so a model edit's HTTP reply can't block until the change is made. The reply only says the request was accepted; the result reaches clients as a WebSocket event once `loop()` has made the change. We chose that over a chunked HTTP response that `loop()` completes, because pushing results to clients is what the WebSocket is for and it needs no pool of pending responses.
- A read has to reply with data, so it does wait, without blocking: its chunked response sends nothing until `loop()` has filled the read's ModelSnapshot.
- MQTT commands already run on `loop()` (`mqtt.loop()` is called from it), but they go through the LoopQueue too. That gives them the radio back-pressure web commands have (`holdWebCommand`): while the radio is full they wait instead of being refused, and they run in the order they arrived with web requests.

View file

@ -0,0 +1,27 @@
---
status: accepted
---
# The web UI and Home Assistant talk to the device over one websocket protocol
The device was aborting while a browser loaded the web UI. Every open HTTP response holds its own TCP send window, the WiFi driver's copies of frames still on air, and a send buffer, which comes to about 14 KB each. A page load opens about ten of them. Six files with three answered at a time took 41 KB, a websocket connect about 10 KB more, and a `/controller` read about 22 KB with its ModelSnapshot. The idle heap is 80–89 KB. Over a weak WiFi link (-73 dBm, about 6 KB/s) these overlap for seconds until an allocation inside the web server fails and aborts.
We decided that the web UI and the Home Assistant integration do everything except the file transfers over one websocket (protocol 2). One connection means one send window per client, however much the client asks for. HTTP keeps only what a socket can't do; [ADR 0006](0006-two-security-modes-eleven-http-routes.md) lists what stays. The wire format is in [PROTOCOL.md](../PROTOCOL.md).
## Considered Options
- **Cap concurrent HTTP responses (a request gate) and budget the heap per response.** We built it and measured it, then dropped it. Each response still costs its full send window, a page load still opens ten requests, and the socket and `/controller` costs come on top of the gate's budget.
- **Keep HTTP, but make `/controller` smaller.** Rejected: it removes one cost and leaves the parallel connections.
- **Push the whole sync without flow control.** Rejected: after a reboot or a WiFi drop, every client reconnects within seconds. Five clients each queuing a full sync (about 20 KB at the model's maximum size) is more than the free heap.
## Consequences
- **Ops run on `loop()`.** async_tcp only parses an op and hands it over through the LoopQueue. `loop()` reads and writes the live shade model and sends the reply, so [ADR 0002](0002-loop-task-is-the-only-writer-of-the-shade-model.md)'s single-writer rule holds and no ModelSnapshot copy is needed. Long work, such as a WiFi scan, starts at once and reports through events, so `loop()` never stalls.
- **Every op has exactly one reply,** to its sender only, including model edits. `requestResult` and the edit's separate "accepted" step go. A command's reply carries no state; the movement arrives as events.
- **Sync on connect.** The device sends its schema (unless the client has it cached), then the shade model and status, in messages of at most 4 KB. Items are never split across messages. The client sends `next` after each message, and the sync ends with `syncEnd`. Each message is formatted from the live model, and events keep flowing during the sync, so the client upserts items by id and the result is exact without a copy. Settings aren't part of sync; a client asks for them when it needs them.
- **The send queue stays at 32 messages per client.** A full queue closes the client, which reconnects and syncs again. Dropping messages would leave a shade showing the wrong position with no way to notice.
- **A JSON envelope that a binary encoding can reuse.** Every frame is `{"v":2,"t":<type>,"id":<n>,"d":{...}}`, with `id` only on ops and replies. Message types and error codes are numbers that fit one byte and are never reused. A later binary encoding would put the same four fields in a fixed header, be asked for at the upgrade, and use binary websocket frames. The `42[...]` socket.io framing goes.
- **Short keys, described by a schema the device sends.** Payload keys are one or two characters from one global dictionary. Booleans are 1/0, and fields at their default are left out. The schema lists every type, field (name, kind, default) and error code. The firmware's table is the only copy, and its writers can only emit declared fields. `hello` carries the schema's hash, and a client that has it cached reconnects with `?schema=<hash>` to skip it. Clients expand frames to full names, so client code stays readable.
- **Feeds.** High-rate events (received frames, frequency scan) go only to clients that subscribed to that feed.
- **A clean break, released as 4.0.0.** The firmware speaks only protocol 2. The new HA integration falls back to the old HTTP API and events when a device doesn't send `hello`; old integrations stop working against 4.0.0. The web UI checks `v` and shows a banner when the firmware and the web application don't match.
- **Tests.** Native tests cover parsing, the op table, permissions, the sync stream and the largest item fitting in 4 KB. A Python e2e suite, run by hand against a device, checks the protocol and survival under a reconnect storm and a page-load replay, and records fixtures. The web UI's client layer is tested with `node --test`. The HA integration tests its protocol module with plain pytest and the integration with pytest-homeassistant-custom-component, against a fake device that replays the fixtures.

View file

@ -0,0 +1,32 @@
---
status: accepted
---
# Two security modes, and eleven HTTP routes on one server
With [ADR 0005](0005-one-websocket-protocol.md) moving everything else to the websocket, HTTP keeps only what has to happen before a socket exists, or moves a file. There is one server, on port 80, and the socket is served from it at `ws://host/ws`. Ports 8081 and 8080 are removed in 4.0.0 with no alias period, because that release breaks the HA integration contract anyway.
| Route | Why it stays HTTP |
|---|---|
| static web assets | the page has to load before it can open a socket |
| `/loginContext` | the security mode and device info, needed before login |
| `/login` | gets the token the socket needs |
| `/upnp.xml` | SSDP description (Windows' Network view, routers) |
| `/backup` | a file the browser saves |
| `/restore`, `/updateFirmware`, `/updateApplication` | file uploads |
| `/shadeCommand`, `/groupCommand`, `/tiltCommand` | one-line commands for scripts and Node-RED |
Static serving covers the web assets only, so `shades.cfg` and the other config files are never served. The script commands keep today's full field names, since people write those requests by hand. `/discovery` goes: Home Assistant's setup reads the device info from `/loginContext`, and mDNS still advertises `serverId`.
There are two security modes. **None** leaves everything open. **Secured** (PIN or password) needs the token on the socket upgrade (`?apikey=`) and on every route except the static files, `/loginContext`, `/login` and `/upnp.xml`. Devices set to "Secure Config Only" become Secured when they upgrade. The rest of the security work (the token key, the login backoff, CORS, the Origin check) is in [SECURITY_PLAN.md](../SECURITY_PLAN.md).
## Considered Options
- **Keep control and config access levels ("Secure Config Only").** Rejected: every op would need a level, and the mode was bypassed in practice, since nothing checked config routes for the config level.
- **Keep port 8081 for scripts.** Rejected: one server costs less memory, and scripts only change a port number.
- **Keep `/discovery` for scripts.** Rejected: it was the only HTTP read left of the shade model, and it alone kept the ModelSnapshot copy alive. A script that needs live state should use the socket.
## Consequences
- The release notes for 4.0.0 must cover: scripts move from port 8081 to 80, the HA integration must be updated, and "Secure Config Only" devices now ask for login on the main page.
- The `_espsomfy_rts._tcp` mDNS record, which Home Assistant's discovery matches, moves from port 8080 to 80 and gains the socket path as a TXT entry.

View file

@ -0,0 +1,17 @@
---
status: accepted
---
# The firmware no longer updates itself from GitHub
The device fetched GitHub's release list, downloaded firmware and filesystem images, and checked daily for a newer release, all over HTTPS on the device itself. A TLS session needs one contiguous block of about 35–40 KB. On IDF 5.5 the heap's largest free block is often smaller (issue #76: "SSL - Memory allocation failed" with a 34.8 KB largest block). The download checked no certificate (`setInsecure()`), so anyone who could intercept it could serve their own firmware. Parts of it ran blocking on the web server's task, and its socket events from there were dropped. The web UI's update buttons sent PUT to routes that accept only GET, so they had been answering 404.
We removed it: the release check and its setting, the release list, download and cancel, `/recoverFilesystem` and the `updateProgress` event. Updates are done by uploading `firmware.bin` and `littlefs.bin` (`/updateFirmware`, `/updateApplication`), or over USB.
## Consequences
- Users download releases from GitHub themselves. The web UI's firmware page shows the installed version and the upload form.
- Home Assistant's update entity shows the installed version only; it can no longer install.
- A broken filesystem is recovered by uploading `littlefs.bin`, or over USB, rather than with `/recoverFilesystem`.
- Issue #76 and PR #72 (GitOTA's per-chip littlefs file name) become obsolete.
- The HTTPS client leaves the firmware, which frees flash in the 1.5 MB OTA slot.