Merge pull request #99 from cjkas/feat/socket-protocol-2

feat: socket protocol 2 core on ws://host/ws

.gitignore

@@ -35,3 +35,7 @@ SomfyController.ino.esp32s2.bin
 # Temporary and Backup Files
 *.orig
 *.bak
+
+# Python (the e2e suite and build scripts)
+__pycache__/
+.pytest_cache/

docs/PROTOCOL.md

@@ -1,51 +1,89 @@
 # Socket protocol 2
 
-**Status:** outline. The type numbers, field keys and payloads are filled in with the firmware protocol core. Why the protocol looks like this: [ADR 0005](adr/0005-one-websocket-protocol.md). Terms: [GLOSSARY.md](../GLOSSARY.md).
+Why the protocol looks like this: [ADR 0005](adr/0005-one-websocket-protocol.md). Terms: [GLOSSARY.md](../GLOSSARY.md).
+
+**The table of types, fields, error codes and feeds is defined once, in [src/ProtoSchema.h](../src/ProtoSchema.h).** The device sends it to every client as the schema, and `test/fixtures/proto2/schema.json` is a copy recorded by the e2e suite (`pytest test/e2e --record`). This page describes how the protocol works, not every field.
 
 ## Connection
 
-- `ws://host/ws` on port 80. Under Secured the upgrade needs `?apikey=<token>` ([ADR 0006](adr/0006-two-security-modes-eleven-http-routes.md)).
+- `ws://host/ws` on port 80. Under Secured the upgrade needs `?apikey=<token>` from `/login`; without a valid one it is refused with 401 ([ADR 0006](adr/0006-two-security-modes-eleven-http-routes.md)).
-- Optional `?schema=<hash>`: the client already has that schema and doesn't need it again.
+- Optional `?schema=<hash>`: the client already has the schema with that hash and doesn't need it again.
-- Text frames only. A binary encoding, if one comes, would be asked for at the upgrade (for example `?enc=bin`) and use binary frames.
+- At most 5 clients. A sixth is closed as soon as it connects.
+- Text frames only, and each message in one frame: a fragmented message is answered with one bad request, without an `id` since it couldn't be read. Until the device puts messages together from their pieces, that includes a frame larger than one TCP segment (about 1.4 KB), which also reaches it in pieces. A binary encoding, if one comes, would be asked for at the upgrade (for example `?enc=bin`) and use binary frames.
 
 ## Envelope
 
 Every frame, in both directions, is one JSON object:
 
 ```json
-{"v":2,"t":20,"id":17,"d":{}}
+{"v":2,"t":129,"id":17,"d":{"s":3,"p":40}}
 ```
 
 | Key | Meaning |
 |---|---|
-| `v` | protocol version |
+| `v` | protocol version, 2 |
-| `t` | message type, a number from the table below |
+| `t` | message type, a number from the schema |
-| `id` | only on an op and its reply: the client's number for the op |
+| `id` | only on a client message that wants a reply, and on that reply |
-| `d` | the payload, with short keys from the schema |
+| `d` | the payload, with keys from the schema's field dictionary |
 
-Within `d`, booleans are `1`/`0` and fields at their default are left out. Keys are one or two characters from one dictionary shared by every type.
+Within `d`:
+- keys are one or two characters, from one dictionary shared by every type;
+- booleans are `1` and `0`;
+- a field at its default is left out: a client fills it in from the schema;
+- lists and objects use the same dictionary for the keys inside them.
+
+The keys of `hello` (`fw`, `sh`) and of the schema itself (`T`, `F`, `E`, `Q`) never change, since a client needs them before it has a schema.
 
 ## Message types
 
-| Range | Sender | Kinds |
+| Range | Sender | Types |
 |---|---|---|
-| 1–15 | either | control: `hello`, `reply`, `next`, `sync`, `syncEnd`, `subscribe`, `unsubscribe` |
+| 1–15 | control | `hello` 1, `reply` 2, `next` 3, `sync` 4, `syncEnd` 5, `subscribe` 6, `unsubscribe` 7 |
-| 16–127 | client | ops: the commands, reads, model edits and `getSettings`/`saveSettings` |
+| 16–127 | client ops | none yet; commands, model edits and settings follow |
-| 128–255 | device | events: `shadeState`, `shade`, `group`, `room`, `*Removed`, `settingsChanged`, status, feeds |
+| 128–255 | device events | `shadeState`, `shade`, `shadeRemoved`, `group`, `groupState`, `groupRemoved`, `room`, `roomRemoved`, `roomOrder`, `shadeOrder`, `groupOrder`, `shadeCommand`, `memory`, `wifi`, `ethernet`, `frame`, `frequencyScan`, `repeaters` |
 
 A number is never reused. A retired type stays reserved.
 
 ## Connect sequence
 
-1. The device sends `hello`: protocol version, firmware version, schema hash.
+1. The device sends `hello`: `{"fw":"<firmware>","sh":"<schema hash>"}`.
-2. Unless the client passed the matching `?schema=`, the sync starts with the schema: types, fields (key, name, kind, default) and error codes.
+2. The device sends the first `sync` frame straight away. Unless the client passed the matching `?schema=`, the sync starts with the schema:
-3. Sync: header, rooms, shades, groups, repeaters, status. Each message is at most 4 KB, items are never split, and the client sends `next` after each one.
+   - `T`: types, as `[number, name]`
-4. `syncEnd`. From then on, events keep the client current. Events also arrive during the sync, so the client upserts by id.
+   - `F`: fields, as `[key, name, kind, default]`, kind being `b` bool, `i` integer, `n` number, `s` string, `l` list, `o` object
+   - `E`: error codes, as `[number, name]`
+   - `Q`: feeds, as `[number, name]`
+3. Then the device: `header` (limits, starting address, what a new shade gets, server id, hostname, model, chip, firmware, connection type), `rooms`, `shades`, `groups`, `repeaters` and `status` (memory, and WiFi or Ethernet). A frame is at most 4 KB, an item is never split across frames, and several sections can share one.
+4. The client sends `next` after each `sync` frame, and the device answers with the following one. With no `next` for 30 s the device closes the socket.
+5. `syncEnd` follows the last `sync` frame.
+
+The client can send `sync` at any time to get the shade model and status again (without the schema).
+
+Events go out to every client from the moment it connects, so they can arrive during the sync, and even before the schema. A client keeps any frame it can't decode yet until it has the schema. Each sync frame is formatted from the live model, so an item in a later frame already holds every change before it. A client treats sync items and events alike: insert or update by id, and ignore removing an id it doesn't have.
+
+## Events
+
+| Event | When | Payload |
+|---|---|---|
+| `shadeState` | a shade moved or its state changed | id, position, target, direction, favorite, tilt, flags, rolling code |
+| `shade` | a shade was added or a setting changed (edit, pairing, linked remote, group, its room deleted) | the whole shade, as in the sync |
+| `group`, `groupState` | the same for a group: its settings or linked shades, or its flags and direction | |
+| `room` | a room was added or saved | the whole room |
+| `*Removed` | a shade, group or room was deleted | its id |
+| `roomOrder`, `shadeOrder`, `groupOrder` | the sort order changed | `is`: the ids in order |
+| `repeaters` | a repeater was linked or unlinked | the list |
+| `shadeCommand` | a remote or the device commanded a shade | id, address, command, source, rolling code, source address |
+| `memory`, `wifi`, `ethernet` | every 30 s, WiFi only when the signal moved | |
 
 ## Replies
 
-Every op gets exactly one reply, to its sender only: `{"ok":1, ...}`, or `{"ok":0,"code":<n>,"desc":"..."}`. The error codes are bad request, unknown op, unauthorized, not found, busy, radio unavailable and internal; their numbers are in the schema.
+A client message with an `id` gets exactly one `reply` with that `id`, to that client only, whether it is an op or a control message such as `next`: `ok` 1 on success, or an error `code` and a `desc` for people. Error codes are in the schema: bad request, unknown op, unauthorized, not found, busy, radio unavailable, internal.
+
+A malformed message gets a bad request, with its `id` when it had one. A type a client may not send gets an unknown op.
 
 ## Feeds
 
-`subscribe`/`unsubscribe` with a feed: `frames` (received remote frames) or `frequencyScan`. Feed events go only to subscribed clients.
+`subscribe` and `unsubscribe` take a feed number in `fe`: `frames` (received remote frames, with their pulses) or `frequencyScan`. Their events go only to clients subscribed to them.
+
+## Inside the device
+
+async_tcp reads client frames and answers with fixed frames that hold nothing of the shade model: `hello`, errors and `ok`. Everything formatted from the shade model goes out from `loop()`, which is the only task that reads and writes it ([ADR 0002](adr/0002-loop-task-is-the-only-writer-of-the-shade-model.md)). Sync frames and events share the socket event buffer, so protocol 2 needs no more static memory.

docs/SECURITY_PLAN.md

@@ -74,12 +74,15 @@ Protocol 2 lands as one PR per step, and `main` keeps working throughout because
 
 | # | Repo | Scope |
 |---|---|---|
-| 1 | firmware | protocol core next to the existing HTTP: `hello`, socket auth, ops through the LoopQueue, sync, events, schema; the e2e suite starts |
+| 1a | firmware | protocol core next to the existing HTTP: `hello`, socket auth, sync, events, feeds, schema; the e2e suite starts |
+| 1b | firmware | command ops through the LoopQueue, with replies from `loop()` |
+| 1c | firmware | model edit ops, replacing `requestResult`; messages reassembled from their pieces |
+| 1d | firmware | settings ops (`getSettings`/`saveSettings` by section), the WiFi and frequency scans, reboot |
 | 2 | firmware | web UI on ops; the CSS files bundled; token in `sessionStorage`; banner on a version mismatch |
 | 3 | HA | protocol 2 with the old-firmware fallback; `/loginContext` setup; runtime login and reauth |
-| 4 | firmware | removals: the old routes, ports 8081 and 8080, `/discovery`, ModelSnapshot, the GitHub update, "Secure Config Only"; static serving limited to web assets; CORS headers; Origin check |
+| 4 | firmware | removals: the old routes, ports 8081 and 8080, `/discovery`, ModelSnapshot, "Secure Config Only"; static serving limited to web assets; CORS headers; Origin check |
 
-PR B (token key) and PR C (login backoff) are separate firmware PRs that can land at any point.
+PR B (token key) and PR C (login backoff) are separate firmware PRs that can land at any point. The GitHub update was removed ahead of the rest ([ADR 0007](adr/0007-no-github-update.md)), as protocol 2 didn't fit beside it.
 
 ## Tests
 

platformio.ini

@@ -108,7 +108,9 @@ build_flags =
 platform = native
 test_framework = unity
 test_build_src = yes
-build_src_filter = -<*> +<Utils.cpp> +<SomfyFrame.cpp> +<JsonFormatter.cpp> +<ShadeMotion.cpp> +<ShadeSensors.cpp> +<ShadeBehaviour.cpp> +<PulseDecoder.cpp> +<TxBuffer.cpp> +<RadioScheduler.cpp> +<ShadeRequests.cpp> +<entryPoints/MqttCommands.cpp> +<ModelSnapshot.cpp> +<ModelEditor.cpp> +<ShadeChanges.cpp>
+build_src_filter = -<*> +<Utils.cpp> +<SomfyFrame.cpp> +<JsonFormatter.cpp> +<ShadeMotion.cpp> +<ShadeSensors.cpp> +<ShadeBehaviour.cpp> +<PulseDecoder.cpp> +<TxBuffer.cpp> +<RadioScheduler.cpp> +<ShadeRequests.cpp> +<entryPoints/MqttCommands.cpp> +<ModelSnapshot.cpp> +<ModelEditor.cpp> +<ShadeChanges.cpp> +<ProtoSchema.cpp> +<ProtoFrame.cpp> +<ProtoSync.cpp> +<ProtoParse.cpp> +<ProtoSlots.cpp>
+lib_deps =
+	bblanchon/ArduinoJson@^7.2.2
 build_flags =
 	-std=gnu++17
 	-Isrc

src/ESPNetwork.cpp

@@ -7,6 +7,7 @@
 #include "ESPNetwork.h"
 #include "entryPoints/Web.h"
 #include "outputs/SocketEmitter.h"
+#include "outputs/ProtoSocket.h"
 #include "Utils.h"
 #include "SSDP.h"
 #include "MqttConnection.h"
@@ -223,6 +224,9 @@ void ESPNetwork::emitSockets(uint8_t num) {
   }
   // A broadcast comes from emitSockets(), which has already sent memStatus.
   if(num != 255) this->emitHeap(num);
+  // Protocol 2 clients get the status in their sync, so only broadcasts.
+  else if(this->connType == conn_types_t::ethernet) protoSocket.ethernet();
+  else protoSocket.wifi();
 }
 // Runs on the WiFi/Ethernet event task, so it sends nothing on the sockets.
 // loop() broadcasts the new status and a client that reconnects gets it from
@@ -690,5 +694,8 @@ void ESPNetwork::emitHeap(uint8_t num) {
   json->addElem("loopMaxUs", loopStats.maxUs());
   json->endObject();
   sockEmit.endEmit(num);
-  if(num == 255) loopStats.publish(millis());
+  if(num == 255) {
+    protoSocket.memory();
+    loopStats.publish(millis());
+  }
 }

src/ESPNetwork.h

@@ -38,6 +38,12 @@ class ESPNetwork {
     bool connecting();
     void clearConnecting();
     conn_types_t preferredConnType();
+    // How the device is connected, as clients are told: Ethernet, Wifi or Unknown.
+    const char *connTypeName() const {
+      if(this->connType == conn_types_t::ethernet) return "Ethernet";
+      if(this->connType == conn_types_t::wifi) return "Wifi";
+      return "Unknown";
+    }
     String ssid;
     String mac;
     int channel;

src/JsonFormatter.cpp

@@ -27,6 +27,14 @@ void JsonFormatter::discard() {
   this->_objects = 0;
   this->_arrays = 0;
 }
+void JsonFormatter::rollback(const json_mark_t &m) {
+  this->_len = m.len;
+  if(this->buff) this->buff[this->_len] = 0;
+  this->_objects = m.objects;
+  this->_arrays = m.arrays;
+  this->_nocomma = m.nocomma;
+  this->_overflow = false;
+}
 void JsonFormatter::beginObject(const char *name) {
   if(name && strlen(name) > 0) this->appendElem(name);
   else if(!this->_nocomma) this->_safecat(",");
@@ -101,7 +109,7 @@ void JsonFormatter::addElem(const char *name, bool bval) { strcpy(this->_numbuff
 void JsonFormatter::_safecat(const char *val, bool escape) {
   const size_t vlen = escape ? this->calcEscapedLength(val) + 2 : strlen(val);
   if(this->_len + vlen >= this->buffSize) {
-    if(!this->_overflow) ESP_LOGE(TAG, "JSON exceeded buffer size %u", (unsigned)this->buffSize);
+    if(!this->_overflow && this->_logOverflow) ESP_LOGE(TAG, "JSON exceeded buffer size %u", (unsigned)this->buffSize);
     this->_overflow = true;
     return;
   }

src/JsonFormatter.h

@@ -2,6 +2,15 @@
 #define json_formatter_h
 #include <Arduino.h>
 
+// Where a formatter's output stands, to go back to when what follows doesn't
+// fit (JsonFormatter::mark()).
+struct json_mark_t {
+  size_t len;
+  uint8_t objects;
+  uint8_t arrays;
+  bool nocomma;
+};
+
 // Streams JSON into a fixed buffer.  Free of the web server so it can be unit
 // tested on the PC (pio test -e native).
 class JsonFormatter {
@@ -16,6 +25,8 @@ class JsonFormatter {
     bool _nocomma = true;
     // Something didn't fit and was left out so the JSON is incomplete.
     bool _overflow = false;
+    // Whether running out of room is logged.
+    bool _logOverflow = true;
     char _numbuff[25] = {0};
     virtual void _safecat(const char *val, bool escape = false);
     // Appends val at buff + _len.  vlen is what it takes, including the quotes
@@ -24,6 +35,12 @@ class JsonFormatter {
     void _appendNumber(const char *name);
   public:
     bool overflowed() const { return _overflow; }
+    json_mark_t mark() const { return {_len, _objects, _arrays, _nocomma}; }
+    // Drops everything written since m, an overflow included.
+    void rollback(const json_mark_t &m);
+    // A writer that tries an item and rolls back when it doesn't fit turns the
+    // overflow log off, as running out of room is expected there.
+    void logOverflow(bool on) { _logOverflow = on; }
     // Drops whatever is written until a buffer is set again.  Nothing is
     // written anywhere, so tasks can share one discarding formatter.
     void discard();

src/ModelEditor.cpp

@@ -1,5 +1,44 @@
 #include "ModelEditor.h"
 
+edit_changes_t editChanges(model_edit_type_t type, const edit_result_t &result) {
+  edit_changes_t c;
+  switch(type) {
+    case model_edit_type_t::add_shade:
+    case model_edit_type_t::save_shade:
+    case model_edit_type_t::set_paired:
+    case model_edit_type_t::link_remote:
+    case model_edit_type_t::unlink_remote:
+      c.shadeId = result.id;
+      break;
+    case model_edit_type_t::add_group:
+    case model_edit_type_t::save_group:
+      c.groupId = result.id;
+      break;
+    // The shade's inGroup goes with it.
+    case model_edit_type_t::link_to_group:
+    case model_edit_type_t::unlink_from_group:
+      c.groupId = result.id;
+      c.shadeId = result.shadeId;
+      break;
+    case model_edit_type_t::add_room:
+    case model_edit_type_t::save_room:
+      c.roomId = result.id;
+      break;
+    case model_edit_type_t::room_order: c.order = sort_list_t::rooms; break;
+    case model_edit_type_t::shade_order: c.order = sort_list_t::shades; break;
+    case model_edit_type_t::group_order: c.order = sort_list_t::groups; break;
+    case model_edit_type_t::link_repeater:
+    case model_edit_type_t::unlink_repeater:
+      c.repeaters = true;
+      break;
+    case model_edit_type_t::delete_room:
+    case model_edit_type_t::delete_shade:
+    case model_edit_type_t::delete_group:
+      break;
+  }
+  return c;
+}
+
 uint8_t firstFreeId(const uint8_t *usedIds, size_t count, uint8_t lastId) {
   for(uint16_t id = 1; id <= lastId; id++) {
     bool used = false;

src/ModelEditor.h

@@ -24,11 +24,13 @@ enum class model_edit_type_t : uint8_t {
   room_order, shade_order, group_order
 };
 // What an edit did, sent to the clients as requestResult.  id is the room,
-// shade or group it added or changed, 0 when none.  code is the validation
+// shade or group it added or changed, 0 when none.  shadeId is the shade a
-// error of a shade that wasn't saved, 0 otherwise.
+// group gained or lost.  code is the validation error of a shade that wasn't
+// saved, 0 otherwise.
 struct edit_result_t {
   bool ok = true;
   uint8_t id = 0;
+  uint8_t shadeId = 0;
   int8_t code = 0;
   const char *desc = "";
   static edit_result_t error(const char *desc, int8_t code = 0) {
@@ -40,6 +42,20 @@ struct edit_result_t {
   }
 };
 
+// The lists an edit can put in a new order.
+enum class sort_list_t : uint8_t { none, rooms, shades, groups };
+// What protocol 2 clients are sent once an edit has gone through: the whole
+// items it changed (0 for none), a list's new order, the repeaters.  Deletes
+// send their removals as they happen, so they need nothing here.
+struct edit_changes_t {
+  uint8_t shadeId = 0;
+  uint8_t groupId = 0;
+  uint8_t roomId = 0;
+  sort_list_t order = sort_list_t::none;
+  bool repeaters = false;
+};
+edit_changes_t editChanges(model_edit_type_t type, const edit_result_t &result);
+
 // The lowest id from 1 to lastId that isn't in usedIds, or 0 when every one
 // is taken.
 uint8_t firstFreeId(const uint8_t *usedIds, size_t count, uint8_t lastId);

src/ProtoFrame.cpp

@@ -0,0 +1,33 @@
+#include "ProtoFrame.h"
+
+void beginFrame(JsonFormatter &json, proto_type_t type, uint32_t id) {
+  json.beginObject();
+  json.addElem("v", (uint8_t)PROTO_VERSION);
+  json.addElem("t", static_cast<uint8_t>(type));
+  if(id) json.addElem("id", id);
+  json.beginObject("d");
+}
+void endFrame(JsonFormatter &json) {
+  json.endObject();
+  json.endObject();
+}
+void writeOkReply(JsonFormatter &json, uint32_t id) {
+  beginFrame(json, proto_type_t::reply, id);
+  ProtoWriter(json).add(proto_field_t::ok, true);
+  endFrame(json);
+}
+void writeErrorReply(JsonFormatter &json, uint32_t id, proto_error_t code, const char *desc) {
+  beginFrame(json, proto_type_t::reply, id);
+  ProtoWriter w(json);
+  w.add(proto_field_t::ok, false);
+  w.add(proto_field_t::code, static_cast<uint8_t>(code));
+  w.add(proto_field_t::desc, desc);
+  endFrame(json);
+}
+void writeHello(JsonFormatter &json, const char *fw) {
+  beginFrame(json, proto_type_t::hello);
+  ProtoWriter w(json);
+  w.add(proto_field_t::fw, fw);
+  w.add(proto_field_t::schemaHash, protoSchemaHash());
+  endFrame(json);
+}

src/ProtoFrame.h

@@ -0,0 +1,20 @@
+#ifndef PROTO_FRAME_H
+#define PROTO_FRAME_H
+#include <stdint.h>
+#include "JsonFormatter.h"
+#include "ProtoSchema.h"
+
+// The envelope of every socket protocol 2 frame, and the frames that hold
+// nothing of the shade model (docs/PROTOCOL.md).  Kept free of the shade model
+// and the web server so it can be unit tested on the PC (pio test -e native).
+
+// A frame: {"v":2,"t":<type>,"id":<id>,"d":{ ... }}.  id 0 leaves it out.
+void beginFrame(JsonFormatter &json, proto_type_t type, uint32_t id = 0);
+void endFrame(JsonFormatter &json);
+// A reply that says ok.
+void writeOkReply(JsonFormatter &json, uint32_t id);
+// A reply carrying an error.
+void writeErrorReply(JsonFormatter &json, uint32_t id, proto_error_t code, const char *desc);
+// hello: the firmware version and the schema hash.
+void writeHello(JsonFormatter &json, const char *fw);
+#endif

src/ProtoParse.cpp

@@ -0,0 +1,36 @@
+#include "ProtoParse.h"
+
+proto_error_t parseClientFrame(const char *text, size_t len, JsonDocument &doc, proto_message_t &out) {
+  out = proto_message_t();
+  if(len > PROTO_MAX_CLIENT_FRAME) return proto_error_t::badRequest;
+  if(deserializeJson(doc, text, len) != DeserializationError::Ok) return proto_error_t::badRequest;
+  if(!doc.is<JsonObject>()) return proto_error_t::badRequest;
+  JsonVariantConst id = doc["id"];
+  if(id.is<uint32_t>()) out.id = id.as<uint32_t>();
+  else if(!id.isNull()) return proto_error_t::badRequest;
+  if(!doc["v"].is<int>() || doc["v"].as<int>() != PROTO_VERSION) return proto_error_t::badRequest;
+  JsonVariantConst t = doc["t"];
+  if(!t.is<int>() || t.as<int>() < 1 || t.as<int>() > 255) return proto_error_t::badRequest;
+  out.type = static_cast<uint8_t>(t.as<int>());
+  if(!protoClientMaySend(out.type)) return proto_error_t::unknownOp;
+  if(protoIsOp(out.type) && out.id == 0) return proto_error_t::badRequest;
+  JsonVariantConst d = doc["d"];
+  if(!d.isNull() && !d.is<JsonObjectConst>()) return proto_error_t::badRequest;
+  return proto_error_t::none;
+}
+
+int32_t protoInt(JsonObjectConst d, proto_field_t field) {
+  const proto_field_info_t &f = protoField(field);
+  JsonVariantConst v = d[f.key];
+  return v.is<int32_t>() ? v.as<int32_t>() : f.def;
+}
+bool protoBool(JsonObjectConst d, proto_field_t field) {
+  const proto_field_info_t &f = protoField(field);
+  JsonVariantConst v = d[f.key];
+  return v.is<int>() ? v.as<int>() != 0 : f.def != 0;
+}
+const char *protoString(JsonObjectConst d, proto_field_t field) {
+  JsonVariantConst v = d[protoField(field).key];
+  return v.is<const char *>() ? v.as<const char *>() : nullptr;
+}
+bool protoHas(JsonObjectConst d, proto_field_t field) { return !d[protoField(field).key].isNull(); }

src/ProtoParse.h

@@ -0,0 +1,32 @@
+#ifndef PROTO_PARSE_H
+#define PROTO_PARSE_H
+#include <stddef.h>
+#include <stdint.h>
+#include <ArduinoJson.h>
+#include "ProtoSchema.h"
+
+// Reading the frames a socket protocol 2 client sends (docs/PROTOCOL.md).
+// Kept free of the web server so it can be unit tested on the PC
+// (pio test -e native).
+
+// The largest frame a client may send; a longer one is refused unread.
+#define PROTO_MAX_CLIENT_FRAME 4096
+
+struct proto_message_t {
+  uint8_t type = 0;
+  uint32_t id = 0;  // 0 when the frame had none
+};
+// Reads one frame into doc and out.  none when it is a message a client may
+// send; otherwise the error to reply with.  out.id is set whenever the frame
+// had one, so the reply can name it.  The payload is doc["d"].
+proto_error_t parseClientFrame(const char *text, size_t len, JsonDocument &doc, proto_message_t &out);
+// An op (16-127) as opposed to a control message.
+inline bool protoIsOp(uint8_t type) { return type >= 16 && type <= 127; }
+
+// A field of a payload, or its default when it was left out.
+int32_t protoInt(JsonObjectConst d, proto_field_t field);
+bool protoBool(JsonObjectConst d, proto_field_t field);
+// nullptr when it was left out, as a string's default is "".
+const char *protoString(JsonObjectConst d, proto_field_t field);
+bool protoHas(JsonObjectConst d, proto_field_t field);
+#endif

src/ProtoSchema.cpp

@@ -0,0 +1,79 @@
+#include <stdio.h>
+#include <string.h>
+#include "ProtoSchema.h"
+
+#define PROTO_FIELD_ROW(name, key, kind, def) {#name, key, kind, def},
+static const proto_field_info_t FIELDS[] = { PROTO_FIELDS(PROTO_FIELD_ROW) };
+#undef PROTO_FIELD_ROW
+// Errors and feeds have no sender, so it is left empty and starts as 0.
+#define PROTO_NAMED_ROW(name, number, ...) {number, #name, PROTO_SENDER(__VA_ARGS__, 0)},
+#define PROTO_SENDER(sender, ...) sender
+static const proto_named_t TYPES[] = { PROTO_TYPES(PROTO_NAMED_ROW) };
+static const proto_named_t ERRORS[] = { PROTO_ERRORS(PROTO_NAMED_ROW) };
+static const proto_named_t FEEDS[] = { PROTO_FEEDS(PROTO_NAMED_ROW) };
+#undef PROTO_SENDER
+#undef PROTO_NAMED_ROW
+static_assert(sizeof(FIELDS) / sizeof(FIELDS[0]) == static_cast<size_t>(proto_field_t::count), "a field without a row");
+
+const proto_field_info_t &protoField(proto_field_t field) { return FIELDS[static_cast<uint8_t>(field)]; }
+const proto_field_info_t *protoFieldByKey(const char *key) {
+  for(const proto_field_info_t &f : FIELDS)
+    if(strcmp(f.key, key) == 0) return &f;
+  return nullptr;
+}
+uint8_t protoTypeCount() { return sizeof(TYPES) / sizeof(TYPES[0]); }
+const proto_named_t &protoTypeAt(uint8_t index) { return TYPES[index]; }
+uint8_t protoErrorCount() { return sizeof(ERRORS) / sizeof(ERRORS[0]); }
+const proto_named_t &protoErrorAt(uint8_t index) { return ERRORS[index]; }
+uint8_t protoFeedCount() { return sizeof(FEEDS) / sizeof(FEEDS[0]); }
+const proto_named_t &protoFeedAt(uint8_t index) { return FEEDS[index]; }
+const char *protoTypeName(uint8_t number) {
+  for(const proto_named_t &t : TYPES)
+    if(t.number == number) return t.name;
+  return nullptr;
+}
+bool protoClientMaySend(uint8_t type) {
+  for(const proto_named_t &t : TYPES)
+    if(t.number == type) return t.sender == 'c' || t.sender == 'b';
+  return false;
+}
+
+static uint32_t fnv(uint32_t hash, const char *text) {
+  for(; *text; text++) {
+    hash ^= (uint8_t)*text;
+    hash *= 16777619u;
+  }
+  return hash;
+}
+static uint32_t fnvNamed(uint32_t hash, const proto_named_t &row) {
+  char num[12];
+  snprintf(num, sizeof(num), "%u:", row.number);
+  hash = fnv(hash, num);
+  hash = fnv(hash, row.name);
+  return fnv(hash, ";");
+}
+const char *protoSchemaHash() {
+  static char hex[9] = {0};
+  if(hex[0]) return hex;
+  uint32_t hash = 2166136261u;
+  for(const proto_named_t &t : TYPES) hash = fnvNamed(hash, t);
+  for(const proto_field_info_t &f : FIELDS) {
+    char row[64];
+    snprintf(row, sizeof(row), "%s:%s:%c:%ld;", f.name, f.key, f.kind, (long)f.def);
+    hash = fnv(hash, row);
+  }
+  for(const proto_named_t &e : ERRORS) hash = fnvNamed(hash, e);
+  for(const proto_named_t &q : FEEDS) hash = fnvNamed(hash, q);
+  snprintf(hex, sizeof(hex), "%08lx", (unsigned long)hash);
+  return hex;
+}
+
+void ProtoWriter::addInt(const proto_field_info_t &f, int64_t value) {
+  if(value == f.def) return;
+  if(value >= INT32_MIN && value <= INT32_MAX) this->json.addElem(f.key, (int32_t)value);
+  else this->json.addElem(f.key, (uint32_t)value);
+}
+void ProtoWriter::item(int64_t value) {
+  if(value >= INT32_MIN && value <= INT32_MAX) this->json.addElem((int32_t)value);
+  else this->json.addElem((uint32_t)value);
+}

src/ProtoSchema.h

@@ -0,0 +1,239 @@
+#ifndef PROTO_SCHEMA_H
+#define PROTO_SCHEMA_H
+#include <stdint.h>
+#include <type_traits>
+#include "JsonFormatter.h"
+
+// Socket protocol 2 (docs/PROTOCOL.md, docs/adr/0005-one-websocket-protocol.md):
+// the one table of message types, fields, error codes and feeds.  The device
+// sends it to clients as the schema, so it is the only copy anywhere.  Writers
+// name fields by proto_field_t, so a field that isn't declared here can't be
+// sent.  Kept free of the shade model and the web server so it can be unit
+// tested on the PC (pio test -e native).
+
+#define PROTO_VERSION 2
+
+// Message types: 1-15 control, 16-127 client ops, 128-255 device events.  A
+// number is never reused; a retired type stays reserved.
+// TYPE(name, number, sender): sender is 'c' client, 'd' device, 'b' both.
+#define PROTO_TYPES(TYPE) \
+  TYPE(hello, 1, 'd') \
+  TYPE(reply, 2, 'd') \
+  TYPE(next, 3, 'c') \
+  TYPE(sync, 4, 'b') \
+  TYPE(syncEnd, 5, 'd') \
+  TYPE(subscribe, 6, 'c') \
+  TYPE(unsubscribe, 7, 'c') \
+  TYPE(shadeState, 128, 'd') \
+  TYPE(shade, 129, 'd') \
+  TYPE(shadeRemoved, 130, 'd') \
+  TYPE(group, 131, 'd') \
+  TYPE(groupState, 132, 'd') \
+  TYPE(groupRemoved, 133, 'd') \
+  TYPE(room, 134, 'd') \
+  TYPE(roomRemoved, 135, 'd') \
+  TYPE(roomOrder, 136, 'd') \
+  TYPE(shadeOrder, 137, 'd') \
+  TYPE(groupOrder, 138, 'd') \
+  TYPE(shadeCommand, 139, 'd') \
+  TYPE(memory, 140, 'd') \
+  TYPE(wifi, 141, 'd') \
+  TYPE(ethernet, 142, 'd') \
+  TYPE(frame, 143, 'd') \
+  TYPE(frequencyScan, 144, 'd') \
+  TYPE(repeaters, 145, 'd')
+
+// Fields, in one dictionary for every type.  FIELD(name, key, kind, default):
+// kind is b bool (sent as 1/0), i integer, n number with a fraction, s string,
+// l list, o object.  A field at its default is left out; a string's default is
+// "" and a list's is empty.  hello's and the schema's own keys (fw, sh, T, F,
+// E, Q) are fixed, as a client needs them before it has the schema.
+#define PROTO_FIELDS(FIELD) \
+  /* the schema and the sync */ \
+  FIELD(types, "T", 'l', 0) \
+  FIELD(fields, "F", 'l', 0) \
+  FIELD(errors, "E", 'l', 0) \
+  FIELD(feeds, "Q", 'l', 0) \
+  FIELD(header, "h", 'o', 0) \
+  FIELD(rooms, "R", 'l', 0) \
+  FIELD(shades, "S", 'l', 0) \
+  FIELD(groups, "G", 'l', 0) \
+  FIELD(repeaters, "X", 'l', 0) \
+  FIELD(status, "Z", 'o', 0) \
+  FIELD(fw, "fw", 's', 0) \
+  FIELD(schemaHash, "sh", 's', 0) \
+  /* replies and control */ \
+  FIELD(ok, "k", 'b', 0) \
+  FIELD(code, "e", 'i', 0) \
+  FIELD(desc, "x", 's', 0) \
+  FIELD(feed, "fe", 'i', 0) \
+  FIELD(ids, "is", 'l', 0) \
+  /* rooms, shades and groups */ \
+  FIELD(shadeId, "s", 'i', 0) \
+  FIELD(groupId, "g", 'i', 0) \
+  FIELD(roomId, "r", 'i', 0) \
+  FIELD(name, "n", 's', 0) \
+  FIELD(remoteAddress, "a", 'i', 0) \
+  FIELD(lastRollingCode, "c", 'i', 0) \
+  FIELD(sortOrder, "o", 'i', 0) \
+  FIELD(flags, "f", 'i', 0) \
+  FIELD(position, "p", 'i', 0) \
+  FIELD(target, "t", 'i', 0) \
+  FIELD(direction, "d", 'i', 0) \
+  FIELD(myPos, "m", 'i', -1) \
+  FIELD(tiltType, "y", 'i', 0) \
+  FIELD(tiltPosition, "q", 'i', 0) \
+  FIELD(tiltTarget, "u", 'i', 0) \
+  FIELD(tiltDirection, "w", 'i', 0) \
+  FIELD(myTiltPos, "M", 'i', -1) \
+  FIELD(upTime, "U", 'i', 0) \
+  FIELD(downTime, "D", 'i', 0) \
+  FIELD(tiltTime, "Y", 'i', 0) \
+  FIELD(stepSize, "z", 'i', 100) \
+  FIELD(paired, "i", 'b', 0) \
+  FIELD(shadeType, "st", 'i', 0) \
+  FIELD(bitLength, "bl", 'i', 56) \
+  FIELD(proto, "pr", 'i', 0) \
+  FIELD(flipCommands, "fc", 'b', 0) \
+  FIELD(flipPosition, "fp", 'b', 0) \
+  FIELD(inGroup, "ig", 'b', 0) \
+  FIELD(sunSensor, "ss", 'b', 0) \
+  FIELD(light, "li", 'b', 0) \
+  FIELD(simMy, "sm", 'b', 0) \
+  FIELD(repeats, "rp", 'i', 1) \
+  FIELD(gpioUp, "gu", 'i', 0) \
+  FIELD(gpioDown, "gd", 'i', 0) \
+  FIELD(gpioMy, "gm", 'i', 0) \
+  FIELD(gpioLLTrigger, "gl", 'b', 0) \
+  FIELD(linkedRemotes, "lk", 'l', 0) \
+  FIELD(linkedShades, "ls", 'l', 0) \
+  /* shadeCommand */ \
+  FIELD(command, "cm", 's', 0) \
+  FIELD(source, "so", 's', 0) \
+  FIELD(sourceAddress, "sr", 'i', 0) \
+  /* the header */ \
+  FIELD(maxRooms, "xr", 'i', 0) \
+  FIELD(maxShades, "xs", 'i', 0) \
+  FIELD(maxGroups, "xg", 'i', 0) \
+  FIELD(maxGroupedShades, "xm", 'i', 0) \
+  FIELD(maxLinkedRemotes, "xl", 'i', 0) \
+  FIELD(startingAddress, "sa", 'i', 0) \
+  FIELD(serverId, "sv", 's', 0) \
+  FIELD(hostname, "hn", 's', 0) \
+  FIELD(model, "md", 's', 0) \
+  FIELD(chipModel, "cp", 's', 0) \
+  FIELD(connType, "ct", 's', 0) \
+  /* memory, wifi and ethernet */ \
+  FIELD(freeHeap, "fr", 'i', 0) \
+  FIELD(maxHeap, "mx", 'i', 0) \
+  FIELD(minHeap, "mn", 'i', 0) \
+  FIELD(totalHeap, "to", 'i', 0) \
+  FIELD(uptime, "up", 'i', 0) \
+  FIELD(loopRate, "lp", 'i', 0) \
+  FIELD(loopMaxUs, "lx", 'i', 0) \
+  FIELD(ssid, "sd", 's', 0) \
+  FIELD(strength, "sg", 'i', 0) \
+  FIELD(channel, "ch", 'i', 0) \
+  FIELD(connected, "co", 'b', 0) \
+  FIELD(speed, "sp", 'i', 0) \
+  FIELD(fullDuplex, "fd", 'b', 0) \
+  /* the frame and frequency scan feeds */ \
+  FIELD(encKey, "ek", 'i', 0) \
+  FIELD(rawCmd, "rw", 'i', 0) \
+  FIELD(rssi, "rs", 'i', 0) \
+  FIELD(bits, "bt", 'i', 0) \
+  FIELD(valid, "va", 'b', 0) \
+  FIELD(hwsync, "hs", 'i', 0) \
+  FIELD(pulses, "pu", 'l', 0) \
+  FIELD(scanning, "sc", 'b', 0) \
+  FIELD(testFreq, "tf", 'n', 0) \
+  FIELD(testRssi, "tr", 'i', 0) \
+  FIELD(frequency, "fq", 'n', 0)
+
+// Error codes in a reply.  ERR(name, number)
+#define PROTO_ERRORS(ERR) \
+  ERR(badRequest, 1) \
+  ERR(unknownOp, 2) \
+  ERR(unauthorized, 3) \
+  ERR(notFound, 4) \
+  ERR(busy, 5) \
+  ERR(radioUnavailable, 6) \
+  ERR(internal, 7)
+
+// Feeds a client can subscribe to.  FEED(name, number)
+#define PROTO_FEEDS(FEED) \
+  FEED(frames, 1) \
+  FEED(frequencyScan, 2)
+
+#define PROTO_ENUM_ENTRY(name, number, ...) name = number,
+enum class proto_type_t : uint8_t { PROTO_TYPES(PROTO_ENUM_ENTRY) };
+enum class proto_error_t : uint8_t { none = 0, PROTO_ERRORS(PROTO_ENUM_ENTRY) };
+enum class proto_feed_t : uint8_t { PROTO_FEEDS(PROTO_ENUM_ENTRY) };
+#undef PROTO_ENUM_ENTRY
+#define PROTO_FIELD_ENTRY(name, ...) name,
+enum class proto_field_t : uint8_t { PROTO_FIELDS(PROTO_FIELD_ENTRY) count };
+#undef PROTO_FIELD_ENTRY
+
+struct proto_field_info_t {
+  const char *name;
+  const char *key;
+  char kind;
+  int32_t def;
+};
+struct proto_named_t {
+  uint8_t number;
+  const char *name;
+  char sender;  // types only
+};
+const proto_field_info_t &protoField(proto_field_t field);
+const proto_field_info_t *protoFieldByKey(const char *key);
+// The tables as the schema lists them, in declaration order.
+uint8_t protoTypeCount();
+const proto_named_t &protoTypeAt(uint8_t index);
+uint8_t protoErrorCount();
+const proto_named_t &protoErrorAt(uint8_t index);
+uint8_t protoFeedCount();
+const proto_named_t &protoFeedAt(uint8_t index);
+const char *protoTypeName(uint8_t number);
+// What a client may send: control messages it sends and the ops.
+bool protoClientMaySend(uint8_t type);
+// Hex FNV-1a of every row of the schema: 8 characters and a terminator.
+const char *protoSchemaHash();
+
+// Writes fields by their short keys into the open object of json: booleans as
+// 1/0, and nothing at all for a value at the field's default.
+class ProtoWriter {
+  public:
+    explicit ProtoWriter(JsonFormatter &json) : json(json) {}
+    template<typename T> void add(proto_field_t field, T value) {
+      const proto_field_info_t &f = protoField(field);
+      if constexpr(std::is_same<T, bool>::value) {
+        if((value ? 1 : 0) == f.def) return;
+        this->json.addElem(f.key, (uint8_t)(value ? 1 : 0));
+      }
+      else if constexpr(std::is_floating_point<T>::value) {
+        if(value == (T)f.def) return;
+        this->json.addElem(f.key, (float)value);
+      }
+      else if constexpr(std::is_integral<T>::value) {
+        this->addInt(f, (int64_t)value);
+      }
+      else {
+        const char *s = value;
+        if(!s || !s[0]) return;
+        this->json.addElem(f.key, s);
+      }
+    }
+    void beginList(proto_field_t field) { this->json.beginArray(protoField(field).key); }
+    void endList() { this->json.endArray(); }
+    void beginObject(proto_field_t field) { this->json.beginObject(protoField(field).key); }
+    void endObject() { this->json.endObject(); }
+    // An object or a value inside a list.
+    void beginItem() { this->json.beginObject(); }
+    void endItem() { this->json.endObject(); }
+    void item(int64_t value);
+    JsonFormatter &json;
+  private:
+    void addInt(const proto_field_info_t &f, int64_t value);
+};
+#endif

src/ProtoSlots.cpp

@@ -0,0 +1,62 @@
+#include "ProtoSlots.h"
+
+proto_slot_t *claimSlot(proto_slot_t *slots, uint8_t count, uint32_t clientId, bool withSchema) {
+  for(uint8_t i = 0; i < count; i++) {
+    uint32_t expected = 0;
+    if(!slots[i].clientId.compare_exchange_strong(expected, clientId)) continue;
+    slots[i].withSchema = withSchema;
+    slots[i].feeds = 0;
+    return &slots[i];
+  }
+  return nullptr;
+}
+void startSync(proto_slot_t &slot) { slot.wants.fetch_or(WANTS_SYNC); }
+proto_slot_t *slotOf(proto_slot_t *slots, uint8_t count, uint32_t clientId) {
+  for(uint8_t i = 0; i < count; i++)
+    if(slots[i].clientId.load() == clientId) return &slots[i];
+  return nullptr;
+}
+void freeSlot(proto_slot_t &slot) {
+  // Cleared before the slot is free, so the next client starts clean.
+  slot.wants = 0;
+  slot.feeds = 0;
+  slot.clientId = 0;
+}
+
+uint32_t takeWants(proto_slot_t &slot, uint32_t seenId) {
+  const uint32_t wants = slot.wants.exchange(0);
+  if(slot.clientId.load() == seenId) return wants;
+  // The client left and another took the slot after loop() read it: what was
+  // taken may be the new client's, so it goes back for the next pass.  A bit
+  // the old client left behind does no harm, as the new one wants a sync.
+  if(wants) slot.wants.fetch_or(wants);
+  return 0;
+}
+sync_step_t nextSyncStep(proto_slot_t &slot, uint32_t now) {
+  const uint32_t id = slot.clientId.load();
+  if(id != slot.loopId) {
+    // A client left or another took the slot.
+    slot.loopId = id;
+    slot.cur = sync_cursor_t();
+    slot.awaitingNext = false;
+  }
+  if(!id) return sync_step_t::none;
+  const uint32_t wants = takeWants(slot, id);
+  if(wants & (WANTS_SYNC | WANTS_SYNC_AGAIN)) {
+    // A client that just connected may skip the schema; one that asks again
+    // never gets it.
+    slot.cur = beginSync((wants & WANTS_SYNC) && slot.withSchema.load());
+    return sync_step_t::send;
+  }
+  if((wants & WANTS_NEXT) && slot.awaitingNext) return sync_step_t::send;
+  if(slot.awaitingNext && now - slot.sentAt > PROTO_SYNC_TIMEOUT_MS) {
+    slot.awaitingNext = false;
+    slot.cur = sync_cursor_t();
+    return sync_step_t::timedOut;
+  }
+  return sync_step_t::none;
+}
+void syncFrameSent(proto_slot_t &slot, uint32_t now) {
+  slot.awaitingNext = slot.cur.active();
+  slot.sentAt = now;
+}

src/ProtoSlots.h

@@ -0,0 +1,59 @@
+#ifndef PROTO_SLOTS_H
+#define PROTO_SLOTS_H
+#include <stdint.h>
+#include <atomic>
+#include "ProtoSync.h"
+
+// The protocol 2 clients and their syncs (docs/PROTOCOL.md).  async_tcp
+// claims and frees a slot and records what its client asked for; loop() moves
+// each client's sync on.  The two sides share only the atomics.  Kept free of
+// the web server so it can be unit tested on the PC (pio test -e native).
+
+#define PROTO_MAX_CLIENTS 5
+// How long a client may take to ask for the next sync frame.
+#define PROTO_SYNC_TIMEOUT_MS 30000
+
+// What a client asked loop() for, as bits.
+enum : uint32_t {
+  WANTS_SYNC = 0x01,        // just connected: a sync, with the schema unless it has it
+  WANTS_NEXT = 0x02,        // the next sync frame
+  WANTS_SYNC_AGAIN = 0x04   // a sync without the schema
+};
+struct proto_slot_t {
+  // Set and cleared on async_tcp; clientId is 0 when the slot is free.
+  std::atomic<uint32_t> clientId{0};
+  std::atomic<uint32_t> wants{0};
+  std::atomic<uint8_t> feeds{0};     // a bit per feed number
+  std::atomic<bool> withSchema{true};
+  // loop() only.
+  uint32_t loopId = 0;               // the client loop() is serving here
+  sync_cursor_t cur;
+  bool awaitingNext = false;
+  uint32_t sentAt = 0;
+};
+
+// async_tcp: the slot a new client takes, or nullptr when every slot is
+// taken.
+proto_slot_t *claimSlot(proto_slot_t *slots, uint8_t count, uint32_t clientId, bool withSchema);
+// async_tcp: hello has gone out, so loop() may start the client's sync.
+void startSync(proto_slot_t &slot);
+proto_slot_t *slotOf(proto_slot_t *slots, uint8_t count, uint32_t clientId);
+// async_tcp: the client has gone.
+void freeSlot(proto_slot_t &slot);
+
+// What loop() does for a slot on this pass.
+enum class sync_step_t : uint8_t {
+  none,
+  send,      // send the sync frame at slot.cur to slot.loopId
+  timedOut   // close slot.loopId: it sent no next in time
+};
+// loop(): picks up a new client, takes what it asked for and moves its sync
+// on.  When the step is send, slot.cur is where the frame starts.
+sync_step_t nextSyncStep(proto_slot_t &slot, uint32_t now);
+// loop(): a sync frame went out, and more follow while slot.cur is active.
+void syncFrameSent(proto_slot_t &slot, uint32_t now);
+// loop(): what the slot's client asked for, provided it is still the client
+// seenId that loop() read.  When another client took the slot in between,
+// what it asked for stays for the next pass and 0 is returned.
+uint32_t takeWants(proto_slot_t &slot, uint32_t seenId);
+#endif

src/ProtoSync.cpp

@@ -0,0 +1,249 @@
+#include "esp_log.h"
+#include "ProtoSync.h"
+#include "ProtoFrame.h"
+
+static const char *TAG = "Proto";
+
+/*********************************************************************
+ * Items
+ ********************************************************************/
+void protoRoom(ProtoWriter &w, const room_snapshot_t &room) {
+  w.add(proto_field_t::roomId, room.roomId);
+  w.add(proto_field_t::name, room.name);
+  w.add(proto_field_t::sortOrder, room.sortOrder);
+}
+void protoShadeState(ProtoWriter &w, const shade_snapshot_t &shade) {
+  w.add(proto_field_t::shadeId, shade.shadeId);
+  w.add(proto_field_t::position, shade.position);
+  w.add(proto_field_t::target, shade.target);
+  w.add(proto_field_t::direction, shade.direction);
+  w.add(proto_field_t::myPos, shade.myPos);
+  w.add(proto_field_t::tiltPosition, shade.tiltPosition);
+  w.add(proto_field_t::tiltTarget, shade.tiltTarget);
+  w.add(proto_field_t::tiltDirection, shade.tiltDirection);
+  w.add(proto_field_t::myTiltPos, shade.myTiltPos);
+  w.add(proto_field_t::flags, shade.flags);
+  w.add(proto_field_t::lastRollingCode, shade.lastRollingCode);
+}
+void protoShade(ProtoWriter &w, const shade_snapshot_t &shade) {
+  protoShadeState(w, shade);
+  w.add(proto_field_t::roomId, shade.roomId);
+  w.add(proto_field_t::name, shade.name);
+  w.add(proto_field_t::remoteAddress, shade.remoteAddress);
+  w.add(proto_field_t::upTime, shade.upTime);
+  w.add(proto_field_t::downTime, shade.downTime);
+  w.add(proto_field_t::tiltTime, shade.tiltTime);
+  w.add(proto_field_t::stepSize, shade.stepSize);
+  w.add(proto_field_t::paired, shade.paired);
+  w.add(proto_field_t::tiltType, shade.tiltType);
+  w.add(proto_field_t::shadeType, shade.shadeType);
+  w.add(proto_field_t::bitLength, shade.bitLength);
+  w.add(proto_field_t::proto, shade.proto);
+  w.add(proto_field_t::flipCommands, shade.flipCommands);
+  w.add(proto_field_t::flipPosition, shade.flipPosition);
+  w.add(proto_field_t::inGroup, shade.inGroup);
+  w.add(proto_field_t::sunSensor, shade.sunSensor);
+  w.add(proto_field_t::light, shade.light);
+  w.add(proto_field_t::simMy, shade.simMy);
+  w.add(proto_field_t::repeats, shade.repeats);
+  w.add(proto_field_t::sortOrder, shade.sortOrder);
+  w.add(proto_field_t::gpioUp, shade.gpioUp);
+  w.add(proto_field_t::gpioDown, shade.gpioDown);
+  w.add(proto_field_t::gpioMy, shade.gpioMy);
+  w.add(proto_field_t::gpioLLTrigger, shade.gpioLLTrigger);
+  if(shade.linkedRemoteCount == 0) return;
+  w.beginList(proto_field_t::linkedRemotes);
+  for(uint8_t i = 0; i < shade.linkedRemoteCount; i++) {
+    w.beginItem();
+    w.add(proto_field_t::remoteAddress, shade.linkedRemotes[i].remoteAddress);
+    w.add(proto_field_t::lastRollingCode, shade.linkedRemotes[i].lastRollingCode);
+    w.endItem();
+  }
+  w.endList();
+}
+void protoGroup(ProtoWriter &w, const group_snapshot_t &group) {
+  w.add(proto_field_t::groupId, group.groupId);
+  w.add(proto_field_t::roomId, group.roomId);
+  w.add(proto_field_t::name, group.name);
+  w.add(proto_field_t::remoteAddress, group.remoteAddress);
+  w.add(proto_field_t::lastRollingCode, group.lastRollingCode);
+  w.add(proto_field_t::bitLength, group.bitLength);
+  w.add(proto_field_t::proto, group.proto);
+  w.add(proto_field_t::sunSensor, group.sunSensor);
+  w.add(proto_field_t::flipCommands, group.flipCommands);
+  w.add(proto_field_t::flags, group.flags);
+  w.add(proto_field_t::repeats, group.repeats);
+  w.add(proto_field_t::sortOrder, group.sortOrder);
+  if(group.linkedShadeCount == 0) return;
+  w.beginList(proto_field_t::linkedShades);
+  for(uint8_t i = 0; i < group.linkedShadeCount; i++) w.item(group.linkedShades[i]);
+  w.endList();
+}
+void protoGroupState(ProtoWriter &w, uint8_t groupId, uint8_t flags, int8_t direction) {
+  w.add(proto_field_t::groupId, groupId);
+  w.add(proto_field_t::flags, flags);
+  w.add(proto_field_t::direction, direction);
+}
+
+/*********************************************************************
+ * Sync
+ ********************************************************************/
+// What closing a frame cut short takes: the list's ], the }} of d and the
+// frame, and the terminator.
+#define SYNC_CLOSE_RESERVE 4
+
+sync_cursor_t beginSync(bool withSchema) {
+  sync_cursor_t cur;
+  cur.section = withSchema ? sync_section_t::types : sync_section_t::header;
+  cur.index = 0;
+  return cur;
+}
+
+static proto_field_t sectionField(sync_section_t section) {
+  switch(section) {
+    case sync_section_t::types: return proto_field_t::types;
+    case sync_section_t::fields: return proto_field_t::fields;
+    case sync_section_t::errors: return proto_field_t::errors;
+    case sync_section_t::feeds: return proto_field_t::feeds;
+    case sync_section_t::header: return proto_field_t::header;
+    case sync_section_t::rooms: return proto_field_t::rooms;
+    case sync_section_t::shades: return proto_field_t::shades;
+    case sync_section_t::groups: return proto_field_t::groups;
+    case sync_section_t::repeaters: return proto_field_t::repeaters;
+    default: return proto_field_t::status;
+  }
+}
+static uint8_t sectionSlots(sync_section_t section) {
+  switch(section) {
+    case sync_section_t::types: return protoTypeCount();
+    case sync_section_t::fields: return static_cast<uint8_t>(proto_field_t::count);
+    case sync_section_t::errors: return protoErrorCount();
+    case sync_section_t::feeds: return protoFeedCount();
+    case sync_section_t::rooms: return SOMFY_MAX_ROOMS;
+    case sync_section_t::shades: return SOMFY_MAX_SHADES;
+    case sync_section_t::groups: return SOMFY_MAX_GROUPS;
+    case sync_section_t::repeaters: return SOMFY_MAX_REPEATERS;
+    default: return 0;
+  }
+}
+static void namedRow(JsonFormatter &json, const proto_named_t &row) {
+  json.beginArray();
+  json.addElem(row.number);
+  json.addElem(row.name);
+  json.endArray();
+}
+// Writes the item in slot of a list section.  False when the slot is empty.
+static bool writeItem(SyncSource &src, ProtoWriter &w, sync_section_t section, uint8_t slot) {
+  JsonFormatter &json = w.json;
+  switch(section) {
+    case sync_section_t::types: namedRow(json, protoTypeAt(slot)); return true;
+    case sync_section_t::errors: namedRow(json, protoErrorAt(slot)); return true;
+    case sync_section_t::feeds: namedRow(json, protoFeedAt(slot)); return true;
+    case sync_section_t::fields: {
+      const proto_field_info_t &f = protoField(static_cast<proto_field_t>(slot));
+      const char kind[2] = {f.kind, '\0'};
+      json.beginArray();
+      json.addElem(f.key);
+      json.addElem(f.name);
+      json.addElem(kind);
+      json.addElem(f.def);
+      json.endArray();
+      return true;
+    }
+    case sync_section_t::rooms: {
+      room_snapshot_t room;
+      if(!src.room(slot, room)) return false;
+      w.beginItem();
+      protoRoom(w, room);
+      w.endItem();
+      return true;
+    }
+    case sync_section_t::shades: {
+      shade_snapshot_t shade;
+      if(!src.shade(slot, shade)) return false;
+      w.beginItem();
+      protoShade(w, shade);
+      w.endItem();
+      return true;
+    }
+    case sync_section_t::groups: {
+      group_snapshot_t group;
+      if(!src.group(slot, group)) return false;
+      w.beginItem();
+      protoGroup(w, group);
+      w.endItem();
+      return true;
+    }
+    case sync_section_t::repeaters: {
+      uint32_t address = 0;
+      if(!src.repeater(slot, address)) return false;
+      w.item(address);
+      return true;
+    }
+    default:
+      return false;
+  }
+}
+static void nextSection(sync_cursor_t &cur) {
+  cur.section = static_cast<sync_section_t>(static_cast<uint8_t>(cur.section) + 1);
+  cur.index = 0;
+}
+
+size_t writeSyncFrame(SyncSource &src, sync_cursor_t &cur, char *buff, size_t size) {
+  if(!cur.active()) return 0;
+  BufferedJsonFormatter json;
+  json.setBuffer(buff, size);
+  json.logOverflow(false);
+  ProtoWriter w(json);
+  beginFrame(json, proto_type_t::sync);
+  bool any = false;
+  auto fits = [&json, size]() { return !json.overflowed() && json.length() + SYNC_CLOSE_RESERVE < size; };
+  while(cur.active()) {
+    if(cur.section == sync_section_t::header || cur.section == sync_section_t::status) {
+      const json_mark_t m = json.mark();
+      w.beginObject(sectionField(cur.section));
+      if(cur.section == sync_section_t::header) src.header(w);
+      else src.status(w);
+      w.endObject();
+      if(!fits()) {
+        json.rollback(m);
+        // Only fixed size values, so this can't happen short of a bug.
+        if(!any) { ESP_LOGE(TAG, "Sync section %u doesn't fit in %u bytes", static_cast<uint8_t>(cur.section), (unsigned)size); nextSection(cur); continue; }
+        break;
+      }
+      any = true;
+      nextSection(cur);
+      continue;
+    }
+    bool open = false;
+    bool full = false;
+    const uint8_t slots = sectionSlots(cur.section);
+    while(cur.index < slots) {
+      const json_mark_t m = json.mark();
+      if(!open) w.beginList(sectionField(cur.section));
+      if(!writeItem(src, w, cur.section, cur.index)) {
+        json.rollback(m);
+        cur.index++;
+        continue;
+      }
+      if(!fits()) {
+        json.rollback(m);
+        if(!any) {
+          ESP_LOGE(TAG, "Sync item %u of section %u doesn't fit in %u bytes, leaving it out", cur.index, static_cast<uint8_t>(cur.section), (unsigned)size);
+          cur.index++;
+          continue;
+        }
+        full = true;
+        break;
+      }
+      open = true;
+      any = true;
+      cur.index++;
+    }
+    if(open) w.endList();
+    if(full) break;
+    nextSection(cur);
+  }
+  endFrame(json);
+  return json.length();
+}

src/ProtoSync.h

@@ -0,0 +1,61 @@
+#ifndef PROTO_SYNC_H
+#define PROTO_SYNC_H
+#include <stddef.h>
+#include <stdint.h>
+#include "ModelSnapshot.h"
+#include "ProtoSchema.h"
+
+// The shade model in socket protocol 2: how each room, shade and group is
+// written, and the sync that sends a client all of them (docs/PROTOCOL.md).
+// Kept free of the shade model and the web server so it can be unit tested on
+// the PC (pio test -e native).
+
+// The largest frame: what a sync message, or any event, is formatted into.
+#define PROTO_MAX_FRAME 4096
+
+// The fields of one item, into the object the caller opened.
+void protoRoom(ProtoWriter &w, const room_snapshot_t &room);
+// Everything about a shade: its settings and its state.
+void protoShade(ProtoWriter &w, const shade_snapshot_t &shade);
+// Only what changes while a shade moves or is commanded.
+void protoShadeState(ProtoWriter &w, const shade_snapshot_t &shade);
+// A group, with the ids of its shades.
+void protoGroup(ProtoWriter &w, const group_snapshot_t &group);
+void protoGroupState(ProtoWriter &w, uint8_t groupId, uint8_t flags, int8_t direction);
+
+// Where a sync gets the shade model from: the live model on the device, a
+// filled snapshot in the tests.  Slots run from 0 to the SOMFY_MAX_* of their
+// kind; an empty slot returns false.
+class SyncSource {
+  public:
+    virtual ~SyncSource() {}
+    virtual bool room(uint8_t slot, room_snapshot_t &out) = 0;
+    virtual bool shade(uint8_t slot, shade_snapshot_t &out) = 0;
+    virtual bool group(uint8_t slot, group_snapshot_t &out) = 0;
+    virtual bool repeater(uint8_t slot, uint32_t &address) = 0;
+    // The fields of the header and status objects.
+    virtual void header(ProtoWriter &w) = 0;
+    virtual void status(ProtoWriter &w) = 0;
+};
+
+// The parts of a sync, in the order they are sent.
+enum class sync_section_t : uint8_t {
+  types, fields, errors, feeds,                         // the schema
+  header, rooms, shades, groups, repeaters, status,     // the device
+  done
+};
+struct sync_cursor_t {
+  sync_section_t section = sync_section_t::done;
+  uint8_t index = 0;
+  bool active() const { return this->section != sync_section_t::done; }
+};
+// A sync from the start: with the schema, or from the header when the client
+// already has it.
+sync_cursor_t beginSync(bool withSchema);
+// Writes the next sync frame into buff: as many whole items as fit in size
+// bytes, never one split in two.  An item too big for an empty frame is left
+// out.  Returns the frame's length, or 0 once the sync is done and syncEnd is
+// due.  Each call reads the source afresh, so a sync shows each item as it is
+// when it goes out, not one moment.
+size_t writeSyncFrame(SyncSource &src, sync_cursor_t &cur, char *buff, size_t size);
+#endif

src/ShadeChanges.h

@@ -22,14 +22,19 @@ enum : uint16_t {
   SHADE_CHANGED_SUNNY          = 0x0400,
   SHADE_CHANGED_WINDY          = 0x0800,
   // Something only the socket's shadeState carries changed.
-  SHADE_CHANGED_STATE          = 0x1000
+  SHADE_CHANGED_STATE          = 0x1000,
+  // A setting changed (an edit, pairing, its room).  Protocol 2 clients get the
+  // whole shade rather than its state.
+  SHADE_CHANGED_CONFIG         = 0x2000
 };
 // The bits that have an MQTT topic.
 #define SHADE_CHANGED_TOPICS 0x0FFF
 // The same for a group: its direction topic, and groupState for the socket.
 enum : uint8_t {
   GROUP_CHANGED_DIRECTION = 0x01,
-  GROUP_CHANGED_STATE     = 0x02
+  GROUP_CHANGED_STATE     = 0x02,
+  // A setting or its shades changed: protocol 2 clients get the whole group.
+  GROUP_CHANGED_CONFIG    = 0x04
 };
 
 // Publishes one MQTT topic: its name under shades/<id>/, its value and whether

src/ShadeController.cpp

@@ -5,6 +5,8 @@
 #include "Somfy.h"
 #include "ConfigFile.h"
 #include "ModelEditor.h"
+#include "ShadeChanges.h"
+#include "outputs/ProtoSocket.h"
 
 static const char *TAG = "Somfy";
 
@@ -280,10 +282,15 @@ bool SomfyShadeController::deleteShade(uint8_t shadeId) {
   for(uint8_t i = 0; i < SOMFY_MAX_SHADES; i++) {
     if(this->shades[i].getShadeId() == shadeId) {
       shades[i].emitState("shadeRemoved");
+      protoSocket.shadeRemoved(shadeId);
       shades[i].unpublish();
       this->shades[i].clear();
     }
   }
+  // Its groups lose it, or a new shade given its id would join them.
+  for(uint8_t i = 0; i < SOMFY_MAX_GROUPS; i++) {
+    if(this->groups[i].getGroupId() != 255 && this->groups[i].unlinkShade(shadeId)) this->groups[i].configChanged();
+  }
   this->commit();
   return true;
 }
@@ -295,15 +302,18 @@ bool SomfyShadeController::deleteRoom(uint8_t roomId) {
         if(shades[j].roomId == roomId) {
           shades[j].roomId = 0;
           shades[j].emitState();
+          shades[j].configChanged();
         }
       }
       for(uint8_t j = 0; j < SOMFY_MAX_GROUPS; j++) {
         if(groups[j].roomId == roomId) {
           groups[j].roomId = 0;
           groups[j].emitState();
+          groups[j].configChanged();
         }
       }
       rooms[i].emitState("roomRemoved");
+      protoSocket.roomRemoved(roomId);
       this->rooms[i].clear();
     }
   }
@@ -313,7 +323,13 @@ bool SomfyShadeController::deleteRoom(uint8_t roomId) {
 bool SomfyShadeController::deleteGroup(uint8_t groupId) {
   for(uint8_t i = 0; i < SOMFY_MAX_GROUPS; i++) {
     if(this->groups[i].getGroupId() == groupId) {
+      // Its shades are no longer in a group, which their inGroup shows.
+      for(uint8_t j = 0; j < SOMFY_MAX_GROUPED_SHADES && groups[i].linkedShades[j] != 0; j++) {
+        SomfyShade *shade = this->getShadeById(groups[i].linkedShades[j]);
+        if(shade) shade->configChanged();
+      }
       groups[i].emitState("groupRemoved");
+      protoSocket.groupRemoved(groupId);
       groups[i].unpublish();
       this->groups[i].clear();
     }

src/Somfy.h

@@ -140,6 +140,8 @@ class SomfyShade : public SomfyRemote, public ShadeSettings, private MotorLink,
     void emitState(uint8_t num, const char *evt = "shadeState");
     // What changed since the fan-out last sent the shade (SHADE_CHANGED_* bits).
     uint16_t changed = 0;
+    // A setting changed: protocol 2 clients get the whole shade.
+    void configChanged() { this->changed |= SHADE_CHANGED_CONFIG; }
     void emitCommand(somfy_commands cmd, const char *source, uint32_t sourceAddress, const char *evt = "shadeCommand");
     void emitCommand(uint8_t num, somfy_commands cmd, const char *source, uint32_t sourceAddress, const char *evt = "shadeCommand");
     // False when the radio refused a command it sent.
@@ -215,6 +217,8 @@ class SomfyGroup : public SomfyRemote {
     void emitState(uint8_t num, const char *evt = "groupState");
     // What changed since the fan-out last sent the group (GROUP_CHANGED_* bits).
     uint8_t changed = 0;
+    // A setting or its shades changed: protocol 2 clients get the whole group.
+    void configChanged() { this->changed |= GROUP_CHANGED_CONFIG; }
     bool sendCommand(somfy_commands cmd);
     bool sendCommand(somfy_commands cmd, uint8_t repeat, uint8_t stepSize = 0);
     int8_t p_direction(int8_t dir);

src/SomfyController.cpp

@@ -6,6 +6,7 @@
 #include "ESPNetwork.h"
 #include "entryPoints/Web.h"
 #include "outputs/SocketEmitter.h"
+#include "outputs/ProtoSocket.h"
 #include "Utils.h"
 #include "Somfy.h"
 #include "MqttConnection.h"
@@ -187,6 +188,7 @@ void loop() {
     logIfSlow(TAG, "Loop web server", timing.lap(), LOOP_SLOW_MS);
     esp_task_wdt_reset();
     sockEmit.loop();
+    protoSocket.loop();
     logIfSlow(TAG, "Loop sockets", timing.lap(), LOOP_SLOW_MS);
     esp_task_wdt_reset();
   }

src/SomfyJson.cpp

@@ -4,6 +4,8 @@
 #include <atomic>
 #include "ModelEditor.h"
 #include "SomfyEditor.h"
+#include "ShadeChanges.h"
+#include "outputs/ProtoSocket.h"
 
 static const char *TAG = "Somfy";
 
@@ -226,9 +228,24 @@ void cancelModelEdit(model_edit_t *edit) {
   delete edit;
   editBusy.store(false);
 }
+// What protocol 2 clients get after an edit (editChanges).  A shade or group
+// goes out whole from the change fan-out, by its config bit, so it is sent
+// once with whatever else changed; the rest goes out now.
+static void markEdited(const model_edit_t &edit, const edit_result_t &result) {
+  const edit_changes_t c = editChanges(edit.type, result);
+  SomfyShade *shade = c.shadeId ? somfy.getShadeById(c.shadeId) : nullptr;
+  if(shade) shade->configChanged();
+  SomfyGroup *group = c.groupId ? somfy.getGroupById(c.groupId) : nullptr;
+  if(group) group->configChanged();
+  SomfyRoom *room = c.roomId ? somfy.getRoomById(c.roomId) : nullptr;
+  if(room) protoSocket.room(*room);
+  if(c.order != sort_list_t::none) protoSocket.order(c.order);
+  if(c.repeaters) protoSocket.repeaters();
+}
 void runModelEdit(model_edit_t *edit) {
   const edit_result_t result = applyModelEdit(*edit);
   if(!result.ok) ESP_LOGW(TAG, "Edit %u refused: %s", static_cast<uint8_t>(edit->type), result.desc);
+  else markEdited(*edit, result);
   emitRequestResult(edit->requestId, result);
   cancelModelEdit(edit);
 }
@@ -358,7 +375,9 @@ edit_result_t applyModelEdit(model_edit_t &edit) {
       else if(!group->linkShade(shadeId)) return edit_result_t::error("This group already has the most shades it can.");
       somfy.commit();
       group->emitState();
-      return done(groupId);
+      edit_result_t r = done(groupId);
+      r.shadeId = shadeId;
+      return r;
     }
     case model_edit_type_t::link_repeater:
     case model_edit_type_t::unlink_repeater: {

src/SomfyShadeOps.cpp

@@ -1,5 +1,6 @@
 #include "SomfyShadeOps.h"
 #include "Somfy.h"
+#include "ShadeChanges.h"
 
 extern SomfyShadeController somfy;
 
@@ -47,6 +48,7 @@ void SomfyShadeOps::setUnpaired(uint8_t shadeId) {
   SomfyShade *shade = somfy.getShadeById(shadeId);
   shade->paired = false;
   shade->save();
+  shade->configChanged();
 }
 void SomfyShadeOps::setRollingCode(uint8_t shadeId, uint16_t code) { somfy.getShadeById(shadeId)->setRollingCode(code); }
 bool SomfyShadeOps::sendGroupCommand(uint8_t groupId, somfy_commands cmd, uint8_t repeat, uint8_t stepSize) {

src/Transceiver.cpp

@@ -11,6 +11,7 @@
 #include "ConfigSettings.h"
 #include "Somfy.h"
 #include "outputs/SocketEmitter.h"
+#include "outputs/ProtoSocket.h"
 
 static const char *TAG = "Somfy";
 
@@ -319,6 +320,15 @@ void Transceiver::emitFrequencyScan(uint8_t num) {
   json->addElem("RSSI", (int32_t)markRSSI);
   json->endObject();
   sockEmit.endEmit(num);
+  if(num == 255 && protoSocket.subscribed(proto_feed_t::frequencyScan)) {
+    ProtoWriter w = protoSocket.beginFeed(proto_type_t::frequencyScan);
+    w.add(proto_field_t::scanning, rxmode == 3);
+    w.add(proto_field_t::testFreq, currFreq);
+    w.add(proto_field_t::testRssi, (int32_t)currRSSI);
+    w.add(proto_field_t::frequency, markFreq);
+    w.add(proto_field_t::rssi, (int32_t)markRSSI);
+    protoSocket.endFeed(proto_feed_t::frequencyScan);
+  }
 }
 bool Transceiver::receive(somfy_rx_t *rx) {
     // Check to see if there is anything in the buffer
@@ -365,6 +375,26 @@ void Transceiver::emitFrame(somfy_frame_t *frame, somfy_rx_t *rx) {
     json->endObject();
     sockEmit.endEmitRoom(ROOM_EMIT_FRAME);
   }
+  if(protoSocket.subscribed(proto_feed_t::frames)) {
+    ProtoWriter w = protoSocket.beginFeed(proto_type_t::frame);
+    w.add(proto_field_t::encKey, frame->encKey);
+    w.add(proto_field_t::remoteAddress, (uint32_t)frame->remoteAddress);
+    w.add(proto_field_t::lastRollingCode, (uint32_t)frame->rollingCode);
+    w.add(proto_field_t::command, translateSomfyCommand(frame->cmd).c_str());
+    w.add(proto_field_t::rawCmd, frame->rawCmd);
+    w.add(proto_field_t::rssi, (int32_t)frame->rssi);
+    w.add(proto_field_t::bits, rx ? rx->bit_length : (uint8_t)0);
+    w.add(proto_field_t::proto, static_cast<uint8_t>(frame->proto));
+    w.add(proto_field_t::valid, frame->valid);
+    w.add(proto_field_t::hwsync, frame->hwsync);
+    if(frame->cmd == somfy_commands::StepUp || frame->cmd == somfy_commands::StepDown) w.add(proto_field_t::stepSize, frame->stepSize);
+    if(rx && rx->pulseCount) {
+      w.beginList(proto_field_t::pulses);
+      for(uint16_t i = 0; i < rx->pulseCount; i++) w.item(rx->pulses[i]);
+      w.endList();
+    }
+    protoSocket.endFeed(proto_feed_t::frames);
+  }
 }
 void Transceiver::clearReceived(void) {
     //packet_received = false;

src/entryPoints/Web.cpp

@@ -16,6 +16,7 @@
 #include "SomfyEditor.h"
 #include "Web.h"
 #include "WebRoutes.h"
+#include "outputs/ProtoSocket.h"
 #include "MqttConnection.h"
 #include "ESPNetwork.h"
 #include "Timing.h"
@@ -230,9 +231,6 @@ static void controllerHeader(JsonFormatter &json, const model_snapshot_t &snap)
 // The parts of /discovery that aren't in the shade model.
 static void discoveryHeader(JsonFormatter &json, const model_snapshot_t &snap) {
   (void)snap;
-  const char *connType = "Unknown";
-  if(net.connType == conn_types_t::ethernet) connType = "Ethernet";
-  else if(net.connType == conn_types_t::wifi) connType = "Wifi";
   json.addElem("serverId", settings.serverId);
   json.addElem("version", settings.fwVersion.name);
   json.addElem("model", "ESPSomfyRTS");
@@ -240,7 +238,7 @@ static void discoveryHeader(JsonFormatter &json, const model_snapshot_t &snap) {
   json.addElem("authType", static_cast<uint8_t>(settings.Security.type));
   json.addElem("permissions", settings.Security.permissions);
   json.addElem("chipModel", settings.chipModel);
-  json.addElem("connType", connType);
+  json.addElem("connType", net.connTypeName());
   json.beginObject("memory");
   json.addElem("max", ESP.getMaxAllocHeap());
   json.addElem("free", ESP.getFreeHeap());
@@ -747,6 +745,8 @@ void Web::begin() {
   DefaultHeaders::Instance().addHeader("Access-Control-Allow-Methods", "PUT,POST,GET,OPTIONS");
   DefaultHeaders::Instance().addHeader("Access-Control-Allow-Headers", "*");
   registerApiRoutes(asyncApiServer);
+  // Before the routes, so serveStatic never answers /ws.
+  protoSocket.begin(asyncServer);
   registerWebUiRoutes(asyncServer);
   asyncServer.begin();
 }

src/outputs/ProtoSocket.cpp

@@ -0,0 +1,421 @@
+#include <Arduino.h>
+#include <WiFi.h>
+#include <algorithm>
+#include "esp_log.h"
+#include "ProtoSocket.h"
+#include "ProtoSync.h"
+#include "ProtoFrame.h"
+#include "ProtoParse.h"
+#include "ProtoSlots.h"
+#include "SomfySnapshot.h"
+#include "Somfy.h"
+#include "ConfigSettings.h"
+#include "ESPNetwork.h"
+#include "Utils.h"
+#include "SocketEmitter.h"
+#include "../entryPoints/Web.h"
+#if HAS_ETHERNET
+#include <ETH.h>
+#endif
+
+static const char *TAG = "Proto";
+
+extern SomfyShadeController somfy;
+extern ConfigSettings settings;
+extern ESPNetwork net;
+extern Web webServer;
+extern LoopStats loopStats;
+extern SocketEmitter sockEmit;
+
+ProtoSocket protoSocket;
+static AsyncWebSocket ws("/ws");
+static_assert(MAX_SOCK_RESPONSE >= PROTO_MAX_FRAME, "protocol 2 frames share the socket event buffer");
+
+static proto_slot_t slots[PROTO_MAX_CLIENTS];
+// What loop() formats events and syncEnd with, into the socket event buffer.
+static BufferedJsonFormatter eventJson;
+
+/*********************************************************************
+ * The shade model as a sync source
+ ********************************************************************/
+static void writeHeader(ProtoWriter &w) {
+  w.add(proto_field_t::maxRooms, (uint8_t)SOMFY_MAX_ROOMS);
+  w.add(proto_field_t::maxShades, (uint8_t)SOMFY_MAX_SHADES);
+  w.add(proto_field_t::maxGroups, (uint8_t)SOMFY_MAX_GROUPS);
+  w.add(proto_field_t::maxGroupedShades, (uint8_t)SOMFY_MAX_GROUPED_SHADES);
+  w.add(proto_field_t::maxLinkedRemotes, (uint8_t)SOMFY_MAX_LINKED_REMOTES);
+  w.add(proto_field_t::startingAddress, somfy.startingAddress);
+  // What a new shade or group gets.
+  w.add(proto_field_t::bitLength, somfy.transceiver.config.type);
+  w.add(proto_field_t::proto, static_cast<uint8_t>(somfy.transceiver.config.proto));
+  w.add(proto_field_t::serverId, settings.serverId);
+  w.add(proto_field_t::hostname, settings.hostname);
+  w.add(proto_field_t::model, "ESPSomfyRTS");
+  w.add(proto_field_t::chipModel, settings.chipModel);
+  w.add(proto_field_t::fw, settings.fwVersion.name);
+  w.add(proto_field_t::connType, net.connTypeName());
+}
+static void writeMemory(ProtoWriter &w) {
+  w.add(proto_field_t::freeHeap, ESP.getFreeHeap());
+  w.add(proto_field_t::maxHeap, ESP.getMaxAllocHeap());
+  w.add(proto_field_t::minHeap, ESP.getMinFreeHeap());
+  w.add(proto_field_t::totalHeap, ESP.getHeapSize());
+  w.add(proto_field_t::uptime, (uint32_t)millis());
+  w.add(proto_field_t::loopRate, loopStats.publishedRate());
+  w.add(proto_field_t::loopMaxUs, loopStats.publishedMaxUs());
+}
+static void writeWifi(ProtoWriter &w) {
+  if(WiFi.status() == WL_CONNECTED) {
+    w.add(proto_field_t::ssid, WiFi.SSID().c_str());
+    w.add(proto_field_t::strength, (int32_t)WiFi.RSSI());
+    w.add(proto_field_t::channel, (int32_t)net.channel);
+  }
+  else {
+    w.add(proto_field_t::strength, (int32_t)-100);
+    w.add(proto_field_t::channel, (int32_t)-1);
+  }
+}
+static void writeEthernet(ProtoWriter &w) {
+#if HAS_ETHERNET
+  if(net.connType == conn_types_t::ethernet) {
+    w.add(proto_field_t::connected, net.connected());
+    w.add(proto_field_t::speed, (uint32_t)ETH.linkSpeed());
+    w.add(proto_field_t::fullDuplex, ETH.fullDuplex());
+  }
+#else
+  (void)w;
+#endif
+}
+class LiveSource : public SyncSource {
+  public:
+    bool room(uint8_t slot, room_snapshot_t &out) override {
+      if(somfy.rooms[slot].roomId == 0) return false;
+      snapshotRoom(somfy.rooms[slot], out);
+      return true;
+    }
+    bool shade(uint8_t slot, shade_snapshot_t &out) override {
+      if(somfy.shades[slot].getShadeId() == 255) return false;
+      snapshotShade(somfy.shades[slot], out);
+      return true;
+    }
+    bool group(uint8_t slot, group_snapshot_t &out) override {
+      if(somfy.groups[slot].getGroupId() == 255) return false;
+      snapshotGroup(somfy.groups[slot], out);
+      return true;
+    }
+    bool repeater(uint8_t slot, uint32_t &address) override {
+      address = somfy.repeaters[slot];
+      return address != 0;
+    }
+    void header(ProtoWriter &w) override { writeHeader(w); }
+    void status(ProtoWriter &w) override {
+      writeMemory(w);
+      if(net.connType == conn_types_t::ethernet) writeEthernet(w);
+      else writeWifi(w);
+    }
+};
+
+/*********************************************************************
+ * async_tcp: connections and what clients send
+ ********************************************************************/
+// Small fixed frames sent straight from async_tcp: nothing in them comes from
+// the shade model, so ADR 0002 doesn't send them through loop().
+static void sendFixed(AsyncWebSocketClient *client, void (*write)(JsonFormatter &json, void *ctx), void *ctx) {
+  char buf[192];
+  BufferedJsonFormatter json;
+  json.setBuffer(buf, sizeof(buf));
+  write(json, ctx);
+  if(!json.overflowed()) client->text(buf, json.length());
+}
+struct error_reply_t {
+  uint32_t id;
+  proto_error_t code;
+  const char *desc;
+};
+static void replyError(AsyncWebSocketClient *client, uint32_t id, proto_error_t code, const char *desc) {
+  error_reply_t e = {id, code, desc};
+  sendFixed(client, [](JsonFormatter &json, void *ctx) {
+    const error_reply_t &e = *static_cast<error_reply_t *>(ctx);
+    writeErrorReply(json, e.id, e.code, e.desc);
+  }, &e);
+}
+// Only a message with an id wants a reply.
+static void replyOk(AsyncWebSocketClient *client, uint32_t id) {
+  if(id == 0) return;
+  sendFixed(client, [](JsonFormatter &json, void *ctx) { writeOkReply(json, *static_cast<uint32_t *>(ctx)); }, &id);
+}
+static bool authorized(AsyncWebServerRequest *request) {
+  if(settings.Security.type == security_types::None) return true;
+  const AsyncWebParameter *key = request->getParam("apikey");
+  if(!key || !request->client()) return false;
+  char token[65];
+  webServer.createAPIToken(request->client()->remoteIP(), token);
+  return key->value() == token;
+}
+static void onConnect(AsyncWebSocketClient *client, AsyncWebServerRequest *request) {
+  const uint32_t id = client->id();
+  const AsyncWebParameter *schema = request ? request->getParam("schema") : nullptr;
+  const bool withSchema = !(schema && schema->value() == protoSchemaHash());
+  proto_slot_t *slot = claimSlot(slots, PROTO_MAX_CLIENTS, id, withSchema);
+  if(!slot) {
+    ESP_LOGW(TAG, "Socket [%lu] refused: all %u clients connected", (unsigned long)id, PROTO_MAX_CLIENTS);
+    client->close();
+    return;
+  }
+  client->setCloseClientOnQueueFull(true);
+  sendFixed(client, [](JsonFormatter &json, void *) { writeHello(json, settings.fwVersion.name); }, nullptr);
+  startSync(*slot);
+  IPAddress ip = client->remoteIP();
+  ESP_LOGI(TAG, "Socket [%lu] connected from %u.%u.%u.%u%s", (unsigned long)id, ip[0], ip[1], ip[2], ip[3], withSchema ? "" : " (has the schema)");
+}
+static void onDisconnect(AsyncWebSocketClient *client) {
+  proto_slot_t *slot = slotOf(slots, PROTO_MAX_CLIENTS, client->id());
+  if(!slot) return;
+  freeSlot(*slot);
+  ESP_LOGI(TAG, "Socket [%lu] disconnected", (unsigned long)client->id());
+}
+static void onFrame(AsyncWebSocketClient *client, AwsFrameInfo *info, uint8_t *data, size_t len) {
+  if(!(info->final && info->index == 0 && info->len == len && info->opcode == WS_TEXT)) {
+    // One reply for the whole message: its first piece gets it, the rest are
+    // dropped.  A frame larger than a TCP segment arrives in pieces too.
+    if(info->num == 0 && info->index == 0) replyError(client, 0, proto_error_t::badRequest, "Send each message as one text frame");
+    return;
+  }
+  proto_slot_t *slot = slotOf(slots, PROTO_MAX_CLIENTS, client->id());
+  if(!slot) return;
+  JsonDocument doc;
+  proto_message_t msg;
+  const proto_error_t err = parseClientFrame(reinterpret_cast<const char *>(data), len, doc, msg);
+  if(err != proto_error_t::none) {
+    replyError(client, msg.id, err, err == proto_error_t::unknownOp ? "Unknown message type" : "Malformed message");
+    return;
+  }
+  switch(static_cast<proto_type_t>(msg.type)) {
+    case proto_type_t::next:
+      slot->wants.fetch_or(WANTS_NEXT);
+      replyOk(client, msg.id);
+      return;
+    case proto_type_t::sync:
+      slot->wants.fetch_or(WANTS_SYNC_AGAIN);
+      replyOk(client, msg.id);
+      return;
+    case proto_type_t::subscribe:
+    case proto_type_t::unsubscribe: {
+      const int32_t feed = protoInt(doc["d"], proto_field_t::feed);
+      bool known = false;
+      for(uint8_t i = 0; i < protoFeedCount(); i++) known |= protoFeedAt(i).number == feed;
+      if(!known) { replyError(client, msg.id, proto_error_t::badRequest, "Unknown feed"); return; }
+      if(msg.type == static_cast<uint8_t>(proto_type_t::subscribe)) slot->feeds.fetch_or(1 << feed);
+      else slot->feeds.fetch_and(~(1 << feed));
+      replyOk(client, msg.id);
+      return;
+    }
+    default:
+      // The ops arrive with the protocol core's later PRs.
+      replyError(client, msg.id, proto_error_t::unknownOp, "Unknown message type");
+      return;
+  }
+}
+void ProtoSocket::wsEvent(AsyncWebSocket *server, AsyncWebSocketClient *client, AwsEventType type, void *arg, uint8_t *data, size_t len) {
+  (void)server;
+  switch(type) {
+    case WS_EVT_CONNECT: onConnect(client, static_cast<AsyncWebServerRequest *>(arg)); break;
+    case WS_EVT_DISCONNECT: onDisconnect(client); break;
+    case WS_EVT_DATA: onFrame(client, static_cast<AwsFrameInfo *>(arg), data, len); break;
+    default: break;
+  }
+}
+
+void ProtoSocket::begin(AsyncWebServer &server) {
+  ws.handleHandshake(authorized);
+  ws.onEvent(ProtoSocket::wsEvent);
+  server.addHandler(&ws);
+}
+bool ProtoSocket::hasClients() const {
+  for(const proto_slot_t &s : slots)
+    if(s.clientId.load()) return true;
+  return false;
+}
+bool ProtoSocket::sending() const { return sockEmit.onLoopTask() && this->hasClients(); }
+
+/*********************************************************************
+ * loop(): syncs
+ ********************************************************************/
+static void sendSyncFrame(proto_slot_t &s) {
+  LiveSource src;
+  char *buf = socketFrameBuffer();
+  const size_t len = writeSyncFrame(src, s.cur, buf, PROTO_MAX_FRAME);
+  if(len) ws.text(s.loopId, buf, len);
+  syncFrameSent(s, millis());
+  if(s.cur.active()) return;
+  eventJson.setBuffer(buf, PROTO_MAX_FRAME);
+  beginFrame(eventJson, proto_type_t::syncEnd);
+  endFrame(eventJson);
+  ws.text(s.loopId, buf, eventJson.length());
+}
+void ProtoSocket::loop() {
+  if(!sockEmit.onLoopTask()) return;
+  for(proto_slot_t &s : slots) {
+    switch(nextSyncStep(s, millis())) {
+      case sync_step_t::send:
+        sendSyncFrame(s);
+        break;
+      case sync_step_t::timedOut:
+        ESP_LOGW(TAG, "Socket [%lu] sent no next for %us, closing it", (unsigned long)s.loopId, PROTO_SYNC_TIMEOUT_MS / 1000);
+        ws.close(s.loopId);
+        break;
+      case sync_step_t::none:
+        break;
+    }
+  }
+  ws.cleanupClients(PROTO_MAX_CLIENTS);
+}
+
+/*********************************************************************
+ * loop(): events
+ ********************************************************************/
+static JsonFormatter &beginEvent(proto_type_t type) {
+  eventJson.setBuffer(socketFrameBuffer(), PROTO_MAX_FRAME);
+  beginFrame(eventJson, type);
+  return eventJson;
+}
+// Closes the event; false when it didn't fit, which is logged.
+static bool finishEvent() {
+  endFrame(eventJson);
+  if(!eventJson.overflowed()) return true;
+  ESP_LOGE(TAG, "Event too large for %u bytes, not sent", PROTO_MAX_FRAME);
+  return false;
+}
+static void sendEvent() {
+  if(finishEvent()) ws.textAll(socketFrameBuffer(), eventJson.length());
+}
+// An item whole, or only its state: a snapshot of it, written by write.
+template<typename Model, typename Snapshot>
+static void sendItem(proto_type_t type, Model &item, void (*snapshot)(Model &, Snapshot &), void (*write)(ProtoWriter &, const Snapshot &)) {
+  Snapshot s;
+  snapshot(item, s);
+  ProtoWriter w(beginEvent(type));
+  write(w, s);
+  sendEvent();
+}
+
+void ProtoSocket::shadeState(SomfyShade &shade) {
+  if(this->sending()) sendItem(proto_type_t::shadeState, shade, snapshotShade, protoShadeState);
+}
+void ProtoSocket::shade(SomfyShade &shade) {
+  if(this->sending()) sendItem(proto_type_t::shade, shade, snapshotShade, protoShade);
+}
+void ProtoSocket::shadeRemoved(uint8_t shadeId) {
+  if(!this->sending()) return;
+  ProtoWriter(beginEvent(proto_type_t::shadeRemoved)).add(proto_field_t::shadeId, shadeId);
+  sendEvent();
+}
+void ProtoSocket::group(SomfyGroup &group) {
+  if(this->sending()) sendItem(proto_type_t::group, group, snapshotGroup, protoGroup);
+}
+void ProtoSocket::groupState(SomfyGroup &group) {
+  if(!this->sending()) return;
+  group_snapshot_t g;
+  snapshotGroup(group, g);
+  ProtoWriter w(beginEvent(proto_type_t::groupState));
+  protoGroupState(w, g.groupId, g.flags, group.direction);
+  sendEvent();
+}
+void ProtoSocket::groupRemoved(uint8_t groupId) {
+  if(!this->sending()) return;
+  ProtoWriter(beginEvent(proto_type_t::groupRemoved)).add(proto_field_t::groupId, groupId);
+  sendEvent();
+}
+void ProtoSocket::room(SomfyRoom &room) {
+  if(this->sending()) sendItem(proto_type_t::room, room, snapshotRoom, protoRoom);
+}
+void ProtoSocket::roomRemoved(uint8_t roomId) {
+  if(!this->sending()) return;
+  ProtoWriter(beginEvent(proto_type_t::roomRemoved)).add(proto_field_t::roomId, roomId);
+  sendEvent();
+}
+void ProtoSocket::order(sort_list_t list) {
+  if(!this->sending()) return;
+  struct entry_t { int8_t order; uint8_t id; };
+  static constexpr uint8_t MOST = std::max({SOMFY_MAX_ROOMS, SOMFY_MAX_SHADES, SOMFY_MAX_GROUPS});
+  entry_t items[MOST];
+  uint8_t n = 0;
+  proto_type_t type;
+  switch(list) {
+    case sort_list_t::rooms:
+      type = proto_type_t::roomOrder;
+      for(SomfyRoom &r : somfy.rooms) if(r.roomId != 0) items[n++] = {r.sortOrder, r.roomId};
+      break;
+    case sort_list_t::shades:
+      type = proto_type_t::shadeOrder;
+      for(SomfyShade &s : somfy.shades) if(s.getShadeId() != 255) items[n++] = {s.sortOrder, s.getShadeId()};
+      break;
+    case sort_list_t::groups:
+      type = proto_type_t::groupOrder;
+      for(SomfyGroup &g : somfy.groups) if(g.getGroupId() != 255) items[n++] = {g.sortOrder, g.getGroupId()};
+      break;
+    default:
+      return;
+  }
+  for(uint8_t i = 1; i < n; i++)
+    for(uint8_t j = i; j > 0 && items[j].order < items[j - 1].order; j--) std::swap(items[j], items[j - 1]);
+  ProtoWriter w(beginEvent(type));
+  w.beginList(proto_field_t::ids);
+  for(uint8_t i = 0; i < n; i++) w.item(items[i].id);
+  w.endList();
+  sendEvent();
+}
+void ProtoSocket::repeaters() {
+  if(!this->sending()) return;
+  ProtoWriter w(beginEvent(proto_type_t::repeaters));
+  w.beginList(proto_field_t::repeaters);
+  for(uint32_t address : somfy.repeaters) if(address != 0) w.item(address);
+  w.endList();
+  sendEvent();
+}
+void ProtoSocket::shadeCommand(SomfyShade &shade, const char *command, const char *source, uint32_t sourceAddress) {
+  if(!this->sending()) return;
+  ProtoWriter w(beginEvent(proto_type_t::shadeCommand));
+  w.add(proto_field_t::shadeId, shade.getShadeId());
+  w.add(proto_field_t::remoteAddress, shade.getRemoteAddress());
+  w.add(proto_field_t::command, command);
+  w.add(proto_field_t::source, source);
+  w.add(proto_field_t::lastRollingCode, shade.lastRollingCode);
+  w.add(proto_field_t::sourceAddress, sourceAddress);
+  sendEvent();
+}
+void ProtoSocket::memory() {
+  if(!this->sending()) return;
+  ProtoWriter w(beginEvent(proto_type_t::memory));
+  writeMemory(w);
+  sendEvent();
+}
+void ProtoSocket::wifi() {
+  if(!this->sending()) return;
+  ProtoWriter w(beginEvent(proto_type_t::wifi));
+  writeWifi(w);
+  sendEvent();
+}
+void ProtoSocket::ethernet() {
+  if(!this->sending()) return;
+  ProtoWriter w(beginEvent(proto_type_t::ethernet));
+  writeEthernet(w);
+  sendEvent();
+}
+bool ProtoSocket::subscribed(proto_feed_t feed) const {
+  // The feed is formatted into the event buffer, which is loop()'s.
+  if(!sockEmit.onLoopTask()) return false;
+  const uint8_t bit = 1 << static_cast<uint8_t>(feed);
+  for(const proto_slot_t &s : slots)
+    if(s.clientId.load() && (s.feeds.load() & bit)) return true;
+  return false;
+}
+ProtoWriter ProtoSocket::beginFeed(proto_type_t type) { return ProtoWriter(beginEvent(type)); }
+void ProtoSocket::endFeed(proto_feed_t feed) {
+  if(!finishEvent()) return;
+  const uint8_t bit = 1 << static_cast<uint8_t>(feed);
+  for(proto_slot_t &s : slots) {
+    const uint32_t id = s.clientId.load();
+    if(id && (s.feeds.load() & bit)) ws.text(id, socketFrameBuffer(), eventJson.length());
+  }
+}

src/outputs/ProtoSocket.h

@@ -0,0 +1,54 @@
+#ifndef PROTO_SOCKET_H
+#define PROTO_SOCKET_H
+#include <ESPAsyncWebServer.h>
+#include "ProtoSchema.h"
+#include "ModelEditor.h"
+
+class SomfyShade;
+class SomfyGroup;
+class SomfyRoom;
+
+// The socket protocol 2 endpoint, ws://host/ws on port 80 (docs/PROTOCOL.md,
+// docs/adr/0005-one-websocket-protocol.md).  It runs next to the old socket on
+// port 8080 until the web UI and Home Assistant have moved over.
+//
+// async_tcp only reads what clients send and answers with fixed frames that
+// hold nothing of the shade model (hello, errors, ok).  Everything formatted
+// from the model goes out from loop(), in the socket event buffer.  The
+// clients' slots and syncs are in ProtoSlots.
+class ProtoSocket {
+  public:
+    // Adds /ws to the server.  From setup(), on loop()'s task.
+    void begin(AsyncWebServer &server);
+    // Moves each client's sync on.  loop() only.
+    void loop();
+    bool hasClients() const;
+    // Events, loop() only.
+    void shadeState(SomfyShade &shade);
+    void shade(SomfyShade &shade);
+    void shadeRemoved(uint8_t shadeId);
+    void group(SomfyGroup &group);
+    void groupState(SomfyGroup &group);
+    void groupRemoved(uint8_t groupId);
+    void room(SomfyRoom &room);
+    void roomRemoved(uint8_t roomId);
+    // roomOrder, shadeOrder or groupOrder: the list's ids, in sort order.
+    void order(sort_list_t list);
+    void repeaters();
+    void shadeCommand(SomfyShade &shade, const char *command, const char *source, uint32_t sourceAddress);
+    void memory();
+    void wifi();
+    void ethernet();
+    // A feed event: write its fields into the writer between the two.  Only
+    // when subscribed() says someone is, which is also only on loop().
+    bool subscribed(proto_feed_t feed) const;
+    ProtoWriter beginFeed(proto_type_t type);
+    void endFeed(proto_feed_t feed);
+    static void wsEvent(AsyncWebSocket *server, AsyncWebSocketClient *client, AwsEventType type, void *arg, uint8_t *data, size_t len);
+  private:
+    // On loop()'s task, with a client to send to: nothing is formatted for
+    // nobody, or off the task that owns the event buffer.
+    bool sending() const;
+};
+extern ProtoSocket protoSocket;
+#endif

src/outputs/SocketEmitter.cpp

@@ -49,6 +49,7 @@ static void removeClient(uint32_t asyncId) {
 }
 
 static char g_response[MAX_SOCK_RESPONSE];
+char *socketFrameBuffer() { return g_response; }
 
 bool room_t::isJoined(uint8_t num) {
   for(uint8_t i = 0; i < sizeof(this->clients); i++) {

src/outputs/SocketEmitter.h

@@ -9,6 +9,9 @@
 // fit is logged and not sent, so the largest one has to fit: see remoteFrame
 // in Transceiver.cpp.
 #define MAX_SOCK_RESPONSE 4096
+// That buffer, which protocol 2 events and sync frames are formatted into too.
+// loop() only, and only between an event's start and its send.
+char *socketFrameBuffer();
 
 struct room_t {
   uint8_t clients[5] = {255, 255, 255, 255, 255};
@@ -18,6 +21,8 @@ struct room_t {
   bool leave(uint8_t num);
   void clear();
 };
+// The old socket on port 8080.  It goes with step 4 of docs/SECURITY_PLAN.md,
+// and with it every sockEmit call that sits next to a protoSocket one.
 class SocketEmitter {
   protected:
     uint8_t newclients = 0;
@@ -26,9 +31,11 @@ class SocketEmitter {
     TaskHandle_t _owner = nullptr;
     // What beginEmit() hands any other task, so the event goes nowhere.
     JsonSockEvent _discard;
-    bool onLoopTask() const;
     void delayInit(uint8_t num);
   public:
+    // Whether this runs on loop()'s task, the only one that may send socket
+    // events: this socket's and protocol 2's.
+    bool onLoopTask() const;
     JsonSockEvent json;
     room_t rooms[SOCK_MAX_ROOMS];
     uint8_t activeClients(uint8_t room);

src/outputs/SomfyEvents.cpp

@@ -2,6 +2,7 @@
 #include "Somfy.h"
 #include "SomfyEditor.h"
 #include "SocketEmitter.h"
+#include "ProtoSocket.h"
 #include "MqttConnection.h"
 
 extern SomfyShadeController somfy;
@@ -49,6 +50,7 @@ void SomfyShade::emitCommand(uint8_t num, somfy_commands cmd, const char *source
   json->addElem("sourceAddress", (uint32_t)sourceAddress);
   json->endObject();
   sockEmit.endEmit(num);
+  if(num == 255) protoSocket.shadeCommand(*this, translateSomfyCommand(cmd).c_str(), source, sourceAddress);
   if(mqtt.connected()) {
     this->publish("cmdSource", source);
     this->publish("cmdAddress", sourceAddress);
@@ -114,6 +116,8 @@ void SomfyShadeController::sendChanges() {
     if(!changed || shade.getShadeId() == 255) continue;
     if(changed & SHADE_CHANGED_TOPICS) shade.publishTopics(changed);
     shade.emitState(255, "shadeState");
+    if(changed & SHADE_CHANGED_CONFIG) protoSocket.shade(shade);
+    else protoSocket.shadeState(shade);
   }
   for(uint8_t i = 0; i < SOMFY_MAX_GROUPS; i++) {
     SomfyGroup &group = this->groups[i];
@@ -122,6 +126,8 @@ void SomfyShadeController::sendChanges() {
     if(!changed || group.getGroupId() == 255) continue;
     if(changed & GROUP_CHANGED_DIRECTION) group.publish("direction", group.direction);
     group.emitState(255, "groupState");
+    if(changed & GROUP_CHANGED_CONFIG) protoSocket.group(group);
+    else protoSocket.groupState(group);
   }
 }
 void SomfyShadeController::emitState(uint8_t num) {

test/e2e/README.md

@@ -0,0 +1,17 @@
+# e2e suite
+
+Runs against a real device on your network, by hand, never in CI. It checks socket protocol 2 ([PROTOCOL.md](../../docs/PROTOCOL.md)) and that the device survives the loads that used to run its heap out.
+
+```sh
+~/.platformio/penv/Scripts/uv run --no-project --with pytest -- pytest test/e2e --device 192.168.1.120
+```
+
+| Option | Use |
+|---|---|
+| `--device` | the device's address (required) |
+| `--password`, `--username` / `--pin` | the credentials of a Secured device |
+| `--record` | save hello, the sync and the schema to `test/fixtures/proto2/`, for the web UI and Home Assistant tests |
+
+A plain run sends nothing on air, changes no settings and doesn't reboot the device. It does fill every socket slot for a few seconds, and replays a page load.
+
+The tests run in whatever security mode the device is in; the suite never changes it. To test both modes, run the suite once in each.

test/e2e/conftest.py

@@ -0,0 +1,84 @@
+"""Options and fixtures for the e2e suite.  See README.md."""
+import os
+import sys
+import time
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(__file__))
+from proto_client import ProtoClient, Schema, http_json, run_sync  # noqa: E402
+
+
+def pytest_addoption(parser):
+    parser.addoption("--device", required=True, help="the device's address, e.g. 192.168.1.120")
+    parser.addoption("--password", default=None, help="the device's password, when it is Secured with one")
+    parser.addoption("--username", default="", help="the username that goes with --password")
+    parser.addoption("--pin", default=None, help="the device's PIN, when it is Secured with one")
+    parser.addoption("--record", action="store_true", help="save what the device sends to test/fixtures/proto2")
+
+
+@pytest.fixture(scope="session")
+def host(request):
+    return request.config.getoption("--device")
+
+
+@pytest.fixture(scope="session")
+def login_context(host):
+    return http_json(host, "/loginContext")
+
+
+@pytest.fixture(scope="session")
+def secured(login_context):
+    return login_context.get("type", 0) != 0
+
+
+@pytest.fixture(scope="session")
+def token(request, host, secured):
+    """The token for this PC, or None when the device isn't Secured."""
+    if not secured:
+        return None
+    password = request.config.getoption("--password")
+    pin = request.config.getoption("--pin")
+    if password is None and pin is None:
+        pytest.skip("the device is Secured: pass --password or --pin")
+    body = {"username": request.config.getoption("--username"), "password": password or "", "pin": pin or ""}
+    return http_json(host, "/login", method="PUT", body=body)["apiKey"]
+
+
+@pytest.fixture
+def connect(host, token):
+    """Opens protocol 2 clients and closes them after the test."""
+    clients = []
+
+    def open_client(**params):
+        if token and "apikey" not in params:
+            params["apikey"] = token
+        c = ProtoClient(host, params=params or None)
+        clients.append(c)
+        return c
+
+    yield open_client
+    for c in clients:
+        c.close()
+    # The device frees a slot when it sees the disconnect; the next test may
+    # need every slot.
+    time.sleep(0.5)
+
+
+@pytest.fixture(scope="session")
+def schema(host, token):
+    """The device's schema, from one full sync."""
+    s = Schema()
+    c = ProtoClient(host, params={"apikey": token} if token else None)
+    try:
+        assert c.status == 101, "the device has no /ws: is it running protocol 2 firmware?"
+        run_sync(c, s)
+    finally:
+        c.close()
+    assert s.complete() and s.hash
+    return s
+
+
+@pytest.fixture(scope="session")
+def recording(request):
+    return request.config.getoption("--record")

test/e2e/proto_client.py

@@ -0,0 +1,232 @@
+"""A socket protocol 2 client for the e2e suite (docs/PROTOCOL.md).
+
+Only the standard library, so the suite needs nothing but pytest.  Frames are
+read on a background thread into a queue; tests take them with recv().
+"""
+import base64
+import json
+import os
+import queue
+import socket
+import struct
+import threading
+import time
+import urllib.parse
+import urllib.request
+
+PROTO_VERSION = 2
+
+
+class SocketClosed(Exception):
+    pass
+
+
+class Schema:
+    """The device's schema: numbers and short keys to names, and defaults."""
+
+    def __init__(self):
+        self.hash = None
+        self.types = {}      # number -> name
+        self.type_numbers = {}
+        self.fields = {}     # key -> (name, kind, default)
+        self.keys = {}       # name -> key
+        self.errors = {}     # number -> name
+        self.error_numbers = {}
+        self.feeds = {}
+        self.feed_numbers = {}
+
+    def add(self, d):
+        for number, name in d.get("T", []):
+            self.types[number] = name
+            self.type_numbers[name] = number
+        for key, name, kind, default in d.get("F", []):
+            self.fields[key] = (name, kind, default)
+            self.keys[name] = key
+        for number, name in d.get("E", []):
+            self.errors[number] = name
+            self.error_numbers[name] = number
+        for number, name in d.get("Q", []):
+            self.feeds[number] = name
+            self.feed_numbers[name] = number
+
+    def complete(self):
+        return bool(self.types and self.fields and self.errors)
+
+    def expand(self, value):
+        """Short keys to names, 1/0 to booleans; omitted fields stay omitted."""
+        if isinstance(value, list):
+            return [self.expand(v) for v in value]
+        if not isinstance(value, dict):
+            return value
+        out = {}
+        for key, v in value.items():
+            name, kind, _ = self.fields.get(key, (key, None, None))
+            if kind == "b" and isinstance(v, int):
+                v = bool(v)
+            out[name] = self.expand(v)
+        return out
+
+    def default(self, name):
+        key = self.keys[name]
+        _, kind, default = self.fields[key]
+        if kind == "b":
+            return bool(default)
+        if kind == "s":
+            return ""
+        if kind == "l":
+            return []
+        return default
+
+
+class ProtoClient:
+    def __init__(self, host, port=80, path="/ws", params=None, timeout=10):
+        self.host = host
+        query = ("?" + urllib.parse.urlencode(params)) if params else ""
+        self.sock = socket.create_connection((host, port), timeout=timeout)
+        key = base64.b64encode(os.urandom(16)).decode()
+        self.sock.sendall((
+            f"GET {path}{query} HTTP/1.1\r\nHost: {host}:{port}\r\nUpgrade: websocket\r\n"
+            f"Connection: Upgrade\r\nSec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n").encode())
+        head = b""
+        while b"\r\n\r\n" not in head:
+            chunk = self.sock.recv(1024)
+            if not chunk:
+                raise SocketClosed("closed during the handshake")
+            head += chunk
+        status_line, self._rest = head.split(b"\r\n\r\n", 1)
+        self.status = int(status_line.split(b" ")[1])
+        self.frames = queue.Queue()
+        self.raw = []  # every text frame as received, for fixtures
+        self.closed = threading.Event()
+        if self.status == 101:
+            threading.Thread(target=self._reader, daemon=True).start()
+        else:
+            self.sock.close()
+            self.closed.set()
+        self._next_id = 1
+
+    def _recvn(self, n):
+        while len(self._rest) < n:
+            chunk = self.sock.recv(4096)
+            if not chunk:
+                raise SocketClosed()
+            self._rest += chunk
+        out, self._rest = self._rest[:n], self._rest[n:]
+        return out
+
+    def _reader(self):
+        self.sock.settimeout(None)
+        try:
+            while True:
+                b1, b2 = self._recvn(2)
+                n = b2 & 0x7F
+                if n == 126:
+                    n = struct.unpack(">H", self._recvn(2))[0]
+                elif n == 127:
+                    n = struct.unpack(">Q", self._recvn(8))[0]
+                data = self._recvn(n)
+                opcode = b1 & 0x0F
+                if opcode == 1:
+                    text = data.decode()
+                    self.raw.append(text)
+                    self.frames.put(json.loads(text))
+                elif opcode == 8:
+                    break
+        except (OSError, SocketClosed, ValueError):
+            pass
+        self.closed.set()
+
+    def send_text(self, text):
+        """A client frame must be masked."""
+        payload = text.encode()
+        mask = os.urandom(4)
+        header = bytes([0x81])
+        n = len(payload)
+        if n < 126:
+            header += bytes([0x80 | n])
+        elif n < 65536:
+            header += bytes([0x80 | 126]) + struct.pack(">H", n)
+        else:
+            header += bytes([0x80 | 127]) + struct.pack(">Q", n)
+        masked = bytes(b ^ mask[i % 4] for i, b in enumerate(payload))
+        self.sock.sendall(header + mask + masked)
+
+    def send(self, type_number, d=None, with_id=False):
+        frame = {"v": PROTO_VERSION, "t": type_number}
+        msg_id = None
+        if with_id:
+            msg_id = self._next_id
+            self._next_id += 1
+            frame["id"] = msg_id
+        if d is not None:
+            frame["d"] = d
+        self.send_text(json.dumps(frame, separators=(",", ":")))
+        return msg_id
+
+    def recv(self, timeout=10):
+        try:
+            return self.frames.get(timeout=timeout)
+        except queue.Empty:
+            raise TimeoutError("no frame in %ss" % timeout)
+
+    def recv_until(self, predicate, timeout=10):
+        """Frames up to and including the first one predicate accepts."""
+        end = time.time() + timeout
+        seen = []
+        while True:
+            left = end - time.time()
+            if left <= 0:
+                raise TimeoutError("not seen in %ss; got types %s" % (timeout, [f.get("t") for f in seen]))
+            f = self.recv(left)
+            seen.append(f)
+            if predicate(f):
+                return seen
+
+    def close(self):
+        try:
+            self.sock.sendall(bytes([0x88, 0x80]) + os.urandom(4))
+        except OSError:
+            pass
+        self.sock.close()
+
+
+def run_sync(client, schema=None, timeout=20, hello=True):
+    """Takes hello (unless hello=False, for a resync) and a whole sync,
+    sending next after each sync frame.
+
+    Returns (hello, sync frames, events seen meanwhile).  Fills schema, and
+    its hash, from the sync when it carries one.
+    """
+    first = client.recv(timeout) if hello else None
+    if hello:
+        assert first["t"] == 1, first
+        if schema is not None:
+            schema.hash = first["d"]["sh"]
+    sync_frames, events = [], []
+    end = time.time() + timeout
+    while True:
+        f = client.recv(max(0.1, end - time.time()))
+        if f["t"] == 4:
+            sync_frames.append(f)
+            if schema is not None:
+                schema.add(f["d"])
+            client.send(3)
+        elif f["t"] == 5:
+            return first, sync_frames, events
+        else:
+            events.append(f)
+
+
+def http_json(host, path, port=80, method="GET", body=None, headers=None, timeout=20):
+    data = json.dumps(body).encode() if body is not None else None
+    req = urllib.request.Request(f"http://{host}:{port}{path}", data=data, method=method, headers=headers or {})
+    if data is not None:
+        req.add_header("Content-Type", "application/json")
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.loads(r.read())
+
+
+def heap(host):
+    """free, min and uptime from /discovery on 8081 (protocol 1, until it goes)."""
+    m = http_json(host, "/discovery", port=8081)["memory"]
+    return {"free": m["free"], "min": m["min"], "max": m["max"], "uptime": m["uptime"]}

test/e2e/test_auth.py

@@ -0,0 +1,38 @@
+"""The socket under the device's security mode (docs/adr/0006-two-security-modes-eleven-http-routes.md).
+
+The suite never changes the device's security itself, so it tests the mode it
+finds: pass --password or --pin for a Secured device.
+"""
+import pytest
+
+from proto_client import ProtoClient
+
+
+def test_none_lets_anyone_connect(host, secured):
+    if secured:
+        pytest.skip("the device is Secured")
+    c = ProtoClient(host)
+    assert c.status == 101
+    c.close()
+
+
+def test_secured_refuses_a_socket_without_a_token(host, secured):
+    if not secured:
+        pytest.skip("the device's security is None")
+    c = ProtoClient(host)
+    assert c.status == 401
+
+
+def test_secured_refuses_a_wrong_token(host, secured):
+    if not secured:
+        pytest.skip("the device's security is None")
+    c = ProtoClient(host, params={"apikey": "0" * 64})
+    assert c.status == 401
+
+
+def test_secured_accepts_the_token(host, secured, token):
+    if not secured:
+        pytest.skip("the device's security is None")
+    c = ProtoClient(host, params={"apikey": token})
+    assert c.status == 101
+    c.close()

test/e2e/test_messages.py

@@ -0,0 +1,88 @@
+"""What the device answers to client frames: errors and feed subscriptions."""
+from proto_client import run_sync
+
+
+def synced(connect, schema):
+    c = connect(schema=schema.hash)
+    run_sync(c)
+    return c
+
+
+def reply_to(c, msg_id, timeout=5):
+    return c.recv_until(lambda f: f["t"] == 2 and f.get("id") == msg_id, timeout)[-1]
+
+
+def error_name(schema, reply):
+    return schema.errors[reply["d"][schema.keys["code"]]]
+
+
+def test_malformed_json_is_a_bad_request(connect, schema):
+    c = synced(connect, schema)
+    c.send_text("not json")
+    reply = c.recv_until(lambda f: f["t"] == 2)[-1]
+    assert error_name(schema, reply) == "badRequest"
+
+
+def test_a_wrong_version_is_a_bad_request(connect, schema):
+    c = synced(connect, schema)
+    c.send_text('{"v":1,"t":3,"id":7}')
+    assert error_name(schema, reply_to(c, 7)) == "badRequest"
+
+
+def test_an_event_type_from_a_client_is_unknown(connect, schema):
+    c = synced(connect, schema)
+    msg_id = c.send(schema.type_numbers["shadeState"], {}, with_id=True)
+    assert error_name(schema, reply_to(c, msg_id)) == "unknownOp"
+
+
+def test_next_and_sync_with_an_id_are_answered(connect, schema):
+    """A message with an id gets one reply, control messages too."""
+    c = synced(connect, schema)
+    msg_id = c.send(schema.type_numbers["next"], with_id=True)
+    assert reply_to(c, msg_id)["d"].get(schema.keys["ok"]) == 1
+    msg_id = c.send(schema.type_numbers["sync"], with_id=True)
+    assert reply_to(c, msg_id)["d"].get(schema.keys["ok"]) == 1
+
+
+def test_a_long_message_gets_one_reply(connect, schema):
+    """A frame bigger than a TCP segment reaches the device in pieces, which it
+    can't put together yet: still one reply for the whole message.  The message
+    is wrong either way, so a reply is due however it arrives."""
+    c = synced(connect, schema)
+    c.send_text('{"v":2,"t":%d,"id":5,"x":"%s"}' % (schema.type_numbers["shadeState"], "a" * 3000))
+    replies = []
+    try:
+        while True:
+            f = c.recv(timeout=3)
+            if f["t"] == 2:
+                replies.append(f)
+    except TimeoutError:
+        pass
+    assert len(replies) == 1, replies
+    # In pieces it can't be read, so its id isn't known; whole, it is an event.
+    assert error_name(schema, replies[0]) in ("badRequest", "unknownOp")
+
+
+def test_subscribing_to_a_feed_is_ok(connect, schema):
+    c = synced(connect, schema)
+    for feed in ("frames", "frequencyScan"):
+        msg_id = c.send(schema.type_numbers["subscribe"], {schema.keys["feed"]: schema.feed_numbers[feed]}, with_id=True)
+        reply = reply_to(c, msg_id)
+        assert reply["d"].get(schema.keys["ok"]) == 1
+        msg_id = c.send(schema.type_numbers["unsubscribe"], {schema.keys["feed"]: schema.feed_numbers[feed]}, with_id=True)
+        assert reply_to(c, msg_id)["d"].get(schema.keys["ok"]) == 1
+
+
+def test_an_unknown_feed_is_a_bad_request(connect, schema):
+    c = synced(connect, schema)
+    msg_id = c.send(schema.type_numbers["subscribe"], {schema.keys["feed"]: 99}, with_id=True)
+    assert error_name(schema, reply_to(c, msg_id)) == "badRequest"
+
+
+def test_memory_status_arrives_within_the_status_period(connect, schema):
+    """The device sends its status every 30 s."""
+    c = synced(connect, schema)
+    f = c.recv_until(lambda f: f["t"] == schema.type_numbers["memory"], timeout=40)[-1]
+    memory = schema.expand(f["d"])
+    assert memory["freeHeap"] > 0
+    assert memory["uptime"] > 0

test/e2e/test_survival.py

@@ -0,0 +1,72 @@
+"""The device must survive the loads that ran its heap out
+(docs/adr/0005-one-websocket-protocol.md): every client reconnecting at once,
+and a browser loading the page."""
+import threading
+import urllib.request
+
+import pytest
+
+from proto_client import ProtoClient, heap, run_sync
+
+# What the heap may drop to under load.  Below it the next allocation in the
+# web server is close to failing, which aborts.
+HEAP_FLOOR = 20 * 1024
+MAX_CLIENTS = 5
+
+
+@pytest.fixture
+def watch_heap(host):
+    """Fails the test when the device rebooted or its heap fell under the floor.
+
+    The device reports its lowest heap since it booted, so a dip under the
+    floor only shows while that low is still above it."""
+    before = heap(host)
+    if before["min"] < HEAP_FLOOR:
+        pytest.skip("the heap has been down to %u bytes since boot: reboot the device to check the floor" % before["min"])
+    yield before
+    after = heap(host)
+    assert after["uptime"] > before["uptime"], "the device rebooted"
+    assert after["min"] >= HEAP_FLOOR, "the heap fell to %u bytes (floor %u)" % (after["min"], HEAP_FLOOR)
+
+
+def test_every_client_reconnecting_at_once(connect, schema, watch_heap):
+    clients = [connect() for _ in range(MAX_CLIENTS)]
+    errors = []
+
+    def sync(c):
+        try:
+            run_sync(c, timeout=60)
+        except Exception as e:  # reported below
+            errors.append(e)
+    threads = [threading.Thread(target=sync, args=(c,)) for c in clients]
+    [t.start() for t in threads]
+    [t.join() for t in threads]
+    assert not errors, errors
+
+
+def test_one_client_too_many_is_closed(connect, schema):
+    clients = [connect(schema=schema.hash) for _ in range(MAX_CLIENTS)]
+    for c in clients:
+        run_sync(c)
+    extra = connect(schema=schema.hash)
+    assert extra.closed.wait(5), "a sixth client stayed connected"
+
+
+def test_page_load(host, token, connect, watch_heap):
+    """The static files in parallel, as a browser fetches them, then the socket."""
+    files = ["/", "/main.css", "/widgets.css", "/icons.css", "/index.js", "/icon.png"]
+    errors = []
+
+    def get(path):
+        try:
+            req = urllib.request.Request(f"http://{host}{path}", headers={"Accept-Encoding": "gzip"})
+            with urllib.request.urlopen(req, timeout=60) as r:
+                r.read()
+        except Exception as e:  # reported below
+            errors.append((path, e))
+    threads = [threading.Thread(target=get, args=(p,)) for p in files]
+    [t.start() for t in threads]
+    c = connect()
+    run_sync(c, timeout=60)
+    [t.join() for t in threads]
+    assert not errors, errors

test/e2e/test_sync.py

@@ -0,0 +1,161 @@
+"""hello, the schema and the sync (docs/PROTOCOL.md)."""
+import json
+import os
+
+from proto_client import Schema, http_json, run_sync
+
+FIXTURES = os.path.join(os.path.dirname(__file__), "..", "fixtures", "proto2")
+
+
+def items(frames, schema, section):
+    key = schema.keys[section]
+    out = []
+    for f in frames:
+        v = f["d"].get(key)
+        if isinstance(v, list):
+            out.extend(v)
+        elif v is not None:
+            out.append(v)
+    return out
+
+
+def test_hello_names_the_firmware_and_schema(connect):
+    c = connect()
+    assert c.status == 101
+    hello = c.recv()
+    assert hello["v"] == 2
+    assert hello["t"] == 1
+    assert hello["d"]["fw"]
+    assert len(hello["d"]["sh"]) == 8
+
+
+def test_first_sync_carries_the_schema(connect):
+    c = connect()
+    schema = Schema()
+    hello, frames, _ = run_sync(c, schema)
+    assert schema.complete()
+    assert schema.type_numbers["hello"] == 1
+    assert schema.keys["shadeId"] == "s"
+    for f in frames:
+        assert len(json.dumps(f, separators=(",", ":"))) < 4096
+
+
+def test_a_client_with_the_schema_gets_none(connect, schema):
+    c = connect(schema=schema.hash)
+    _, frames, _ = run_sync(c)
+    for f in frames:
+        for k in ("T", "F", "E", "Q"):
+            assert k not in f["d"]
+
+
+def test_an_old_schema_hash_gets_the_schema_again(connect):
+    c = connect(schema="00000000")
+    schema = Schema()
+    run_sync(c, schema)
+    assert schema.complete()
+
+
+def test_sync_holds_the_same_model_as_controller(connect, schema, host, token):
+    c = connect(schema=schema.hash)
+    _, frames, _ = run_sync(c)
+    headers = {"apikey": token} if token else {}
+    controller = http_json(host, "/controller", headers=headers)
+    shades = [schema.expand(s) for s in items(frames, schema, "shades")]
+    groups = [schema.expand(g) for g in items(frames, schema, "groups")]
+    rooms = items(frames, schema, "rooms")
+    assert sorted(s["shadeId"] for s in shades) == sorted(s["shadeId"] for s in controller["shades"])
+    assert sorted(g["groupId"] for g in groups) == sorted(g["groupId"] for g in controller["groups"])
+    assert len(rooms) == len(controller["rooms"])
+    by_id = {s["shadeId"]: s for s in controller["shades"]}
+    for s in shades:
+        assert s.get("name", "") == by_id[s["shadeId"]]["name"]
+        assert s.get("remoteAddress", 0) == by_id[s["shadeId"]]["remoteAddress"]
+    for g in groups:
+        linked = sorted(x["shadeId"] for x in next(cg for cg in controller["groups"] if cg["groupId"] == g["groupId"])["linkedShades"])
+        assert sorted(g.get("linkedShades", [])) == linked
+    assert len(items(frames, schema, "header")) == 1
+    assert len(items(frames, schema, "status")) == 1
+
+
+def test_sync_waits_for_next(connect):
+    c = connect()
+    c.recv()  # hello
+    first = c.recv()
+    assert first["t"] == 4
+    # Without next no more sync comes, unless the whole sync fit one frame.
+    # Events may come meanwhile, such as the 30 s memory status.
+    while True:
+        try:
+            more = c.recv(timeout=2)
+        except TimeoutError:
+            return
+        if more["t"] >= 128:
+            continue
+        assert more["t"] == 5, "a second sync frame came without next"
+        return
+
+
+def test_resync_sends_it_all_again(connect, schema):
+    c = connect(schema=schema.hash)
+    _, first, _ = run_sync(c)
+    c.send(schema.type_numbers["sync"])
+    _, again, _ = run_sync(c, hello=False)
+    assert len(items(again, schema, "shades")) == len(items(first, schema, "shades"))
+    for f in again:
+        assert "T" not in f["d"]
+
+
+# What a recording must not keep from a real installation: the remote
+# addresses and rolling codes would let anyone forge frames for its shades.
+SCRUBBED = {"remoteAddress", "lastRollingCode", "sourceAddress", "startingAddress", "repeaters",
+            "name", "serverId", "hostname", "ssid"}
+
+
+def scrub(value, schema, counter):
+    """Fake values of the same kind in place of anything identifying."""
+    if isinstance(value, list):
+        return [scrub(v, schema, counter) for v in value]
+    if not isinstance(value, dict):
+        return value
+    out = {}
+    for key, v in value.items():
+        name = schema.fields.get(key, (key,))[0]
+        if name in SCRUBBED:
+            counter[0] += 1
+            if isinstance(v, list):
+                v = [0x100000 + counter[0] * 16 + i for i in range(len(v))]
+            elif isinstance(v, str):
+                v = "%s %u" % (name, counter[0])
+            else:
+                v = 0x100000 + counter[0]
+        out[key] = scrub(v, schema, counter)
+    return out
+
+
+def test_record_fixtures(connect, recording):
+    """With --record, saves hello and a whole first sync for the other suites,
+    with the installation's identifying values replaced."""
+    if not recording:
+        return
+    c = connect()
+    schema = Schema()
+    run_sync(c, schema)
+    os.makedirs(FIXTURES, exist_ok=True)
+    hello = json.loads(c.raw[0])
+    with open(os.path.join(FIXTURES, "hello.json"), "w", encoding="utf-8") as f:
+        json.dump(hello, f, indent=1)
+    counter = [0]
+    frames = [json.loads(r) for r in c.raw[1:]]
+    for frame in frames:
+        frame["d"] = scrub(frame["d"], schema, counter)
+    with open(os.path.join(FIXTURES, "sync.json"), "w", encoding="utf-8") as f:
+        json.dump(frames, f, indent=1)
+    expanded = {
+        "hash": hello["d"]["sh"],
+        "types": {str(k): v for k, v in sorted(schema.types.items())},
+        "fields": {k: {"name": n, "kind": kind, "default": d} for k, (n, kind, d) in sorted(schema.fields.items())},
+        "errors": {str(k): v for k, v in sorted(schema.errors.items())},
+        "feeds": {str(k): v for k, v in sorted(schema.feeds.items())},
+    }
+    with open(os.path.join(FIXTURES, "schema.json"), "w", encoding="utf-8") as f:
+        json.dump(expanded, f, indent=1)

test/fixtures/proto2/hello.json

@@ -0,0 +1,8 @@
+{
+ "v": 2,
+ "t": 1,
+ "d": {
+  "fw": "v3.0.14",
+  "sh": "8198c564"
+ }
+}

test/fixtures/proto2/schema.json

@@ -0,0 +1,510 @@
+{
+ "hash": "8198c564",
+ "types": {
+  "1": "hello",
+  "2": "reply",
+  "3": "next",
+  "4": "sync",
+  "5": "syncEnd",
+  "6": "subscribe",
+  "7": "unsubscribe",
+  "128": "shadeState",
+  "129": "shade",
+  "130": "shadeRemoved",
+  "131": "group",
+  "132": "groupState",
+  "133": "groupRemoved",
+  "134": "room",
+  "135": "roomRemoved",
+  "136": "roomOrder",
+  "137": "shadeOrder",
+  "138": "groupOrder",
+  "139": "shadeCommand",
+  "140": "memory",
+  "141": "wifi",
+  "142": "ethernet",
+  "143": "frame",
+  "144": "frequencyScan",
+  "145": "repeaters"
+ },
+ "fields": {
+  "D": {
+   "name": "downTime",
+   "kind": "i",
+   "default": 0
+  },
+  "E": {
+   "name": "errors",
+   "kind": "l",
+   "default": 0
+  },
+  "F": {
+   "name": "fields",
+   "kind": "l",
+   "default": 0
+  },
+  "G": {
+   "name": "groups",
+   "kind": "l",
+   "default": 0
+  },
+  "M": {
+   "name": "myTiltPos",
+   "kind": "i",
+   "default": -1
+  },
+  "Q": {
+   "name": "feeds",
+   "kind": "l",
+   "default": 0
+  },
+  "R": {
+   "name": "rooms",
+   "kind": "l",
+   "default": 0
+  },
+  "S": {
+   "name": "shades",
+   "kind": "l",
+   "default": 0
+  },
+  "T": {
+   "name": "types",
+   "kind": "l",
+   "default": 0
+  },
+  "U": {
+   "name": "upTime",
+   "kind": "i",
+   "default": 0
+  },
+  "X": {
+   "name": "repeaters",
+   "kind": "l",
+   "default": 0
+  },
+  "Y": {
+   "name": "tiltTime",
+   "kind": "i",
+   "default": 0
+  },
+  "Z": {
+   "name": "status",
+   "kind": "o",
+   "default": 0
+  },
+  "a": {
+   "name": "remoteAddress",
+   "kind": "i",
+   "default": 0
+  },
+  "bl": {
+   "name": "bitLength",
+   "kind": "i",
+   "default": 56
+  },
+  "bt": {
+   "name": "bits",
+   "kind": "i",
+   "default": 0
+  },
+  "c": {
+   "name": "lastRollingCode",
+   "kind": "i",
+   "default": 0
+  },
+  "ch": {
+   "name": "channel",
+   "kind": "i",
+   "default": 0
+  },
+  "cm": {
+   "name": "command",
+   "kind": "s",
+   "default": 0
+  },
+  "co": {
+   "name": "connected",
+   "kind": "b",
+   "default": 0
+  },
+  "cp": {
+   "name": "chipModel",
+   "kind": "s",
+   "default": 0
+  },
+  "ct": {
+   "name": "connType",
+   "kind": "s",
+   "default": 0
+  },
+  "d": {
+   "name": "direction",
+   "kind": "i",
+   "default": 0
+  },
+  "e": {
+   "name": "code",
+   "kind": "i",
+   "default": 0
+  },
+  "ek": {
+   "name": "encKey",
+   "kind": "i",
+   "default": 0
+  },
+  "f": {
+   "name": "flags",
+   "kind": "i",
+   "default": 0
+  },
+  "fc": {
+   "name": "flipCommands",
+   "kind": "b",
+   "default": 0
+  },
+  "fd": {
+   "name": "fullDuplex",
+   "kind": "b",
+   "default": 0
+  },
+  "fe": {
+   "name": "feed",
+   "kind": "i",
+   "default": 0
+  },
+  "fp": {
+   "name": "flipPosition",
+   "kind": "b",
+   "default": 0
+  },
+  "fq": {
+   "name": "frequency",
+   "kind": "n",
+   "default": 0
+  },
+  "fr": {
+   "name": "freeHeap",
+   "kind": "i",
+   "default": 0
+  },
+  "fw": {
+   "name": "fw",
+   "kind": "s",
+   "default": 0
+  },
+  "g": {
+   "name": "groupId",
+   "kind": "i",
+   "default": 0
+  },
+  "gd": {
+   "name": "gpioDown",
+   "kind": "i",
+   "default": 0
+  },
+  "gl": {
+   "name": "gpioLLTrigger",
+   "kind": "b",
+   "default": 0
+  },
+  "gm": {
+   "name": "gpioMy",
+   "kind": "i",
+   "default": 0
+  },
+  "gu": {
+   "name": "gpioUp",
+   "kind": "i",
+   "default": 0
+  },
+  "h": {
+   "name": "header",
+   "kind": "o",
+   "default": 0
+  },
+  "hn": {
+   "name": "hostname",
+   "kind": "s",
+   "default": 0
+  },
+  "hs": {
+   "name": "hwsync",
+   "kind": "i",
+   "default": 0
+  },
+  "i": {
+   "name": "paired",
+   "kind": "b",
+   "default": 0
+  },
+  "ig": {
+   "name": "inGroup",
+   "kind": "b",
+   "default": 0
+  },
+  "is": {
+   "name": "ids",
+   "kind": "l",
+   "default": 0
+  },
+  "k": {
+   "name": "ok",
+   "kind": "b",
+   "default": 0
+  },
+  "li": {
+   "name": "light",
+   "kind": "b",
+   "default": 0
+  },
+  "lk": {
+   "name": "linkedRemotes",
+   "kind": "l",
+   "default": 0
+  },
+  "lp": {
+   "name": "loopRate",
+   "kind": "i",
+   "default": 0
+  },
+  "ls": {
+   "name": "linkedShades",
+   "kind": "l",
+   "default": 0
+  },
+  "lx": {
+   "name": "loopMaxUs",
+   "kind": "i",
+   "default": 0
+  },
+  "m": {
+   "name": "myPos",
+   "kind": "i",
+   "default": -1
+  },
+  "md": {
+   "name": "model",
+   "kind": "s",
+   "default": 0
+  },
+  "mn": {
+   "name": "minHeap",
+   "kind": "i",
+   "default": 0
+  },
+  "mx": {
+   "name": "maxHeap",
+   "kind": "i",
+   "default": 0
+  },
+  "n": {
+   "name": "name",
+   "kind": "s",
+   "default": 0
+  },
+  "o": {
+   "name": "sortOrder",
+   "kind": "i",
+   "default": 0
+  },
+  "p": {
+   "name": "position",
+   "kind": "i",
+   "default": 0
+  },
+  "pr": {
+   "name": "proto",
+   "kind": "i",
+   "default": 0
+  },
+  "pu": {
+   "name": "pulses",
+   "kind": "l",
+   "default": 0
+  },
+  "q": {
+   "name": "tiltPosition",
+   "kind": "i",
+   "default": 0
+  },
+  "r": {
+   "name": "roomId",
+   "kind": "i",
+   "default": 0
+  },
+  "rp": {
+   "name": "repeats",
+   "kind": "i",
+   "default": 1
+  },
+  "rs": {
+   "name": "rssi",
+   "kind": "i",
+   "default": 0
+  },
+  "rw": {
+   "name": "rawCmd",
+   "kind": "i",
+   "default": 0
+  },
+  "s": {
+   "name": "shadeId",
+   "kind": "i",
+   "default": 0
+  },
+  "sa": {
+   "name": "startingAddress",
+   "kind": "i",
+   "default": 0
+  },
+  "sc": {
+   "name": "scanning",
+   "kind": "b",
+   "default": 0
+  },
+  "sd": {
+   "name": "ssid",
+   "kind": "s",
+   "default": 0
+  },
+  "sg": {
+   "name": "strength",
+   "kind": "i",
+   "default": 0
+  },
+  "sh": {
+   "name": "schemaHash",
+   "kind": "s",
+   "default": 0
+  },
+  "sm": {
+   "name": "simMy",
+   "kind": "b",
+   "default": 0
+  },
+  "so": {
+   "name": "source",
+   "kind": "s",
+   "default": 0
+  },
+  "sp": {
+   "name": "speed",
+   "kind": "i",
+   "default": 0
+  },
+  "sr": {
+   "name": "sourceAddress",
+   "kind": "i",
+   "default": 0
+  },
+  "ss": {
+   "name": "sunSensor",
+   "kind": "b",
+   "default": 0
+  },
+  "st": {
+   "name": "shadeType",
+   "kind": "i",
+   "default": 0
+  },
+  "sv": {
+   "name": "serverId",
+   "kind": "s",
+   "default": 0
+  },
+  "t": {
+   "name": "target",
+   "kind": "i",
+   "default": 0
+  },
+  "tf": {
+   "name": "testFreq",
+   "kind": "n",
+   "default": 0
+  },
+  "to": {
+   "name": "totalHeap",
+   "kind": "i",
+   "default": 0
+  },
+  "tr": {
+   "name": "testRssi",
+   "kind": "i",
+   "default": 0
+  },
+  "u": {
+   "name": "tiltTarget",
+   "kind": "i",
+   "default": 0
+  },
+  "up": {
+   "name": "uptime",
+   "kind": "i",
+   "default": 0
+  },
+  "va": {
+   "name": "valid",
+   "kind": "b",
+   "default": 0
+  },
+  "w": {
+   "name": "tiltDirection",
+   "kind": "i",
+   "default": 0
+  },
+  "x": {
+   "name": "desc",
+   "kind": "s",
+   "default": 0
+  },
+  "xg": {
+   "name": "maxGroups",
+   "kind": "i",
+   "default": 0
+  },
+  "xl": {
+   "name": "maxLinkedRemotes",
+   "kind": "i",
+   "default": 0
+  },
+  "xm": {
+   "name": "maxGroupedShades",
+   "kind": "i",
+   "default": 0
+  },
+  "xr": {
+   "name": "maxRooms",
+   "kind": "i",
+   "default": 0
+  },
+  "xs": {
+   "name": "maxShades",
+   "kind": "i",
+   "default": 0
+  },
+  "y": {
+   "name": "tiltType",
+   "kind": "i",
+   "default": 0
+  },
+  "z": {
+   "name": "stepSize",
+   "kind": "i",
+   "default": 100
+  }
+ },
+ "errors": {
+  "1": "badRequest",
+  "2": "unknownOp",
+  "3": "unauthorized",
+  "4": "notFound",
+  "5": "busy",
+  "6": "radioUnavailable",
+  "7": "internal"
+ },
+ "feeds": {
+  "1": "frames",
+  "2": "frequencyScan"
+ }
+}

test/test_json/test_json.cpp

@@ -228,6 +228,42 @@ static void test_setBuffer_after_discard_writes_again() {
   TEST_ASSERT_EQUAL_STRING("{}", storage);
 }
 
+/*********************************************************************
+ * Mark and roll back
+ ********************************************************************/
+static void test_rollback_drops_what_was_written_since_the_mark() {
+  fmt.beginObject();
+  fmt.addElem("a", (uint8_t)1);
+  const json_mark_t m = fmt.mark();
+  fmt.addElem("b", (uint8_t)2);
+  fmt.beginArray("c");
+  fmt.rollback(m);
+  fmt.addElem("d", (uint8_t)3);
+  fmt.endObject();
+  TEST_ASSERT_EQUAL_STRING("{\"a\":1,\"d\":3}", storage);
+}
+static void test_rollback_clears_an_overflow() {
+  fmt.beginObject();
+  const json_mark_t m = fmt.mark();
+  char big[BUFF_SIZE + 1];
+  memset(big, 'z', BUFF_SIZE);
+  big[BUFF_SIZE] = '\0';
+  fmt.addElem("big", big);
+  TEST_ASSERT_TRUE(fmt.overflowed());
+  fmt.rollback(m);
+  TEST_ASSERT_FALSE(fmt.overflowed());
+  fmt.addElem("ok", true);
+  fmt.endObject();
+  TEST_ASSERT_EQUAL_STRING("{\"ok\":true}", storage);
+}
+static void test_rollback_at_the_start_leaves_an_empty_buffer() {
+  const json_mark_t m = fmt.mark();
+  fmt.beginObject();
+  fmt.rollback(m);
+  TEST_ASSERT_EQUAL_UINT32(0, fmt.length());
+  TEST_ASSERT_EQUAL_STRING("", storage);
+}
+
 int main(int argc, char **argv) {
   (void)argc;
   (void)argv;
@@ -251,5 +287,8 @@ int main(int argc, char **argv) {
   RUN_TEST(test_setBuffer_resets_length);
   RUN_TEST(test_discard_writes_nothing);
   RUN_TEST(test_setBuffer_after_discard_writes_again);
+  RUN_TEST(test_rollback_drops_what_was_written_since_the_mark);
+  RUN_TEST(test_rollback_clears_an_overflow);
+  RUN_TEST(test_rollback_at_the_start_leaves_an_empty_buffer);
   return UNITY_END();
 }

test/test_modeleditor/test_modeleditor.cpp

@@ -134,6 +134,66 @@ static void test_255_means_no_pin() {
   TEST_ASSERT_EQUAL_INT8(0, conflict(shade_types::roller, radio_proto::GP_Remote, 255, 26, 255));
 }
 
+/*********************************************************************
+ * What protocol 2 clients are sent after an edit
+ ********************************************************************/
+static edit_result_t resultFor(uint8_t id, uint8_t shadeId = 0) {
+  edit_result_t r;
+  r.id = id;
+  r.shadeId = shadeId;
+  return r;
+}
+static void test_a_shade_edit_sends_the_whole_shade() {
+  const model_edit_type_t types[] = {
+    model_edit_type_t::add_shade, model_edit_type_t::save_shade, model_edit_type_t::set_paired,
+    model_edit_type_t::link_remote, model_edit_type_t::unlink_remote
+  };
+  for(model_edit_type_t t : types) {
+    const edit_changes_t c = editChanges(t, resultFor(7));
+    TEST_ASSERT_EQUAL_UINT8(7, c.shadeId);
+    TEST_ASSERT_EQUAL_UINT8(0, c.groupId);
+    TEST_ASSERT_EQUAL_UINT8(0, c.roomId);
+  }
+}
+static void test_a_group_edit_sends_the_whole_group() {
+  TEST_ASSERT_EQUAL_UINT8(3, editChanges(model_edit_type_t::add_group, resultFor(3)).groupId);
+  TEST_ASSERT_EQUAL_UINT8(3, editChanges(model_edit_type_t::save_group, resultFor(3)).groupId);
+}
+static void test_linking_a_shade_to_a_group_sends_both() {
+  // The shade's inGroup changes with the group's shades.
+  for(model_edit_type_t t : {model_edit_type_t::link_to_group, model_edit_type_t::unlink_from_group}) {
+    const edit_changes_t c = editChanges(t, resultFor(3, 9));
+    TEST_ASSERT_EQUAL_UINT8(3, c.groupId);
+    TEST_ASSERT_EQUAL_UINT8(9, c.shadeId);
+  }
+}
+static void test_a_room_edit_sends_the_room() {
+  TEST_ASSERT_EQUAL_UINT8(2, editChanges(model_edit_type_t::add_room, resultFor(2)).roomId);
+  TEST_ASSERT_EQUAL_UINT8(2, editChanges(model_edit_type_t::save_room, resultFor(2)).roomId);
+}
+static void test_a_sort_edit_sends_its_list() {
+  TEST_ASSERT_EQUAL(sort_list_t::rooms, editChanges(model_edit_type_t::room_order, resultFor(0)).order);
+  TEST_ASSERT_EQUAL(sort_list_t::shades, editChanges(model_edit_type_t::shade_order, resultFor(0)).order);
+  TEST_ASSERT_EQUAL(sort_list_t::groups, editChanges(model_edit_type_t::group_order, resultFor(0)).order);
+  TEST_ASSERT_EQUAL(sort_list_t::none, editChanges(model_edit_type_t::save_shade, resultFor(1)).order);
+}
+static void test_a_repeater_edit_sends_the_repeaters() {
+  TEST_ASSERT_TRUE(editChanges(model_edit_type_t::link_repeater, resultFor(0)).repeaters);
+  TEST_ASSERT_TRUE(editChanges(model_edit_type_t::unlink_repeater, resultFor(0)).repeaters);
+  TEST_ASSERT_FALSE(editChanges(model_edit_type_t::save_room, resultFor(1)).repeaters);
+}
+static void test_deletes_send_nothing_more() {
+  // Their removals go out as they happen.
+  for(model_edit_type_t t : {model_edit_type_t::delete_room, model_edit_type_t::delete_shade, model_edit_type_t::delete_group}) {
+    const edit_changes_t c = editChanges(t, resultFor(4));
+    TEST_ASSERT_EQUAL_UINT8(0, c.shadeId);
+    TEST_ASSERT_EQUAL_UINT8(0, c.groupId);
+    TEST_ASSERT_EQUAL_UINT8(0, c.roomId);
+    TEST_ASSERT_EQUAL(sort_list_t::none, c.order);
+    TEST_ASSERT_FALSE(c.repeaters);
+  }
+}
+
 int main(int argc, char **argv) {
   (void)argc;
   (void)argv;
@@ -157,5 +217,12 @@ int main(int argc, char **argv) {
   RUN_TEST(test_a_pin_another_shade_uses);
   RUN_TEST(test_only_the_pins_a_shade_drives_are_checked);
   RUN_TEST(test_255_means_no_pin);
+  RUN_TEST(test_a_shade_edit_sends_the_whole_shade);
+  RUN_TEST(test_a_group_edit_sends_the_whole_group);
+  RUN_TEST(test_linking_a_shade_to_a_group_sends_both);
+  RUN_TEST(test_a_room_edit_sends_the_room);
+  RUN_TEST(test_a_sort_edit_sends_its_list);
+  RUN_TEST(test_a_repeater_edit_sends_the_repeaters);
+  RUN_TEST(test_deletes_send_nothing_more);
   return UNITY_END();
 }

test/test_proto/test_proto.cpp

@@ -0,0 +1,406 @@
+#include <unity.h>
+#include <set>
+#include <string>
+#include <vector>
+#include <ArduinoJson.h>
+#include "ProtoSchema.h"
+#include "ProtoFrame.h"
+#include "ProtoSync.h"
+#include "ProtoParse.h"
+
+// Socket protocol 2: the schema table, the writer, the sync and reading what
+// clients send (docs/PROTOCOL.md).
+
+void setUp() {}
+void tearDown() {}
+
+static char buff[PROTO_MAX_FRAME];
+static BufferedJsonFormatter fmt;
+
+/*********************************************************************
+ * The schema table
+ ********************************************************************/
+static void test_field_keys_and_names_are_unique() {
+  std::set<std::string> keys, names;
+  for(uint8_t i = 0; i < static_cast<uint8_t>(proto_field_t::count); i++) {
+    const proto_field_info_t &f = protoField(static_cast<proto_field_t>(i));
+    TEST_ASSERT_TRUE_MESSAGE(keys.insert(f.key).second, f.key);
+    TEST_ASSERT_TRUE_MESSAGE(names.insert(f.name).second, f.name);
+    TEST_ASSERT_TRUE_MESSAGE(strlen(f.key) >= 1 && strlen(f.key) <= 2, f.key);
+  }
+}
+static void test_every_field_has_a_known_kind() {
+  for(uint8_t i = 0; i < static_cast<uint8_t>(proto_field_t::count); i++) {
+    const proto_field_info_t &f = protoField(static_cast<proto_field_t>(i));
+    TEST_ASSERT_NOT_NULL_MESSAGE(strchr("binslo", f.kind), f.name);
+  }
+}
+static void test_type_numbers_are_unique_and_in_their_sender_range() {
+  std::set<uint8_t> numbers;
+  for(uint8_t i = 0; i < protoTypeCount(); i++) {
+    const proto_named_t &t = protoTypeAt(i);
+    TEST_ASSERT_TRUE_MESSAGE(numbers.insert(t.number).second, t.name);
+    if(t.number >= 128) TEST_ASSERT_EQUAL_CHAR_MESSAGE('d', t.sender, t.name);
+    else if(t.number >= 16) TEST_ASSERT_EQUAL_CHAR_MESSAGE('c', t.sender, t.name);
+    else TEST_ASSERT_TRUE_MESSAGE(t.number >= 1, t.name);
+  }
+}
+static void test_error_and_feed_numbers_are_unique() {
+  std::set<uint8_t> errors, feeds;
+  for(uint8_t i = 0; i < protoErrorCount(); i++) TEST_ASSERT_TRUE(errors.insert(protoErrorAt(i).number).second);
+  for(uint8_t i = 0; i < protoFeedCount(); i++) TEST_ASSERT_TRUE(feeds.insert(protoFeedAt(i).number).second);
+  TEST_ASSERT_FALSE(errors.count(0));
+}
+static void test_bootstrap_keys_are_fixed() {
+  // A client needs these before it has the schema.
+  TEST_ASSERT_EQUAL_STRING("fw", protoField(proto_field_t::fw).key);
+  TEST_ASSERT_EQUAL_STRING("sh", protoField(proto_field_t::schemaHash).key);
+  TEST_ASSERT_EQUAL_STRING("T", protoField(proto_field_t::types).key);
+  TEST_ASSERT_EQUAL_STRING("F", protoField(proto_field_t::fields).key);
+  TEST_ASSERT_EQUAL_STRING("E", protoField(proto_field_t::errors).key);
+  TEST_ASSERT_EQUAL_STRING("Q", protoField(proto_field_t::feeds).key);
+}
+static void test_schema_hash_is_eight_hex_digits_and_stable() {
+  const std::string first = protoSchemaHash();
+  TEST_ASSERT_EQUAL_UINT32(8, first.size());
+  TEST_ASSERT_EQUAL_UINT32(std::string::npos, first.find_first_not_of("0123456789abcdef"));
+  TEST_ASSERT_EQUAL_STRING(first.c_str(), protoSchemaHash());
+}
+static void test_field_by_key() {
+  TEST_ASSERT_EQUAL_STRING("shadeId", protoFieldByKey("s")->name);
+  TEST_ASSERT_NULL(protoFieldByKey("??"));
+}
+
+/*********************************************************************
+ * The writer
+ ********************************************************************/
+static std::string writeObject(void (*fill)(ProtoWriter &)) {
+  fmt.setBuffer(buff, sizeof(buff));
+  ProtoWriter w(fmt);
+  fmt.beginObject();
+  fill(w);
+  fmt.endObject();
+  return buff;
+}
+static void test_values_at_their_default_are_left_out() {
+  TEST_ASSERT_EQUAL_STRING("{}", writeObject([](ProtoWriter &w) {
+    w.add(proto_field_t::position, (int8_t)0);
+    w.add(proto_field_t::myPos, (int8_t)-1);
+    w.add(proto_field_t::repeats, (uint8_t)1);
+    w.add(proto_field_t::paired, false);
+    w.add(proto_field_t::name, "");
+    w.add(proto_field_t::name, (const char *)nullptr);
+  }).c_str());
+}
+static void test_values_off_their_default_are_written_by_key() {
+  TEST_ASSERT_EQUAL_STRING("{\"p\":50,\"m\":0,\"i\":1,\"n\":\"Office\",\"a\":4294967295,\"q\":-5}", writeObject([](ProtoWriter &w) {
+    w.add(proto_field_t::position, (int8_t)50);
+    w.add(proto_field_t::myPos, (int8_t)0);
+    w.add(proto_field_t::paired, true);
+    w.add(proto_field_t::name, "Office");
+    w.add(proto_field_t::remoteAddress, (uint32_t)0xFFFFFFFF);
+    w.add(proto_field_t::tiltPosition, (int8_t)-5);
+  }).c_str());
+}
+
+/*********************************************************************
+ * Frames
+ ********************************************************************/
+static void test_hello_frame() {
+  fmt.setBuffer(buff, sizeof(buff));
+  writeHello(fmt, "3.1.0");
+  const std::string expected = std::string("{\"v\":2,\"t\":1,\"d\":{\"fw\":\"3.1.0\",\"sh\":\"") + protoSchemaHash() + "\"}}";
+  TEST_ASSERT_EQUAL_STRING(expected.c_str(), buff);
+}
+static void test_error_reply_frame() {
+  fmt.setBuffer(buff, sizeof(buff));
+  writeErrorReply(fmt, 17, proto_error_t::notFound, "Shade not found");
+  // ok is 0, its default, so it is left out.
+  TEST_ASSERT_EQUAL_STRING("{\"v\":2,\"t\":2,\"id\":17,\"d\":{\"e\":4,\"x\":\"Shade not found\"}}", buff);
+}
+static void test_ok_reply_frame() {
+  fmt.setBuffer(buff, sizeof(buff));
+  writeOkReply(fmt, 9);
+  TEST_ASSERT_EQUAL_STRING("{\"v\":2,\"t\":2,\"id\":9,\"d\":{\"k\":1}}", buff);
+}
+
+/*********************************************************************
+ * Items
+ ********************************************************************/
+static shade_snapshot_t restingShade(uint8_t id) {
+  shade_snapshot_t s = {};
+  s.shadeId = id;
+  s.myPos = s.myTiltPos = -1;
+  s.stepSize = 100;
+  s.bitLength = 56;
+  s.repeats = 1;
+  return s;
+}
+static void test_shade_state_of_a_resting_shade_is_its_id() {
+  const shade_snapshot_t s = restingShade(3);
+  fmt.setBuffer(buff, sizeof(buff));
+  ProtoWriter w(fmt);
+  fmt.beginObject();
+  protoShadeState(w, s);
+  fmt.endObject();
+  TEST_ASSERT_EQUAL_STRING("{\"s\":3}", buff);
+}
+static void test_group_lists_shade_ids() {
+  group_snapshot_t g = {};
+  g.groupId = 2;
+  g.bitLength = 56;
+  g.repeats = 1;
+  g.linkedShadeCount = 3;
+  g.linkedShades[0] = 4;
+  g.linkedShades[1] = 7;
+  g.linkedShades[2] = 9;
+  fmt.setBuffer(buff, sizeof(buff));
+  ProtoWriter w(fmt);
+  fmt.beginObject();
+  protoGroup(w, g);
+  fmt.endObject();
+  TEST_ASSERT_EQUAL_STRING("{\"g\":2,\"ls\":[4,7,9]}", buff);
+}
+
+/*********************************************************************
+ * Sync
+ ********************************************************************/
+// The widest name: 20 control characters, each escaped to \u00XX.
+static void widestName(char *name) {
+  memset(name, 0x01, 20);
+  name[20] = '\0';
+}
+static shade_snapshot_t widestShade(uint8_t id) {
+  shade_snapshot_t s = {};
+  s.shadeId = id;
+  s.roomId = 255;
+  widestName(s.name);
+  s.remoteAddress = s.upTime = s.downTime = s.tiltTime = 0xFFFFFFFF;
+  s.stepSize = s.lastRollingCode = 0xFFFF;
+  s.position = s.target = s.myPos = s.direction = -128;
+  s.tiltPosition = s.tiltTarget = s.myTiltPos = s.tiltDirection = -128;
+  s.tiltType = s.shadeType = s.bitLength = s.proto = s.flags = s.repeats = 255;
+  s.gpioUp = s.gpioDown = s.gpioMy = 255;
+  s.sortOrder = -128;
+  s.paired = s.flipCommands = s.flipPosition = s.inGroup = true;
+  s.sunSensor = s.light = s.simMy = s.gpioLLTrigger = true;
+  s.linkedRemoteCount = SOMFY_MAX_LINKED_REMOTES;
+  for(uint8_t j = 0; j < SOMFY_MAX_LINKED_REMOTES; j++) s.linkedRemotes[j] = {0xFFFFFFFF, 0xFFFF};
+  return s;
+}
+// A source over plain arrays; a slot with id 0 is empty.
+class ArraySource : public SyncSource {
+  public:
+    room_snapshot_t rooms[SOMFY_MAX_ROOMS] = {};
+    shade_snapshot_t shades[SOMFY_MAX_SHADES] = {};
+    group_snapshot_t groups[SOMFY_MAX_GROUPS] = {};
+    uint32_t repeaters[SOMFY_MAX_REPEATERS] = {};
+    bool room(uint8_t slot, room_snapshot_t &out) override { out = rooms[slot]; return out.roomId != 0; }
+    bool shade(uint8_t slot, shade_snapshot_t &out) override { out = shades[slot]; return out.shadeId != 0; }
+    bool group(uint8_t slot, group_snapshot_t &out) override { out = groups[slot]; return out.groupId != 0; }
+    bool repeater(uint8_t slot, uint32_t &address) override { address = repeaters[slot]; return address != 0; }
+    void header(ProtoWriter &w) override { w.add(proto_field_t::maxShades, (uint8_t)SOMFY_MAX_SHADES); }
+    void status(ProtoWriter &w) override { w.add(proto_field_t::freeHeap, (uint32_t)80000); }
+};
+static void fillWidest(ArraySource &src) {
+  for(uint8_t i = 0; i < SOMFY_MAX_ROOMS; i++) {
+    src.rooms[i].roomId = i + 1;
+    widestName(src.rooms[i].name);
+    src.rooms[i].sortOrder = -128;
+  }
+  for(uint8_t i = 0; i < SOMFY_MAX_SHADES; i++) src.shades[i] = widestShade(i + 1);
+  for(uint8_t i = 0; i < SOMFY_MAX_GROUPS; i++) {
+    group_snapshot_t &g = src.groups[i];
+    g.groupId = i + 1;
+    g.roomId = 255;
+    widestName(g.name);
+    g.remoteAddress = 0xFFFFFFFF;
+    g.lastRollingCode = 0xFFFF;
+    g.bitLength = g.proto = g.flags = g.repeats = 255;
+    g.sortOrder = -128;
+    g.linkedShadeCount = SOMFY_MAX_GROUPED_SHADES;
+    for(uint8_t j = 0; j < SOMFY_MAX_GROUPED_SHADES; j++) g.linkedShades[j] = j + 1;
+  }
+  for(uint8_t i = 0; i < SOMFY_MAX_REPEATERS; i++) src.repeaters[i] = 0xFFFFFFFF;
+}
+// Every frame of a sync, parsed.  Each must be valid JSON within size bytes.
+static std::vector<JsonDocument> runSync(SyncSource &src, bool withSchema, size_t size) {
+  std::vector<JsonDocument> frames;
+  static char frame[PROTO_MAX_FRAME];
+  sync_cursor_t cur = beginSync(withSchema);
+  for(int guard = 0; guard < 1000; guard++) {
+    const size_t len = writeSyncFrame(src, cur, frame, size);
+    if(len == 0) return frames;
+    TEST_ASSERT_LESS_THAN_UINT32(size, len);
+    JsonDocument doc;
+    TEST_ASSERT_EQUAL_MESSAGE(DeserializationError::Ok, deserializeJson(doc, frame, len).code(), frame);
+    TEST_ASSERT_EQUAL(2, doc["v"].as<int>());
+    TEST_ASSERT_EQUAL(static_cast<int>(proto_type_t::sync), doc["t"].as<int>());
+    frames.push_back(std::move(doc));
+  }
+  TEST_FAIL_MESSAGE("the sync never ended");
+  return frames;
+}
+// How many items a section has over every frame.
+static size_t count(const std::vector<JsonDocument> &frames, proto_field_t section) {
+  size_t n = 0;
+  for(const JsonDocument &doc : frames) {
+    JsonVariantConst v = doc["d"][protoField(section).key];
+    if(v.is<JsonArrayConst>()) n += v.as<JsonArrayConst>().size();
+    else if(!v.isNull()) n++;
+  }
+  return n;
+}
+static void test_full_sync_sends_every_item_whole() {
+  static ArraySource src;
+  fillWidest(src);
+  const std::vector<JsonDocument> frames = runSync(src, true, PROTO_MAX_FRAME);
+  TEST_ASSERT_GREATER_THAN_UINT32(1, frames.size());
+  TEST_ASSERT_EQUAL_UINT32(protoTypeCount(), count(frames, proto_field_t::types));
+  TEST_ASSERT_EQUAL_UINT32(static_cast<uint8_t>(proto_field_t::count), count(frames, proto_field_t::fields));
+  TEST_ASSERT_EQUAL_UINT32(protoErrorCount(), count(frames, proto_field_t::errors));
+  TEST_ASSERT_EQUAL_UINT32(protoFeedCount(), count(frames, proto_field_t::feeds));
+  TEST_ASSERT_EQUAL_UINT32(1, count(frames, proto_field_t::header));
+  TEST_ASSERT_EQUAL_UINT32(SOMFY_MAX_ROOMS, count(frames, proto_field_t::rooms));
+  TEST_ASSERT_EQUAL_UINT32(SOMFY_MAX_SHADES, count(frames, proto_field_t::shades));
+  TEST_ASSERT_EQUAL_UINT32(SOMFY_MAX_GROUPS, count(frames, proto_field_t::groups));
+  TEST_ASSERT_EQUAL_UINT32(SOMFY_MAX_REPEATERS, count(frames, proto_field_t::repeaters));
+  TEST_ASSERT_EQUAL_UINT32(1, count(frames, proto_field_t::status));
+  // Every shade comes out with all its linked remotes.
+  for(const JsonDocument &doc : frames)
+    for(JsonVariantConst s : doc["d"]["S"].as<JsonArrayConst>())
+      TEST_ASSERT_EQUAL_UINT32(SOMFY_MAX_LINKED_REMOTES, s["lk"].as<JsonArrayConst>().size());
+}
+static void test_a_client_with_the_schema_skips_it() {
+  static ArraySource src;
+  src = ArraySource();
+  src.shades[0] = restingShade(1);
+  const std::vector<JsonDocument> frames = runSync(src, false, PROTO_MAX_FRAME);
+  TEST_ASSERT_EQUAL_UINT32(0, count(frames, proto_field_t::types));
+  TEST_ASSERT_EQUAL_UINT32(0, count(frames, proto_field_t::fields));
+  TEST_ASSERT_EQUAL_UINT32(1, frames.size());
+  TEST_ASSERT_EQUAL(1, frames[0]["d"]["S"][0]["s"].as<int>());
+}
+static void test_empty_slots_are_skipped() {
+  static ArraySource src;
+  src = ArraySource();
+  src.shades[5] = restingShade(6);
+  src.shades[20] = restingShade(21);
+  src.rooms[3].roomId = 4;
+  const std::vector<JsonDocument> frames = runSync(src, false, PROTO_MAX_FRAME);
+  TEST_ASSERT_EQUAL_UINT32(2, count(frames, proto_field_t::shades));
+  TEST_ASSERT_EQUAL_UINT32(1, count(frames, proto_field_t::rooms));
+  TEST_ASSERT_EQUAL_UINT32(0, count(frames, proto_field_t::groups));
+  TEST_ASSERT_TRUE(frames[0]["d"]["X"].isNull());
+}
+static void test_a_small_frame_takes_more_frames_and_keeps_items_whole() {
+  static ArraySource src;
+  fillWidest(src);
+  const std::vector<JsonDocument> big = runSync(src, false, PROTO_MAX_FRAME);
+  const std::vector<JsonDocument> small = runSync(src, false, 1400);
+  TEST_ASSERT_GREATER_THAN_UINT32(big.size(), small.size());
+  TEST_ASSERT_EQUAL_UINT32(SOMFY_MAX_SHADES, count(small, proto_field_t::shades));
+  TEST_ASSERT_EQUAL_UINT32(SOMFY_MAX_GROUPS, count(small, proto_field_t::groups));
+}
+static void test_an_item_too_big_for_a_frame_is_left_out_and_the_rest_go() {
+  static ArraySource src;
+  src = ArraySource();
+  src.shades[0] = widestShade(1);
+  src.shades[1] = restingShade(2);
+  // Too small for the widest shade, big enough for a resting one.
+  const std::vector<JsonDocument> frames = runSync(src, false, 200);
+  TEST_ASSERT_EQUAL_UINT32(1, count(frames, proto_field_t::shades));
+}
+static void test_the_largest_shade_fits_one_frame() {
+  // test_full_sync covers it in the sync; this is the shade event on its own.
+  const shade_snapshot_t s = widestShade(1);
+  fmt.setBuffer(buff, sizeof(buff));
+  ProtoWriter w(fmt);
+  beginFrame(fmt, proto_type_t::shade);
+  protoShade(w, s);
+  endFrame(fmt);
+  TEST_ASSERT_FALSE(fmt.overflowed());
+  TEST_ASSERT_LESS_THAN_UINT32(PROTO_MAX_FRAME / 2, fmt.length());
+}
+
+/*********************************************************************
+ * What clients send
+ ********************************************************************/
+static proto_error_t parse(const char *text, proto_message_t &req) {
+  JsonDocument doc;
+  return parseClientFrame(text, strlen(text), doc, req);
+}
+static void test_next_is_read() {
+  proto_message_t req;
+  TEST_ASSERT_EQUAL(proto_error_t::none, parse("{\"v\":2,\"t\":3}", req));
+  TEST_ASSERT_EQUAL_UINT8(static_cast<uint8_t>(proto_type_t::next), req.type);
+  TEST_ASSERT_EQUAL_UINT32(0, req.id);
+}
+static void test_subscribe_carries_its_id_and_payload() {
+  proto_message_t req;
+  JsonDocument doc;
+  const char *text = "{\"v\":2,\"t\":6,\"id\":9,\"d\":{\"fe\":1}}";
+  TEST_ASSERT_EQUAL(proto_error_t::none, parseClientFrame(text, strlen(text), doc, req));
+  TEST_ASSERT_EQUAL_UINT32(9, req.id);
+  TEST_ASSERT_EQUAL_INT32(1, protoInt(doc["d"], proto_field_t::feed));
+}
+static void test_bad_frames_are_refused() {
+  proto_message_t req;
+  TEST_ASSERT_EQUAL(proto_error_t::badRequest, parse("not json", req));
+  TEST_ASSERT_EQUAL(proto_error_t::badRequest, parse("[2,3]", req));
+  TEST_ASSERT_EQUAL(proto_error_t::badRequest, parse("{\"v\":1,\"t\":3}", req));
+  TEST_ASSERT_EQUAL(proto_error_t::badRequest, parse("{\"v\":2}", req));
+  TEST_ASSERT_EQUAL(proto_error_t::badRequest, parse("{\"v\":2,\"t\":\"next\"}", req));
+  TEST_ASSERT_EQUAL(proto_error_t::badRequest, parse("{\"v\":2,\"t\":3,\"d\":[]}", req));
+  TEST_ASSERT_EQUAL(proto_error_t::badRequest, parse("{\"v\":2,\"t\":3,\"id\":\"x\"}", req));
+}
+static void test_types_a_client_may_not_send_are_unknown() {
+  proto_message_t req;
+  // A device event, and a number nobody uses.
+  TEST_ASSERT_EQUAL(proto_error_t::unknownOp, parse("{\"v\":2,\"t\":128,\"id\":5}", req));
+  TEST_ASSERT_EQUAL_UINT32(5, req.id);
+  TEST_ASSERT_EQUAL(proto_error_t::unknownOp, parse("{\"v\":2,\"t\":99,\"id\":6}", req));
+}
+static void test_a_left_out_field_reads_as_its_default() {
+  JsonDocument doc;
+  deserializeJson(doc, "{\"p\":40}");
+  JsonObjectConst d = doc.as<JsonObjectConst>();
+  TEST_ASSERT_EQUAL_INT32(40, protoInt(d, proto_field_t::position));
+  TEST_ASSERT_EQUAL_INT32(-1, protoInt(d, proto_field_t::myPos));
+  TEST_ASSERT_FALSE(protoBool(d, proto_field_t::paired));
+  TEST_ASSERT_NULL(protoString(d, proto_field_t::name));
+  TEST_ASSERT_TRUE(protoHas(d, proto_field_t::position));
+  TEST_ASSERT_FALSE(protoHas(d, proto_field_t::target));
+}
+
+int main(int argc, char **argv) {
+  (void)argc;
+  (void)argv;
+  UNITY_BEGIN();
+  RUN_TEST(test_field_keys_and_names_are_unique);
+  RUN_TEST(test_every_field_has_a_known_kind);
+  RUN_TEST(test_type_numbers_are_unique_and_in_their_sender_range);
+  RUN_TEST(test_error_and_feed_numbers_are_unique);
+  RUN_TEST(test_bootstrap_keys_are_fixed);
+  RUN_TEST(test_schema_hash_is_eight_hex_digits_and_stable);
+  RUN_TEST(test_field_by_key);
+  RUN_TEST(test_values_at_their_default_are_left_out);
+  RUN_TEST(test_values_off_their_default_are_written_by_key);
+  RUN_TEST(test_hello_frame);
+  RUN_TEST(test_error_reply_frame);
+  RUN_TEST(test_ok_reply_frame);
+  RUN_TEST(test_shade_state_of_a_resting_shade_is_its_id);
+  RUN_TEST(test_group_lists_shade_ids);
+  RUN_TEST(test_full_sync_sends_every_item_whole);
+  RUN_TEST(test_a_client_with_the_schema_skips_it);
+  RUN_TEST(test_empty_slots_are_skipped);
+  RUN_TEST(test_a_small_frame_takes_more_frames_and_keeps_items_whole);
+  RUN_TEST(test_an_item_too_big_for_a_frame_is_left_out_and_the_rest_go);
+  RUN_TEST(test_the_largest_shade_fits_one_frame);
+  RUN_TEST(test_next_is_read);
+  RUN_TEST(test_subscribe_carries_its_id_and_payload);
+  RUN_TEST(test_bad_frames_are_refused);
+  RUN_TEST(test_types_a_client_may_not_send_are_unknown);
+  RUN_TEST(test_a_left_out_field_reads_as_its_default);
+  return UNITY_END();
+}

test/test_protoslots/test_protoslots.cpp

@@ -0,0 +1,172 @@
+#include <unity.h>
+#include "ProtoSlots.h"
+
+// Protocol 2's client slots and the pace of their syncs: what async_tcp and
+// loop() each do with a slot, in the orders the two tasks can interleave.
+
+static proto_slot_t slots[PROTO_MAX_CLIENTS];
+
+void setUp() {
+  for(proto_slot_t &s : slots) {
+    freeSlot(s);
+    s.loopId = 0;
+    s.cur = sync_cursor_t();
+    s.awaitingNext = false;
+    s.sentAt = 0;
+  }
+}
+void tearDown() {}
+
+// A client as onConnect sets it up: a slot, then hello, then the sync.
+static proto_slot_t &connect(uint32_t id, bool withSchema = true) {
+  proto_slot_t *slot = claimSlot(slots, PROTO_MAX_CLIENTS, id, withSchema);
+  TEST_ASSERT_NOT_NULL(slot);
+  startSync(*slot);
+  return *slot;
+}
+// What sendSyncFrame does with the cursor: on to the end, or one frame in.
+static void sendAll(proto_slot_t &slot, uint32_t now) {
+  slot.cur.section = sync_section_t::done;
+  syncFrameSent(slot, now);
+}
+static void sendSome(proto_slot_t &slot, uint32_t now) {
+  slot.cur.section = sync_section_t::shades;
+  syncFrameSent(slot, now);
+}
+
+/*********************************************************************
+ * Slots
+ ********************************************************************/
+static void test_a_sixth_client_gets_no_slot() {
+  for(uint32_t id = 1; id <= PROTO_MAX_CLIENTS; id++) connect(id);
+  TEST_ASSERT_NULL(claimSlot(slots, PROTO_MAX_CLIENTS, 99, true));
+}
+static void test_a_freed_slot_is_taken_again_clean() {
+  proto_slot_t &a = connect(1);
+  a.feeds.fetch_or(0x02);
+  a.wants.fetch_or(WANTS_NEXT);
+  freeSlot(a);
+  TEST_ASSERT_NULL(slotOf(slots, PROTO_MAX_CLIENTS, 1));
+  proto_slot_t *b = claimSlot(slots, PROTO_MAX_CLIENTS, 2, false);
+  TEST_ASSERT_EQUAL_PTR(&a, b);
+  TEST_ASSERT_EQUAL_UINT8(0, b->feeds.load());
+  TEST_ASSERT_EQUAL_UINT32(0, b->wants.load());
+  TEST_ASSERT_FALSE(b->withSchema.load());
+}
+static void test_no_sync_before_hello() {
+  // claimSlot alone, as onConnect has it before hello has gone out.
+  proto_slot_t *slot = claimSlot(slots, PROTO_MAX_CLIENTS, 1, true);
+  TEST_ASSERT_EQUAL(sync_step_t::none, nextSyncStep(*slot, 0));
+  startSync(*slot);
+  TEST_ASSERT_EQUAL(sync_step_t::send, nextSyncStep(*slot, 0));
+}
+
+/*********************************************************************
+ * The sync's pace
+ ********************************************************************/
+static void test_a_new_client_gets_the_schema_unless_it_has_it() {
+  proto_slot_t &a = connect(1, true);
+  proto_slot_t &b = connect(2, false);
+  TEST_ASSERT_EQUAL(sync_step_t::send, nextSyncStep(a, 0));
+  TEST_ASSERT_EQUAL(sync_section_t::types, a.cur.section);
+  TEST_ASSERT_EQUAL(sync_step_t::send, nextSyncStep(b, 0));
+  TEST_ASSERT_EQUAL(sync_section_t::header, b.cur.section);
+  TEST_ASSERT_EQUAL_UINT32(2, b.loopId);
+}
+static void test_the_next_frame_waits_for_next() {
+  proto_slot_t &a = connect(1);
+  nextSyncStep(a, 0);
+  sendSome(a, 100);
+  TEST_ASSERT_EQUAL(sync_step_t::none, nextSyncStep(a, 200));
+  a.wants.fetch_or(WANTS_NEXT);
+  TEST_ASSERT_EQUAL(sync_step_t::send, nextSyncStep(a, 300));
+}
+static void test_next_after_the_sync_has_ended_does_nothing() {
+  proto_slot_t &a = connect(1);
+  nextSyncStep(a, 0);
+  sendAll(a, 100);
+  a.wants.fetch_or(WANTS_NEXT);
+  TEST_ASSERT_EQUAL(sync_step_t::none, nextSyncStep(a, 200));
+}
+static void test_a_sync_again_starts_over_without_the_schema() {
+  proto_slot_t &a = connect(1);
+  nextSyncStep(a, 0);
+  sendSome(a, 100);
+  a.wants.fetch_or(WANTS_SYNC_AGAIN);
+  TEST_ASSERT_EQUAL(sync_step_t::send, nextSyncStep(a, 200));
+  TEST_ASSERT_EQUAL(sync_section_t::header, a.cur.section);
+}
+static void test_no_next_for_30s_closes_the_client_once() {
+  proto_slot_t &a = connect(1);
+  nextSyncStep(a, 0);
+  sendSome(a, 1000);
+  TEST_ASSERT_EQUAL(sync_step_t::none, nextSyncStep(a, 1000 + PROTO_SYNC_TIMEOUT_MS));
+  TEST_ASSERT_EQUAL(sync_step_t::timedOut, nextSyncStep(a, 1001 + PROTO_SYNC_TIMEOUT_MS));
+  TEST_ASSERT_EQUAL(sync_step_t::none, nextSyncStep(a, 2000 + PROTO_SYNC_TIMEOUT_MS));
+}
+static void test_a_finished_sync_never_times_out() {
+  proto_slot_t &a = connect(1);
+  nextSyncStep(a, 0);
+  sendAll(a, 0);
+  TEST_ASSERT_EQUAL(sync_step_t::none, nextSyncStep(a, 10 * PROTO_SYNC_TIMEOUT_MS));
+}
+
+/*********************************************************************
+ * A slot changing hands under loop()
+ ********************************************************************/
+static void test_a_client_that_left_mid_sync_is_forgotten() {
+  proto_slot_t &a = connect(1);
+  nextSyncStep(a, 0);
+  sendSome(a, 0);
+  freeSlot(a);
+  TEST_ASSERT_EQUAL(sync_step_t::none, nextSyncStep(a, 1));
+  TEST_ASSERT_FALSE(a.awaitingNext);
+  TEST_ASSERT_EQUAL_UINT32(0, a.loopId);
+}
+static void test_the_next_client_in_the_slot_gets_its_own_sync() {
+  proto_slot_t &a = connect(1);
+  nextSyncStep(a, 0);
+  sendSome(a, 0);
+  freeSlot(a);
+  connect(2);
+  TEST_ASSERT_EQUAL(sync_step_t::send, nextSyncStep(a, 1));
+  TEST_ASSERT_EQUAL_UINT32(2, a.loopId);
+  TEST_ASSERT_EQUAL(sync_section_t::types, a.cur.section);
+}
+static void test_wants_taken_for_a_client_that_left_stay_for_the_next() {
+  // loop() read client 1, then 1 left and 2 took the slot before loop() took
+  // what it wanted: 2's sync must not be used up on 1.
+  proto_slot_t &a = connect(1);
+  nextSyncStep(a, 0);
+  freeSlot(a);
+  connect(2);
+  TEST_ASSERT_EQUAL_UINT32(0, takeWants(a, 1));
+  TEST_ASSERT_EQUAL_UINT32(WANTS_SYNC, a.wants.load());
+  TEST_ASSERT_EQUAL(sync_step_t::send, nextSyncStep(a, 1));
+  TEST_ASSERT_EQUAL_UINT32(2, a.loopId);
+}
+static void test_wants_of_the_client_loop_read_are_taken() {
+  proto_slot_t &a = connect(1);
+  TEST_ASSERT_EQUAL_UINT32(WANTS_SYNC, takeWants(a, 1));
+  TEST_ASSERT_EQUAL_UINT32(0, a.wants.load());
+}
+
+int main(int argc, char **argv) {
+  (void)argc;
+  (void)argv;
+  UNITY_BEGIN();
+  RUN_TEST(test_a_sixth_client_gets_no_slot);
+  RUN_TEST(test_a_freed_slot_is_taken_again_clean);
+  RUN_TEST(test_no_sync_before_hello);
+  RUN_TEST(test_a_new_client_gets_the_schema_unless_it_has_it);
+  RUN_TEST(test_the_next_frame_waits_for_next);
+  RUN_TEST(test_next_after_the_sync_has_ended_does_nothing);
+  RUN_TEST(test_a_sync_again_starts_over_without_the_schema);
+  RUN_TEST(test_no_next_for_30s_closes_the_client_once);
+  RUN_TEST(test_a_finished_sync_never_times_out);
+  RUN_TEST(test_a_client_that_left_mid_sync_is_forgotten);
+  RUN_TEST(test_the_next_client_in_the_slot_gets_its_own_sync);
+  RUN_TEST(test_wants_taken_for_a_client_that_left_stay_for_the_next);
+  RUN_TEST(test_wants_of_the_client_loop_read_are_taken);
+  return UNITY_END();
+}