ESPSomfy-RTS/docs/LOOP_QUEUE_PLAN.md

Plan: domain modules behind the LoopQueue

Status: all steps built (PRs #39 to #44). Written: 2026-10-15, against commit e624a41 on refactor/entry-points-outputs. Terms are defined in GLOSSARY.md; the threading rule is ADR-0002.

Why

• Two translators for the same requests. HTTP (Web::processQueue) and MQTT (MQTTClass::receive) each turn a request into shade calls, and they have drifted apart:
  • MQTT tiltTarget, mypos and tiltPosition ignore flipPosition;
  • /setPositions ignores it the other way round;
  • repeat=0 means "default" for shades but "no repeats" for groups.
• Writers on the web task. Model edits (/addShade, /linkRemote, /setPaired, ...) and every read (/controller, /shades, /discovery, ...) touch the shade model on the async_tcp task while loop() changes it. See KNOWN_ISSUES.md and ADR-0002.
• Publishing depends on discipline. Each p_* setter publishes its own MQTT topic, and emitState() calls are placed by hand. Writes that skip a setter are never published, and MQTT position has been a constant 50 since c94819e.
• No native tests. None of this compiles in the native env, so it has no tests.

Shape

WebUiRoutes ─┐                       ┌─► ShadeRequests ─► shades/groups ─► radio
ApiRoutes  ──┼─► LoopQueue ─► loop() ┼─► ModelEditor   ─► shade model
MqttCommands ┘   (one FIFO)          └─► ModelSnapshot fill ─► read reply (chunked)
                                               │ dirty bits, once per loop pass
                                               ▼
                                    MqttPublisher · SocketEmitter

Module	Covers	Called by
ShadeRequests	shade and group commands; favorite and pairing; position and rolling-code correction	WebUiRoutes, ApiRoutes, MqttCommands
ModelEditor	add, save, delete, link and sort rooms, shades, groups and repeaters	WebUiRoutes
ModelSnapshot	the eight read routes: /discovery /controller /rooms /shades /groups /room /shade /group	WebUiRoutes, ApiRoutes

Settings, network, radio config, firmware and discovery have one transport each and little logic of their own. They keep calling the modules they use today (ConfigSettings, ESPNetwork, Transceiver, GitOTA, SSDP); a new module would only pass calls through. The exception is pure validation, such as the pin-conflict check, which moves into a function the native tests can reach.

Decisions

LoopQueue

• One FIFO from every other task into loop(). It carries shade requests, model edits and snapshot fills.
• No item jumps the queue, model edits included.
• MQTT goes through it too, although it already runs on loop(). That gives MQTT the radio back-pressure that web commands have (holdWebCommand) and keeps arrival order.

ShadeRequests

• Interface: one request value, grown out of web_command_t. Positions are given as the client sees them.
• It owns:
  • looking up the shade or group;
  • flipping positions;
  • range checks;
  • repeat defaults;
  • refusing a request when the radio can't take it.
• Test seam: the shade sits behind exactly the operations processQueue calls today:
  • moveToTarget, moveToTiltTarget;
  • sendCommand, sendTiltCommand, repeatFrame/isLastCommand, sendSensorCommand;
  • setMyPosition, unpair, setRollingCode, set positions;
  • a lookup giving the shade's or group's existence, flip and tilt type.
• Two implementations behind it: the real one forwards to SomfyShade/SomfyGroup; a fake records the calls in native tests.
• Later: when the shade behaviour itself is deepened (candidate #1 of the architecture review), only the real implementation changes.

ModelEditor

• What it takes over from the Web.cpp handlers and SomfyShadeController:
  • id and sort-order allocation (the six getNext*Id/getMax*Order copies);
  • limits;
  • validation;
  • linking rules: remote to shade, shade to group, repeaters;
  • what is committed and which events go out after each edit.
• What SomfyShadeController keeps: storage and lookup.
• Request bodies:
  • The handler copies the JSON body to the heap and queues a pointer to it.
  • loop() parses and validates it with the code in SomfyJson.cpp; keeping it there avoids a second ArduinoJson file.
  • At most one edit is in flight; another gets 503. A body over 4 KB gets 413.
• Reply and result:
  • The HTTP reply is {status: "accepted", requestId}.
  • Once loop() has made the change, it sends the existing …Added/…Removed/…State events, with their payloads unchanged (Home Assistant reads shadeRemoved).
  • It also broadcasts requestResult {requestId, ok, desc, id}. The UI that sent the request picks it out by requestId.
• Web UI:
  • It fetches the details it needs through a read route, for example /shade?shadeId= after an add.
  • If no requestResult arrives within about 5 s, it shows a popup that says the change wasn't confirmed and offers a reload.
• Socket-handler fixes in index.js, part of this step:
  • shadeAdded/shadeRemoved are no-ops;
  • procRoomRemoved keeps the removed room;
  • the reply body is read even on errors.

ModelSnapshot

• One snapshot type with the reported fields of rooms, shades, groups and repeaters. That is about 5.5 KB at 32 shades, 16 groups and 16 rooms, against roughly 24 KB for a copy of the full SomfyShade objects.
• One snapshot per read:
  1. The handler allocates the snapshot (one item for /shade, /group and /room) and queues a fill.
  2. The chunked response returns RESPONSE_TRY_AGAIN until loop() has filled the snapshot.
  3. It then streams JSON from that private copy.
• Not in the snapshot: the parts outside the shade model are built on the fly. That covers the /discovery device info and memory figures, and the /controller header (limits, radio config, firmware version).
• Limit: at most 2 reads in flight; a third gets 503.
• Serializers work from the snapshot, not from somfy. They lose group->updateFlags(), which sends events from the web task today, and they become testable natively through JsonFormatter.

Change fan-out

• Marking:
  • Each shade and group gets a bitmask with one bit per reported value: position, target, direction, tilt, favorite, sun flags, rolling code, and so on.
  • The p_* setters set the bit, and only when the displayed value changes, as today.
• Sending: once per loop() pass, for each marked shade or group:
  • the socket gets one shadeState/groupState;
  • MQTT gets only the topics whose bits are set;
  • then the bits are cleared.
• Reconnects: what an output misses while disconnected comes back on reconnect. MQTT connect() publishes everything, and a new socket client gets initClients().
• Events that aren't state (shadeCommand, "remote X pressed Up") stay explicit calls.

Flash

• Debug build: esp32devdbg moves to esp32_huge_app.csv (one 3 MB app, no OTA) and stays there. It isn't shipped, and at 98.3% of the OTA slot it had 26 KB left.
• Release builds: the release envs keep their partition tables, and CI builds them. esp32dev is at 1,396,957 bytes (88.8%), so 176 KB is free.
• Firmware-update testing has to use a release build.

Steps

Each step is a PR of its own that can be tested on the device. Each one runs pio test -e native and the esp32devdbg and esp32dev builds.

#	Step	Closes
0	Move esp32devdbg to esp32_huge_app.csv, in a commit of its own. The device then needs one USB flash (see below).	flash pressure during the refactor
1	LoopQueue + ShadeRequests. Web and MQTT both submit shade requests. Native tests run through the fake shade.	MQTT flip drift, repeat defaults, MQTT commands lost while the radio is full
2	ModelSnapshot for all eight read routes. Native tests cover the serializers.	inconsistent /controller documents, events sent from the web task
3	ModelEditor, requestResult and the index.js changes.	the rest of the web-task writers in KNOWN_ISSUES
4	Change fan-out with dirty bits, replacing publishing from the p_* setters.	writes that skip MQTT, position stuck at 50
5	Split Web.cpp into WebUiRoutes and ApiRoutes, and MQTT.cpp/SomfyMqtt.cpp into MqttCommands, an MQTT connection and MqttPublisher. Edit parsing stays in SomfyJson.cpp.	the glossary names land in the code

Step 1: what changed in behaviour

• Flip handling is the same for every entry point. MQTT tiltTarget, mypos, myTiltPos and tiltPosition, and /setPositions (which Home Assistant calls), now turn the position round for a shade with flipPosition, as the other routes always did.
• Setting the favorite over MQTT keeps the other half. mypos on a shade with tilt keeps the favorite tilt; before, it set it to 0. myTiltPos keeps the favorite lift, or uses where the shade is when it has none.
• repeat means the same everywhere. Left out or negative is the shade's or group's own repeats; 0 is no repeats. /shadeCommand used to read 0 as "the shade's own". A JSON repeat over 127 is capped; it used to wrap round to "the shade's own".
• Position corrections reach MQTT. /setPositions and MQTT position/tiltPosition go through the setters, so they are published.
• MQTT commands go through the LoopQueue. While the radio is full they wait instead of being refused, and they run in order with web requests. An MQTT command topic that isn't understood is logged.
• The LoopQueue is drained on every loop() pass, with or without a network. It is no longer drained from inside a GitHub firmware download, so a restore can't reboot the device in the middle of one.

WebCommand.h became LoopQueue.h (the item and the hold rule) and ShadeRequests.h (the request, the ShadeOps seam and the module). SomfyShadeOps is the device side of the seam. entryPoints/MqttCommands holds the topic parsing. There are 45 new native tests (271 in all).

Flash: esp32devdbg 1,546,405 → 1,547,381 bytes, esp32dev 1,396,957 → 1,398,337 bytes.

Step 2: what changed in behaviour

• Each read describes one moment. /controller, /discovery, /rooms, /shades, /groups, /room, /shade and /group stream a ModelSnapshot that loop() fills. The first two KNOWN_ISSUES items about /controller are fixed for reads.
• Every read is chunked. /discovery, /shades and the rest used to be built whole into a growing response buffer, the path of the old cbuf::resize crash.
• A read waits for loop(). The headers go out at once; the body follows once the snapshot is filled. That is usually one loop() pass, but if the headers' ACK comes back first the library only asks again on its next poll, up to about 500 ms later. A read queued behind a command that waits for the radio waits with it.
• At most two reads at once. A third gets 503 "Busy, try the read again". Each read holds a 5.6 KB snapshot and a 3 KB unit on the heap only while it runs.
• An item too big for the 3 KB unit is left out of every read route, not only /controller (see KNOWN_ISSUES).
• /room, /shade and /group still answer 500 for an id that doesn't exist. If the item disappears between that check and the snapshot, the reply is 200 with {"status":"ERROR","desc":"… Id not found."}.
• Group flags in replies are worked out without side effects. Serializing a group no longer calls updateFlags(), which sent groupState and published MQTT from the web server task.
• /discovery no longer logs where its time went. Building it is now spread over the chunks.

ModelSnapshot.h/.cpp holds the snapshot types, the JSON for each item, SnapshotWriter (the eight documents, built a unit at a time) and SnapshotSlots. SomfySnapshot.cpp copies the shade model into it, for loop()'s fills and for the replies still written on the web server task. The SOMFY_MAX_* limits moved to ShadeTypes.h so the native tests can use them. 21 new native tests, 292 in all.

Flash: esp32devdbg 1,547,381 → 1,547,821 bytes, esp32dev 1,398,337 → 1,400,401 bytes.

Step 3: what changed in behaviour

• Every edit of the shade model runs on loop(). The 19 edit routes reply {"status":"accepted","requestId":N} and queue a copy of the request. loop() applies it and broadcasts requestResult {requestId, ok, id, code, desc}. The errors that used to come in the HTTP reply ("Shade Id not found.", "Data Error." with its code, and so on) come in requestResult instead.
• One edit at a time. A second edit gets 503 "Busy, try the change again", and the web UI retries it. A body over 4 KB gets 413.
• The web UI waits for the result and then reads the item back (/shade, /group, /room, /repeaters), so the code that used the old reply body is unchanged. If no result comes within 5 s, it offers to reload the page. Reads that get 503 "Busy" are retried too.
• Added, removed and saved items reach every client. Saving sends shadeState, groupState or roomState. The web UI now acts on shadeAdded, shadeRemoved, groupAdded and groupRemoved from other clients.
• A shade that fails validation is no longer added. /addShade used to keep the new shade with defaults when its pins clashed.
• Sort orders and linked repeaters are saved. Linking a remote, a repeater or a group member reports when the list is full.
• /getNextRoom, /getNextShade, /getNextGroup, /groupOptions and the new /repeaters are snapshot reads.
• ModelEditor's rules (ModelEditor.h/.cpp) replace the six getNext*Id/getMax*Order copies, the remote-address search, the repeater and group-member list code, and the pin-conflict check. The pin check now uses the protocol being saved and checks every pin against Ethernet.

SomfyEditor.h declares the edit handed to loop(). The code that applies it lives in SomfyJson.cpp, next to fromJSON, so ArduinoJson stays in one file. 27 new native tests, 319 in all.

Flash: esp32devdbg 1,547,821 → 1,536,337 bytes, esp32dev 1,400,401 → 1,393,221 bytes. The 19 handler bodies became one path.

Step 4: what changed in behaviour

• MQTT gets the shade's real position and target. Both were a constant 50 since c94819e, and direction was a constant 0 when MQTT connected.
• Each pass sends a changed shade or group once. The p_* setters and emitState() only mark the shade (SHADE_CHANGED_* bits in ShadeChanges.h). At the end of each loop() pass, SomfyShadeController::sendChanges() publishes the changed MQTT topics and sends one shadeState or groupState.
  • A group whose flags change now sends one groupState, not two.
  • emitState(evt) with an event name (shadeAdded, ...) and emitState(num) for one client still go out at once.
• position and tiltPosition are retained by the broker when they change, as they already were when MQTT connected. A client that subscribes later gets the current position.
• Topic values come from one tested place. publishShadeTopics() writes the topics from a shade snapshot, and publishState() (all topics, on connect) uses it too.

8 new native tests, 327 in all. Flash: esp32devdbg 1,536,337 → 1,536,533 bytes, esp32dev 1,393,221 → 1,393,449 bytes.

Step 5: the file split

Code moved unchanged; only files and names changed.

File	Holds
entryPoints/WebUiRoutes.cpp	registerWebUiRoutes(): the port 80 routes and the UI's files, with the helpers only they use
entryPoints/ApiRoutes.cpp	registerApiRoutes(): the port 8081 routes
entryPoints/Web.cpp	the web servers' lifecycle, auth, and the handlers both route sets call
entryPoints/WebRoutes.h	what those three share
entryPoints/MqttCommands.*	+ MQTT_COMMAND_TOPICS, the topics the connection subscribes to
MqttConnection.*	was entryPoints/MQTT.*; class MQTTClass → MqttConnection
outputs/MqttPublisher.cpp	was outputs/SomfyMqtt.cpp
outputs/SocketEmitter.*	was outputs/Sockets.*

The subscribe and unsubscribe lists, which repeated groups/+/direction/set, became the one MQTT_COMMAND_TOPICS list, and a native test checks that the parser takes every topic in it. Flash: esp32devdbg 1,536,533 → 1,539,645 bytes (+3.1 KB, since the routes are a second file that uses ArduinoJson, as SOMFY_SPLIT_PLAN.md warned), esp32dev 1,393,449 → 1,391,113 bytes (−2.3 KB, where -Os inlines less).

Step 0 on a device

No longer needed: esp32_huge_app.csv now has the release file system (0xE0000 at 0x310000) and puts the coredump in the 64 KB that was unused at the end, so the debug and release builds share one data layout and either can be flashed over the other. A device formatted at 0xD0000 grows to 0xE0000 on its next mount, which LittleFS can do. What follows was the procedure while the debug table had the smaller file system.

NVS stays at 0x9000, so the rolling codes survive. The file system partition shrinks from 0xE0000 to 0xD0000, and the existing LittleFS won't mount, because LittleFS.begin() doesn't format. So:

1. Take a backup through /backup.
2. Flash the partition table, firmware and file system over USB: pio run -e esp32devdbg -t upload -t uploadfs.
3. Restore the backup.

Risks

• async_tcp must never wait. Every handler only queues and returns. The two kinds of waiting reply, the read routes, poll with RESPONSE_TRY_AGAIN and never block.
• Heap. Edit bodies (one at a time, up to 4 KB) and snapshots (two at a time, about 5.5 KB each) come from the heap. Check the memStatus max figure on the device after steps 2 and 3.
• The 2 KB socket event limit (see KNOWN_ISSUES). That is why requestResult carries an id and not the object.
• The UI changes ship with the firmware in step 3, because the UI is served from LittleFS. An older UI against the new firmware would wait for HTTP reply bodies that no longer come.

Found during the review, not part of this plan

Still open:

• The Home Assistant integration sends no apikey header on /shades, /groups and the shade, group and tilt commands, so they fail with security on. The fix is ESPSomfy-RTS-HA#4, not yet merged.

Fixed by step 3:

• The Ethernet pin conflict was only checked for the up pin, and the check used the shade's current protocol instead of the one being saved.
• linkRepeater never set isDirty, so a newly linked repeater wasn't saved. The sort orders weren't saved either.
• /linkRemote and /unlinkRemote sent a 500 and then a second response.
• The web UI's procRoomRemoved kept the removed room and dropped the others, and deleteShade never closed its prompt (prompt.remove;).
• The web UI called /repeaters, which didn't exist.

Fixed by step 4:

• MQTT position/target were published as a constant 50, and direction as 0 on connect.
• A group whose flags changed sent groupState twice.

Fixed later, by SHADE_BEHAVIOUR_PLAN.md (reconcile Q18):

• settingPos was never cleared when the lift arrived, while settingTiltPos was, so a later step or group move also sent a stop when it arrived.

Fixed later, each on its own:

• A restore without shades skipped the shade and group records but not the room records, so it read the rest of the file from the wrong place. The skip is now shadeModelBytes() in ConfigHeader.h, with native tests (#54).
• New ids stopped two short of the maximum: 30 shades, 14 groups and 14 rooms instead of 32, 16 and 16, an off-by-one from the first commit. They now run to the maximum (#57).
• SomfyGroup::fromJSON read linkedShades into a local array and dropped it, and wrote past that array when a request listed more than 32 shades. The code is gone; members change only through /linkToGroup and /unlinkFromGroup (#58).