Both sides added a helper at the end of test/e2e/test_settings.py
(boot_time here, header on main); both are kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The records move to ConfigRecords, free of the file system and the
shade model, with 33 unit tests: each record's size against the header,
a round trip of each, the layouts of versions 2 to 24, and a real backup
when CONFIG_BACKUP names one.
The tests found, and this fixes:
- A quoted value as long as its field (a hostname of 30 or more
characters, a full NTP server, MQTT host or time zone) was read short,
and every value after it shifted.
- A comma in a shade, group or room name shifted the rest of its record.
Names are read at their fixed width.
- 4.0 boards without Ethernet declared the network record 28 bytes longer
than they wrote it, so a restore of the radio without the network read
the radio from the middle of its record. Records are read and skipped
to their newline, and backup() measures what it writes.
- A restore cleared rooms beyond the backup's only on load.
A restore now takes the backup's rolling codes even when they are lower
than NVS's; a load still keeps the higher.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ADR 0003: the C5 gets its own consequence instead of sitting in the
ADR 0006 amendment. It names the single-slot table, the chip revision
minimum (v1.0, CONFIG_ESP32C5_REV_MIN_FULL=100) and the longer WiFi
scans with 5 GHz.
- platformio.ini: the [esp32] comment names the C5 and C6 as released
debug builds; the C5 comment no longer repeats the C6's.
- Transceiver: the C5 comment says why TX and RX are 4 and 5.
- CI: the addr_fs comment names both partition tables.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The C5 builds like the C6: a debug build with one 3 MB app slot, so it
updates over USB. At CORE_DEBUG_LEVEL=2 its image left 3.4 KB of the
1.5 MB OTA slot, under CI's 8 KB.
- platformio.ini: env esp32c5 (esp32-c5-devkitc-1).
- Chip model "c5", so the updater and releases name its files
SomfyController.*.esp32c5.bin.
- Radio defaults: the framework's SPI pins (CSN 6, MOSI 8, MISO 9,
SCK 10), TX 4, RX 5.
- Web UI: GPIO 0-28, without 13-14 (USB) and 15-22 (flash, PSRAM).
- CI builds and releases it; its bootloader sits at 0x2000.
- ADR 0003: amended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the old API gone, the C3 image is 1,498,512 bytes at
CORE_DEBUG_LEVEL=2 and leaves 72.6 KB of its 1.5 MB OTA slot. It was
76.5 KB over when ESP-IDF 5.5 dropped it.
- platformio.ini: env esp32c3, single core, esp32_3MB.csv, level 2.
- CI builds and releases it with the other chips.
- ADR 0003: amended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The routes matched any path below them, and a web asset name there
passed the token check: GET /backup/a.js returned the backup under
Secured. Every route is now an exact match, and a web asset sits in
the root only.
- e2e: those paths answer 401 or 404; the restore test now fails when
the chunks skip the token check (checked with the check removed).
- WEB_CORS lets the socket take Origin: null instead of skipping the
Origin check.
- ADR 0006, GLOSSARY and README: the Origin check in either mode, /ws
as a public route, the Host header behind a reverse proxy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under Secured every HTTP route but the public ones needs the token
(docs/adr/0006-two-security-modes-eleven-http-routes.md):
- isPublicRoute, isWebAsset and originMatchesHost in src/HttpAccess.h,
with native tests.
- One middleware checks the token. It runs once the body is in, so
/restore, /updateFirmware and /updateApplication check it on their
first chunk and drop a refused upload.
- Static serving answers the web UI's files only: shades.cfg and
shades.tmp are no longer served.
- The wildcard CORS headers go; WEB_CORS turns them back on for UI work
from file://.
- The socket refuses a handshake whose Origin doesn't name its Host.
- "Secure Config Only" goes: a device set to it is Secured, and
/loginContext drops permissions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 4a of docs/SECURITY_PLAN.md. Port 80 keeps the eleven HTTP routes of
ADR 0006 and the socket at /ws; everything else moved to protocol 2.
- Port 8081 (ApiRoutes) and the socket on 8080 (SocketEmitter) are gone.
Protocol 2 owns its frame buffer and loop-task check.
- Port 80 drops every route the old web UI called: reads, edits,
settings, scans, reboot, /shades.cfg, /discovery. HttpRoutes.cpp holds
what stays.
- ModelSnapshot, SnapshotSlots and the read documents are gone. The item
snapshot types stay, as ItemSnapshot, for events, MQTT and the command
replies.
- Model edits reply only to their protocol 2 op: requestResult goes.
- The settings' toJSON writers, used only by the settings routes, go.
- The _espsomfy_rts._tcp mDNS record moves to port 80, with path=/ws.
- The memory event reports the loop-rate window it closes.
- A deleted room no longer re-publishes its retained MQTT topics.
- e2e: the heap check reads the sync's status; the sync is checked on
its own instead of against /controller.
- README gets an API and ports section.
BREAKING CHANGE: ports 8081 and 8080 and the old HTTP routes are gone.
Home Assistant needs integration 3.0.0 or newer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1d of the protocol 2 core (docs/SECURITY_PLAN.md).
- getSettings/saveSettings by section: general, network, mqtt, security,
radio. A save is the settings route's request, so loop() applies it with
the settings' own fromJSON(). ProtoSettings checks every field first:
kind, range, string length, IP addresses.
- Secrets are only reported as has*. A security save replies with the
client's new token and closes the other sockets; other saves send
settingsChanged to the other clients.
- scanWifi runs an async scan from ESPNetwork, kept apart from the roaming
scan, and wifiNetworks goes to the clients that asked.
- beginFrequencyScan, endFrequencyScan and reboot run at once on async_tcp,
as their routes do: a scan's end mustn't wait in the loop queue behind an
unpair that waits for the scan to end (new op kind 'a').
- e2e: test_settings.py and a --reboot flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edit routes become ops 26-44 (addRoom ... groupSortOrder): ProtoEdits
turns an op's payload into the route's request, full names and real
booleans, and loop() applies it with applyModelEdit() as the route
does. The reply goes to the client that sent it, with an add's new id,
or the route's reason and a shade's validation code; the HTTP routes
keep sending requestResult.
getNextRoom, getNextShade, getNextGroup and groupOptions (45-48) are
answered by loop() from the shade model, through a new LoopQueue item.
Messages that arrive in pieces, a frame over one TCP segment or several
frames, are put back together by ProtoAssembly: one at a time across
clients, up to 4 KB, in a buffer held only while one is on its way.
A message in pieces is now told from its first frame's opcode, which
also stops a fragmented message's last frame drawing a second reply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shadeCommand, groupCommand, tiltCommand, repeatCommand, setPositions,
setSensor, setMyPosition, unpairShade, setRollingCode (types 16-24) are
read on async_tcp by ProtoOps and handed to loop() through the
LoopQueue, which replies once the request has run: ok, not found or
radio unavailable. A bad payload, a paused radio or a full queue is
answered straight away. sendRemoteCommand (25) touches nothing of the
shade model, so it goes to the radio from async_tcp like its route.
ShadeRequests::run() now says how a request went, not only whether
the radio took it. An op reads only the fields that were sent, so
setPositions can leave the tilt as it is. The shadeCommand event is
renamed shadeCommanded, since the op has the route's name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the review of #99:
- A sync could stall when a client took a slot the moment another left:
loop() used up the new client's sync on the old one. The slots move
to ProtoSlots, a pure module with native tests, and loop() takes a
slot's wants only for the client it read (takeWants).
- Deleting a group sends its shades, whose inGroup changed. Deleting a
shade unlinks it from its groups and sends them, so a new shade given
its id doesn't join them.
- next and sync with an id get an ok reply, and a message in pieces
gets one bad request, not one per piece.
- Feeds check for loop() before they write the event buffer, and log
an event too large, as events do.
- editChanges() (ModelEditor, native tests) says what clients get after
an edit; the edit result carries the shade a group gained or lost.
order() takes a sort_list_t.
- The frame writers move to ProtoFrame. Renames: pf to proto_field_t,
the X-macro letters to TYPE/FIELD/ERR/FEED, proto_request_t to
proto_message_t, REQ_* to WANTS_*, fmt to eventJson. The event
bodies share sendItem(), the connection type name lives in
ESPNetwork, and the loop() task check in SocketEmitter.
- e2e: the heap floor check skips when the low since boot is already
under it; tests for replies to next and sync and a long message.
- SECURITY_PLAN.md splits step 1 into 1a-1d.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The /ws endpoint runs next to the old socket on port 8080. The
handshake checks the token under security, the device sends hello,
then a sync of the schema (unless the client has it), header, rooms,
shades, groups, repeaters and status, paced by the client's next.
Events go out with the old ones: shadeState or the whole shade on a
config change, groups, rooms, removals, orders, repeaters,
shadeCommand, memory, wifi, ethernet, and the frames and
frequencyScan feeds.
The table of types, fields, errors and feeds is defined once and is
sent as the schema. The writer, item writers, sync and client frame
parser are pure and natively tested.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The debug build logs at info level, and each line holds up the task that
logs it for ~9ms at 115200. At 921600 that drops to ~1ms. SERIAL_BAUD
defaults to 115200, so the other envs are unchanged. The ROM's boot
banner still prints at 115200 and shows as garbage in the monitor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copied from the registry package lsatan/SmartRC-CC1101-Driver-Lib@2.5.7
(upstream unmaintained since 2021) so it can be fixed for Arduino 3.x in
the next commit. This commit is the untouched copy, so that diff shows
every change. Licence: LICENSE.txt ("permission to modify and publish").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IDF 4.4 gets no more security fixes, and the planned RMT radio needs
IDF 5's RMT driver. See docs/adr/0003-pioarduino-idf55.md.
- Every env on one pinned pioarduino release. C6 drops its own
library list and flags but stays a debug build on huge_app.
- Release envs build at CORE_DEBUG_LEVEL=2 so the ESP32 image fits its
1.5 MB OTA slot (17 KB free); debug envs stay at 3.
- Drop the C3: its image is 76.5 KB over the OTA slot on IDF 5.5.
- CI installs pioarduino's Core and checks firmware.bin against the app
partition, failing under 8 KB free. PlatformIO's "Flash:" figure
reads ~45 KB low and passed a C3 image that doesn't fit.
- esp_task_wdt_reconfigure(): on 3.x the core already starts the watchdog
and esp_task_wdt_init() fails, so the C6 build kept the 5 s default.
- An updated firmware is kept only once it has run 60 s with the
network up; a reset before then rolls back to the previous one.
- ETH.begin() takes the 3.x argument order; Arduino 2.x branches gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both are minor updates with no API change for us. The espressif32 7.x
platform and SmartRC-CC1101-Driver-Lib 3.x major updates are left out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR B of SHADE_BEHAVIOUR_PLAN.md. What a shade does with frames, commands,
group commands and radio events moves unchanged from SomfyShade into
ShadeBehaviour, which builds natively and is pinned by test_shadebehaviour.
ShadeBehaviour reads ShadeSettings and RemoteSettings, gets the time on
every input, and holds the motion state that other files now read through
state(). It sends through MotorLink and reports through ShadeEvents
(changes, commands, save wanted, sensor flags) as things happen.
SomfyShade wraps it: its MotorLink is SomfyRemote::sendCommandAfter, which
now leaves the sender out of the echo and radio hold because the module
hears its own frame. lastFrame stays shared with the send path.
ConfigFile writes through restore() and clearTilt(), SomfyShadeOps through
setPosition(). The pinning tests label where the copies of a rule disagree
"today:", for the reconcile phase.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The p_* setters and emitState() no longer publish or emit straight away:
they set SHADE_CHANGED_* / GROUP_CHANGED_* bits on the shade or group.
At the end of every loop() pass SomfyShadeController::sendChanges()
publishes the MQTT topics whose bits are set and sends one shadeState or
groupState, then clears the bits. emitState(evt) with an event name and
emitState(num) for one client still go out at once.
publishShadeTopics() (ShadeChanges.cpp, tested natively) writes the topics
from a shade snapshot; publishState() on connect uses it too. That fixes
MQTT position and target, a constant 50 since c94819e, and direction, a
constant 0 on connect. position and tiltPosition are now retained when
they change, as they were on connect, and a group whose flags change sends
one groupState instead of two. somfy_flags_t moves to ShadeTypes.h.
8 new native tests, 327 in all. Flash: esp32devdbg +196 bytes, esp32dev
+228 bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 19 routes that changed rooms, shades, groups and repeaters on the web
server task (add, save, delete, link and unlink, setPaired, the sort
orders) now copy their request into a model_edit_t, queue it on the
LoopQueue and reply {status:accepted,requestId:N}. loop() applies it
(SomfyJson.cpp, next to fromJSON) and broadcasts requestResult {requestId,
ok, id, code, desc}. One edit at a time; a second gets 503, a body over
4 KB gets 413.
ModelEditor.h/.cpp holds the rules, tested natively: the lowest free id,
the next sort order, a free remote address, adding to and removing from
the group-member and repeater lists, and the GPIO pin-conflict check. They
replace the getNext*Id/getMax*Order copies, the remote-address search,
the list code and the inline pin checks. The pin check now uses the
protocol being saved and checks every pin against Ethernet, not only up.
/getNextRoom, /getNextShade, /getNextGroup, /groupOptions and a new
/repeaters (the UI already called it) are ModelSnapshot reads.
The web UI's editJSON() sends an edit, waits for its requestResult, then
reads the item back from /shade, /group, /room or /repeaters, so the
callbacks that used the old reply bodies are unchanged. No result within
5 s offers a reload; reads and edits told Busy are retried. It now also
refreshes its lists on shadeAdded/shadeRemoved/groupAdded/groupRemoved
from other clients, and procRoomRemoved keeps the right rooms.
Also fixed: a shade that fails validation is no longer added; linked
repeaters and sort orders are saved; linking reports a full list.
27 new native tests, 319 in all. Flash: esp32devdbg -11,484 bytes,
esp32dev -7,180 bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/controller, /discovery, /rooms, /shades, /groups, /room, /shade and /group
now take a SnapshotSlots slot, queue a fill on the LoopQueue and send a
chunked response whose body waits (RESPONSE_TRY_AGAIN) until loop() has
copied the shade model into the slot. SnapshotWriter then streams the
document from that copy a unit at a time, so each document describes one
moment. At most two reads run at once; a third gets 503. A slot's 5.6 KB
snapshot is allocated only while a read holds it, and its atomic state
decides whether loop() or the read frees it when the read goes away first.
The JSON of each item is unchanged and now lives in ModelSnapshot.cpp,
where the native tests check it. Replies still written on the web server
task copy their one item with SomfySnapshot.cpp, so serializing a group
no longer calls updateFlags(), which emitted groupState and published MQTT
from async_tcp.
Also: every read route is chunked and leaves out an item too big for its
3 KB unit, like /controller did; /discovery drops its timing log; the
SOMFY_MAX_* limits move to ShadeTypes.h.
21 new native tests, 292 in all. Flash: esp32devdbg +440 bytes, esp32dev
+2,064 bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HTTP routes and MQTT commands now both build a shade request and submit it
to the LoopQueue, which loop() drains into ShadeRequests. ShadeRequests
finds the shade or group, turns client positions into stored ones, checks
ranges and fills in default repeats, and reaches the shades through the
ShadeOps seam: SomfyShadeOps on the device, a recording fake in the native
tests. MqttCommands turns a .../set topic into a shade request.
Behaviour that changes:
- MQTT tiltTarget, mypos, myTiltPos and tiltPosition, and /setPositions,
now honour flipPosition like the other routes.
- MQTT mypos keeps the favorite tilt (was set to 0) and myTiltPos keeps
the favorite lift.
- repeat 0 on /shadeCommand means no repeats, as it did for groups; a
JSON repeat over 127 is capped instead of wrapping to the default.
- Position corrections go through the setters, so MQTT publishes them.
- MQTT commands wait while the radio is full instead of being refused.
- The queue is drained every loop() pass and no longer from inside a
GitHub firmware download.
WebCommand.h is replaced by LoopQueue.h and ShadeRequests.h. 45 new native
tests, 271 in all. Flash: esp32devdbg +976 bytes, esp32dev +1,380 bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
esp32devdbg used 98.3% of the 1.5 MB OTA slot. On esp32_huge_app.csv the
same 1,546,405-byte image uses 49.2%, so the refactors ahead have room.
The release envs keep esp32_3MB.csv and CI keeps them within limits.
NVS stays at 0x9000, but the file system shrinks from 0xE0000 to 0xD0000,
so a board moving to this table needs a backup, a USB flash with
uploadfs, and a restore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The decoding of the time between received edges into frame bits moves out
of Transceiver::handleReceive unchanged into decodePulse(), with the RTS
timings and the receive buffer type. It stays in IRAM on the ESP32 and is
now covered by native tests that feed it the edges sendFrame() produces
for 56 and 80-bit first frames and repeats, a glitch and a broken pulse.
handleReceive keeps the noise guard, the edge timing and the RX queue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sun and wind timers were spread over processFrame, checkMovement and
processInternalCommand. ShadeSensors now holds them and decides when the
weather extends or retracts a shade, when turning the sun flag on moves
it, and when wind blocks a remote Down. Each caller still sets the targets
itself, so no shade moves differently yet.
The timers are now 32-bit like millis(). As 64-bit values holding a 32-bit
millis() they fired at once after the rollover every 49.7 days. The debug
logs of each timer go: they were compiled out at CORE_DEBUG_LEVEL 3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At -Og GCC only caps inlining growth in units over large-unit-insns
(10000 instructions). Smaller files inline ArduinoJson's helpers into
every caller: moving the JSON parsing out of Somfy.cpp grew the image by
19 KB, past the 1.5 MB app partition. Treating every file as large keeps
inlining at the level the big files already had and frees 13 KB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The order radio jobs go out in was spread across callers, file statics and
the radio task: each remote tracked its last back job so a stop could decide
whether it may jump the queue, and the task kept a separate count of the back
jobs it had taken. RadioScheduler now owns the waiting jobs, their tickets,
the stop-goes-first rule, delayed jobs, flush and suspend, and the outcome of
every job. It is plain C++, with the time passed in and the caller locking,
so all of it is covered by native tests. Transceiver keeps the FreeRTOS task,
the locks and the CC1101.
sendCommandAfter returns the job's ticket instead of leaving it in lastTxSeq.
Two stops for the same remote now go out in the order they were sent; queued
to the front they used to come out reversed, putting the rolling codes out of
order.
Adds GLOSSARY.md for the radio terms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The frame was popped before queueTx(), so whenever the radio queue was busy
it was lost. Peek it instead, only remove it once it is queued (or TX is
suspended), and don't try while the queue has no room. The repeater buffer
moves to TxBuffer.h/.cpp so it can be unit tested.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Arduino core already links with --wrap=log_printf for the ESP
Insights hook in libesp_diagnostics, so defining __wrap_log_printf
failed to link with a duplicate definition. That hook prints through
log_printfv, so wrap that instead and take the lock there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Arduino core's log_printf() has its UART lock commented out and
writes with ets_printf(), so lines logged at the same time from
async_tcp and loop() came out interleaved character by character. It
also formats short lines into one static buffer shared by every task.
Wrap log_printf at link time and hold a mutex around the core's
log_printfv(). It never blocks before the scheduler runs, in an ISR or
in a critical section, and waits at most 50ms before printing anyway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
checkMovement worked out the position four times, nearly copy-pasted,
for lift down, lift up, tilt down and tilt up. All four now call
travelPosition(), which keeps the same arithmetic, including how the
elapsed time wraps with millis(). Tilt up with no tilt time still
stops the tilt direction in the caller.
transformPosition and transformCommand keep their behaviour through
the new displayPosition() and flipCommand() helpers. The native env
builds ShadeMotion.cpp so all of this can be unit tested on the PC.
No behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JsonFormatter and BufferedJsonFormatter move verbatim out of
WResp.cpp/WResp.h, which now include JsonFormatter.h. They don't need
the web server, so on their own they can be unit tested on the PC; the
native env builds the new file. AsyncJsonResp and JsonSockEvent stay
in WResp. No behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builds SomfyFrame.cpp in the native env, with a minimal String and
no-op ESP_LOGx in the stubs, and tests:
- a fixed 56-bit frame, byte for byte, so what goes on air is pinned
- encode/decode round trips for 56-bit, 80-bit (repeats, Favorite,
Stop, steps, Toggle), RTW and RTV frames
- rejection of bad checksums, addresses, rolling codes, keys and
80-bit parity
- repeat/synonym tracking in copy()
- parsing and naming of commands
The command-name round trip fails: names the firmware reports, such as
"Step Up" and "My+Up", parse back to the wrong command.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds an [env:native] for host-side unit tests (pio test -e native) with
a minimal Arduino.h stub, and covers the Utils helpers: _trim,
toBoolean, SETCHARPROP, sortArray and the Timestamp conversions.
The firmware envs now extend a shared [esp32] section instead of [env]
so the native env doesn't inherit the Arduino framework, the ESP32
libraries or the build scripts.
The half-hour calcTZOffset cases fail: +05:30 comes back as 2300.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Web command handlers ran on the async_tcp task and blocked up to 3s
waiting for loop() to bit-bang radio frames. async_tcp also services the
WebSocket on port 8080, so pongs stalled and Home Assistant
(ping_timeout=15) disconnected when several covers were commanded.
- Add a radio_tx FreeRTOS task with a job queue; sendFrame/repeatFrame
and repeater frames enqueue instead of transmitting inline.
- Serialize CC1101 access with a recursive mutex (TX task, apply, end,
frequency scan, RSSI read).
- Make Web::queueCommand fire-and-forget; drop the done semaphore.
- Route /setMyPosition and /unpairShade through the command queue.
- Clamp repeats to MAX_TX_REPEATS (40).
- Pin AsyncTCP to core 0 on dual-core targets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
