feat(logloader): MAVLink FTP transport, request-driven fetching, and an HTTP API

[dakejahl]

Summary

Supersedes #29, #30 and #31 — this branch contains their commits unchanged, with authorship intact, so merging this merges all three. They can be closed.

Reviewing that stack turned up structural problems in the code underneath it, and fixing those meant rewriting the parts they sit on, so keeping four PRs would have meant reviewing #29's ServerInterface changes and then reviewing their replacement. One branch instead: MAVLink FTP for transport, fetching driven by request rather than by mirroring the SD card, and an HTTP API for the ARK-OS Logs page to drive.

Verified end to end against PX4 SITL and a local flight-review. Adds tests and CI, which the repo had none of.

Problem

Transport. LOG_DATA (msg 120) has no target_system/target_component, so a router between logloader and the autopilot has no addressing to work with and copies every chunk to every endpoint it serves, telemetry radio included. LOG_ENTRY compounds it: it identifies a log by index and timestamp rather than by file, so anything downloading over FTP while listing over the log protocol has to map one onto the other by size, mtime or list ordering — and ArduPilot's numbering, its LOG_MAX_FILES wrap and its .BIN naming all have to be modelled to do it. When that mapping guesses wrong, the wrong file transfers in full and is discarded.

Structure. ServerInterface was 860 lines doing two unrelated jobs: a SQLite repository and a Flight Review HTTP client. About 70% of it was database code and the name described the other 30%. Because the database lived inside the upload target, download state was duplicated across local_server.db and remote_server.db, and LogLoader kept them in step by calling every mutation twice. The download queue was read from whichever happened to be first.

Behaviour. logloader tried to mirror the vehicle's SD card. A companion powered on for the first time, or one whose database had been reset, would pull and upload the entire history over a link with better things to do — and there was no way to see what it had or to ask for a particular log.

Defects. A pending log whose local file was missing or empty was retried forever and blocked every older log behind it; only HTTP 400 escaped the queue. ServerInterface::_should_exit and LogLoader::_loop_disabled were plain bools written by one thread and read by another. The signal handler took a mutex and signalled a condition variable, neither of which is async-signal-safe. CMAKE_BUILD_TYPE was never set, so the shipped binary was built at -O0. generate_uuid hashed with std::hash — implementation-defined — into state persisted on disk.

Solution

Everything over MAVLink FTP (FILE_TRANSFER_PROTOCOL, msg 110), for the listing and the transfer. It is addressed, and both stacks reply to the requesting sysid/compid, so transfers are unicast. Identity becomes path-below-the-log-root plus size, so there is no cross-protocol mapping left to get wrong and ArduPilot needs no special handling — a directory listing just reports files. The root is probed at @MAV_LOG, then /fs/microsd/log, /APM/LOGS and /log. MAVSDK's Ftp plugin discards size and mtime from list replies, so FtpListClient runs the listing over MavlinkPassthrough; worth upstreaming so it can be deleted.

Storage split from transport. LogDatabase owns the inventory and the intent, UploadTarget is one Flight Review endpoint with no state of its own, and LegacyImport quarantines the pre-FTP migration behind a header saying it is deletable once the fleet has upgraded. One database, one place to look.

Fetching is request-driven. A log is fetched because it appeared while logloader was watching — a flight just ended — or because something asked. On a database that has never seen the vehicle, only the newest is taken. A listing with more than download.max_auto_queue new logs is an unfamiliar card rather than a burst of flights, so it queues one and says so. Intent lives in the database, so automatic and requested work share one path.

An HTTP API on 127.0.0.1:3005: GET /logs, /status, /events (SSE), POST /logs/{download,upload,cancel}, DELETE /logs/{id}/file. ARK-Electronics/ARK-OS#109 is the page that drives it.

Uploads stream off disk with a computed Content-Length. httplib's MultipartFormDataItems overload concatenates the whole body into a second string; a 400 MB log needed most of a gigabyte on a Pi, and the resulting OOM kill retried the same log on restart. Measured: 16 MB RSS while uploading a 76 MB log.

Also fixed, most of it found by review or by the SITL run: a use-after-free where FtpListClient handed MAVSDK a this-capturing callback (MAVSDK unsubscribes asynchronously, so a reply arriving after shutdown wrote into freed heap); an ODR violation from defining CPPHTTPLIB_OPENSSL_SUPPORT per-file, which gave two translation units different socket layouts and segfaulted on the first real run; Transaction::commit clearing its flag before COMMIT, so a failed commit left the transaction open and every later write was silently discarded while the daemon looked healthy; a missing ALTER TABLE for a column added to an existing schema; and a log still being written being treated as absent, which marked every row "not on vehicle" for one interval after each restart.

Rows are marked absent rather than deleted when a log leaves the vehicle, so a downloaded log keeps its file and its upload history. Config tables are one level deep on purpose — ARK-OS's shared config editor renders exactly one level, and a setting the operator cannot reach from the web UI may as well not exist. The old flat keys are still read, with a deprecation warning.

Upgrades. Pre-FTP databases keyed logs on the LOG_ENTRY timestamp, which FTP cannot reproduce. local_server.db and remote_server.db are folded into one logloader.db and matched to the listing by size, so a fleet upgrading does not re-download and re-upload its history. A log the old version had downloaded but not yet uploaded keeps its file and its state but is not queued — carrying that intent over would produce exactly the flood this version exists to avoid.

tests/test_log_database.cpp covers 16 cases including the legacy import, which runs once against real user data and cannot be un-run. Writing them found two defects on its own.

Carried over from #30 and #31

The reachability cooldown and the API-key headers are preserved in UploadTarget, with one change: 401/403 no longer blacklist a log. Those statuses mean this account is not authorized yet, not this log is bad, and the expected first-run state returns 403 — blacklisting would silently discard one pending log per cycle. The batch still stops on them, which resumes on its own once the key is valid.

Worth flagging: the API-key client has no server counterpart. The ARK flight_review fork has no api_key.py on any branch, and UploadHandler.prepare() authenticates with a session cookie plus _is_uploader_approved, 403ing otherwise. The plumbing is harmless and forward-compatible so it is kept, but setting remote_api_key currently changes nothing.

Follow-ups

• Bump the ARK-OS submodule pointer — feat(logs-page): add a Logs page for browsing and fetching vehicle logs ARK-OS#109 does this and depends on it.
• Confirm ARK's shipped PX4 configs start the companion MAVLink instance with -x; FTP is now load-bearing.
• Resume interrupted transfers rather than restarting them. A 76 MB log timed out once mid-test and was refetched from scratch; the failure-count ordering keeps it from blocking the queue, but resume is still worth doing.