[LAB_00] IN PROGRESS
LabData from the edge
A home data-engineering lab: two ESP32 boards publish telemetry over MQTT, an old netbook ingests it, and a PC processes the data in layers (bronze to silver to gold).
On top of that pipeline, the goal is to measure sequential execution, local parallelism and distributed processing, and learn when distributing pays off and when it does not. Every result gets logged, even when distributing turns out slower.
Foundation phase: architecture, decision records and the telemetry contract (draft).

[01] Pipeline
Dashed = not built yet (TODO)
- 01TODOESP32 · MicroPythonInternal sensors, message built per contract
- 02TODOMQTT · MosquittoBroker on the netbook, QoS 1, per-device auth
- 03TODOIngestion · bronzeRaw NDJSON, append-only, partitioned by date
- 04TODOSilver · ParquetValidated against the contract, deduplicated
- 05TODOGold · ParquetAggregates per device, metric and window
- 06TODOBenchmarksSequential × Dask local × distributed
[02] Hardware
- PCProcessing and coordinationCore i5 (13th gen) · 32 GB
- NetbookMQTT broker, ingestion, experimental workerIntel Atom · 2 GB · Debian
- ESP32-S3Sensor nodeMicroPython
- ESP32-WROOM-32Sensor nodeMicroPython
[03] Hypotheses
What the benchmarks should confirm or refute.
- H1For the real ESP32 volume, sequential execution is enough and faster than Dask.
- H2Columnar Parquet cuts transformation time more than adding workers.
- H3There is a minimum volume above which local Dask beats sequential.
- H4Adding the netbook as a worker makes small loads slower because of network cost.
- H5DuckDB or Polars on one machine compete with distributed Dask on this hardware.
[04] Data contract
Messages in the telemetry contract v1 format (draft). These lines are simulated; real readings will replace them once the pipeline runs.
{"source":"simulator","schema_version":1,"device_id":"esp32s3-sim01","boot_id":"9f3a1c2e","seq":40,"ts_device_ms":null,"time_synced":false,"site":"lab","readings":[{"metric":"wifi_rssi","value":-61,"unit":"dBm"},{"metric":"free_heap","value":154320,"unit":"B"},{"metric":"uptime","value":3585,"unit":"s"}]}
{"source":"simulator","schema_version":1,"device_id":"esp32wroom-sim02","boot_id":"2b7d04aa","seq":17,"ts_device_ms":null,"time_synced":false,"site":"lab","readings":[{"metric":"wifi_rssi","value":-67,"unit":"dBm"},{"metric":"free_heap","value":98112,"unit":"B"},{"metric":"uptime","value":1530,"unit":"s"}]}
{"source":"simulator","schema_version":1,"device_id":"esp32s3-sim01","boot_id":"9f3a1c2e","seq":41,"ts_device_ms":null,"time_synced":false,"site":"lab","readings":[{"metric":"wifi_rssi","value":-60,"unit":"dBm"},{"metric":"free_heap","value":154208,"unit":"B"},{"metric":"uptime","value":3595,"unit":"s"}]}
{"source":"simulator","schema_version":1,"device_id":"esp32wroom-sim02","boot_id":"2b7d04aa","seq":18,"ts_device_ms":null,"time_synced":false,"site":"lab","readings":[{"metric":"wifi_rssi","value":-68,"unit":"dBm"},{"metric":"free_heap","value":98040,"unit":"B"},{"metric":"uptime","value":1540,"unit":"s"}]}
{"source":"simulator","schema_version":1,"device_id":"esp32s3-sim01","boot_id":"9f3a1c2e","seq":42,"ts_device_ms":null,"time_synced":false,"site":"lab","readings":[{"metric":"wifi_rssi","value":-61,"unit":"dBm"},{"metric":"free_heap","value":154320,"unit":"B"},{"metric":"uptime","value":3605,"unit":"s"}]}
{"source":"simulator","schema_version":1,"device_id":"esp32wroom-sim02","boot_id":"2b7d04aa","seq":19,"ts_device_ms":null,"time_synced":false,"site":"lab","readings":[{"metric":"wifi_rssi","value":-66,"unit":"dBm"},{"metric":"free_heap","value":97984,"unit":"B"},{"metric":"uptime","value":1550,"unit":"s"}]}
{"source":"simulator","schema_version":1,"device_id":"esp32s3-sim01","boot_id":"9f3a1c2e","seq":43,"ts_device_ms":null,"time_synced":false,"site":"lab","readings":[{"metric":"wifi_rssi","value":-59,"unit":"dBm"},{"metric":"free_heap","value":154176,"unit":"B"},{"metric":"uptime","value":3615,"unit":"s"}]}
{"source":"simulator","schema_version":1,"device_id":"esp32wroom-sim02","boot_id":"2b7d04aa","seq":20,"ts_device_ms":null,"time_synced":false,"site":"lab","readings":[{"metric":"wifi_rssi","value":-67,"unit":"dBm"},{"metric":"free_heap","value":98112,"unit":"B"},{"metric":"uptime","value":1560,"unit":"s"}]}
[05] Lab diary
- Lab page published on this site.
- Telemetry data contract v1 drafted: envelope, metric catalog, validation and rejection codes.
- Repository started: architecture overview, 7 decision records (proposed) and testing strategy.
[06] Related work
Mobile projects that already touch telemetry, IoT and edge data.