IoT Data Aggregation Platform

IoT sensor data aggregation system with async processing, local-first infrastructure, and a 5-role team split.

Overview

This project runs as a Docker Compose stack with:

• Flask API for ingestion and retrieval
• RabbitMQ queue for async jobs
• Worker service for summary processing
• DynamoDB Local for persistence
• MinIO for object-storage compatibility
• Nginx-hosted frontend dashboard

Project Structure

api/            Flask API endpoints
backend/        Business logic, models, validation
db/             DynamoDB table access and helpers
worker/         Queue consumer and job processing
frontend/       Static UI served by nginx
shared/         Shared config and queue helpers
tests/          Unit tests
scripts/        Smoke and verification scripts
docker-compose.yml

Quick Start

docker compose up --build -d
docker compose ps

Service URLs:

• Frontend: http://localhost:8080
• API: http://localhost:5000
• RabbitMQ UI: http://localhost:15672
• MinIO Console: http://localhost:9001
• DynamoDB Local: http://localhost:8000

Clean AWS Cloud Deploy (Scripted)

This project now supports clean, no-manual-service deployment through one script.

Prerequisites:

• AWS CLI configured (aws configure)
• AWS SAM CLI installed
• Python 3.10+ available

Optional clean slate:

aws cloudformation delete-stack --stack-name iot-aggregation --region us-east-1

Deploy everything (API Gateway + Lambda + SQS + DynamoDB + S3 frontend):

powershell -ExecutionPolicy Bypass -File scripts/deploy_cloud.ps1

What the script does:

• Cleans .aws-sam build artifacts
• Builds and deploys SAM stack
• Uploads frontend to S3 website bucket
• Writes deployed API URL to frontend/config.js
• Writes deployed API URL to esp8266_sensor_nodes/cloud_endpoint.h
• Runs cloud smoke test (GET /health, POST /data)

Expected outputs include:

• API URL (https://...execute-api.../Prod)
• Frontend bucket path
• Smoke-test data_id

ESP8266 Cloud Integration

Both sketches now use HTTPS and the generated cloud endpoint header:

• esp8266_sensor_nodes/temp_humidity_node.ino
• esp8266_sensor_nodes/pressure_ethanol_node.ino

After deployment, reflash both nodes. They post to https://<api-id>.execute-api.<region>.amazonaws.com/Prod/data.

End-to-End Cloud Flow

1. ESP8266 nodes send JSON metrics to API Gateway /data.
2. API Lambda stores record in DynamoDB and publishes job to SQS.
3. Worker Lambda consumes SQS and computes summaries/alerts.
4. Frontend fetches /list and /alerts from API Gateway.
5. Dashboard displays latest node metrics and alert states.

API Endpoints

• GET /health
• POST /data
• GET /summary?id=<data_id>
• GET /list

Example POST /data body:

{
  "sensor_id": "SENSOR-01",
  "values": [20.5, 21.0, 22.3]
}

Data Flow

1. API validates and stores a new record as pending.
2. API publishes a job to RabbitMQ.
3. Worker consumes the job, computes summary, updates record.
4. Status transitions: pending -> processing -> done (or failed after retries).

Testing and Verification

Run tests:

$env:USE_LOCAL='true'
$env:DYNAMO_ENDPOINT='http://localhost:8000'
.venv/Scripts/python.exe -m pytest -q

Run integration smoke:

powershell -ExecutionPolicy Bypass -File scripts/integration_smoke.ps1

Run all checks (stack + tests + smoke + DB scan):

powershell -ExecutionPolicy Bypass -File scripts/verify_all.ps1

Team Roles

• Member 1: Lead/Integrator + Worker
• Member 2: Backend
• Member 3: API
• Member 4: Database
• Member 5: Frontend/DevOps

Notes

• Current setup is local-first and does not require AWS accounts.
• DynamoDB is emulated with DynamoDB Local.
• Object storage uses MinIO local defaults.
• Frontend uses nginx /api proxy to reach the API container.

Related Docs

• frontend/README.md
• db/schema.md
• scripts/integration_smoke.ps1
• scripts/verify_all.ps1