Calendink/Provider/tdd/backend_architecture.md

Backend Architecture for ESP32-S3 Provider

Authored by Claude Opus 4 Date: 2026-03-02

---

1. Goal

Serve the Svelte web dashboard and expose a REST API from the ESP32-S3 using ESP-IDF's built-in esp_http_server. The backend must:

• Serve static files (the compiled Svelte frontend) from flash
• Provide system information over JSON
• Allow remote reboot
• Support independent frontend/backend development workflows

2. Chosen Stack

Technology	Source	Role
esp_http_server	ESP-IDF built-in	HTTP server daemon
cJSON	ESP-IDF built-in	JSON serialization for API responses
LittleFS	joltwallet/esp_littlefs	Filesystem on flash for serving frontend files

All three are standard in ESP-IDF projects. esp_http_server and cJSON ship with the SDK. LittleFS is the recommended flash filesystem for ESP-IDF — Espressif's own restful_server example uses joltwallet/littlefs.

3. Why LittleFS Over Other Options

LittleFS vs SPIFFS

	LittleFS	SPIFFS
Directory support	✅ Real directories	❌ Flat namespace
Wear leveling	✅ Dynamic	⚠️ Static
Power-loss safe	✅ Copy-on-write	❌ Can corrupt
ESP-IDF status	✅ Recommended	⚠️ Deprecated in recent examples
Performance	Faster for small files	Slower mount, no dir listing

SPIFFS was the original choice in older ESP-IDF examples but has been replaced by LittleFS in the current restful_server example. LittleFS is the better choice going forward.

LittleFS Partition vs EMBED_FILES

We considered two approaches to deploy the frontend:

	LittleFS Partition	EMBED_FILES (binary embedding)
Storage	Separate flash partition	Compiled into firmware binary
Update	Can flash partition independently	Must reflash entire firmware
OTA	Needs separate partition OTA	UI updates with firmware naturally
Dev workflow	Can skip frontend flash during iteration	Always included
Filesystem	Real VFS — open/read/close file APIs	Direct memory pointer

We chose LittleFS partition because:

1. Development speed — a Kconfig toggle (CALENDINK_DEPLOY_WEB_PAGES) lets you skip flashing the frontend entirely when iterating on the backend
2. Separation of concerns — frontend and firmware have independent flash regions
3. Follows ESP-IDF conventions — matches the official restful_server example pattern
4. No RAM overhead — files are read from flash in chunks, not loaded into memory

4. Kconfig Deploy Toggle

The CALENDINK_DEPLOY_WEB_PAGES menuconfig option controls whether the frontend gets flashed:

Setting	Effect	Use case
OFF (default)	Frontend not flashed. API still works.	Backend development — fast flash, use PC dev server for UI
ON	frontend/dist/ is written to the www LittleFS partition	Production deployment — everything runs on ESP32

This mirrors the official ESP-IDF pattern (CONFIG_EXAMPLE_DEPLOY_WEB_PAGES in the restful_server example).

5. API Design

Endpoints

GET  /api/system/info    → JSON with chip, heap, uptime, firmware, connection
POST /api/system/reboot  → JSON acknowledgment, then esp_restart()
GET  /*                   → Static files from LittleFS /www (when deployed)

CORS

During development, the Svelte dev server runs on http://localhost:5173 and API calls go to http://<ESP32_IP>. This is cross-origin, so the backend adds CORS headers:

• Access-Control-Allow-Origin: *
• Access-Control-Allow-Methods: GET, POST, OPTIONS
• Access-Control-Allow-Headers: Content-Type
• OPTIONS preflight handler

In production (frontend served from ESP32), everything is same-origin — CORS headers have no effect but don't hurt.

6. File Organization

Provider/main/
├── main.cpp                    # Entry point, network init, starts HTTP server
├── http_server.cpp             # HTTP server lifecycle, static file handler, CORS
├── api/
│   └── system/
│       ├── info.cpp            # GET /api/system/info
│       └── reboot.cpp          # POST /api/system/reboot
├── led_status.cpp              # Existing — LED management
├── connect.cpp                 # Existing — Ethernet/WiFi
├── types.hpp                   # Existing — type aliases
└── Kconfig.projbuild           # Existing + new web server config

Why This Structure

The project uses a unity build pattern — main.cpp #includes .cpp files directly (e.g. #include "connect.cpp"). This is unconventional but works well for small embedded projects since ESP-IDF only compiles the files listed in idf_component_register(SRCS ...).

We extend this pattern to the HTTP server:

• http_server.cpp #includes the API handler files
• Each handler file is self-contained: it defines its URI struct and registration function
• New endpoint groups get their own folder under api/ (e.g. api/calendar/, api/display/)

7. Partition Table

# Name,   Type, SubType,  Offset,   Size
nvs,      data, nvs,      0x9000,   0x6000
phy_init, data, phy,      0xf000,   0x1000
factory,  app,  factory,  0x10000,  1M
www,      data, littlefs, ,         64K

The www partition is 64KB — more than enough for the 16kB gzipped frontend. Only gets written during idf.py flash when CALENDINK_DEPLOY_WEB_PAGES is enabled.

8. Build Pipeline

Frontend Build (PC)                      ESP-IDF Build
──────────────────                       ──────────────
npm run build                             idf.py build
  ↓                                         ↓
frontend/dist/                           firmware.bin
  index.html (47kB)                          +
  index.html.gz (16kB)                   www.bin (LittleFS image, when deploy=ON)
                                              ↓
                                         idf.py flash
                                              ↓
                                         ESP32-S3 Flash
                                         ├── factory partition → firmware
                                         └── www partition → frontend files

9. Summary

9. Summary

We use esp_http_server + cJSON + LittleFS — all standard ESP-IDF components — to serve the frontend and expose a REST API. A LittleFS partition stores frontend files separately from firmware, with a Kconfig toggle to skip frontend flashing during backend development. The API is structured as modular handler files under api/ for clean scalability.

---

10. Implementation Results

Added 2026-03-03 after implementation and integration was completed.

Refactoring & Architecture Outcomes

• Unity Build Pattern: Successfully adopted for the HTTP server (http_server.cpp includes .cpp API handlers). This simplified the build process, reduced include complexity, and resolved multiple redefinition and linkage errors without needing complex CMake modifications.
• State Management: Created a centralized appstate.hpp to cleanly share global state (g_Ethernet_Initialized, g_Wifi_Initialized) across the project, eliminating ad-hoc extern declarations.

API Capabilities & Analytics

• System Info (GET /api/system/info): Returns real-time JSON payload containing chip type, free heap, uptime, firmware version, and connection status. Data payload is lightweight (~110 bytes).
• Remote Reboot (POST /api/system/reboot): Initiates an async reboot using esp_timer with a 1-second delay, allowing the backend to flush a successful 200 OK JSON response to the client before the processor halts.
• CORS Support: Implemented Access-Control-Allow-Origin: * headers for all API GET and POST responses, along with an OPTIONS preflight handler, to support seamless local UI development against the ESP32.

Stability & Performance Fixes

• Persistent Daemon: Addressed an issue where app_main executed to completion immediately, causing the web server daemon to drop. Implemented a non-blocking vTaskDelay keep-alive loop to persist the application state and keep the HTTP server listening indefinitely without spinning the CPU.
• Static File Fallbacks: The LittleFS static file handler correctly falls back to index.html (and .gz variants) to seamlessly support Svelte's Single Page Application (SPA) routing patterns.

Observability Benchmarks

• Heap Usage: The system info endpoint natively tracks free heap availability. Observed typical runtime footprint leaves roughly 247 KB free heap with active WiFi, API handling, and active HTTP server routing.
• API Response Latency: The minimalist handler approach results in near-instantaneous JSON responses (milliseconds), effortlessly supporting the frontend dashboard's 5-second polling interval without blocking the ESP32-S3 network stack.