Calendink/Provider/tdd/backend_architecture.md

# Backend Architecture for ESP32-S3 Provider

**Authored by Claude Opus 4**
**Date:** 2026-03-02

---

## 1. Goal

Serve the Svelte web dashboard and expose a REST API from the ESP32-S3 using ESP-IDF's built-in `esp_http_server`. The backend must:
- Serve static files (the compiled Svelte frontend) from flash
- Provide system information over JSON
- Allow remote reboot
- Support independent frontend/backend development workflows

## 2. Chosen Stack

| Technology | Source | Role |
|---|---|---|
| **esp_http_server** | ESP-IDF built-in | HTTP server daemon |
| **cJSON** | ESP-IDF built-in | JSON serialization for API responses |
| **LittleFS** | [joltwallet/esp_littlefs](https://github.com/joltwallet/esp_littlefs) | Filesystem on flash for serving frontend files |

All three are standard in ESP-IDF projects. `esp_http_server` and `cJSON` ship with the SDK. LittleFS is the recommended flash filesystem for ESP-IDF — Espressif's own [restful_server example](https://github.com/espressif/esp-idf/blob/master/examples/protocols/http_server/restful_server/main/idf_component.yml) uses `joltwallet/littlefs`.

## 3. Why LittleFS Over Other Options

### LittleFS vs SPIFFS

| | LittleFS | SPIFFS |
|---|---|---|
| **Directory support** | ✅ Real directories | ❌ Flat namespace |
| **Wear leveling** | ✅ Dynamic | ⚠️ Static |
| **Power-loss safe** | ✅ Copy-on-write | ❌ Can corrupt |
| **ESP-IDF status** | ✅ Recommended | ⚠️ Deprecated in recent examples |
| **Performance** | Faster for small files | Slower mount, no dir listing |

SPIFFS was the original choice in older ESP-IDF examples but has been replaced by LittleFS in the current restful_server example. LittleFS is the better choice going forward.

### LittleFS Partition vs EMBED_FILES

We considered two approaches to deploy the frontend:

| | LittleFS Partition | `EMBED_FILES` (binary embedding) |
|---|---|---|
| **Storage** | Separate flash partition | Compiled into firmware binary |
| **Update** | Can flash partition independently | Must reflash entire firmware |
| **OTA** | Needs separate partition OTA | UI updates with firmware naturally |
| **Dev workflow** | Can skip frontend flash during iteration | Always included |
| **Filesystem** | Real VFS — open/read/close file APIs | Direct memory pointer |

**We chose LittleFS partition** because:
1. **Development speed** — a Kconfig toggle (`CALENDINK_DEPLOY_WEB_PAGES`) lets you skip flashing the frontend entirely when iterating on the backend
2. **Separation of concerns** — frontend and firmware have independent flash regions
3. **Follows ESP-IDF conventions** — matches the official restful_server example pattern
4. **No RAM overhead** — files are read from flash in chunks, not loaded into memory

## 4. Kconfig Deploy Toggle

The `CALENDINK_DEPLOY_WEB_PAGES` menuconfig option controls whether the frontend gets flashed:

| Setting | Effect | Use case |
|---|---|---|
| **OFF** (default) | Frontend not flashed. API still works. | Backend development — fast flash, use PC dev server for UI |
| **ON** | `frontend/dist/` is written to the `www` LittleFS partition | Production deployment — everything runs on ESP32 |

This mirrors the official ESP-IDF pattern (`CONFIG_EXAMPLE_DEPLOY_WEB_PAGES` in the restful_server example).

## 5. API Design

### Endpoints

```
GET  /api/system/info    → JSON with chip, heap, uptime, firmware, connection
POST /api/system/reboot  → JSON acknowledgment, then esp_restart()
GET  /*                   → Static files from LittleFS /www (when deployed)
```

### CORS

During development, the Svelte dev server runs on `http://localhost:5173` and API calls go to `http://<ESP32_IP>`. This is cross-origin, so the backend adds CORS headers:

- `Access-Control-Allow-Origin: *`
- `Access-Control-Allow-Methods: GET, POST, OPTIONS`
- `Access-Control-Allow-Headers: Content-Type`
- `OPTIONS` preflight handler

In production (frontend served from ESP32), everything is same-origin — CORS headers have no effect but don't hurt.

## 6. File Organization

```
Provider/main/
├── main.cpp                    # Entry point, network init, starts HTTP server
├── http_server.cpp             # HTTP server lifecycle, static file handler, CORS
├── api/
│   └── system/
│       ├── info.cpp            # GET /api/system/info
│       └── reboot.cpp          # POST /api/system/reboot
├── led_status.cpp              # Existing — LED management
├── connect.cpp                 # Existing — Ethernet/WiFi
├── types.hpp                   # Existing — type aliases
└── Kconfig.projbuild           # Existing + new web server config
```

### Why This Structure

The project uses a **unity build** pattern — `main.cpp` `#include`s `.cpp` files directly (e.g. `#include "connect.cpp"`). This is unconventional but works well for small embedded projects since ESP-IDF only compiles the files listed in `idf_component_register(SRCS ...)`.

We extend this pattern to the HTTP server:
- `http_server.cpp` `#include`s the API handler files
- Each handler file is self-contained: it defines its URI struct and registration function
- New endpoint groups get their own folder under `api/` (e.g. `api/calendar/`, `api/display/`)

## 7. Partition Table

```
# Name,   Type, SubType,  Offset,   Size
nvs,      data, nvs,      0x9000,   0x6000
phy_init, data, phy,      0xf000,   0x1000
factory,  app,  factory,  0x10000,  1M
www,      data, littlefs, ,         64K
```

The `www` partition is 64KB — more than enough for the 16kB gzipped frontend. Only gets written during `idf.py flash` when `CALENDINK_DEPLOY_WEB_PAGES` is enabled.

## 8. Build Pipeline

```
Frontend Build (PC)                      ESP-IDF Build
──────────────────                       ──────────────
npm run build                             idf.py build
  ↓                                         ↓
frontend/dist/                           firmware.bin
  index.html (47kB)                          +
  index.html.gz (16kB)                   www.bin (LittleFS image, when deploy=ON)
                                              ↓
                                         idf.py flash
                                              ↓
                                         ESP32-S3 Flash
                                         ├── factory partition → firmware
                                         └── www partition → frontend files
```

## 9. Summary

## 9. Summary

We use **esp_http_server + cJSON + LittleFS** — all standard ESP-IDF components — to serve the frontend and expose a REST API. A **LittleFS partition** stores frontend files separately from firmware, with a **Kconfig toggle** to skip frontend flashing during backend development. The API is structured as **modular handler files** under `api/` for clean scalability.

---

## 10. Implementation Results

*Added 2026-03-03 after implementation and integration was completed.*

### Refactoring & Architecture Outcomes
- **Unity Build Pattern**: Successfully adopted for the HTTP server (`http_server.cpp` includes `.cpp` API handlers). This simplified the build process, reduced include complexity, and resolved multiple redefinition and linkage errors without needing complex CMake modifications.
- **State Management**: Created a centralized `appstate.hpp` to cleanly share global state (`g_Ethernet_Initialized`, `g_Wifi_Initialized`) across the project, eliminating ad-hoc `extern` declarations.

### API Capabilities & Analytics
- **System Info (`GET /api/system/info`)**: Returns real-time JSON payload containing chip type, free heap, uptime, firmware version, and connection status. Data payload is lightweight (~110 bytes).
- **Remote Reboot (`POST /api/system/reboot`)**: Initiates an async reboot using `esp_timer` with a 1-second delay, allowing the backend to flush a successful `200 OK` JSON response to the client before the processor halts.
- **CORS Support**: Implemented `Access-Control-Allow-Origin: *` headers for all API GET and POST responses, along with an `OPTIONS` preflight handler, to support seamless local UI development against the ESP32.

### Stability & Performance Fixes
- **Persistent Daemon**: Addressed an issue where `app_main` executed to completion immediately, causing the web server daemon to drop. Implemented a non-blocking `vTaskDelay` keep-alive loop to persist the application state and keep the HTTP server listening indefinitely without spinning the CPU.
- **Static File Fallbacks**: The LittleFS static file handler correctly falls back to `index.html` (and `.gz` variants) to seamlessly support Svelte's Single Page Application (SPA) routing patterns.

### Observability Benchmarks
- **Heap Usage**: The system info endpoint natively tracks free heap availability. Observed typical runtime footprint leaves roughly **247 KB free heap** with active WiFi, API handling, and active HTTP server routing.
- **API Response Latency**: The minimalist handler approach results in near-instantaneous JSON responses (milliseconds), effortlessly supporting the frontend dashboard's 5-second polling interval without blocking the ESP32-S3 network stack.