Made everything needed to update firmware. Added the bundle to upload both front and backendin a bundle. Added magic number for more safety

Provider/Documentation/build_bundle.md

@@ -0,0 +1,59 @@
+# Universal OTA Bundle
+
+The Universal OTA Bundle allows you to update both the **Firmware** and the **Frontend** of the Calendink Provider in a single operation. This ensures that your UI and backend logic are always in sync.
+
+## 1. How it Works
+
+The bundle is a custom `.bundle` file that contains:
+1. A **12-byte header** (Magic `BNDL`, FW size, UI size).
+2. The **Firmware binary** (`Provider.bin`).
+3. The **Frontend LittleFS binary** (`www_v*.bin`).
+
+The ESP32 backend streams this file, writing the firmware to the next OTA slot and the frontend to the inactive `www` partition. It only commits the update if both parts are written successfully.
+
+## 2. Prerequisites
+
+- You have a working [Frontend Build Environment](build_frontend.md).
+- You have the [ESP-IDF SDK](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/index.html) installed for firmware compilation.
+
+## 3. Creating a Bundle
+
+To create a new bundle, follow these steps in order:
+
+### Step A: Build the Frontend
+Inside the `frontend/` directory:
+```bash
+npm run build:esp32
+```
+
+### Step B: Build the Firmware
+From the project **root** directory:
+```bash
+idf.py build
+```
+
+### Step C: Generate the Bundle
+Inside the `frontend/` directory:
+```bash
+npm run ota:bundle
+```
+> [!NOTE]
+> `npm run ota:bundle` now automatically runs `npm run ota:package` first to ensure the latest Svelte build is turned into a LittleFS image before bundling.
+
+The output will be saved in `frontend/bundles/` with a name like `universal_v0.1.11.bundle`.
+
+## 4. Flashing the Bundle
+
+1. Open the Calendink Provider Dashboard in your browser.
+2. Navigate to the **System Updates** section.
+3. Click the **Universal Bundle** button.
+4. Drag and drop your `.bundle` file into the upload area.
+5. Click **Update**.
+
+The device will reboot once the upload is complete. You can verify the update by checking the version numbers and the UI changes (like the number of rockets in the header!).
+
+## 5. Troubleshooting
+
+- **"Invalid bundle magic"**: Ensure you are uploading a `.bundle` file, not a `.bin`.
+- **"Firmware part is corrupted"**: The bundle was likely created while the firmware build was incomplete or failed.
+- **Old UI appearing**: Ensure you ran `npm run build:esp32` *before* `npm run ota:bundle`.

Provider/Documentation/build_frontend.md

@@ -80,3 +80,9 @@ Once the backend supports it (Phase 2+), you can update the frontend without usi
    - Go to the **Frontend Update** section.
    - Select the `www.bin` file and click **Flash**.
    - The device will automatically write to the inactive partition and reboot.
+
+## 6. Universal OTA Bundle
+
+For a safer and more convenient update experience, you can bundle both the Firmware and Frontend into a single file. 
+
+See the [Universal OTA Bundle Guide](build_bundle.md) for details.

Provider/frontend/package.json

@@ -8,6 +8,7 @@
     "build": "vite build",
     "build:esp32": "vite build && node scripts/gzip.js",
     "ota:package": "node scripts/package.js",
+    "ota:bundle": "npm run ota:package && node scripts/bundle.js",
     "preview": "vite preview"
   },
   "devDependencies": {

Provider/frontend/scripts/bundle.js

@@ -0,0 +1,79 @@
+/**
+ * Universal OTA Bundle Creator
+ * Packs FW (Provider.bin) and WWW (www_v*.bin) into a single .bundle file.
+ */
+
+import { readFileSync, writeFileSync, readdirSync, existsSync, mkdirSync } from 'fs';
+import { resolve, dirname } from 'path';
+import { fileURLToPath } from 'url';
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const projectRoot = resolve(__dirname, '..');
+const providerRoot = resolve(projectRoot, '..');
+const binDir = resolve(projectRoot, 'bin');
+
+// Paths
+const fwFile = resolve(providerRoot, 'build', 'Provider.bin');
+const bundleDir = resolve(projectRoot, 'bundles');
+
+if (!existsSync(bundleDir)) {
+    mkdirSync(bundleDir, { recursive: true });
+}
+
+console.log('--- Universal Bundle Packaging ---');
+
+// 1. Find the latest www.bin with proper semantic version sorting
+const binFiles = readdirSync(binDir)
+    .filter(f => f.startsWith('www_v') && f.endsWith('.bin'))
+    .sort((a, b) => {
+        const getParts = (s) => {
+            const m = s.match(/v(\d+)\.(\d+)\.(\d+)/);
+            return m ? m.slice(1).map(Number) : [0, 0, 0];
+        };
+        const [aMajor, aMinor, aRev] = getParts(a);
+        const [bMajor, bMinor, bRev] = getParts(b);
+        return (bMajor - aMajor) || (bMinor - aMinor) || (bRev - aRev);
+    });
+
+if (binFiles.length === 0) {
+    console.error('Error: No www_v*.bin found in frontend/bin/. Run "npm run ota:package" first.');
+    process.exit(1);
+}
+
+const wwwFile = resolve(binDir, binFiles[0]);
+
+if (!existsSync(fwFile)) {
+    console.error(`Error: Firmware binary not found at ${fwFile}. Run "idf.py build" first.`);
+    process.exit(1);
+}
+
+try {
+    console.log(`Packing Firmware: ${fwFile}`);
+    console.log(`Packing Frontend: ${wwwFile}`);
+
+    const fwBuf = readFileSync(fwFile);
+    const wwwBuf = readFileSync(wwwFile);
+
+    // Create 12-byte header
+    // Magic: BNDL (4 bytes)
+    // FW Size: uint32 (4 bytes)
+    // WWW Size: uint32 (4 bytes)
+    const header = Buffer.alloc(12);
+    header.write('BNDL', 0);
+    header.writeUInt32LE(fwBuf.length, 4);
+    header.writeUInt32LE(wwwBuf.length, 8);
+
+    const bundleBuf = Buffer.concat([header, fwBuf, wwwBuf]);
+    const outputFile = resolve(bundleDir, `universal_v${binFiles[0].replace('www_v', '').replace('.bin', '')}.bundle`);
+
+    writeFileSync(outputFile, bundleBuf);
+
+    console.log('-------------------------------');
+    console.log(`Success: Bundle created at ${outputFile}`);
+    console.log(`Total size: ${(bundleBuf.length / 1024 / 1024).toFixed(2)} MB`);
+    console.log('-------------------------------');
+
+} catch (e) {
+    console.error('Error creating bundle:', e.message);
+    process.exit(1);
+}

Provider/frontend/src/App.svelte

@@ -108,7 +108,7 @@
   <div class="w-full max-w-6xl space-y-8">
     <!-- Header -->
     <div class="text-center">
-      <h1 class="text-2xl font-bold text-accent">Calendink Provider 🚀🚀🚀</h1>
+      <h1 class="text-2xl font-bold text-accent">Calendink Provider 🚀🚀👑</h1>
       <p class="text-text-secondary text-sm">ESP32-S3 System Dashboard v{__APP_VERSION__}</p>
       
       <!-- Status Badge -->
@@ -197,13 +197,16 @@
                   </div>
                   <div class="text-right flex flex-col items-end">
                     <div class="text-[11px] font-mono text-text-primary">{formatBytes(part.size)}</div>
+                    <div class="flex items-center gap-1.5 mt-0.5">
                       {#if part.app_version}
-                      <div class="text-[9px] text-accent font-bold">v{part.app_version}</div>
-                    {:else if part.free !== undefined}
-                      <div class="text-[9px] {part.free > 0 ? 'text-success' : 'text-text-secondary'} font-bold">
-                        {formatBytes(part.free)} free
-                      </div>
+                        <span class="text-[9px] text-accent font-bold">v{part.app_version}</span>
                       {/if}
+                      {#if part.free !== undefined}
+                        <span class="text-[9px] {part.free > 1024 ? 'text-success' : 'text-text-secondary'} font-bold">
+                          {formatBytes(part.free)} free
+                        </span>
+                      {/if}
+                    </div>
                   </div>
                 </div>
               {/each}

Provider/frontend/src/lib/OTAUpdate.svelte

@@ -1,6 +1,6 @@
 <script>
     let { onReboot = null } = $props();
-    import { getOTAStatus, uploadOTAFrontend, uploadOTAFirmware, getSystemInfo } from "./api.js";
+    import { getOTAStatus, uploadOTAFrontend, uploadOTAFirmware, uploadOTABundle, getSystemInfo } from "./api.js";
   
     const IS_DEV = import.meta.env.DEV;
 
@@ -23,7 +23,7 @@
   
     let selectedFile = $state(null);
     let showAdvanced = $state(false);
-    /** @type {'frontend' | 'firmware'} */
+    /** @type {'frontend' | 'firmware' | 'bundle'} */
     let updateMode = $state("frontend");
     let isDragging = $state(false);
   
@@ -55,7 +55,15 @@
     }
 
     function processFile(file) {
-      if (file.name.endsWith('.bin')) {
+      if (updateMode === 'bundle') {
+        if (file.name.endsWith('.bundle')) {
+          selectedFile = file;
+          errorMsg = "";
+        } else {
+          selectedFile = null;
+          errorMsg = "Please select a valid .bundle file";
+        }
+      } else if (file.name.endsWith('.bin')) {
         selectedFile = file;
         errorMsg = "";
       } else {
@@ -78,8 +86,10 @@
       try {
         if (updateMode === "frontend") {
             await uploadOTAFrontend(selectedFile);
-        } else {
+        } else if (updateMode === "firmware") {
             await uploadOTAFirmware(selectedFile);
+        } else {
+            await uploadOTABundle(selectedFile);
         }
         clearInterval(progressInterval);
         uploadProgress = 100;
@@ -105,6 +115,7 @@
     }
 
     const currentTarget = $derived(() => {
+        if (updateMode === 'bundle') return 'FW + UI';
         if (updateMode === 'frontend') return otaInfo.target_partition;
         // For firmware, target is the slot that is NOT the running one
         const runningLabel = otaInfo.running_firmware_label || 'ota_0';
@@ -131,7 +142,14 @@
             class="text-[10px] font-bold uppercase tracking-tight px-2 py-1 rounded transition-colors
                    {showAdvanced && updateMode === 'firmware' ? 'bg-accent text-white' : 'bg-border text-text-secondary hover:text-text-primary'}"
         >
-            Firmware OTA
+            Firmware
+        </button>
+        <button 
+            onclick={() => toggleMode('bundle')}
+            class="text-[10px] font-bold uppercase tracking-tight px-2 py-1 rounded transition-colors
+                   {showAdvanced && updateMode === 'bundle' ? 'bg-accent text-white' : 'bg-border text-text-secondary hover:text-text-primary'}"
+        >
+            Universal Bundle
         </button>
     </div>
   </div>
@@ -190,13 +208,13 @@
               ondragleave={() => isDragging = false}
               ondrop={handleDrop}
             >
-              <input type="file" accept=".bin" onchange={handleFileChange} class="absolute inset-0 opacity-0 cursor-pointer" />
+              <input type="file" accept="{updateMode === 'bundle' ? '.bundle' : '.bin'}" onchange={handleFileChange} class="absolute inset-0 opacity-0 cursor-pointer" />
               <div class="text-2xl">{updateMode === 'frontend' ? '🎨' : '⚙️'}</div>
               {#if selectedFile}
                 <div class="text-xs font-medium text-text-primary">{selectedFile.name}</div>
                 <div class="text-[10px] text-text-secondary">{(selectedFile.size / 1024).toFixed(1)} KB</div>
               {:else}
-                <div class="text-xs text-text-primary">Drop {updateMode} .bin here</div>
+                <div class="text-xs text-text-primary">Drop {updateMode === 'bundle' ? 'Universal .bundle' : updateMode === 'frontend' ? 'UI .bin' : 'Firmware .bin'} here</div>
                 <div class="text-[10px] text-text-secondary">or click to browse</div>
               {/if}
             </div>
@@ -207,7 +225,7 @@
                 disabled={status === "uploading"}
                 class="w-full py-2 text-xs font-bold rounded-lg transition-colors bg-accent text-white hover:brightness-110 disabled:opacity-40"
               >
-                {status === "uploading" ? 'Flashing...' : `Update ${updateMode === 'frontend' ? 'UI' : 'Firmware'}`}
+                {status === "uploading" ? 'Flashing...' : `Update ${updateMode === 'bundle' ? 'Everything' : updateMode === 'frontend' ? 'UI' : 'Firmware'}`}
               </button>
             {/if}
 

Provider/frontend/src/lib/api.js

@@ -94,3 +94,25 @@ export async function uploadOTAFirmware(file) {
     
     return res.json();
 }
+
+/**
+ * Upload a universal .bundle image (FW + WWW).
+ * @param {File} file The bundle binary file to upload.
+ * @returns {Promise<{status: string, message: string}>}
+ */
+export async function uploadOTABundle(file) {
+    const res = await fetch(`${API_BASE}/api/ota/bundle`, {
+        method: 'POST',
+        body: file,
+        headers: {
+            'Content-Type': 'application/octet-stream'
+        }
+    });
+    
+    if (!res.ok) {
+        const errorText = await res.text();
+        throw new Error(`Upload failed (${res.status}): ${errorText || res.statusText}`);
+    }
+    
+    return res.json();
+}

Provider/frontend/version.json

@@ -1,5 +1,5 @@
 {
   "major": 0,
   "minor": 1,
-  "revision": 12
+  "revision": 14
 }

Provider/main/api/ota/bundle.cpp

@@ -0,0 +1,187 @@
+// SDK
+#include "cJSON.h"
+#include "esp_http_server.h"
+#include "esp_log.h"
+#include "esp_ota_ops.h"
+#include "esp_partition.h"
+#include "esp_system.h"
+#include "esp_timer.h"
+#include "nvs.h"
+#include "nvs_flash.h"
+#include <sys/param.h>
+
+// Project
+#include "appstate.hpp"
+#include "types.hpp"
+
+#define BUNDLE_SCRATCH_BUFSIZE 4096
+
+typedef struct
+{
+  char magic[4];
+  uint32_t fw_size;
+  uint32_t www_size;
+} bundle_header_t;
+
+internal void bundle_ota_restart_timer_callback(void *arg) { esp_restart(); }
+
+internal esp_err_t api_ota_bundle_handler(httpd_req_t *req)
+{
+  httpd_resp_set_hdr(req, "Access-Control-Allow-Origin", "*");
+
+  if (req->content_len < sizeof(bundle_header_t))
+  {
+    ESP_LOGE("OTA_BUNDLE", "Request content too short");
+    httpd_resp_send_err(req, HTTPD_400_BAD_REQUEST, "Content too short");
+    return ESP_FAIL;
+  }
+
+  char *buf = (char *)malloc(BUNDLE_SCRATCH_BUFSIZE);
+  if (!buf)
+  {
+    ESP_LOGE("OTA_BUNDLE", "Failed to allocate buffer");
+    httpd_resp_send_err(req, HTTPD_500_INTERNAL_SERVER_ERROR, "Out of memory");
+    return ESP_FAIL;
+  }
+
+  // 1. Read Header
+  bundle_header_t header;
+  int overhead = httpd_req_recv(req, (char *)&header, sizeof(bundle_header_t));
+  if (overhead <= 0)
+  {
+    free(buf);
+    httpd_resp_send_err(req, HTTPD_500_INTERNAL_SERVER_ERROR,
+                        "Header receive failed");
+    return ESP_FAIL;
+  }
+
+  if (memcmp(header.magic, "BNDL", 4) != 0)
+  {
+    free(buf);
+    ESP_LOGE("OTA_BUNDLE", "Invalid magic: %.4s", header.magic);
+    httpd_resp_send_err(req, HTTPD_400_BAD_REQUEST, "Invalid bundle magic");
+    return ESP_FAIL;
+  }
+
+  ESP_LOGI("OTA_BUNDLE",
+           "Starting Universal Update: FW %lu bytes, WWW %lu bytes",
+           header.fw_size, header.www_size);
+
+  // 2. Prepare Firmware Update
+  const esp_partition_t *fw_part = esp_ota_get_next_update_partition(NULL);
+  esp_ota_handle_t fw_handle = 0;
+  esp_err_t err = esp_ota_begin(fw_part, header.fw_size, &fw_handle);
+  if (err != ESP_OK)
+  {
+    free(buf);
+    ESP_LOGE("OTA_BUNDLE", "esp_ota_begin failed (%s)", esp_err_to_name(err));
+    httpd_resp_send_err(req, HTTPD_500_INTERNAL_SERVER_ERROR,
+                        "OTA Begin failed");
+    return ESP_FAIL;
+  }
+
+  // 3. Stream Firmware
+  uint32_t fw_remaining = header.fw_size;
+  bool fw_first_chunk = true;
+  while (fw_remaining > 0)
+  {
+    int recv_len =
+        httpd_req_recv(req, buf, MIN(fw_remaining, BUNDLE_SCRATCH_BUFSIZE));
+    if (recv_len <= 0)
+    {
+      if (recv_len == HTTPD_SOCK_ERR_TIMEOUT)
+        continue;
+      esp_ota_abort(fw_handle);
+      free(buf);
+      return ESP_FAIL;
+    }
+
+    if (fw_first_chunk && recv_len > 0)
+    {
+      if ((uint8_t)buf[0] != 0xE9)
+      {
+        ESP_LOGE("OTA_BUNDLE", "Invalid FW magic in bundle: %02X",
+                 (uint8_t)buf[0]);
+        esp_ota_abort(fw_handle);
+        free(buf);
+        httpd_resp_send_err(
+            req, HTTPD_400_BAD_REQUEST,
+            "Invalid Bundle: Firmware part is corrupted or invalid.");
+        return ESP_FAIL;
+      }
+      fw_first_chunk = false;
+    }
+
+    esp_ota_write(fw_handle, buf, recv_len);
+    fw_remaining -= recv_len;
+  }
+  esp_ota_end(fw_handle);
+
+  // 4. Prepare WWW Update
+  uint8_t target_www_slot = g_Active_WWW_Partition == 0 ? 1 : 0;
+  const char *www_label = target_www_slot == 0 ? "www_0" : "www_1";
+  const esp_partition_t *www_part = esp_partition_find_first(
+      ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_DATA_LITTLEFS, www_label);
+
+  esp_partition_erase_range(www_part, 0, www_part->size);
+
+  // 5. Stream WWW
+  uint32_t www_remaining = header.www_size;
+  uint32_t www_written = 0;
+  while (www_remaining > 0)
+  {
+    int recv_len =
+        httpd_req_recv(req, buf, MIN(www_remaining, BUNDLE_SCRATCH_BUFSIZE));
+    if (recv_len <= 0)
+    {
+      if (recv_len == HTTPD_SOCK_ERR_TIMEOUT)
+        continue;
+      free(buf);
+      return ESP_FAIL;
+    }
+    esp_partition_write(www_part, www_written, buf, recv_len);
+    www_written += recv_len;
+    www_remaining -= recv_len;
+  }
+
+  free(buf);
+
+  // 6. Commit Updates
+  esp_ota_set_boot_partition(fw_part);
+
+  nvs_handle_t nvs_h;
+  if (nvs_open("storage", NVS_READWRITE, &nvs_h) == ESP_OK)
+  {
+    nvs_set_u8(nvs_h, "www_part", target_www_slot);
+    nvs_commit(nvs_h);
+    nvs_close(nvs_h);
+  }
+
+  ESP_LOGI("OTA_BUNDLE", "Universal Update Complete! Rebooting...");
+
+  httpd_resp_set_type(req, "application/json");
+  cJSON *root = cJSON_CreateObject();
+  cJSON_AddStringToObject(root, "status", "success");
+  cJSON_AddStringToObject(root, "message",
+                          "Universal update successful, rebooting...");
+  const char *resp = cJSON_Print(root);
+  httpd_resp_sendstr(req, resp);
+  free((void *)resp);
+  cJSON_Delete(root);
+
+  // Reboot
+  esp_timer_create_args_t tmr_args = {};
+  tmr_args.callback = &bundle_ota_restart_timer_callback;
+  tmr_args.name = "bundle_reboot";
+  esp_timer_handle_t tmr;
+  esp_timer_create(&tmr_args, &tmr);
+  esp_timer_start_once(tmr, 1'000'000);
+
+  return ESP_OK;
+}
+
+internal const httpd_uri_t api_ota_bundle_uri = {.uri = "/api/ota/bundle",
+                                                 .method = HTTP_POST,
+                                                 .handler =
+                                                     api_ota_bundle_handler,
+                                                 .user_ctx = NULL};

Provider/main/api/ota/firmware.cpp

@@ -72,6 +72,21 @@ internal esp_err_t api_ota_firmware_handler(httpd_req_t *req)
       return ESP_FAIL;
     }
 
+    if (binary_file_len == 0 && recv_len > 0)
+    {
+      if ((uint8_t)buf[0] != 0xE9)
+      {
+        ESP_LOGE("OTA_FW", "Invalid magic: %02X. Expected 0xE9 for Firmware.",
+                 (uint8_t)buf[0]);
+        esp_ota_abort(update_handle);
+        free(buf);
+        httpd_resp_send_err(
+            req, HTTPD_400_BAD_REQUEST,
+            "Invalid file: This does not look like an ESP32 firmware binary.");
+        return ESP_FAIL;
+      }
+    }
+
     err = esp_ota_write(update_handle, (const void *)buf, recv_len);
     if (err != ESP_OK)
     {

Provider/main/api/ota/frontend.cpp

@@ -8,7 +8,6 @@
 #include "nvs.h"
 #include "nvs_flash.h"
 
-
 // Project
 #include "appstate.hpp"
 #include "types.hpp"
@@ -59,6 +58,7 @@ internal esp_err_t api_ota_frontend_handler(httpd_req_t *req)
 
   int total_read = 0;
   int remaining = req->content_len;
+  bool first_chunk = true;
 
   while (remaining > 0)
   {
@@ -77,6 +77,21 @@ internal esp_err_t api_ota_frontend_handler(httpd_req_t *req)
       return ESP_FAIL;
     }
 
+    if (first_chunk)
+    {
+      if ((uint8_t)buf[0] == 0xE9)
+      {
+        ESP_LOGE("OTA", "Magic 0xE9 detected. This looks like a FIRMWARE bin, "
+                        "but you are uploading to FRONTEND slot!");
+        free(buf);
+        httpd_resp_send_err(
+            req, HTTPD_400_BAD_REQUEST,
+            "Invalid file: This is a Firmware binary, not a UI binary.");
+        return ESP_FAIL;
+      }
+      first_chunk = false;
+    }
+
     err = esp_partition_write(partition, total_read, buf, recv_len);
     if (err != ESP_OK)
     {

Provider/main/api/ota/status.cpp

@@ -3,9 +3,13 @@
 // SDK
 #include "cJSON.h"
 #include "esp_http_server.h"
+#include "esp_image_format.h"
 #include "esp_littlefs.h"
+#include "esp_log.h"
 #include "esp_ota_ops.h"
 #include "esp_partition.h"
+#include "esp_vfs.h"
+#include <string.h>
 
 // Project
 #include "appstate.hpp"
@@ -58,11 +62,16 @@ internal esp_err_t api_ota_status_handler(httpd_req_t *req)
       esp_app_desc_t app_desc;
       if (esp_ota_get_partition_description(p, &app_desc) == ESP_OK)
       {
-        // This is a bit of a hack as we don't have a direct "binary size" in
-        // the header but we can at least show it's occupied. For simplicity, if
-        // it's a valid app, we'll mark some space as used. Actually, without a
-        // better way to get the exact bin size, we'll just show it's an App.
         cJSON_AddStringToObject(p_obj, "app_version", app_desc.version);
+
+        // Get the true binary size from image metadata
+        esp_image_metadata_t data;
+        const esp_partition_pos_t pos = {.offset = p->address, .size = p->size};
+        if (esp_image_get_metadata(&pos, &data) == ESP_OK)
+        {
+          cJSON_AddNumberToObject(p_obj, "used", data.image_len);
+          cJSON_AddNumberToObject(p_obj, "free", p->size - data.image_len);
+        }
       }
     }
 

Provider/main/http_server.cpp

@@ -13,6 +13,7 @@
 #endif
 
 // Project
+#include "api/ota/bundle.cpp"
 #include "api/ota/firmware.cpp"
 #include "api/ota/frontend.cpp"
 #include "api/ota/status.cpp"
@@ -235,6 +236,7 @@ internal httpd_handle_t start_webserver(void)
     httpd_register_uri_handler(server, &api_ota_status_uri);
     httpd_register_uri_handler(server, &api_ota_frontend_uri);
     httpd_register_uri_handler(server, &api_ota_firmware_uri);
+    httpd_register_uri_handler(server, &api_ota_bundle_uri);
 
 #ifdef CONFIG_CALENDINK_DEPLOY_WEB_PAGES
     // Register static file handler last as a catch-all wildcard if deployed

Provider/tdd/firmware_ota.md

@@ -7,38 +7,44 @@
 
 ## 1. Goal
 
-Implement a robust Over-The-Air (OTA) update mechanism specifically for the main firmware of the ESP32-S3. The update must:
-- Update the core application logic without requiring a physical USB connection.
+Implement a robust Over-The-Air (OTA) update mechanism for both the main firmware of the ESP32-S3 and the Svelte frontend. The update must:
+- Update the core application logic and the user interface without requiring a physical USB connection.
+- Keep the Firmware and Frontend in sync by allowing them to be updated together atomically.
 - Provide a reliable fallback if an update fails (Rollback capability via A/B slots).
 - Provide a permanent "factory" fallback as an extreme safety measure.
-- Integrate seamlessly with the existing Svelte frontend UI for a push-based update experience.
-- Maintain a clear versioning scheme visible to the user.
+- Prevent accidental cross-flashing (e.g., flashing UI to firmware slots).
+- Maintain a clear versioning scheme visible to the user, with accurate partition space reporting.
 
 ## 2. Chosen Approach
 
-We implemented a **Dual-Partition Image Flash (A/B slots) with Factory Fallback** strategy using ESP-IDF's native OTA mechanisms.
+We implemented a **Universal Dual-Partition OTA system** using ESP-IDF's native OTA mechanisms for the firmware and LittleFS for the frontend.
 
-The build process generates a single `.bin` firmware image. This image is uploaded via the frontend UI and streamed directly to the inactive OTA flash partition (`ota_0` or `ota_1`). Upon successful transfer and validation, the bootloader is instructed to boot from the new partition on the next restart.
+Updates can be performed individually (Firmware only via `.bin`, Frontend only via `.bin`), but the primary and recommended approach is the **Universal OTA Bundle**. 
+The build process generates a single `.bundle` file containing both the firmware image and the compiled frontend filesystem. This bundle is uploaded via the frontend UI, streamed directly to the inactive OTA flash partition (`ota_0` or `ota_1`) and inactive UI partition (`www_0` or `www_1`). Upon successful transfer and validation of both components, the bootloader and NVS are instructed to switch active partitions on the next restart.
 
 ## 3. Design Decisions & Trade-offs
 
 ### 3.1. Why Dual-Partition (A/B) with Factory?
 - **Safety**: A failed or interrupted upload never "bricks" the device.
-- **Factory Fallback**: By maintaining a dedicated 2MB `factory` partition alongside the two 2MB OTA partitions (`ota_0`, `ota_1`), we ensure that even if both OTA slots are irrecoverably corrupted, the device can always boot into a known-good state. This requires an initial USB flash to set up but provides maximum long-term reliability.
-- **Storage Allocation**: With 16MB of total flash on the ESP32-S3, dedicating 6MB to application code (3x 2MB) is a worthwhile trade-off for extreme resilience, while still leaving ample room for the frontend (`www_0`, `www_1`) and NVS.
+- **Factory Fallback**: By maintaining a dedicated 2MB `factory` partition alongside the two 2MB OTA partitions (`ota_0`, `ota_1`), we ensure that even if both OTA slots are irrecoverably corrupted, the device can always boot into a known-good state. 
+- **Frontend Sync**: The frontend also uses a dual-partition layout (`www_0`, `www_1`). The Universal Bundle ensures both FW and UI switch together.
 
 ### 3.2. Automatic App Rollback
 We rely on ESP-IDF's built-in "App Rollback" feature.
-- **The Mechanism**: When the ESP32 boots a newly OTA-flashed firmware, it is marked as "Pending Verify". If the application crashes, resets, or fails to explicitly mark itself as "valid" during this initial boot, the bootloader automatically reverts to the previous working partition on the subsequent boot.
-- **Validation Point**: We consider the firmware "valid" (and stop the rollback timer) only after it successfully establishes a network connection (Ethernet or WiFi). This ensures that a bad update won't permanently disconnect the device from future OTA attempts.
+- **The Mechanism**: When the ESP32 boots a newly OTA-flashed firmware, it is marked as "Pending Verify". If the application crashes or fails to mark itself as "valid", the bootloader reverts to the previous working partition.
+- **Validation Point**: We consider the firmware "valid" only after it successfully establishes a network connection.
 
-### 3.3. Push vs. Pull Updates
-- **Decision**: We implemented a "Push" mechanism where the user manually uploads the `.bin` file via the web UI.
-- **Rationale**: This matches the existing frontend OTA workflow and is simpler to implement initially. It avoids the need for external update servers, manifest files, and polling mechanisms. A "Pull" mechanism can be added later if fleet management becomes a requirement.
+### 3.3. Universal Bundle Format & Automation
+- **Format**: A custom 12-byte header (`BNDL` magic + 4-byte FW size + 4-byte UI size) followed by the FW binary and UI binary.
+- **Automation**: The Svelte build chain automates packaging. Running `npm run ota:bundle` automatically triggers Vite production build, LittleFS frontend packaging, applies proper semantic version sorting (to always pick the latest compiled UI), and generates the `.bundle` payload.
 
-### 3.4. Versioning Strategy
-- **Decision**: We extract the firmware version directly from the ESP-IDF natively embedded `esp_app_desc_t` structure.
-- **Rationale**: This ensures the version reported by the API (`GET /api/system/info`) is exactly the version compiled by CMake (`PROJECT_VER`), eliminating the risk of manual mismatches or external version files getting out of sync.
+### 3.4. Safety & Validation
+- **Magic Number Checks**: The backend enforces strict validation before writing to flash. Firmware endpoints and bundle streams check for the ESP32 image magic byte (`0xE9`), and Bundle endpoints check for the `BNDL` magic header. This prevents a user from accidentally uploading a LittleFS image to the Firmware slot, avoiding immediate boot loops.
+- **Atomic Commits**: The Universal Bundle handler only sets the new boot partition and updates the NVS UI partition index if *both* firmware and frontend streams complete successfully. 
+
+### 3.5. Versioning & Partition Metadata
+- **Firmware Versioning**: Extracted natively from `esp_app_desc_t`, syncing API version with CMake `PROJECT_VER`.
+- **Space Reporting**: The system dynamically scans App partitions using `esp_image_get_metadata()` to determine the exact binary size flashed in each slot. This allows the UI to display accurate "used" and "free" space per partition, regardless of the fixed partition size.
 
 ## 4. Final Architecture
 
@@ -56,17 +62,18 @@ www_1,    data, littlefs, ,         1M
 ```
 
 ### 4.2. Backend Components
-- `main/api/ota/firmware.cpp`: The endpoint (`POST /api/ota/firmware`) handling the streaming ingestion of the `.bin` file using standard ESP-IDF `esp_ota` functions.
-- `main/api/system/system.cpp`: The endpoint querying `esp_app_get_description()` to expose the unified version payload to the frontend.
-- `main/main.cpp`: The orchestrator that calls `esp_ota_mark_app_valid_cancel_rollback()` post-network connection.
+- `bundle.cpp`: Handles `POST /api/ota/bundle`. Streams the file, splitting it on the fly into the inactive `ota` and `www` partitions.
+- `firmware.cpp` & `frontend.cpp`: Handles individual component updates.
+- `status.cpp`: Uses `esp_partition_find` and `esp_image_get_metadata` to report partition sizes and active slots.
+- `main.cpp`: Calls `esp_ota_mark_app_valid_cancel_rollback()` post-network connection and manages NVS synchronization for the UI slot when booting from Factory.
 
 ### 4.3. UI/UX Implementation
-- The Frontend OTA update component (`OTAUpdate.svelte`) is expanded to include a parallel "Firmware Update" section.
-- This UI section handles file selection, upload progress visualization, and system reboot confirmation, providing parity with the existing frontend update UX.
+- The Svelte Dashboard features a comprehensive "Update System" component supporting individual (FW/UI) and combined (Bundle) uploads.
+- A "Partition Table" view provides real-time visibility into the exact binary size, available free space, and version hash of every system and app partition.
 
 ## 5. Summary
 
-We use **ESP-IDF's native OTA APIs** with a **Factory + Dual A/B Partition** layout for maximum reliability. The system leverages **Automatic App Rollback** to prevent network lockouts from bad firmware. Versioning is natively controlled via the **CMake build descriptions**, and the entire update process is driven centrally from the **Svelte Frontend UI** via a Push-based REST endpoint.
+We use **ESP-IDF's native OTA APIs** with a **Factory + Dual A/B Partition** layout, synchronized with a **Dual LittleFS Partition** layout for the frontend. The system relies on custom **Universal Bundles** to guarantee atomic FW+UI upgrades, protected by **Magic Number validations** and **Automatic App Rollbacks**. The entire process is driven from a highly integrated Svelte UI that leverages backend metadata extraction to provide accurate system insights.
 
 ---
 *Created by Antigravity - Last Updated: 2026-03-03*