Juliet/Game/Plans/01_Serialization_And_Text_Archive.md

# Juliet Game Engine: Serialization Core & .jasset Text Archive
## Technical Specification & Implementation Plan

**Document ID**: SPEC-001-SERIALIZATION-CORE  
**Component**: Juliet Engine Core / Asset Pipeline  
**Target Architecture**: Juliet Game Engine (C++20, x64, D3D12)  
**File Location**: `Game/Plans/01_Serialization_And_Text_Archive.md`  

---

## 1. Executive Summary & Architecture Goals

### 1.1 Context & Current Limitations
The Juliet game engine previously utilized packed binary records for level and entity serialization (`Assets/world.bin`, `WorldFileHeader`, `WorldEntityDiskRecord`). While packed binary formats are compact, they present severe architectural roadblocks during collaborative development:
1. **Merge Incompatibility**: Binary assets cannot be merged or diffed across version control systems (Git / Perforce). Concurrent edits by multiple designers or engineers result in unresolvable binary conflicts and data loss.
2. **Schema Rigidity**: Adding, removing, or reordering a single struct field invalidates all previously serialized binary blobs unless complex, manual byte-offset mapping tables are maintained.
3. **Opacity**: Programmers and technical artists cannot inspect, debug, or patch asset properties in a standard text editor.

### 1.2 Architectural Goals
The `.jasset` text archive framework is engineered to replace legacy binary blobs with a robust, human-readable, diff-friendly property serialization pipeline while strictly adhering to Juliet's performance and memory constraints:
- **Zero Dynamic Heap Allocations**: All parsing, tokenization, formatting, and buffer transformations execute entirely within Juliet memory arenas (`Arena`, `TempArena`, `scratch_begin` / `scratch_end`). Standard library containers (`std::string`, `std::vector`, `std::map`) and raw heap allocators (`malloc`, `new`) are forbidden.
- **Zero-Copy In-Memory Tokenization**: Files are loaded into arena memory once via `LoadFile`. The tokenizer parses properties into lightweight slices represented by Juliet's `String` (`char* Str; size_t Size;`), referencing existing file buffer bytes without string replication or null-termination overhead.
- **$O(1)$ Property Lookup via Compile-Time & Runtime CRC32**: Property keys in text files are hashed once during tokenization into 32-bit CRC values. In code, lookups utilize compile-time hashed literals (`operator""_crc32`). Property resolution reduces to a single 32-bit integer comparison, avoiding string comparison overhead in serialization loops.
- **Symmetric Single-Function Serialization**: A single `Serialize` implementation per entity or data structure handles both Save and Load paths, guaranteeing that write and read schemas never diverge.
- **Graceful Forward/Backward Compatibility**: Missing keys during load automatically preserve struct default values. Unknown keys present in newer asset files are safely ignored without parse failure.
- **Two-Tier Decoupled Versioning**: Core engine entity properties (`kEntityBaseVersion`) and derived gameplay class properties (`Class::Version`) are versioned independently. Engine-level updates never bump derived entity class versions.
- **Post-Serialization In-Place Migration**: Deprecated fields can be read through specialized primitives (`ReadDeprecated*`) to migrate legacy data structures in-place upon loading, producing cleaned, modern schemas on subsequent saves.
- **Zero Exceptions & Total Warning Cleanliness**: Fully conforming to Juliet coding guidelines: strict assertions (`Assert`), `[[nodiscard]]`, `auto*`/`auto&`, mandatory braces, and `static_cast`/`reinterpret_cast`.

---

## 2. The `.jasset` Text Format Specification

### 2.1 Grammar & Structural Rules
The `.jasset` format uses a line-oriented, key-value property hierarchy designed for visual clarity and clean Git diffs.

```ebnf
AssetFile       ::= { CommentLine | EmptyLine | PropertyDeclaration } ;
CommentLine     ::= ( "#" | "//" ) { Character } LineEnding ;
EmptyLine       ::= { Whitespace } LineEnding ;
PropertyDeclaration ::= KeyHeader LineEnding ValueBlock ;
KeyHeader       ::= ";" { Whitespace } Identifier ;
ValueBlock      ::= { ValueLine LineEnding } ;
ValueLine       ::= { Whitespace } ValueString { Whitespace } ;
LineEnding      ::= "\r\n" | "\n" ;
Identifier      ::= [a-zA-Z_][a-zA-Z0-9_]* ;
```

1. **Property Key Header (`;\n`)**:
   - Every property declaration begins with a semicolon `;` as the first non-whitespace character.
   - The semicolon is followed by optional whitespace and a CamelCase identifier: `; PropertyName`.
   - Keys are case-sensitive and must be valid C++ identifier tokens.
2. **Value Block**:
   - The line immediately following the property header contains the property's serialized payload.
   - For multi-line values or arrays, lines continue until the next property header `;` or end-of-file.
3. **Comments & Ignored Tokens**:
   - Any line starting with `#` or `//` (after optional leading whitespace) is treated as a comment.
   - Empty lines and lines containing only whitespace are ignored.
4. **Line Termination Handling**:
   - The parser natively accepts both Windows (`\r\n`) and POSIX (`\n`) line endings.
   - Trailing carriage returns (`\r`) are automatically stripped during tokenization.
5. **Whitespace Tolerance**:
   - Leading and trailing spaces and horizontal tabs (`\t`) on both key and value lines are stripped during tokenization.

### 2.2 Formatting Specifications & Examples

#### Scalar Types
- **Floating Point (`float`, `float32`)**: Formatted via `%f` (default 6 decimal digits) or `%.9g` for full single-precision round-trip fidelity.
  ```
  ; Health
  100.000000

  ; Mass
  14.250000
  ```
- **32-Bit Signed Integer (`int32`)**: Formatted as decimal integer `%d`.
  ```
  ; AmmoCount
  45

  ; TeamIndex
  -1
  ```
- **64-Bit Unsigned Integer (`uint64`, `EntityID`)**: Formatted as a 16-character padded hexadecimal integer prefixed with `0x`. Hexadecimal ensures 64-bit handle readability and exact bit-pattern preservation.
  ```
  ; EntityID
  0x00000000DEADBEEF

  ; LayerMask
  0x0000000000000001
  ```
- **Boolean (`bool`)**: Formatted as `true` or `false`. For backwards tolerance, the parser also accepts `1` and `0`.
  ```
  ; IsStatic
  true

  ; CastShadows
  false
  ```

#### Vector Types
- **3D Vector (`Vector3` / `float x, y, z`)**: Formatted as three space-delimited floating-point values on a single line.
  ```
  ; Position
  10.500000 0.000000 -25.250000

  ; Scale
  1.000000 1.000000 1.000000
  ```
- **2D Vector (`Vector2` / `float x, y`)**: Formatted as two space-delimited floating-point values.
  ```
  ; UVOffset
  0.000000 0.500000
  ```

#### String Types
- **String (`String` / `String8`)**:
  - Strings without spaces can be serialized as raw text tokens.
  - Strings containing spaces or symbols are enclosed in double quotes `"..."`.
  ```
  ; AssetName
  Character_Mesh_Hero

  ; DisplayName
  "Grand Citadel Knight"
  ```

#### Comprehensive Entity Asset File Example (`Entity_01.jasset`)
```
# Juliet Game Engine Asset File
# Generated automatically by AssetPipeline. Do not manually corrupt keys.

; AssetType
Entity

; BaseVersion
1

; EntityID
0x0000000000000042

; Kind
Inert

; Position
100.000000 25.500000 -50.000000

; ClassVersion
2

; MeshInstance
4

; MaterialOverride
"Materials/M_Granite_Polished"

; IsVisible
true
```

---

## 3. The In-Memory Zero-Copy Parser

### 3.1 Memory Layout & Property Nodes
To eliminate heap fragmentation and per-object allocation overhead during level loading, the parser loads the entire `.jasset` file into contiguous arena memory via `LoadFile`. The tokenizer then constructs a flat array of `TextPropertyNode` structures on a `TempArena`.

```cpp
struct TextPropertyNode
{
    uint32 KeyCRC;
    String Value;
    bool   Consumed;
};
```

- `KeyCRC`: The 32-bit CRC hash of the trimmed property name.
- `Value`: A `String` (`String8`) struct containing a `char* Str` pointer directly into the file buffer and `size_t Size`. No new string allocations are performed.
- `Consumed`: A boolean flag initialized to `false`. When a property is queried and read via `SerializeProp`, `Consumed` is set to `true`.

```
File Buffer in Arena:
+-------------------------------------------------------------------------------+
| ; Position\n10.0 20.0 30.0\n; Health\n100.0\n                                |
+-------------------------------------------------------------------------------+
       ^              ^              ^        ^
       |              |              |        |
TextPropertyNode[0]:  |              |        |
  KeyCRC: CRC("Position")            |        |
  Value: { Str --------+ , Size: 14 }         |
  Consumed: true                              |
                                              |
TextPropertyNode[1]:                          |
  KeyCRC: CRC("Health")                       |
  Value: { Str -------------------------------+ , Size: 5 }
  Consumed: true
```

### 3.2 Dual-Mode CRC32: Compile-Time & Runtime
Juliet's `CRC32.h` defines `consteval uint32 crc32(const char* str, size_t length)`. However, `consteval` guarantees compilation failure when called on runtime input (such as keys extracted from a text file during parsing).

The engine requires a unified `Crc32` implementation that functions at runtime for parsed text tokens while retaining `constexpr` / `consteval` capability for compile-time string literals.

```cpp
// Core/Common/CRC32.h additions
[[nodiscard]] constexpr uint32 Crc32(const char* str, size_t length)
{
    Assert(str != nullptr || length == 0);
    const char* p   = str;
    uint32      crc = ~0U;
    while (length--)
    {
        crc = details::crc32_tab[(crc ^ static_cast<uint8>(*p++)) & 0xFF] ^ (crc >> 8);
    }
    return crc ^ ~0U;
}

[[nodiscard]] constexpr uint32 Crc32(String str)
{
    return Crc32(str.Str, str.Size);
}

[[nodiscard]] consteval uint32 operator""_crc32(const char* str, size_t length)
{
    return Crc32(str, length);
}
```

### 3.3 Zero-Copy Tokenization Algorithm
The tokenization algorithm scans the memory buffer in a single pass. It first counts property keys to allocate the exact array size on the arena, then populates the `TextPropertyNode` array.

```cpp
[[nodiscard]] inline String TrimWhitespace(String str)
{
    while (str.Size > 0 && (*str.Str == ' ' || *str.Str == '\t' || *str.Str == '\r' || *str.Str == '\n'))
    {
        str.Str++;
        str.Size--;
    }
    while (str.Size > 0 && (str.Str[str.Size - 1] == ' ' || str.Str[str.Size - 1] == '\t' || 
                            str.Str[str.Size - 1] == '\r' || str.Str[str.Size - 1] == '\n'))
    {
        str.Size--;
    }
    return str;
}

struct ParsedTextArchive
{
    TextPropertyNode* Nodes         = nullptr;
    uint32            PropertyCount = 0;
};

[[nodiscard]] ParsedTextArchive TokenizeTextArchive(NonNullPtr<Arena> arena, ByteBuffer fileBuffer)
{
    Assert(fileBuffer.Data != nullptr);

    char* cursor = reinterpret_cast<char*>(fileBuffer.Data);
    char* end    = cursor + fileBuffer.Size;

    // Pass 1: Count properties to allocate exactly on arena
    uint32 propertyCount = 0;
    char*  scan          = cursor;
    while (scan < end)
    {
        if (*scan == ';')
        {
            if (scan == cursor || *(scan - 1) == '\n')
            {
                propertyCount++;
            }
        }
        scan++;
    }

    if (propertyCount == 0)
    {
        return { nullptr, 0 };
    }

    auto* nodes = ArenaPushArray<TextPropertyNode>(arena, propertyCount);
    Assert(nodes != nullptr);

    // Pass 2: Extract keys and value slices
    uint32 nodeIndex = 0;
    scan             = cursor;

    while (scan < end && nodeIndex < propertyCount)
    {
        // Skip leading whitespace / empty lines / comments
        while (scan < end && (*scan == '\r' || *scan == '\n' || *scan == ' ' || *scan == '\t'))
        {
            scan++;
        }

        if (scan >= end)
        {
            break;
        }

        // Check for comment line
        if (*scan == '#' || (*scan == '/' && scan + 1 < end && *(scan + 1) == '/'))
        {
            while (scan < end && *scan != '\n')
            {
                scan++;
            }
            continue;
        }

        // Check for property declaration ';'
        if (*scan == ';')
        {
            scan++; // skip ';'

            // Extract key name
            char* keyStart = scan;
            while (scan < end && *scan != '\r' && *scan != '\n')
            {
                scan++;
            }

            String rawKey = { .Str = keyStart, .Size = static_cast<size_t>(scan - keyStart) };
            String key    = TrimWhitespace(rawKey);

            // Skip newline after key
            while (scan < end && (*scan == '\r' || *scan == '\n'))
            {
                scan++;
            }

            // Extract value block (everything until next ';' at start of line or EOF)
            char* valStart = scan;
            char* valEnd   = scan;

            while (scan < end)
            {
                if (*scan == ';' && (scan == cursor || *(scan - 1) == '\n'))
                {
                    break;
                }
                scan++;
                valEnd = scan;
            }

            String rawVal = { .Str = valStart, .Size = static_cast<size_t>(valEnd - valStart) };
            String val    = TrimWhitespace(rawVal);

            nodes[nodeIndex].KeyCRC   = Crc32(key);
            nodes[nodeIndex].Value    = val;
            nodes[nodeIndex].Consumed = false;
            nodeIndex++;
        }
        else
        {
            // Advance unexpected character
            scan++;
        }
    }

    return { nodes, nodeIndex };
}
```

### 3.4 Key Lookup & Unconsumed Key Audit
Lookup performs a fast linear scan over the contiguous `TextPropertyNode` array. Because typical game entities possess between 5 and 50 properties, a cache-coherent linear scan over contiguous 16-byte structs executes in single-digit nanoseconds, comfortably fitting within CPU L1/L2 data cache.

```cpp
[[nodiscard]] inline TextPropertyNode* FindProperty(TextPropertyNode* nodes, uint32 count, uint32 keyCRC)
{
    Assert(nodes != nullptr || count == 0);
    for (uint32 index = 0; index < count; ++index)
    {
        if (nodes[index].KeyCRC == keyCRC)
        {
            nodes[index].Consumed = true;
            return &nodes[index];
        }
    }
    return nullptr;
}

#if JULIET_DEBUG
inline void AuditUnconsumedProperties(const TextPropertyNode* nodes, uint32 count, const char* contextName)
{
    Assert(nodes != nullptr || count == 0);
    for (uint32 index = 0; index < count; ++index)
    {
        if (!nodes[index].Consumed)
        {
            LogWarning(LogCategory::Core, 
                       "[%s] Unconsumed or obsolete property detected: CRC 0x%08X (Value: '%.*s')",
                       contextName, nodes[index].KeyCRC, 
                       static_cast<int>(nodes[index].Value.Size), nodes[index].Value.Str);
        }
    }
}
#endif
```

---

## 4. The `archive` Struct & Mode Handling

### 4.1 Struct Definition & Mode Flags
Juliet's existing `archive` struct in `Core/Common/serialization.h` is restricted to binary offsets and raw arena pointers. We upgrade `archive` into a unified serialization context supporting both text `.jasset` and binary streams.

```cpp
enum class ArchiveMode : uint8
{
    SavingText,
    LoadingText,
    SavingBinary,
    LoadingBinary
};

struct TextPropertyNode;

struct archive
{
    Arena*            ArenaInstance = nullptr;
    IOStream*         Stream        = nullptr;
    TextPropertyNode* Properties    = nullptr;
    uint32            PropertyCount = 0;
    ArchiveMode       Mode          = ArchiveMode::LoadingText;
    uint32            BaseVersion   = 0;
    uint16            ClassVersion  = 0;

    // Legacy binary support fields
    void*             BasePtr       = nullptr;
    index_t           Offset        = 0;

    [[nodiscard]] bool IsLoading() const
    {
        return Mode == ArchiveMode::LoadingText || Mode == ArchiveMode::LoadingBinary;
    }

    [[nodiscard]] bool IsSaving() const
    {
        return Mode == ArchiveMode::SavingText || Mode == ArchiveMode::SavingBinary;
    }

    [[nodiscard]] bool IsText() const
    {
        return Mode == ArchiveMode::LoadingText || Mode == ArchiveMode::SavingText;
    }

    [[nodiscard]] bool IsBinary() const
    {
        return Mode == ArchiveMode::LoadingBinary || Mode == ArchiveMode::SavingBinary;
    }
};
```

### 4.2 Output Stream Formatting via `IOPrintf`
When saving in `ArchiveMode::SavingText`, `archive` outputs directly to an open `IOStream` using `IOPrintf`.

```cpp
inline void WritePropertyHeader(archive& ar, const char* keyName)
{
    Assert(ar.IsSaving());
    Assert(ar.Stream != nullptr);
    Assert(keyName != nullptr);
    IOPrintf(ar.Stream, "; %s\n", keyName);
}
```

This design provides:
1. Direct stream output with zero heap buffer allocations.
2. Canonical spacing and formatting across all entity serializers.
3. Formatted outputs immediately flushed or buffered according to `IOStreamInterface` configuration.

---

## 5. `SerializeProp` API & Implementation

### 5.1 Unified Serialization Idiom
The `SerializeProp` function family encapsulates both loading and saving behind a single call. If an asset file lacks a given property (e.g. an older asset file loaded by newer code), `SerializeProp` returns `false` during loading, and the destination variable retains its existing default value.

```cpp
#define SERIALIZE_PROP(ar, var) SerializeProp((ar), #var, #var##_crc32, (var))
```

### 5.2 Primitive Type Helpers

#### Float (`float`)
```cpp
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, float& value)
{
    Assert(keyName != nullptr);
    if (ar.IsSaving())
    {
        WritePropertyHeader(ar, keyName);
        IOPrintf(ar.Stream, "%f\n\n", value);
        return true;
    }

    auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
    if (node == nullptr)
    {
        return false;
    }

    // Fast float conversion from String slice
    char buffer[64];
    size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
    MemCopy(buffer, node->Value.Str, copySize);
    buffer[copySize] = '\0';

    char* endPtr = nullptr;
    float parsed = strtof(buffer, &endPtr);
    if (endPtr != buffer)
    {
        value = parsed;
        return true;
    }
    return false;
}
```

#### 32-Bit Signed Integer (`int32`)
```cpp
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, int32& value)
{
    Assert(keyName != nullptr);
    if (ar.IsSaving())
    {
        WritePropertyHeader(ar, keyName);
        IOPrintf(ar.Stream, "%d\n\n", value);
        return true;
    }

    auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
    if (node == nullptr)
    {
        return false;
    }

    char buffer[32];
    size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
    MemCopy(buffer, node->Value.Str, copySize);
    buffer[copySize] = '\0';

    char* endPtr = nullptr;
    int32 parsed = static_cast<int32>(strtol(buffer, &endPtr, 10));
    if (endPtr != buffer)
    {
        value = parsed;
        return true;
    }
    return false;
}
```

#### 64-Bit Unsigned Integer (`uint64`, `EntityID`)
```cpp
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, uint64& value)
{
    Assert(keyName != nullptr);
    if (ar.IsSaving())
    {
        WritePropertyHeader(ar, keyName);
        IOPrintf(ar.Stream, "0x%016llX\n\n", value);
        return true;
    }

    auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
    if (node == nullptr)
    {
        return false;
    }

    char buffer[32];
    size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
    MemCopy(buffer, node->Value.Str, copySize);
    buffer[copySize] = '\0';

    char* endPtr = nullptr;
    int base = (buffer[0] == '0' && (buffer[1] == 'x' || buffer[1] == 'X')) ? 16 : 10;
    uint64 parsed = strtoull(buffer, &endPtr, base);
    if (endPtr != buffer)
    {
        value = parsed;
        return true;
    }
    return false;
}
```

#### Boolean (`bool`)
```cpp
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, bool& value)
{
    Assert(keyName != nullptr);
    if (ar.IsSaving())
    {
        WritePropertyHeader(ar, keyName);
        IOPrintf(ar.Stream, "%s\n\n", value ? "true" : "false");
        return true;
    }

    auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
    if (node == nullptr)
    {
        return false;
    }

    if (StringCompare(node->Value, WrapString("true")) == 0 ||
        StringCompare(node->Value, WrapString("1")) == 0)
    {
        value = true;
        return true;
    }
    if (StringCompare(node->Value, WrapString("false")) == 0 ||
        StringCompare(node->Value, WrapString("0")) == 0)
    {
        value = false;
        return true;
    }
    return false;
}
```

### 5.3 Vector Helpers (`float x, y, z`)
```cpp
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, float& x, float& y, float& z)
{
    Assert(keyName != nullptr);
    if (ar.IsSaving())
    {
        WritePropertyHeader(ar, keyName);
        IOPrintf(ar.Stream, "%f %f %f\n\n", x, y, z);
        return true;
    }

    auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
    if (node == nullptr)
    {
        return false;
    }

    char buffer[128];
    size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
    MemCopy(buffer, node->Value.Str, copySize);
    buffer[copySize] = '\0';

    float parsedX = 0.0f;
    float parsedY = 0.0f;
    float parsedZ = 0.0f;
    int matches = sscanf_s(buffer, "%f %f %f", &parsedX, &parsedY, &parsedZ);
    if (matches == 3)
    {
        x = parsedX;
        y = parsedY;
        z = parsedZ;
        return true;
    }
    return false;
}
```

### 5.4 String Helpers
```cpp
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, String& value)
{
    Assert(keyName != nullptr);
    if (ar.IsSaving())
    {
        WritePropertyHeader(ar, keyName);
        bool hasSpace = ContainsChar(value, ' ');
        if (hasSpace)
        {
            IOPrintf(ar.Stream, "\"%.*s\"\n\n", static_cast<int>(value.Size), value.Str);
        }
        else
        {
            IOPrintf(ar.Stream, "%.*s\n\n", static_cast<int>(value.Size), value.Str);
        }
        return true;
    }

    auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
    if (node == nullptr)
    {
        return false;
    }

    String parsed = node->Value;
    // Strip optional surrounding double quotes
    if (parsed.Size >= 2 && parsed.Str[0] == '"' && parsed.Str[parsed.Size - 1] == '"')
    {
        parsed.Str++;
        parsed.Size -= 2;
    }

    // Allocate persistent string copy on the archive arena
    Assert(ar.ArenaInstance != nullptr);
    value = StringCopy(ar.ArenaInstance, parsed);
    return true;
}
```

---

## 6. Two-Tier Versioning Architecture

### 6.1 Architectural Rationale: Base vs Derived Decoupling
In entity-component systems or object-oriented engine hierarchies, entities consist of two distinct domains:
1. **Core Engine Identity (Base Entity)**: Position, Rotation, Scale, EntityID, Class Kind, Render Flags, Layer Masks. Managed by engine architects.
2. **Gameplay Specialization (Derived Class)**: Ammo, Health, AI State, Mesh Instance ID, Patrol Paths. Managed by gameplay programmers.

#### The Fragility of Monolithic Versioning
In naive serialization architectures, a single `uint32 Version` governs the entire file. When an engine programmer updates the base `Entity` struct (e.g. adding a `uint32 LayerMask`), bumping the global version invalidates or forces schema changes across every single derived entity type in the project.

#### The Two-Tier Solution
Juliet decouples versioning into two independent tiers:
- **Tier 1: Base Engine Version (`kEntityBaseVersion`)**: Declared centrally in `Entity.h`. Governs `Entity` base fields.
- **Tier 2: Derived Class Version (`Class::Version`)**: Declared per-entity class in `Class.h` and initialized in `DEFINE_ENTITY_VERSIONED`.

```
========================================================================
                          .jasset Text Archive
========================================================================
 ; AssetType
 Entity
 ; BaseVersion  ------> Governed by kEntityBaseVersion in Entity.h
 1
 ; EntityID
 0x0000000000000001
 ; Kind
 Inert
 ; Position
 0.0 0.0 0.0
------------------------------------------------------------------------
 ; ClassVersion ------> Governed by Class::Version in Class.h
 2
 ; MeshInstance
 42
========================================================================
```

When an engine programmer bumps `kEntityBaseVersion` from `1` to `2` to add `LayerMask`, no derived game classes (`Inert`, `Monster`, `Vehicle`) need version increments or code modifications.

### 6.2 Implementation Details

#### Engine Base Version (`Game/Entity/Entity.h`)
```cpp
constexpr uint32 kEntityBaseVersion = 1;

void SerializeEntityBase(archive& ar, NonNullPtr<Entity> entity);
```

#### Derived Class Version (`Juliet/include/Engine/Class.h`)
```cpp
struct Class
{
    uint32             CRC;
    uint8              kind;
    uint16             Version; // Added derived class version
    serialize_fct_type serialize_fct;
    size_t             size_of;
    size_t             alignment;

#if JULIET_DEBUG
    String             Name;
#endif
};

consteval Class MakeClass(String name, uint8 kind, uint16 version, size_t size, size_t align, serialize_fct_type fct)
{
    Class cls         = {};
    cls.CRC           = crc32(name.Str, name.Size);
    cls.kind          = kind;
    cls.Version       = version;
    cls.size_of       = size;
    cls.alignment     = align;
    cls.serialize_fct = fct;

#if JULIET_DEBUG
    cls.Name          = name;
#endif

    return cls;
}
```

#### Class Registration Macros (`Game/Entity/Entity.h`)
```cpp
#define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct)                                                        \
    Class entityKind##entity =                                                                                         \
        MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version), sizeof(entity), alignof(entity), serialize_fct); \
    Class* entity::Kind = &entityKind##entity;
```

### 6.3 Execution Flow in `SerializeEntity`
```cpp
void Serialize(archive& ar, NonNullPtr<Entity> entity)
{
    Assert(entity.Get() != nullptr);

    // --- Tier 1: Base Entity Serialization ---
    if (ar.IsSaving())
    {
        uint32 baseVer = kEntityBaseVersion;
        SERIALIZE_PROP(ar, baseVer);
        SERIALIZE_PROP(ar, entity->ID);

        String kindStr = WrapString(kEntity_type_names[entity->Kind->kind]);
        SerializeProp(ar, "Kind", "Kind"_crc32, kindStr);

        SerializeProp(ar, "Position", "Position"_crc32, entity->X, entity->Y, entity->Z);
    }
    else
    {
        uint32 baseVer = 0;
        if (!SerializeProp(ar, "BaseVersion", "BaseVersion"_crc32, baseVer))
        {
            baseVer = 1; // Default to initial schema if absent
        }
        ar.BaseVersion = baseVer;

        SERIALIZE_PROP(ar, entity->ID);

        String kindStr = {};
        if (SerializeProp(ar, "Kind", "Kind"_crc32, kindStr))
        {
            // Resolve class pointer from name
            for (uint8 i = 0; i < ToUnderlying(Entity_Type::Count); ++i)
            {
                if (StringCompare(kindStr, WrapString(kEntity_type_names[i])) == 0)
                {
                    entity->Kind = kEntity_type_class_ptr[i];
                    break;
                }
            }
        }
        Assert(entity->Kind != nullptr);

        SerializeProp(ar, "Position", "Position"_crc32, entity->X, entity->Y, entity->Z);
    }

    // --- Tier 2: Derived Entity Serialization ---
    if (entity->Kind->serialize_fct != nullptr && entity->Derived != nullptr)
    {
        if (ar.IsSaving())
        {
            uint32 classVer = entity->Kind->Version;
            SerializeProp(ar, "ClassVersion", "ClassVersion"_crc32, classVer);
        }
        else
        {
            uint32 classVer = 0;
            if (!SerializeProp(ar, "ClassVersion", "ClassVersion"_crc32, classVer))
            {
                classVer = entity->Kind->Version;
            }
            ar.ClassVersion = static_cast<uint16>(classVer);
        }

        entity->Kind->serialize_fct(&ar, entity->Derived);
    }
}
```

---

## 7. Post-Serialization Deprecation & Migration

### 7.1 Schema Evolution Challenge
Over the lifecycle of a game, gameplay mechanics evolve:
- A scalar float `Speed` is replaced by a directional 2D vector `Velocity`.
- A single texture index `TextureID` is replaced by an asset path string `DiffuseTexture`.
- Obsolete properties are deleted entirely.

Retaining deprecated members in active C++ structs creates code clutter, wastes memory, and invites bugs.

### 7.2 Deprecation Primitives (`ReadDeprecated*`)
Deprecation primitives allow serializers to ingest obsolete properties exclusively during loading without polluting modern structs or writing deprecated keys back to disk during saving.

```cpp
bool ReadDeprecated(archive& ar, const char* keyName, uint32 keyCRC, float& outVal)
{
    Assert(keyName != nullptr);
    if (ar.IsSaving())
    {
        return false; // Deprecated fields are never saved
    }

    auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
    if (node == nullptr)
    {
        return false;
    }

    char buffer[64];
    size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
    MemCopy(buffer, node->Value.Str, copySize);
    buffer[copySize] = '\0';

    char* endPtr = nullptr;
    float parsed = strtof(buffer, &endPtr);
    if (endPtr != buffer)
    {
        outVal = parsed;
        return true;
    }
    return false;
}

bool ReadDeprecatedVec2(archive& ar, const char* keyName, uint32 keyCRC, float& outX, float& outY)
{
    Assert(keyName != nullptr);
    if (ar.IsSaving())
    {
        return false;
    }

    auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
    if (node == nullptr)
    {
        return false;
    }

    char buffer[128];
    size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
    MemCopy(buffer, node->Value.Str, copySize);
    buffer[copySize] = '\0';

    float x = 0.0f;
    float y = 0.0f;
    if (sscanf_s(buffer, "%f %f", &x, &y) == 2)
    {
        outX = x;
        outY = y;
        return true;
    }
    return false;
}

bool ReadDeprecatedString(archive& ar, const char* keyName, uint32 keyCRC, NonNullPtr<Arena> arena, String& outVal)
{
    Assert(keyName != nullptr);
    if (ar.IsSaving())
    {
        return false;
    }

    auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
    if (node == nullptr)
    {
        return false;
    }

    String parsed = node->Value;
    if (parsed.Size >= 2 && parsed.Str[0] == '"' && parsed.Str[parsed.Size - 1] == '"')
    {
        parsed.Str++;
        parsed.Size -= 2;
    }
    outVal = StringCopy(arena, parsed);
    return true;
}
```

### 7.3 In-Place Migration Pattern
When an asset file with an older `ClassVersion` is loaded, the derived serializer detects `ar.ClassVersion < N`, calls `ReadDeprecated*` to read obsolete fields, maps the legacy data into the modern struct, and continues. On the subsequent save, the asset file is emitted using the modern schema without deprecated keys.

```cpp
struct Projectile
{
    DECLARE_ENTITY()

    // Modern Schema (v2)
    float VelocityX = 0.0f;
    float VelocityY = 0.0f;
    float Damage    = 50.0f;
};

void SerializeProjectile(archive* arPtr, void* payload)
{
    Assert(arPtr != nullptr);
    Assert(payload != nullptr);
    auto& ar         = *arPtr;
    auto* projectile = static_cast<Projectile*>(payload);

    if (ar.IsSaving())
    {
        SERIALIZE_PROP(ar, projectile->VelocityX);
        SERIALIZE_PROP(ar, projectile->VelocityY);
        SERIALIZE_PROP(ar, projectile->Damage);
    }
    else
    {
        SERIALIZE_PROP(ar, projectile->Damage);

        if (ar.ClassVersion < 2)
        {
            // Migration from v1: scalar 'Speed' converted to 'VelocityX'
            float legacySpeed = 0.0f;
            if (ReadDeprecated(ar, "Speed", "Speed"_crc32, legacySpeed))
            {
                projectile->VelocityX = legacySpeed;
                projectile->VelocityY = 0.0f;
            }
        }
        else
        {
            SERIALIZE_PROP(ar, projectile->VelocityX);
            SERIALIZE_PROP(ar, projectile->VelocityY);
        }
    }
}
```

---

## 8. Step-by-Step Implementation Roadmap & Unit Testing Plan

### 8.1 Implementation Roadmap

```
+-------------------------------------------------------------------------+
| Phase 1: Core Utilities & Dual-Mode CRC32                               |
|   - Update CRC32.h with constexpr Crc32(String) and Crc32(char*, len)   |
|   - Extend struct Class in Class.h with uint16 Version                  |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| Phase 2: In-Memory Zero-Copy Parser                                     |
|   - Implement TextArchiveParser.h / .cpp                                |
|   - TokenizeTextArchive, TrimWhitespace, FindProperty                   |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| Phase 3: Archive Struct & SerializeProp Helpers                         |
|   - Upgrade archive in serialization.h with ArchiveMode                 |
|   - Implement primitive, vector, and string SerializeProp helpers       |
|   - Implement ReadDeprecated* primitives                                |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| Phase 4: Entity & World Integration                                     |
|   - Update Entity.h / Entity.cpp with two-tier serialization            |
|   - Refactor World.cpp to load/save .jasset files                       |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| Phase 5: Verification & Unit Testing Suite                              |
|   - Build and run SerializationUnitTest.cpp                             |
+-------------------------------------------------------------------------+
```

#### Phase 1: Core Utilities & Dual-Mode CRC32
1. **Target**: `Juliet/include/Core/Common/CRC32.h`
   - Add `constexpr uint32 Crc32(const char* str, size_t length)` and `constexpr uint32 Crc32(String str)`.
   - Ensure existing `operator""_crc32` calls `Crc32`.
2. **Target**: `Juliet/include/Engine/Class.h`
   - Add `uint16 Version` to `struct Class`.
   - Update `MakeClass` to accept `uint16 version = 1`.

#### Phase 2: In-Memory Zero-Copy Parser
1. **Target**: `Juliet/include/Core/Common/TextArchiveParser.h` (and `src/Core/Common/TextArchiveParser.cpp`)
   - Define `struct TextPropertyNode { uint32 KeyCRC; String Value; bool Consumed; }`.
   - Implement `TokenizeTextArchive(NonNullPtr<Arena> arena, ByteBuffer buffer)`.
   - Implement `FindProperty` and `AuditUnconsumedProperties`.

#### Phase 3: Archive Struct & `SerializeProp` Helpers
1. **Target**: `Juliet/include/Core/Common/serialization.h`
   - Introduce `enum class ArchiveMode : uint8`.
   - Upgrade `struct archive` with stream pointer, property array, mode, and versions.
   - Implement overloaded `SerializeProp` for `float`, `int32`, `uint64`, `bool`, vectors, and `String`.
   - Implement `ReadDeprecated`, `ReadDeprecatedVec2`, `ReadDeprecatedString`.

#### Phase 4: Entity & World Integration
1. **Target**: `Game/Entity/Entity.h` and `Game/Entity/Entity.cpp`
   - Define `constexpr uint32 kEntityBaseVersion = 1;`.
   - Update `DEFINE_ENTITY` and add `DEFINE_ENTITY_VERSIONED`.
   - Implement `Serialize(archive& ar, NonNullPtr<Entity> entity)` supporting `.jasset` text format.
2. **Target**: `Game/Data/World.h` and `Game/Data/World.cpp`
   - Implement text-based world saving and loading using `.jasset` formatting.

#### Phase 5: Comprehensive Unit Testing
1. **Target**: `Game/UnitTest/SerializationUnitTest.h` and `Game/UnitTest/SerializationUnitTest.cpp`
   - Add unit tests verifying parsing, round-trip serialization, defaults preservation, versioning, and deprecation.

---

### 8.2 Comprehensive Unit Testing Plan (`SerializationUnitTest.cpp`)

The unit test suite validates all architectural requirements without modifying engine framework code for test-specific cases.

```cpp
// Game/UnitTest/SerializationUnitTest.h
#pragma once

#include <Juliet.h>

#if JULIET_DEBUG
namespace UnitTest
{
    void RunSerializationUnitTests();
}
#endif
```

```cpp
// Game/UnitTest/SerializationUnitTest.cpp
#include <UnitTest/SerializationUnitTest.h>

#if JULIET_DEBUG

#include <Core/Common/CRC32.h>
#include <Core/Common/String.h>
#include <Core/Common/serialization.h>
#include <Core/HAL/IO/IOStream.h>
#include <Core/Logging/LogManager.h>
#include <Core/Logging/LogTypes.h>
#include <Core/Memory/MemoryArena.h>
#include <Core/Thread/ThreadContext.h>
#include <Entity/Entity.h>

namespace UnitTest
{
    // Test Struct for Derived Entity Testing
    struct DummyVehicle
    {
        DECLARE_ENTITY()

        float  MaxSpeed    = 120.0f;
        int32  GearCount   = 6;
        uint64 ChassisUUID = 0xABCDEF0123456789ULL;
        bool   Turbo       = true;
        float  PosX        = 10.0f;
        float  PosY        = 20.0f;
        float  PosZ        = 30.0f;
    };

    void SerializeDummyVehicle(archive* arPtr, void* payload)
    {
        Assert(arPtr != nullptr);
        Assert(payload != nullptr);
        auto& ar      = *arPtr;
        auto* vehicle = static_cast<DummyVehicle*>(payload);

        SERIALIZE_PROP(ar, vehicle->MaxSpeed);
        SERIALIZE_PROP(ar, vehicle->GearCount);
        SERIALIZE_PROP(ar, vehicle->ChassisUUID);
        SERIALIZE_PROP(ar, vehicle->Turbo);
        SerializeProp(ar, "Position", "Position"_crc32, vehicle->PosX, vehicle->PosY, vehicle->PosZ);
    }

    DEFINE_ENTITY_VERSIONED(DummyVehicle, 1, SerializeDummyVehicle);

    // Test 1: Parser tokenization with whitespace, comments, and mixed line endings
    static void TestParserTokenization()
    {
        TempArena temp = scratch_begin(nullptr, 0);

        const char* testContent =
            "# Header Comment\r\n"
            "// Secondary comment\n"
            "\n"
            "; Health\r\n"
            "   100.500000   \r\n"
            "\n"
            "; Name\n"
            "\"Paladin Hero\"\n"
            "\n"
            "; Position\r\n"
            "1.0 2.0 3.0\r\n";

        ByteBuffer buffer = {
            .Data = reinterpret_cast<Byte*>(const_cast<char*>(testContent)),
            .Size = strlen(testContent)
        };

        ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer);
        Assert(parsed.PropertyCount == 3);

        auto* healthNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Health"_crc32);
        Assert(healthNode != nullptr);
        Assert(StringCompare(healthNode->Value, WrapString("100.500000")) == 0);

        auto* nameNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Name"_crc32);
        Assert(nameNode != nullptr);
        Assert(StringCompare(nameNode->Value, WrapString("\"Paladin Hero\"")) == 0);

        auto* posNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Position"_crc32);
        Assert(posNode != nullptr);
        Assert(StringCompare(posNode->Value, WrapString("1.0 2.0 3.0")) == 0);

        scratch_end(temp);
        LogMessage(LogCategory::Core, "TestParserTokenization passed.");
    }

    // Test 2: Missing properties retain default struct values
    static void TestDefaultValueRetention()
    {
        TempArena temp = scratch_begin(nullptr, 0);

        const char* incompleteContent =
            "; MaxSpeed\n"
            "180.0\n";

        ByteBuffer buffer = {
            .Data = reinterpret_cast<Byte*>(const_cast<char*>(incompleteContent)),
            .Size = strlen(incompleteContent)
        };

        ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer);

        archive ar = {};
        ar.ArenaInstance = temp.Arena;
        ar.Mode          = ArchiveMode::LoadingText;
        ar.Properties    = parsed.Nodes;
        ar.PropertyCount = parsed.PropertyCount;

        DummyVehicle vehicle;
        // Defaults: MaxSpeed=120, GearCount=6, Turbo=true
        SerializeDummyVehicle(&ar, &vehicle);

        Assert(vehicle.MaxSpeed == 180.0f); // Overwritten by archive
        Assert(vehicle.GearCount == 6);      // Preserved default
        Assert(vehicle.Turbo == true);       // Preserved default
        Assert(vehicle.PosX == 10.0f);       // Preserved default

        scratch_end(temp);
        LogMessage(LogCategory::Core, "TestDefaultValueRetention passed.");
    }

    // Test 3: Deprecation migration from v1 to v2
    struct LegacyWeapon
    {
        DECLARE_ENTITY()
        float VelocityX = 0.0f;
        float VelocityY = 0.0f;
    };

    void SerializeLegacyWeapon(archive* arPtr, void* payload)
    {
        Assert(arPtr != nullptr);
        Assert(payload != nullptr);
        auto& ar     = *arPtr;
        auto* weapon = static_cast<LegacyWeapon*>(payload);

        if (ar.IsSaving())
        {
            SERIALIZE_PROP(ar, weapon->VelocityX);
            SERIALIZE_PROP(ar, weapon->VelocityY);
        }
        else
        {
            if (ar.ClassVersion < 2)
            {
                float oldSpeed = 0.0f;
                if (ReadDeprecated(ar, "Speed", "Speed"_crc32, oldSpeed))
                {
                    weapon->VelocityX = oldSpeed;
                    weapon->VelocityY = 0.0f;
                }
            }
            else
            {
                SERIALIZE_PROP(ar, weapon->VelocityX);
                SERIALIZE_PROP(ar, weapon->VelocityY);
            }
        }
    }

    DEFINE_ENTITY_VERSIONED(LegacyWeapon, 2, SerializeLegacyWeapon);

    static void TestDeprecationMigration()
    {
        TempArena temp = scratch_begin(nullptr, 0);

        // Simulated v1 file containing obsolete 'Speed'
        const char* v1Content =
            "; ClassVersion\n"
            "1\n"
            "; Speed\n"
            "75.500000\n";

        ByteBuffer buffer = {
            .Data = reinterpret_cast<Byte*>(const_cast<char*>(v1Content)),
            .Size = strlen(v1Content)
        };

        ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer);

        archive ar = {};
        ar.ArenaInstance = temp.Arena;
        ar.Mode          = ArchiveMode::LoadingText;
        ar.Properties    = parsed.Nodes;
        ar.PropertyCount = parsed.PropertyCount;
        ar.ClassVersion  = 1;

        LegacyWeapon weapon;
        SerializeLegacyWeapon(&ar, &weapon);

        Assert(weapon.VelocityX == 75.5f);
        Assert(weapon.VelocityY == 0.0f);

        scratch_end(temp);
        LogMessage(LogCategory::Core, "TestDeprecationMigration passed.");
    }

    void RunSerializationUnitTests()
    {
        LogMessage(LogCategory::Core, "=== Running Serialization & .jasset Unit Tests ===");
        TestParserTokenization();
        TestDefaultValueRetention();
        TestDeprecationMigration();
        LogMessage(LogCategory::Core, "=== All Serialization Unit Tests Passed Successfully ===");
    }
} // namespace UnitTest

#endif