Juliet/Game/Plans/01_Serialization_And_Text_Archive.md

# Juliet Engine: Serialization & Text Archive Architecture

## Technical Specification & Architectural Design

---

## 1. Executive Summary & Architecture Goals

### 1.1 Context & Motivation
Juliet historically relied on monolithic, packed binary blobs for world and entity persistence. While fast to read as raw byte offsets, binary serialization suffers from three major flaws:
1. **Merge Incompatibility**: Binary assets cannot be merged or diffed in version control systems (Git / Perforce), causing unresolvable binary conflicts and data loss.
2. **Schema Rigidity**: Adding, removing, or reordering a single struct field invalidates all existing binary files unless complex manual byte-offset mapping tables are maintained.
3. **Opacity**: Designers and engineers cannot inspect, debug, or patch asset properties in a standard text editor.

### 1.2 Architectural Goals
The `.jasset` text archive framework replaces legacy binary blobs with a human-readable, diff-friendly property serialization pipeline adhering to Juliet's systems programming principles:
- **Zero Dynamic Heap Allocations**: Parsing, tokenization, formatting, and buffer transformations execute entirely within Juliet memory arenas (`Arena`, `TempArena`, `scratch_begin` / `scratch_end`). Standard library containers (`std::string`, `std::vector`) and raw heap allocators (`malloc`, `new`) are forbidden.
- **Zero-Copy In-Memory Tokenization**: Files are loaded into arena memory once via `LoadFile`. The tokenizer parses properties into lightweight slices represented by Juliet's `String` (`char* Str; size_t Size;`), referencing existing file buffer bytes without string duplication.
- **$O(1)$ Property Lookup via Compile-Time & Runtime CRC32**: Property keys in text files are hashed once during tokenization into 32-bit CRC values. In code, lookups utilize compile-time hashed literals (`operator""_crc32`). Property resolution is a single 32-bit integer comparison.
- **Symmetric Single-Function Serialization**: A single `serialize` implementation per entity or data structure handles both Save and Load paths, guaranteeing read and write schemas never diverge.
- **Graceful Forward/Backward Compatibility**: Missing keys during load automatically preserve struct default values. Unknown keys present in newer asset files are safely ignored without parse failure.
- **Two-Tier Decoupled Versioning via Generalized `Class`**: Core engine entity properties (`; version`) and derived gameplay class properties (`; class_version`) are versioned independently through their respective `Class` descriptors.
- **In-Place Schema Migration**: Deprecated fields no longer present in C++ structs are read into temporary stack variables during load using standard `SERIALIZE` calls guarded by `if (ar.loading && version < N)`, seamlessly converting legacy data without struct pollution or persisting obsolete keys on subsequent saves.
- **Clean Warning-Free C++**: Fully conforming to Juliet coding guidelines: strict assertions (`Assert`), `[[nodiscard]]`, `auto*`/`auto&`, mandatory braces, and `static_cast`/`reinterpret_cast`.

---

## 2. The `.jasset` Text Format Specification

### 2.1 Grammar & Structural Rules
The `.jasset` format uses a line-oriented, key-value property hierarchy designed for visual clarity and clean Git diffs.

```ebnf
AssetFile             ::= { CommentLine | EmptyLine | PropertyDeclaration } ;
CommentLine           ::= ( "#" | "//" ) { Character } LineEnding ;
EmptyLine             ::= { Whitespace } LineEnding ;
PropertyDeclaration   ::= KeyHeader LineEnding ValueBlock ;
KeyHeader             ::= ";" { Whitespace } Identifier ;
ValueBlock            ::= { ValueLine LineEnding } ;
ValueLine             ::= { Whitespace } ValueString { Whitespace } ;
LineEnding            ::= "\r\n" | "\n" ;
Identifier            ::= [a-zA-Z_][a-zA-Z0-9_]* ;
```

#### Formatting Rules:
1. **Key Declarations**: A property begins with a semicolon `;` followed by optional whitespace and a case-sensitive identifier (e.g. `; position`).
2. **Value Blocks**: The line(s) immediately following a key header contain its value payload.
3. **Comments**: Any line whose first non-whitespace character is `#` or `//` is treated as a comment and ignored. Inline comments on property lines are forbidden.
4. **Whitespace**: Leading and trailing spaces or tabs on both keys and values are stripped during tokenization.
5. **Line Endings**: Both Windows CRLF (`\r\n`) and Linux LF (`\n`) are transparently accepted.

### 2.2 Formatting Specifications

#### Scalar Types
Scalars are formatted as decimal representations on a single line:
```
; max_speed
180.500000

; gear_count
6

; turbo
true
```
- Floats: Output via `%.9g` or `%f`.
- Integers: Signed (`%d`, `%lld`) and unsigned (`%u`, `%llu`).
- Booleans: Case-insensitive `true` / `false` or `1` / `0`.

#### Vector Types (`Vector4`)
Multi-component vectors are space-delimited on a single value line:
```
; position
10.0 20.0 30.0 1.0
```

#### String Types (`String`)
Strings containing spaces are wrapped in double quotes `"..."`. Quotes are automatically stripped upon loading and emitted during saving when spaces are present:
```
; name
"Paladin Hero"
```

#### Asset File Example (`Entity_01.jasset`)
```
# Juliet Entity Asset File
; version
1

; id
1001

; position
12.500000 0.000000 45.200000 1.000000

; class
Inert

; class_version
1

; mesh_instance
42
```

---

## 3. Zero-Copy Tokenization & Fast Property Lookup

### 3.1 Data Structures (`Core/Common/serialization.h`)

To eliminate heap fragmentation, the parser loads the entire `.jasset` file into contiguous arena memory and parses it into a flat array of lightweight slices.

```cpp
struct ArchivePropertyNode
{
    String key;
    String value;
    uint32 key_crc;
    bool   consumed;
};

struct ParsedArchive
{
    ArchivePropertyNode* nodes          = nullptr;
    uint32               property_count = 0;
};
```

- `key`: Sliced `String` referencing the key name.
- `value`: Sliced `String` directly referencing file buffer bytes (zero-copy).
- `key_crc`: 32-bit CRC hash computed once during tokenization.
- `consumed`: Initialized to `false`. Set to `true` whenever queried by `find_property`.

### 3.2 Dual-Mode CRC32 (`Core/Common/CRC32.h`)
Lookups rely on compile-time string hashing via `constexpr` / `consteval`:
```cpp
[[nodiscard]] constexpr uint32 crc32(const char* str, size_t length);
[[nodiscard]] constexpr uint32 crc32(String str);
[[nodiscard]] consteval uint32 operator""_crc32(const char* str, size_t length);
```

### 3.3 Tokenization API (`tokenize_archive`)
```cpp
JULIET_API ParsedArchive tokenize_archive(NonNullPtr<Arena> arena, ByteBuffer file_buffer);
```
**Algorithm Invariants**:
1. **Pass 1 (Count)**: Scans the buffer to count `;` key headers at line starts, allocating the exact node array in `arena`.
2. **Pass 2 (Extract)**: Slices key and value `String`s, trims whitespace, computes `key_crc = crc32(key)`, and populates nodes. Skips `#` and `//` comments.

### 3.4 Property Lookup & Audit API
```cpp
JULIET_API ArchivePropertyNode* find_property(NonNullPtr<ParsedArchive> archive, uint32 property_crc);

#if JULIET_DEBUG
JULIET_API void audit_unconsumed_properties(NonNullPtr<ParsedArchive> archive, String context_name);
#endif
```
- `find_property`: Performs an $O(1)$ integer comparison against `key_crc`. When found, marks `node->consumed = true`.
- `audit_unconsumed_properties`: Iterates through all nodes in debug builds and logs warnings for any property with `consumed == false`, catching typos or abandoned schema fields.

---

## 4. The `Archive` Context Struct & Streaming I/O

### 4.1 Struct Definition (`Core/Common/serialization.h`)
The `Archive` struct unifies loading and saving state into a single decoupled context:

```cpp
struct Archive
{
    Arena*        arena;
    bool          loading;
    ParsedArchive base   = {};
    IOStream*     stream = nullptr;

    // Legacy binary support fields (to be deprecated)
    void*   base_ptr = nullptr;
    index_t offset   = 0;
};
```

- When `loading == true`: Reads properties from `base.nodes` via `find_property`.
- When `loading == false`: Writes formatted key-value pairs directly to `stream`.

### 4.2 Property Header Formatting
```cpp
JULIET_API void write_property_header(Archive& archive, String property_name);
```
Emits `; <property_name>\n` directly to `ar.stream` with zero intermediate heap buffers.

---

## 5. Property Serialization API & Helpers

### 5.1 Unified Serialization Idiom
All property serialization uses a single template function:

```cpp
template <typename Type>
bool serialize(Archive& ar, String property_name, uint32 property_crc, Type& value)
{
    Assert(IsValid(property_name));

    bool result = false;
    if (ar.loading)
    {
        if (auto* prop = find_property(&ar.base, property_crc))
        {
            if (read_prop(ar, prop->value, value))
            {
                result = true;
            }
        }
    }
    else
    {
        write_property_header(ar, property_name);
        write(ar.stream, value);
        result = true;
    }

    return result;
}
```

### 5.2 Convenience Macros
```cpp
#define SERIALIZE(ar, name, var) serialize((ar), ConstString(#name), #name##_crc32, (var))
#define SERIALIZE_SIMPLE(ar, var) SERIALIZE(ar, var, var)
```
- `SERIALIZE(ar, id, entity->ID)`: Serializes property named `"id"` with `"id"_crc32`.
- `SERIALIZE_SIMPLE(ar, position)`: Uses variable identifier as property name.

### 5.3 Supported Type Conversions
Conversion between text and memory is handled by overloaded `read` and `write` primitives:

| C++ Type | Text Format | Conversion Primitive |
| :--- | :--- | :--- |
| `float` | `180.500000` | `strtof` / `IOPrintf("%.9g")` |
| `int8`, `int16`, `int32`, `int64` | `42` / `-100` | `strtol`, `strtoll` / `IOPrintf("%d")` |
| `uint8`, `uint16`, `uint32`, `uint64` | `1001` / `0x...` | `strtoul`, `strtoull` / `IOPrintf("%u")` |
| `bool` | `true` / `false` | `true/false/1/0` string compare / `IOPrintf` |
| `Vector4` | `10.0 20.0 30.0 1.0` | Space-delimited float parse / `IOPrintf` |
| `String` | `"Paladin Hero"` | Arena-allocated copy, quote strip / `IOPrintf` |

---

## 6. Two-Tier Versioning & Generalized `Class` Architecture

### 6.1 Architectural Principle
To prevent monolithic engine updates from forcing all gameplay assets to re-version, schema versions are decoupled into two tiers:
1. **Base Version (`; version`)**: Managed by root classes (e.g. `Entity::kind->version`). Governs core engine properties (`id`, `position`).
2. **Derived Version (`; class_version`)**: Managed by derived classes (e.g. `Inert::kind->version`). Governs gameplay-specific component properties.

### 6.2 The `Class` Descriptor (`Engine/Class.h`)
Every serializable entity or component is described by an immutable `Class` instance:

```cpp
using serialize_fct_type    = void (*)(Archive& ar, uint16 version, void* payload);
using default_init_fct_type = void (*)(void* payload);

struct Class
{
    uint32                CRC;
    uint8                 kind;
    uint16                version;
    const Class*          base_class;
    serialize_fct_type    serialize_fct;
    default_init_fct_type default_init_fct;
    size_t                size_of;
    size_t                alignment;

#if JULIET_DEBUG
    String Name;
#endif
};
```

### 6.3 Class Registration Macros
```cpp
#define DECLARE_CLASS() \
    static Class* kind;

#define DEFINE_CLASS_VERSIONED(cls, version, base_class, serialize_fct)                                                \
    inline void default_init_##cls(void* payload) { *static_cast<cls*>(payload) = cls{}; }                            \
    constexpr Class classKind##cls =                                                                                   \
        MakeClass(ConstString(#cls), 0, (version), (base_class), sizeof(cls), alignof(cls),                           \
                  (serialize_fct), default_init_##cls);                                                                \
    Class* cls::kind = const_cast<Class*>(&classKind##cls);
```

For derived entity types, `DECLARE_ENTITY()` and `DEFINE_ENTITY_VERSIONED` compose cleanly:
```cpp
#define DECLARE_ENTITY() \
    Entity* base;        \
    DECLARE_CLASS()

#define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct)                                                         \
    inline void default_init_##entity(void* payload) { *static_cast<entity*>(payload) = entity{}; }                    \
    constexpr Class entityKind##entity = MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version),         \
                                                   &classKindEntity, sizeof(entity), alignof(entity),                   \
                                                   (serialize_fct), default_init_##entity);                             \
    Class*          entity::kind       = const_cast<Class*>(&entityKind##entity);
```

### 6.4 Universal Class Serializer (`Engine/class.cpp`)
```cpp
void serialize(Archive& ar, NonNullPtr<Class> cls, void* instance)
{
    Assert(instance != nullptr);

    uint16 version = cls->version;
    if (cls->base_class)
    {
        serialize(ar, ConstString("class_version"), "class_version"_crc32, version);
    }
    else
    {
        serialize(ar, ConstString("version"), "version"_crc32, version);
    }

    if (cls->serialize_fct)
    {
        cls->serialize_fct(ar, version, instance);
    }
}
```

### 6.5 Runtime Type Queries (`IsA`)
Polymorphic type safety is resolved without virtual tables or RTTI:
```cpp
bool IsA(const Class& query, const Class* target);

template <typename TargetType>
bool IsA(const Class& cls)
{
    return IsA(cls, TargetType::kind);
}
```

### 6.6 Entity Serialization Composition
An entity instance composes base `Entity` properties and derived component properties:
```cpp
void serialize_entity(Archive& ar, uint16 /*version*/, void* payload)
{
    Assert(payload != nullptr);
    auto* entity = static_cast<Entity*>(payload);

    SERIALIZE(ar, id, entity->ID);
    SERIALIZE(ar, position, entity->position);
}

DEFINE_CLASS_VERSIONED(Entity, 1, nullptr, serialize_entity)

void serialize(Archive& ar, NonNullPtr<Entity> entity)
{
    // 1. Serialize base Entity properties (reads/writes '; version')
    serialize(ar, Entity::kind, entity.Get());

    // 2. Serialize derived component properties (reads/writes '; class_version')
    if (entity->derived_kind != nullptr && entity->derived != nullptr)
    {
        serialize(ar, entity->derived_kind, entity->derived);
    }
}
```

---

## 7. In-Place Schema Migration & Deprecation

### 7.1 Deprecation Principle
When gameplay code evolves, obsolete member variables are deleted from active C++ structs to avoid memory waste and code clutter. Obsolete properties are migrated exclusively during loading using temporary local stack variables.

### 7.2 The Stack-Allocated Migration Idiom
In the class's `serialize_fct(Archive& ar, uint16 version, void* payload)`:
1. When `ar.loading == true` and `version < N`:
   - Declare a temporary variable on the stack matching the legacy type.
   - Call `SERIALIZE(ar, old_field_name, deprecated_var)`.
   - If present, transform the legacy data into the modern struct field(s).
2. When saving (`ar.loading == false`):
   - The migration block is skipped. Only modern struct properties are written.
   - On the next save, obsolete keys are automatically purged from disk.

```cpp
void serialize_projectile(Archive& ar, uint16 version, void* payload)
{
    Assert(payload != nullptr);
    auto* projectile = static_cast<Projectile*>(payload);

    SERIALIZE(ar, damage, projectile->damage);

    if (ar.loading && version < 2)
    {
        // Migrating v1 scalar 'speed' into modern Vector4 'velocity'
        float deprecated_speed = 0.0f;
        if (SERIALIZE(ar, speed, deprecated_speed))
        {
            projectile->velocity = Vector4{ deprecated_speed, 0.0f, 0.0f, 0.0f };
        }
    }
    else
    {
        SERIALIZE(ar, velocity, projectile->velocity);
    }
}
```

---

## 8. Verification & Unit Testing Framework

### 8.1 Engine-Level Test Runner (`Juliet/src/UnitTest/`)
Unit testing lives inside the Juliet engine layer (`Juliet/src/UnitTest/serialization_test.cpp`) and executes during engine startup in debug builds via `UnitTest::RunUnitTests()` in `RunUnitTests.cpp`.

### 8.2 Test Coverage Matrix

| Test Function | Target Feature | Validation Criteria |
| :--- | :--- | :--- |
| `test_parser_tokenization` | Zero-copy text parser | Validates handling of `#` and `//` comments, whitespace trimming, mixed CRLF/LF, and fast CRC property lookups. |
| `test_default_value_retention` | Partial schema loading | Validates that missing properties in partial files preserve existing struct default values without corruption. |
| `test_deprecation_migration` | Stack-based schema migration | Loads a v1 asset containing obsolete `; speed`, verifies `version = 1` is received, and validates migration into modern fields. |
| `test_class_inheritance` | Runtime `IsA` queries | Validates polymorphic inheritance checks across base and derived `Class` instances. |
| `test_string_and_vector4` | Primitives & text quoting | Verifies parsing and quote handling of `String` and multi-component `Vector4`. |

---

## 9. Deliverables & File Summary

| File | Responsibilities |
| :--- | :--- |
| [`Juliet/include/Core/Common/serialization.h`](file:///w:/Classified/Juliet/Juliet/include/Core/Common/serialization.h) | `ArchivePropertyNode`, `ParsedArchive`, `Archive` struct, `serialize` template, and `SERIALIZE` macros. |
| [`Juliet/src/Core/Common/serialization.cpp`](file:///w:/Classified/Juliet/Juliet/src/Core/Common/serialization.cpp) | `tokenize_archive`, `find_property`, `audit_unconsumed_properties`, `read_prop`, `read`, and `write` primitives. |
| [`Juliet/include/Engine/Class.h`](file:///w:/Classified/Juliet/Juliet/include/Engine/Class.h) | `Class` struct, `MakeClass`, `DECLARE_CLASS()`, `DEFINE_CLASS_VERSIONED`, and `IsA` declarations. |
| [`Juliet/src/Engine/class.cpp`](file:///w:/Classified/Juliet/Juliet/src/Engine/class.cpp) | Universal `serialize(Archive&, NonNullPtr<Class>, void*)` and runtime `IsA` traversal. |
| [`Juliet/src/UnitTest/serialization_test.cpp`](file:///w:/Classified/Juliet/Juliet/src/UnitTest/serialization_test.cpp) | Exhaustive unit tests for tokenization, defaults retention, version migration, and type queries. |
| [`Game/Entity/Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h) | `DECLARE_ENTITY()`, `DEFINE_ENTITY_VERSIONED`, and `Entity` struct definition. |
| [`Game/Entity/Entity.cpp`](file:///w:/Classified/Juliet/Game/Entity/Entity.cpp) | `serialize_entity` registration and two-tier `serialize(Archive&, NonNullPtr<Entity>)` composition. |