# Juliet Game Engine: Serialization Core & .jasset Text Archive ## Technical Specification & Implementation Plan **Document ID**: SPEC-001-SERIALIZATION-CORE **Component**: Juliet Engine Core / Asset Pipeline **Target Architecture**: Juliet Game Engine (C++20, x64, D3D12) **File Location**: `Game/Plans/01_Serialization_And_Text_Archive.md` --- ## 1. Executive Summary & Architecture Goals ### 1.1 Context & Current Limitations The Juliet game engine previously utilized packed binary records for level and entity serialization (`Assets/world.bin`, `WorldFileHeader`, `WorldEntityDiskRecord`). While packed binary formats are compact, they present severe architectural roadblocks during collaborative development: 1. **Merge Incompatibility**: Binary assets cannot be merged or diffed across version control systems (Git / Perforce). Concurrent edits by multiple designers or engineers result in unresolvable binary conflicts and data loss. 2. **Schema Rigidity**: Adding, removing, or reordering a single struct field invalidates all previously serialized binary blobs unless complex, manual byte-offset mapping tables are maintained. 3. **Opacity**: Programmers and technical artists cannot inspect, debug, or patch asset properties in a standard text editor. ### 1.2 Architectural Goals The `.jasset` text archive framework is engineered to replace legacy binary blobs with a robust, human-readable, diff-friendly property serialization pipeline while strictly adhering to Juliet's performance and memory constraints: - **Zero Dynamic Heap Allocations**: All parsing, tokenization, formatting, and buffer transformations execute entirely within Juliet memory arenas (`Arena`, `TempArena`, `scratch_begin` / `scratch_end`). Standard library containers (`std::string`, `std::vector`, `std::map`) and raw heap allocators (`malloc`, `new`) are forbidden. - **Zero-Copy In-Memory Tokenization**: Files are loaded into arena memory once via `LoadFile`. The tokenizer parses properties into lightweight slices represented by Juliet's `String` (`char* Str; size_t Size;`), referencing existing file buffer bytes without string replication or null-termination overhead. - **$O(1)$ Property Lookup via Compile-Time & Runtime CRC32**: Property keys in text files are hashed once during tokenization into 32-bit CRC values. In code, lookups utilize compile-time hashed literals (`operator""_crc32`). Property resolution reduces to a single 32-bit integer comparison, avoiding string comparison overhead in serialization loops. - **Symmetric Single-Function Serialization**: A single `Serialize` implementation per entity or data structure handles both Save and Load paths, guaranteeing that write and read schemas never diverge. - **Graceful Forward/Backward Compatibility**: Missing keys during load automatically preserve struct default values. Unknown keys present in newer asset files are safely ignored without parse failure. - **Two-Tier Decoupled Versioning**: Core engine entity properties (`kEntityBaseVersion`) and derived gameplay class properties (`Class::Version`) are versioned independently. Engine-level updates never bump derived entity class versions. - **In-Place Schema Migration**: Deprecated properties no longer present in modern structs can be read during loading into local stack variables using standard serialization (`serialize` / `SERIALIZE`) guarded by version checks (`if (ar.loading && ar.class_version < N)`), seamlessly transforming legacy values without struct pollution or persisting obsolete keys on subsequent saves. - **Zero Exceptions & Total Warning Cleanliness**: Fully conforming to Juliet coding guidelines: strict assertions (`Assert`), `[[nodiscard]]`, `auto*`/`auto&`, mandatory braces, and `static_cast`/`reinterpret_cast`. --- ## 2. The `.jasset` Text Format Specification ### 2.1 Grammar & Structural Rules The `.jasset` format uses a line-oriented, key-value property hierarchy designed for visual clarity and clean Git diffs. ```ebnf AssetFile ::= { CommentLine | EmptyLine | PropertyDeclaration } ; CommentLine ::= ( "#" | "//" ) { Character } LineEnding ; EmptyLine ::= { Whitespace } LineEnding ; PropertyDeclaration ::= KeyHeader LineEnding ValueBlock ; KeyHeader ::= ";" { Whitespace } Identifier ; ValueBlock ::= { ValueLine LineEnding } ; ValueLine ::= { Whitespace } ValueString { Whitespace } ; LineEnding ::= "\r\n" | "\n" ; Identifier ::= [a-zA-Z_][a-zA-Z0-9_]* ; ``` 1. **Property Key Header (`;\n`)**: - Every property declaration begins with a semicolon `;` as the first non-whitespace character. - The semicolon is followed by optional whitespace and a CamelCase identifier: `; PropertyName`. - Keys are case-sensitive and must be valid C++ identifier tokens. 2. **Value Block**: - The line immediately following the property header contains the property's serialized payload. - For multi-line values or arrays, lines continue until the next property header `;` or end-of-file. 3. **Comments & Ignored Tokens**: - Any line starting with `#` or `//` (after optional leading whitespace) is treated as a comment. - Empty lines and lines containing only whitespace are ignored. 4. **Line Termination Handling**: - The parser natively accepts both Windows (`\r\n`) and POSIX (`\n`) line endings. - Trailing carriage returns (`\r`) are automatically stripped during tokenization. 5. **Whitespace Tolerance**: - Leading and trailing spaces and horizontal tabs (`\t`) on both key and value lines are stripped during tokenization. ### 2.2 Formatting Specifications & Examples #### Scalar Types - **Floating Point (`float`, `float32`)**: Formatted via `%f` (default 6 decimal digits) or `%.9g` for full single-precision round-trip fidelity. ``` ; Health 100.000000 ; Mass 14.250000 ``` - **32-Bit Signed Integer (`int32`)**: Formatted as decimal integer `%d`. ``` ; AmmoCount 45 ; TeamIndex -1 ``` - **64-Bit Unsigned Integer (`uint64`, `EntityID`)**: Formatted as a 16-character padded hexadecimal integer prefixed with `0x`. Hexadecimal ensures 64-bit handle readability and exact bit-pattern preservation. ``` ; EntityID 0x00000000DEADBEEF ; LayerMask 0x0000000000000001 ``` - **Boolean (`bool`)**: Formatted as `true` or `false`. For backwards tolerance, the parser also accepts `1` and `0`. ``` ; IsStatic true ; CastShadows false ``` #### Vector Types - **3D Vector (`Vector3` / `float x, y, z`)**: Formatted as three space-delimited floating-point values on a single line. ``` ; Position 10.500000 0.000000 -25.250000 ; Scale 1.000000 1.000000 1.000000 ``` - **2D Vector (`Vector2` / `float x, y`)**: Formatted as two space-delimited floating-point values. ``` ; UVOffset 0.000000 0.500000 ``` #### String Types - **String (`String` / `String8`)**: - Strings without spaces can be serialized as raw text tokens. - Strings containing spaces or symbols are enclosed in double quotes `"..."`. ``` ; AssetName Character_Mesh_Hero ; DisplayName "Grand Citadel Knight" ``` #### Comprehensive Entity Asset File Example (`Entity_01.jasset`) ``` # Juliet Game Engine Asset File # Generated automatically by AssetPipeline. Do not manually corrupt keys. ; AssetType Entity ; Version 1 ; EntityID 0x0000000000000042 ; Kind Inert ; Position 100.000000 25.500000 -50.000000 ; ClassVersion 2 ; MeshInstance 4 ; MaterialOverride "Materials/M_Granite_Polished" ; IsVisible true ``` --- ## 3. The In-Memory Zero-Copy Parser ### 3.1 Memory Layout & Property Nodes To eliminate heap fragmentation and per-object allocation overhead during level loading, the parser loads the entire `.jasset` file into contiguous arena memory via `LoadFile`. The tokenizer then constructs a flat array of `TextPropertyNode` structures on a `TempArena`. ```cpp struct TextPropertyNode { uint32 KeyCRC; String Value; bool Consumed; }; ``` - `KeyCRC`: The 32-bit CRC hash of the trimmed property name. - `Value`: A `String` (`String8`) struct containing a `char* Str` pointer directly into the file buffer and `size_t Size`. No new string allocations are performed. - `Consumed`: A boolean flag initialized to `false`. When a property is queried and read via `SerializeProp`, `Consumed` is set to `true`. ``` File Buffer in Arena: +-------------------------------------------------------------------------------+ | ; Position\n10.0 20.0 30.0\n; Health\n100.0\n | +-------------------------------------------------------------------------------+ ^ ^ ^ ^ | | | | TextPropertyNode[0]: | | | KeyCRC: CRC("Position") | | Value: { Str --------+ , Size: 14 } | Consumed: true | | TextPropertyNode[1]: | KeyCRC: CRC("Health") | Value: { Str -------------------------------+ , Size: 5 } Consumed: true ``` ### 3.2 Dual-Mode CRC32: Compile-Time & Runtime Juliet's `CRC32.h` defines `consteval uint32 crc32(const char* str, size_t length)`. However, `consteval` guarantees compilation failure when called on runtime input (such as keys extracted from a text file during parsing). The engine requires a unified `Crc32` implementation that functions at runtime for parsed text tokens while retaining `constexpr` / `consteval` capability for compile-time string literals. ```cpp // Core/Common/CRC32.h additions [[nodiscard]] constexpr uint32 Crc32(const char* str, size_t length) { Assert(str != nullptr || length == 0); const char* p = str; uint32 crc = ~0U; while (length--) { crc = details::crc32_tab[(crc ^ static_cast(*p++)) & 0xFF] ^ (crc >> 8); } return crc ^ ~0U; } [[nodiscard]] constexpr uint32 Crc32(String str) { return Crc32(str.Str, str.Size); } [[nodiscard]] consteval uint32 operator""_crc32(const char* str, size_t length) { return Crc32(str, length); } ``` ### 3.3 Zero-Copy Tokenization Algorithm The tokenization algorithm scans the memory buffer in a single pass. It first counts property keys to allocate the exact array size on the arena, then populates the `TextPropertyNode` array. ```cpp [[nodiscard]] inline String TrimWhitespace(String str) { while (str.Size > 0 && (*str.Str == ' ' || *str.Str == '\t' || *str.Str == '\r' || *str.Str == '\n')) { str.Str++; str.Size--; } while (str.Size > 0 && (str.Str[str.Size - 1] == ' ' || str.Str[str.Size - 1] == '\t' || str.Str[str.Size - 1] == '\r' || str.Str[str.Size - 1] == '\n')) { str.Size--; } return str; } struct ParsedTextArchive { TextPropertyNode* Nodes = nullptr; uint32 PropertyCount = 0; }; [[nodiscard]] ParsedTextArchive TokenizeTextArchive(NonNullPtr arena, ByteBuffer fileBuffer) { Assert(fileBuffer.Data != nullptr); char* cursor = reinterpret_cast(fileBuffer.Data); char* end = cursor + fileBuffer.Size; // Pass 1: Count properties to allocate exactly on arena uint32 propertyCount = 0; char* scan = cursor; while (scan < end) { if (*scan == ';') { if (scan == cursor || *(scan - 1) == '\n') { propertyCount++; } } scan++; } if (propertyCount == 0) { return { nullptr, 0 }; } auto* nodes = ArenaPushArray(arena, propertyCount); Assert(nodes != nullptr); // Pass 2: Extract keys and value slices uint32 nodeIndex = 0; scan = cursor; while (scan < end && nodeIndex < propertyCount) { // Skip leading whitespace / empty lines / comments while (scan < end && (*scan == '\r' || *scan == '\n' || *scan == ' ' || *scan == '\t')) { scan++; } if (scan >= end) { break; } // Check for comment line if (*scan == '#' || (*scan == '/' && scan + 1 < end && *(scan + 1) == '/')) { while (scan < end && *scan != '\n') { scan++; } continue; } // Check for property declaration ';' if (*scan == ';') { scan++; // skip ';' // Extract key name char* keyStart = scan; while (scan < end && *scan != '\r' && *scan != '\n') { scan++; } String rawKey = { .Str = keyStart, .Size = static_cast(scan - keyStart) }; String key = TrimWhitespace(rawKey); // Skip newline after key while (scan < end && (*scan == '\r' || *scan == '\n')) { scan++; } // Extract value block (everything until next ';' at start of line or EOF) char* valStart = scan; char* valEnd = scan; while (scan < end) { if (*scan == ';' && (scan == cursor || *(scan - 1) == '\n')) { break; } scan++; valEnd = scan; } String rawVal = { .Str = valStart, .Size = static_cast(valEnd - valStart) }; String val = TrimWhitespace(rawVal); nodes[nodeIndex].KeyCRC = Crc32(key); nodes[nodeIndex].Value = val; nodes[nodeIndex].Consumed = false; nodeIndex++; } else { // Advance unexpected character scan++; } } return { nodes, nodeIndex }; } ``` ### 3.4 Key Lookup & Unconsumed Key Audit Lookup performs a fast linear scan over the contiguous `TextPropertyNode` array. Because typical game entities possess between 5 and 50 properties, a cache-coherent linear scan over contiguous 16-byte structs executes in single-digit nanoseconds, comfortably fitting within CPU L1/L2 data cache. ```cpp [[nodiscard]] inline TextPropertyNode* FindProperty(TextPropertyNode* nodes, uint32 count, uint32 keyCRC) { Assert(nodes != nullptr || count == 0); for (uint32 index = 0; index < count; ++index) { if (nodes[index].KeyCRC == keyCRC) { nodes[index].Consumed = true; return &nodes[index]; } } return nullptr; } #if JULIET_DEBUG inline void AuditUnconsumedProperties(const TextPropertyNode* nodes, uint32 count, const char* contextName) { Assert(nodes != nullptr || count == 0); for (uint32 index = 0; index < count; ++index) { if (!nodes[index].Consumed) { LogWarning(LogCategory::Core, "[%s] Unconsumed or obsolete property detected: CRC 0x%08X (Value: '%.*s')", contextName, nodes[index].KeyCRC, static_cast(nodes[index].Value.Size), nodes[index].Value.Str); } } } #endif ``` --- ## 4. The `archive` Struct & Mode Handling ### 4.1 Struct Definition & Mode Flags Juliet's existing `archive` struct in `Core/Common/serialization.h` is restricted to binary offsets and raw arena pointers. We upgrade `archive` into a unified serialization context supporting both text `.jasset` and binary streams. ```cpp enum class ArchiveMode : uint8 { SavingText, LoadingText, SavingBinary, LoadingBinary }; struct TextPropertyNode; struct archive { Arena* ArenaInstance = nullptr; IOStream* Stream = nullptr; TextPropertyNode* Properties = nullptr; uint32 PropertyCount = 0; ArchiveMode Mode = ArchiveMode::LoadingText; uint32 BaseVersion = 0; uint16 ClassVersion = 0; // Legacy binary support fields void* BasePtr = nullptr; index_t Offset = 0; [[nodiscard]] bool IsLoading() const { return Mode == ArchiveMode::LoadingText || Mode == ArchiveMode::LoadingBinary; } [[nodiscard]] bool IsSaving() const { return Mode == ArchiveMode::SavingText || Mode == ArchiveMode::SavingBinary; } [[nodiscard]] bool IsText() const { return Mode == ArchiveMode::LoadingText || Mode == ArchiveMode::SavingText; } [[nodiscard]] bool IsBinary() const { return Mode == ArchiveMode::LoadingBinary || Mode == ArchiveMode::SavingBinary; } }; ``` ### 4.2 Output Stream Formatting via `IOPrintf` When saving in `ArchiveMode::SavingText`, `archive` outputs directly to an open `IOStream` using `IOPrintf`. ```cpp inline void WritePropertyHeader(archive& ar, const char* keyName) { Assert(ar.IsSaving()); Assert(ar.Stream != nullptr); Assert(keyName != nullptr); IOPrintf(ar.Stream, "; %s\n", keyName); } ``` This design provides: 1. Direct stream output with zero heap buffer allocations. 2. Canonical spacing and formatting across all entity serializers. 3. Formatted outputs immediately flushed or buffered according to `IOStreamInterface` configuration. --- ## 5. `SerializeProp` API & Implementation ### 5.1 Unified Serialization Idiom The `SerializeProp` function family encapsulates both loading and saving behind a single call. If an asset file lacks a given property (e.g. an older asset file loaded by newer code), `SerializeProp` returns `false` during loading, and the destination variable retains its existing default value. ```cpp #define SERIALIZE_PROP(ar, var) SerializeProp((ar), #var, #var##_crc32, (var)) ``` ### 5.2 Primitive Type Helpers #### Float (`float`) ```cpp bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, float& value) { Assert(keyName != nullptr); if (ar.IsSaving()) { WritePropertyHeader(ar, keyName); IOPrintf(ar.Stream, "%f\n\n", value); return true; } auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC); if (node == nullptr) { return false; } // Fast float conversion from String slice char buffer[64]; size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1); MemCopy(buffer, node->Value.Str, copySize); buffer[copySize] = '\0'; char* endPtr = nullptr; float parsed = strtof(buffer, &endPtr); if (endPtr != buffer) { value = parsed; return true; } return false; } ``` #### 32-Bit Signed Integer (`int32`) ```cpp bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, int32& value) { Assert(keyName != nullptr); if (ar.IsSaving()) { WritePropertyHeader(ar, keyName); IOPrintf(ar.Stream, "%d\n\n", value); return true; } auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC); if (node == nullptr) { return false; } char buffer[32]; size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1); MemCopy(buffer, node->Value.Str, copySize); buffer[copySize] = '\0'; char* endPtr = nullptr; int32 parsed = static_cast(strtol(buffer, &endPtr, 10)); if (endPtr != buffer) { value = parsed; return true; } return false; } ``` #### 64-Bit Unsigned Integer (`uint64`, `EntityID`) ```cpp bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, uint64& value) { Assert(keyName != nullptr); if (ar.IsSaving()) { WritePropertyHeader(ar, keyName); IOPrintf(ar.Stream, "0x%016llX\n\n", value); return true; } auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC); if (node == nullptr) { return false; } char buffer[32]; size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1); MemCopy(buffer, node->Value.Str, copySize); buffer[copySize] = '\0'; char* endPtr = nullptr; int base = (buffer[0] == '0' && (buffer[1] == 'x' || buffer[1] == 'X')) ? 16 : 10; uint64 parsed = strtoull(buffer, &endPtr, base); if (endPtr != buffer) { value = parsed; return true; } return false; } ``` #### Boolean (`bool`) ```cpp bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, bool& value) { Assert(keyName != nullptr); if (ar.IsSaving()) { WritePropertyHeader(ar, keyName); IOPrintf(ar.Stream, "%s\n\n", value ? "true" : "false"); return true; } auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC); if (node == nullptr) { return false; } if (StringCompare(node->Value, WrapString("true")) == 0 || StringCompare(node->Value, WrapString("1")) == 0) { value = true; return true; } if (StringCompare(node->Value, WrapString("false")) == 0 || StringCompare(node->Value, WrapString("0")) == 0) { value = false; return true; } return false; } ``` ### 5.3 Vector Helpers (`float x, y, z`) ```cpp bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, float& x, float& y, float& z) { Assert(keyName != nullptr); if (ar.IsSaving()) { WritePropertyHeader(ar, keyName); IOPrintf(ar.Stream, "%f %f %f\n\n", x, y, z); return true; } auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC); if (node == nullptr) { return false; } char buffer[128]; size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1); MemCopy(buffer, node->Value.Str, copySize); buffer[copySize] = '\0'; float parsedX = 0.0f; float parsedY = 0.0f; float parsedZ = 0.0f; int matches = sscanf_s(buffer, "%f %f %f", &parsedX, &parsedY, &parsedZ); if (matches == 3) { x = parsedX; y = parsedY; z = parsedZ; return true; } return false; } ``` ### 5.4 String Helpers ```cpp bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, String& value) { Assert(keyName != nullptr); if (ar.IsSaving()) { WritePropertyHeader(ar, keyName); bool hasSpace = ContainsChar(value, ' '); if (hasSpace) { IOPrintf(ar.Stream, "\"%.*s\"\n\n", static_cast(value.Size), value.Str); } else { IOPrintf(ar.Stream, "%.*s\n\n", static_cast(value.Size), value.Str); } return true; } auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC); if (node == nullptr) { return false; } String parsed = node->Value; // Strip optional surrounding double quotes if (parsed.Size >= 2 && parsed.Str[0] == '"' && parsed.Str[parsed.Size - 1] == '"') { parsed.Str++; parsed.Size -= 2; } // Allocate persistent string copy on the archive arena Assert(ar.ArenaInstance != nullptr); value = StringCopy(ar.ArenaInstance, parsed); return true; } ``` --- ## 6. Two-Tier Versioning Architecture ### 6.1 Architectural Rationale: Base vs Derived Decoupling In entity-component systems or object-oriented engine hierarchies, entities consist of two distinct domains: 1. **Core Engine Identity (Base Entity)**: Position, Rotation, Scale, EntityID, Class Kind, Render Flags, Layer Masks. Managed by engine architects. 2. **Gameplay Specialization (Derived Class)**: Ammo, Health, AI State, Mesh Instance ID, Patrol Paths. Managed by gameplay programmers. #### The Fragility of Monolithic Versioning In naive serialization architectures, a single `uint32 Version` governs the entire file. When an engine programmer updates the base `Entity` struct (e.g. adding a `uint32 LayerMask`), bumping the global version invalidates or forces schema changes across every single derived entity type in the project. #### The Two-Tier Solution: Universal `; Version` + Optional `; ClassVersion` To keep `.jasset` files clean and unified across all asset types: - **All `.jasset` files declare a universal `; Version` tag**: - In simple standalone assets (e.g., `WorldSettings.jasset`, `Material.jasset`), `; Version` is the single asset schema version. - In entity assets, `; Version` maps to the Base Engine Version (`kEntityBaseVersion` in `Entity.h`), governing core engine fields (`Position`, `Rotation`, `Scale`, etc.). - **Entity classes with derived versioning add an optional `; ClassVersion`**: - Governed by `Class::Version` in `Class.h`. - Only written/read for entity classes that define custom versions. If omitted in the file, it defaults to `1`. - Non-entity assets never see or use `; ClassVersion`. ``` ======================================================================== .jasset Text Archive ======================================================================== ; AssetType Entity ; Version ------> Universal asset version (kEntityBaseVersion for Entity) 1 ; EntityID 0x0000000000000001 ; Kind Inert ; Position 0.0 0.0 0.0 ------------------------------------------------------------------------ ; ClassVersion ------> Optional derived version (Class::Version in Class.h) 2 ; MeshInstance 42 ======================================================================== ``` When an engine programmer bumps `kEntityBaseVersion` from `1` to `2` to add `LayerMask`, no derived game classes (`Inert`, `Monster`, `Vehicle`) need version increments or code modifications. Conversely, non-entity assets (like `WorldSettings.jasset`) simply use `; Version \n 1` without carrying entity-specific terminology. ### 6.2 Implementation Details: The Generalized `Class` Model In this architecture, **every serializable struct in Juliet has a `Class` descriptor**: #### Generalized `Class` Struct (`Juliet/include/Engine/Class.h`) ```cpp using serialize_fct_type = void (*)(archive& ar, void* payload, uint16 version); struct Class { uint32 CRC; uint8 kind; uint16 Version; // Struct schema version Class* BaseClass; // Pointer to parent class (or nullptr if root) serialize_fct_type serialize_fct; // Type-specific serialization callback size_t size_of; size_t alignment; #if JULIET_DEBUG String Name; #endif }; consteval Class MakeClass(String name, uint8 kind, uint16 version, Class* baseClass, size_t size, size_t align, serialize_fct_type fct) { Class cls = {}; cls.CRC = crc32(name.Str, name.Size); cls.kind = kind; cls.Version = version; cls.BaseClass = baseClass; cls.size_of = size; cls.alignment = align; cls.serialize_fct = fct; #if JULIET_DEBUG cls.Name = name; #endif return cls; } ``` #### Registration Macros ```cpp // General class registration (e.g. for Entity, WorldSettings, Materials) #define DEFINE_CLASS_VERSIONED(cls, version, base_class, serialize_fct) \ Class classKind##cls = \ MakeClass(ConstString(#cls), 0, (version), (base_class), sizeof(cls), alignof(cls), (serialize_fct)); \ Class* cls::Kind = &classKind##cls; // Entity derived class registration (automatically sets BaseClass = Entity::Kind) #define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct) \ Class entityKind##entity = \ MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version), Entity::Kind, sizeof(entity), \ alignof(entity), (serialize_fct)); \ Class* entity::Kind = &entityKind##entity; ``` #### Universal Class Instance Serializer Because every type is a `Class`, serializing **any** struct in the engine is unified into a single function: ```cpp void SerializeClassInstance(archive& ar, Class* cls, void* instance) { Assert(cls != nullptr); Assert(instance != nullptr); uint16 version = cls->Version; if (ar.IsSaving()) { // If this is a derived entity class, write as class_version; otherwise write universal version if (cls->BaseClass != nullptr) { SerializeProp(ar, "class_version", "class_version"_crc32, version); } else { SerializeProp(ar, "version", "version"_crc32, version); } } else { if (cls->BaseClass != nullptr) { (void)SerializeProp(ar, "class_version", "class_version"_crc32, version); } else { (void)SerializeProp(ar, "version", "version"_crc32, version); } } if (cls->serialize_fct != nullptr) { cls->serialize_fct(ar, instance, version); } } ``` #### Hierarchical Type Queries via `BaseClass` ```cpp [[nodiscard]] inline bool IsA(const Class* queryClass, const Class* targetClass) { const Class* current = queryClass; while (current != nullptr) { if (current == targetClass) { return true; } current = current->BaseClass; } return false; } ``` ### 6.3 Entity Serialization with Generalized Classes An entity instance is cleanly composed of its base `Entity` class and its derived `DerivedKind` class: ```cpp void Serialize(archive& ar, NonNullPtr entity) { Assert(entity.Get() != nullptr); // 1. Serialize Base Entity using Entity's own Class descriptor (Entity::Kind) SerializeClassInstance(ar, Entity::Kind, entity.Get()); // 2. Serialize Derived Entity using its Class descriptor (entity->DerivedKind) if (entity->DerivedKind != nullptr && entity->Derived != nullptr) { SerializeClassInstance(ar, entity->DerivedKind, entity->Derived); } } ``` --- ## 7. Post-Serialization Deprecation & Migration ### 7.1 Schema Evolution Challenge Over the lifecycle of a game, gameplay mechanics evolve: - A scalar float `Speed` is replaced by a directional 2D vector `Velocity`. - A single texture index `TextureID` is replaced by an asset path string `DiffuseTexture`. - Obsolete properties are deleted entirely. Retaining deprecated members in active C++ structs creates code clutter, wastes memory, and invites bugs. ### 7.2 Stack-Allocated Migration via Standard Serialization Rather than maintaining dedicated deprecation primitives or polluting C++ structs with obsolete members, deprecated properties are migrated using the standard `serialize` / `SERIALIZE` function with temporary local variables allocated on the stack. When an asset property is deprecated, restructured, or renamed: 1. The obsolete field is completely removed from the modern C++ struct definition. 2. In the entity/component serializer, an `if (ar.loading && ar.class_version < N)` block is added. 3. A local variable of the legacy type is declared on the stack. 4. The standard `SERIALIZE(ar, OldPropName, deprecated_val)` (or `serialize(ar, ConstString("OldPropName"), "OldPropName"_crc32, deprecated_val)`) is called. If the property exists in the loaded asset, it parses into `deprecated_val` and returns `true`. 5. The serializer performs whatever mapping or transformation is required into the modern struct fields. 6. Because the migration block is strictly guarded by `ar.loading`, it never executes during save operations (`ar.loading == false`). The serializer writes only modern struct fields, automatically purging obsolete keys on subsequent saves without requiring dedicated cleanup routines. ### 7.3 In-Place Migration Pattern When an asset file with an older `class_version` is loaded, the serializer detects `ar.class_version < N`, reads obsolete fields into stack variables using standard serialization, maps the legacy data into the modern struct, and completes loading. On the subsequent save, the asset file is emitted using the modern schema without deprecated keys. ```cpp struct Projectile { DECLARE_ENTITY() // Modern Schema (v2) float VelocityX = 0.0f; float VelocityY = 0.0f; float Damage = 50.0f; }; void SerializeProjectile(Archive* arPtr, void* payload) { Assert(arPtr != nullptr); Assert(payload != nullptr); auto& ar = *arPtr; auto* projectile = static_cast(payload); if (ar.loading) { SERIALIZE(ar, Damage, projectile->Damage); if (ar.class_version < 2) { // Migration from v1: scalar 'Speed' converted to 'VelocityX' on the stack float deprecated_speed = 0.0f; if (SERIALIZE(ar, Speed, deprecated_speed)) { projectile->VelocityX = deprecated_speed; projectile->VelocityY = 0.0f; } } else { SERIALIZE(ar, VelocityX, projectile->VelocityX); SERIALIZE(ar, VelocityY, projectile->VelocityY); } } else { // Save modern schema SERIALIZE(ar, VelocityX, projectile->VelocityX); SERIALIZE(ar, VelocityY, projectile->VelocityY); SERIALIZE(ar, Damage, projectile->Damage); } } ``` --- ## 8. Step-by-Step Implementation Roadmap & Unit Testing Plan ### 8.1 Implementation Roadmap ``` +-------------------------------------------------------------------------+ | Phase 1: Core Utilities & Dual-Mode CRC32 | | - Update CRC32.h with constexpr Crc32(String) and Crc32(char*, len) | | - Extend struct Class in Class.h with uint16 Version | +-------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------+ | Phase 2: In-Memory Zero-Copy Parser | | - Implement TextArchiveParser.h / .cpp | | - TokenizeTextArchive, TrimWhitespace, FindProperty | +-------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------+ | Phase 3: Archive Struct & Serialize Helpers | | - Upgrade Archive in serialization.h with loading / version state | | - Implement primitive, vector, and string serialize helpers | | - Support stack-allocated schema migration | +-------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------+ | Phase 4: Entity & World Integration | | - Update Entity.h / Entity.cpp with two-tier serialization | | - Refactor World.cpp to load/save .jasset files | +-------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------+ | Phase 5: Verification & Unit Testing Suite | | - Build and run SerializationUnitTest.cpp | +-------------------------------------------------------------------------+ ``` #### Phase 1: Core Utilities & Dual-Mode CRC32 1. **Target**: `Juliet/include/Core/Common/CRC32.h` - Add `constexpr uint32 Crc32(const char* str, size_t length)` and `constexpr uint32 Crc32(String str)`. - Ensure existing `operator""_crc32` calls `Crc32`. 2. **Target**: `Juliet/include/Engine/Class.h` - Add `uint16 Version` to `struct Class`. - Update `MakeClass` to accept `uint16 version = 1`. #### Phase 2: In-Memory Zero-Copy Parser 1. **Target**: `Juliet/include/Core/Common/TextArchiveParser.h` (and `src/Core/Common/TextArchiveParser.cpp`) - Define `struct TextPropertyNode { uint32 KeyCRC; String Value; bool Consumed; }`. - Implement `TokenizeTextArchive(NonNullPtr arena, ByteBuffer buffer)`. - Implement `FindProperty` and `AuditUnconsumedProperties`. #### Phase 3: Archive Struct & `serialize` Helpers 1. **Target**: `Juliet/include/Core/Common/serialization.h` - Upgrade `struct Archive` with stream pointer, property array, `loading` flag, and versions (`base_version`, `class_version`). - Implement overloaded `serialize`, `read_prop`, and `write` for `float`, `int32`, `uint64`, `bool`, vectors, and `String`. - Support deprecation migration using standard `serialize` / `SERIALIZE` with local stack variables under `if (ar.loading && ar.class_version < N)`. #### Phase 4: Entity & World Integration 1. **Target**: `Game/Entity/Entity.h` and `Game/Entity/Entity.cpp` - Define `constexpr uint32 kEntityBaseVersion = 1;`. - Update `DEFINE_ENTITY` and add `DEFINE_ENTITY_VERSIONED`. - Implement `Serialize(archive& ar, NonNullPtr entity)` supporting `.jasset` text format. 2. **Target**: `Game/Data/World.h` and `Game/Data/World.cpp` - Implement text-based world saving and loading using `.jasset` formatting. #### Phase 5: Comprehensive Unit Testing 1. **Target**: `Game/UnitTest/SerializationUnitTest.h` and `Game/UnitTest/SerializationUnitTest.cpp` - Add unit tests verifying parsing, round-trip serialization, defaults preservation, versioning, and deprecation. --- ### 8.2 Comprehensive Unit Testing Plan (`SerializationUnitTest.cpp`) The unit test suite validates all architectural requirements without modifying engine framework code for test-specific cases. ```cpp // Game/UnitTest/SerializationUnitTest.h #pragma once #include #if JULIET_DEBUG namespace UnitTest { void RunSerializationUnitTests(); } #endif ``` ```cpp // Game/UnitTest/SerializationUnitTest.cpp #include #if JULIET_DEBUG #include #include #include #include #include #include #include #include #include namespace UnitTest { // Test Struct for Derived Entity Testing struct DummyVehicle { DECLARE_ENTITY() float MaxSpeed = 120.0f; int32 GearCount = 6; uint64 ChassisUUID = 0xABCDEF0123456789ULL; bool Turbo = true; float PosX = 10.0f; float PosY = 20.0f; float PosZ = 30.0f; }; void SerializeDummyVehicle(archive* arPtr, void* payload) { Assert(arPtr != nullptr); Assert(payload != nullptr); auto& ar = *arPtr; auto* vehicle = static_cast(payload); SERIALIZE_PROP(ar, vehicle->MaxSpeed); SERIALIZE_PROP(ar, vehicle->GearCount); SERIALIZE_PROP(ar, vehicle->ChassisUUID); SERIALIZE_PROP(ar, vehicle->Turbo); SerializeProp(ar, "Position", "Position"_crc32, vehicle->PosX, vehicle->PosY, vehicle->PosZ); } DEFINE_ENTITY_VERSIONED(DummyVehicle, 1, SerializeDummyVehicle); // Test 1: Parser tokenization with whitespace, comments, and mixed line endings static void TestParserTokenization() { TempArena temp = scratch_begin(nullptr, 0); const char* testContent = "# Header Comment\r\n" "// Secondary comment\n" "\n" "; Health\r\n" " 100.500000 \r\n" "\n" "; Name\n" "\"Paladin Hero\"\n" "\n" "; Position\r\n" "1.0 2.0 3.0\r\n"; ByteBuffer buffer = { .Data = reinterpret_cast(const_cast(testContent)), .Size = strlen(testContent) }; ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer); Assert(parsed.PropertyCount == 3); auto* healthNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Health"_crc32); Assert(healthNode != nullptr); Assert(StringCompare(healthNode->Value, WrapString("100.500000")) == 0); auto* nameNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Name"_crc32); Assert(nameNode != nullptr); Assert(StringCompare(nameNode->Value, WrapString("\"Paladin Hero\"")) == 0); auto* posNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Position"_crc32); Assert(posNode != nullptr); Assert(StringCompare(posNode->Value, WrapString("1.0 2.0 3.0")) == 0); scratch_end(temp); LogMessage(LogCategory::Core, "TestParserTokenization passed."); } // Test 2: Missing properties retain default struct values static void TestDefaultValueRetention() { TempArena temp = scratch_begin(nullptr, 0); const char* incompleteContent = "; MaxSpeed\n" "180.0\n"; ByteBuffer buffer = { .Data = reinterpret_cast(const_cast(incompleteContent)), .Size = strlen(incompleteContent) }; ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer); archive ar = {}; ar.ArenaInstance = temp.Arena; ar.Mode = ArchiveMode::LoadingText; ar.Properties = parsed.Nodes; ar.PropertyCount = parsed.PropertyCount; DummyVehicle vehicle; // Defaults: MaxSpeed=120, GearCount=6, Turbo=true SerializeDummyVehicle(&ar, &vehicle); Assert(vehicle.MaxSpeed == 180.0f); // Overwritten by archive Assert(vehicle.GearCount == 6); // Preserved default Assert(vehicle.Turbo == true); // Preserved default Assert(vehicle.PosX == 10.0f); // Preserved default scratch_end(temp); LogMessage(LogCategory::Core, "TestDefaultValueRetention passed."); } // Test 3: Deprecation migration from v1 to v2 struct LegacyWeapon { DECLARE_ENTITY() float VelocityX = 0.0f; float VelocityY = 0.0f; }; void SerializeLegacyWeapon(Archive* arPtr, void* payload) { Assert(arPtr != nullptr); Assert(payload != nullptr); auto& ar = *arPtr; auto* weapon = static_cast(payload); if (ar.loading) { if (ar.class_version < 2) { float deprecated_speed = 0.0f; if (SERIALIZE(ar, Speed, deprecated_speed)) { weapon->VelocityX = deprecated_speed; weapon->VelocityY = 0.0f; } } else { SERIALIZE(ar, VelocityX, weapon->VelocityX); SERIALIZE(ar, VelocityY, weapon->VelocityY); } } else { SERIALIZE(ar, VelocityX, weapon->VelocityX); SERIALIZE(ar, VelocityY, weapon->VelocityY); } } DEFINE_ENTITY_VERSIONED(LegacyWeapon, 2, SerializeLegacyWeapon); static void TestDeprecationMigration() { TempArena temp = scratch_begin(nullptr, 0); // Simulated v1 file containing obsolete 'Speed' const char* v1Content = "; class_version\n" "1\n" "; Speed\n" "75.500000\n"; ByteBuffer buffer = { .Data = reinterpret_cast(const_cast(v1Content)), .Size = strlen(v1Content) }; ParsedArchive parsed = tokenize_archive(temp.Arena, buffer); Archive ar = {}; ar.arena = temp.Arena; ar.loading = true; ar.base = parsed; ar.class_version = 1; LegacyWeapon weapon; SerializeLegacyWeapon(&ar, &weapon); Assert(weapon.VelocityX == 75.5f); Assert(weapon.VelocityY == 0.0f); scratch_end(temp); LogMessage(LogCategory::Core, "TestDeprecationMigration passed."); } void RunSerializationUnitTests() { LogMessage(LogCategory::Core, "=== Running Serialization & .jasset Unit Tests ==="); TestParserTokenization(); TestDefaultValueRetention(); TestDeprecationMigration(); LogMessage(LogCategory::Core, "=== All Serialization Unit Tests Passed Successfully ==="); } } // namespace UnitTest #endif