Juliet/Game/Plans/01_Serialization_And_Text_Archive.md

Juliet Engine: Serialization & Text Archive Architecture

Technical Specification & Architectural Design

---

1. Executive Summary & Architecture Goals

1.1 Context & Motivation

Juliet historically relied on monolithic, packed binary blobs for world and entity persistence. While fast to read as raw byte offsets, binary serialization suffers from three major flaws:

1. Merge Incompatibility: Binary assets cannot be merged or diffed in version control systems (Git / Perforce), causing unresolvable binary conflicts and data loss.
2. Schema Rigidity: Adding, removing, or reordering a single struct field invalidates all existing binary files unless complex manual byte-offset mapping tables are maintained.
3. Opacity: Designers and engineers cannot inspect, debug, or patch asset properties in a standard text editor.

1.2 Architectural Goals

The .jasset text archive framework replaces legacy binary blobs with a human-readable, diff-friendly property serialization pipeline adhering to Juliet's systems programming principles:

• Zero Dynamic Heap Allocations: Parsing, tokenization, formatting, and buffer transformations execute entirely within Juliet memory arenas (Arena, TempArena, scratch_begin / scratch_end). Standard library containers (std::string, std::vector) and raw heap allocators (malloc, new) are forbidden.
• Zero-Copy In-Memory Tokenization: Files are loaded into arena memory once via LoadFile. The tokenizer parses properties into lightweight slices represented by Juliet's String (char* Str; size_t Size;), referencing existing file buffer bytes without string duplication.
• O(1) Property Lookup via Compile-Time & Runtime CRC32: Property keys in text files are hashed once during tokenization into 32-bit CRC values. In code, lookups utilize compile-time hashed literals (operator""_crc32). Property resolution is a single 32-bit integer comparison.
• Symmetric Single-Function Serialization: A single serialize implementation per entity or data structure handles both Save and Load paths, guaranteeing read and write schemas never diverge.
• Graceful Forward/Backward Compatibility: Missing keys during load automatically preserve struct default values. Unknown keys present in newer asset files are safely ignored without parse failure.
• Two-Tier Decoupled Versioning via Generalized Class: Core engine entity properties (; version) and derived gameplay class properties (; class_version) are versioned independently through their respective Class descriptors.
• In-Place Schema Migration: Deprecated fields no longer present in C++ structs are read into temporary stack variables during load using standard SERIALIZE calls guarded by if (ar.loading && version < N), seamlessly converting legacy data without struct pollution or persisting obsolete keys on subsequent saves.
• Clean Warning-Free C++: Fully conforming to Juliet coding guidelines: strict assertions (Assert), [[nodiscard]], auto*/auto&, mandatory braces, and static_cast/reinterpret_cast.

---

2. The .jasset Text Format Specification

2.1 Grammar & Structural Rules

The .jasset format uses a line-oriented, key-value property hierarchy designed for visual clarity and clean Git diffs.

AssetFile             ::= { CommentLine | EmptyLine | PropertyDeclaration } ;
CommentLine           ::= ( "#" | "//" ) { Character } LineEnding ;
EmptyLine             ::= { Whitespace } LineEnding ;
PropertyDeclaration   ::= KeyHeader LineEnding ValueBlock ;
KeyHeader             ::= ";" { Whitespace } Identifier ;
ValueBlock            ::= { ValueLine LineEnding } ;
ValueLine             ::= { Whitespace } ValueString { Whitespace } ;
LineEnding            ::= "\r\n" | "\n" ;
Identifier            ::= [a-zA-Z_][a-zA-Z0-9_]* ;

Formatting Rules:

1. Key Declarations: A property begins with a semicolon ; followed by optional whitespace and a case-sensitive identifier (e.g. ; position).
2. Value Blocks: The line(s) immediately following a key header contain its value payload.
3. Comments: Any line whose first non-whitespace character is # or // is treated as a comment and ignored. Inline comments on property lines are forbidden.
4. Whitespace: Leading and trailing spaces or tabs on both keys and values are stripped during tokenization.
5. Line Endings: Both Windows CRLF (\r\n) and Linux LF (\n) are transparently accepted.

2.2 Formatting Specifications

Scalar Types

Scalars are formatted as decimal representations on a single line:

; max_speed
180.500000

; gear_count
6

; turbo
true

• Floats: Output via %.9g or %f.
• Integers: Signed (%d, %lld) and unsigned (%u, %llu).
• Booleans: Case-insensitive true / false or 1 / 0.

Vector Types (Vector4)

Multi-component vectors are space-delimited on a single value line:

; position
10.0 20.0 30.0 1.0

String Types (String)

Strings containing spaces are wrapped in double quotes "...". Quotes are automatically stripped upon loading and emitted during saving when spaces are present:

; name
"Paladin Hero"

Asset File Example (Entity_01.jasset)

# Juliet Entity Asset File
; version
1

; id
1001

; position
12.500000 0.000000 45.200000 1.000000

; class
Inert

; class_version
1

; mesh_instance
42

---

3. Zero-Copy Tokenization & Fast Property Lookup

3.1 Data Structures (Core/Common/serialization.h)

To eliminate heap fragmentation, the parser loads the entire .jasset file into contiguous arena memory and parses it into a flat array of lightweight slices.

struct ArchivePropertyNode
{
    String key;
    String value;
    uint32 key_crc;
    bool   consumed;
};

struct ParsedArchive
{
    ArchivePropertyNode* nodes          = nullptr;
    uint32               property_count = 0;
};

• key: Sliced String referencing the key name.
• value: Sliced String directly referencing file buffer bytes (zero-copy).
• key_crc: 32-bit CRC hash computed once during tokenization.
• consumed: Initialized to false. Set to true whenever queried by find_property.

3.2 Dual-Mode CRC32 (Core/Common/CRC32.h)

Lookups rely on compile-time string hashing via constexpr / consteval:

[[nodiscard]] constexpr uint32 crc32(const char* str, size_t length);
[[nodiscard]] constexpr uint32 crc32(String str);
[[nodiscard]] consteval uint32 operator""_crc32(const char* str, size_t length);

3.3 Tokenization API (tokenize_archive)

JULIET_API ParsedArchive tokenize_archive(NonNullPtr<Arena> arena, ByteBuffer file_buffer);

Algorithm Invariants:

1. Pass 1 (Count): Scans the buffer to count ; key headers at line starts, allocating the exact node array in arena.
2. Pass 2 (Extract): Slices key and value Strings, trims whitespace, computes key_crc = crc32(key), and populates nodes. Skips # and // comments.

3.4 Property Lookup & Audit API

JULIET_API ArchivePropertyNode* find_property(NonNullPtr<ParsedArchive> archive, uint32 property_crc);

#if JULIET_DEBUG
JULIET_API void audit_unconsumed_properties(NonNullPtr<ParsedArchive> archive, String context_name);
#endif

• find_property: Performs an O(1) integer comparison against key_crc. When found, marks node->consumed = true.
• audit_unconsumed_properties: Iterates through all nodes in debug builds and logs warnings for any property with consumed == false, catching typos or abandoned schema fields.

---

4. The Archive Context Struct & Streaming I/O

4.1 Struct Definition (Core/Common/serialization.h)

The Archive struct unifies loading and saving state into a single decoupled context:

struct Archive
{
    Arena*        arena;
    bool          loading;
    ParsedArchive base   = {};
    IOStream*     stream = nullptr;

    // Legacy binary support fields (to be deprecated)
    void*   base_ptr = nullptr;
    index_t offset   = 0;
};

• When loading == true: Reads properties from base.nodes via find_property.
• When loading == false: Writes formatted key-value pairs directly to stream.

4.2 Property Header Formatting

JULIET_API void write_property_header(Archive& archive, String property_name);

Emits ; <property_name>\n directly to ar.stream with zero intermediate heap buffers.

---

5. Property Serialization API & Helpers

5.1 Unified Serialization Idiom

All property serialization uses a single template function:

template <typename Type>
bool serialize(Archive& ar, String property_name, uint32 property_crc, Type& value)
{
    Assert(IsValid(property_name));

    bool result = false;
    if (ar.loading)
    {
        if (auto* prop = find_property(&ar.base, property_crc))
        {
            if (read_prop(ar, prop->value, value))
            {
                result = true;
            }
        }
    }
    else
    {
        write_property_header(ar, property_name);
        write(ar.stream, value);
        result = true;
    }

    return result;
}

5.2 Convenience Macros

#define SERIALIZE(ar, name, var) serialize((ar), ConstString(#name), #name##_crc32, (var))
#define SERIALIZE_SIMPLE(ar, var) SERIALIZE(ar, var, var)

• SERIALIZE(ar, id, entity->ID): Serializes property named "id" with "id"_crc32.
• SERIALIZE_SIMPLE(ar, position): Uses variable identifier as property name.

5.3 Supported Type Conversions

Conversion between text and memory is handled by overloaded read and write primitives:

C++ Type	Text Format	Conversion Primitive
float	180.500000	strtof / IOPrintf("%.9g")
int8, int16, int32, int64	42 / -100	strtol, strtoll / IOPrintf("%d")
uint8, uint16, uint32, uint64	1001 / 0x...	strtoul, strtoull / IOPrintf("%u")
bool	true / false	true/false/1/0 string compare / IOPrintf
Vector4	10.0 20.0 30.0 1.0	Space-delimited float parse / IOPrintf
String	"Paladin Hero"	Arena-allocated copy, quote strip / IOPrintf

---

6. Two-Tier Versioning & Generalized Class Architecture

6.1 Architectural Principle

To prevent monolithic engine updates from forcing all gameplay assets to re-version, schema versions are decoupled into two tiers:

1. Base Version (; version): Managed by root classes (e.g. Entity::kind->version). Governs core engine properties (id, position).
2. Derived Version (; class_version): Managed by derived classes (e.g. Inert::kind->version). Governs gameplay-specific component properties.

6.2 The Class Descriptor (Engine/Class.h)

Every serializable entity or component is described by an immutable Class instance:

using serialize_fct_type = void (*)(Archive& ar, uint16 version, void* payload);

struct Class
{
    uint32             CRC;
    uint8              kind;
    uint16             version;
    const Class*       base_class;
    serialize_fct_type serialize_fct;
    size_t             size_of;
    size_t             alignment;

#if JULIET_DEBUG
    String Name;
#endif
};

6.3 Class Registration Macros

#define DECLARE_CLASS() \
    static Class* kind;

#define DEFINE_CLASS_VERSIONED(cls, version, base_class, serialize_fct)                                                \
    constexpr Class classKind##cls =                                                                                   \
        MakeClass(ConstString(#cls), 0, (version), (base_class), sizeof(cls), alignof(cls), (serialize_fct));          \
    Class* cls::kind = const_cast<Class*>(&classKind##cls);

For derived entity types, DECLARE_ENTITY() and DEFINE_ENTITY_VERSIONED compose cleanly:

#define DECLARE_ENTITY() \
    Entity* base;        \
    DECLARE_CLASS()

#define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct)                                                         \
    constexpr Class entityKind##entity = MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version),         \
                                                   &classKindEntity, sizeof(entity), alignof(entity), (serialize_fct)); \
    Class*          entity::kind       = const_cast<Class*>(&entityKind##entity);

6.4 Universal Class Serializer (Engine/class.cpp)

void serialize(Archive& ar, NonNullPtr<Class> cls, void* instance)
{
    Assert(instance != nullptr);

    uint16 version = cls->version;
    if (cls->base_class)
    {
        serialize(ar, ConstString("class_version"), "class_version"_crc32, version);
    }
    else
    {
        serialize(ar, ConstString("version"), "version"_crc32, version);
    }

    if (cls->serialize_fct)
    {
        cls->serialize_fct(ar, version, instance);
    }
}

6.5 Runtime Type Queries (IsA)

Polymorphic type safety is resolved without virtual tables or RTTI:

bool IsA(const Class& query, const Class* target);

template <typename TargetType>
bool IsA(const Class& cls)
{
    return IsA(cls, TargetType::kind);
}

6.6 Entity Serialization Composition

An entity instance composes base Entity properties and derived component properties:

void serialize_entity(Archive& ar, uint16 /*version*/, void* payload)
{
    Assert(payload != nullptr);
    auto* entity = static_cast<Entity*>(payload);

    SERIALIZE(ar, id, entity->ID);
    SERIALIZE(ar, position, entity->position);
}

DEFINE_CLASS_VERSIONED(Entity, 1, nullptr, serialize_entity)

void serialize(Archive& ar, NonNullPtr<Entity> entity)
{
    // 1. Serialize base Entity properties (reads/writes '; version')
    serialize(ar, Entity::kind, entity.Get());

    // 2. Serialize derived component properties (reads/writes '; class_version')
    if (entity->derived_kind != nullptr && entity->derived != nullptr)
    {
        serialize(ar, entity->derived_kind, entity->derived);
    }
}

---

7. In-Place Schema Migration & Deprecation

7.1 Deprecation Principle

When gameplay code evolves, obsolete member variables are deleted from active C++ structs to avoid memory waste and code clutter. Obsolete properties are migrated exclusively during loading using temporary local stack variables.

7.2 The Stack-Allocated Migration Idiom

In the class's serialize_fct(Archive& ar, uint16 version, void* payload):

1. When ar.loading == true and version < N:
   • Declare a temporary variable on the stack matching the legacy type.
   • Call SERIALIZE(ar, old_field_name, deprecated_var).
   • If present, transform the legacy data into the modern struct field(s).
2. When saving (ar.loading == false):
   • The migration block is skipped. Only modern struct properties are written.
   • On the next save, obsolete keys are automatically purged from disk.

void serialize_projectile(Archive& ar, uint16 version, void* payload)
{
    Assert(payload != nullptr);
    auto* projectile = static_cast<Projectile*>(payload);

    SERIALIZE(ar, damage, projectile->damage);

    if (ar.loading && version < 2)
    {
        // Migrating v1 scalar 'speed' into modern Vector4 'velocity'
        float deprecated_speed = 0.0f;
        if (SERIALIZE(ar, speed, deprecated_speed))
        {
            projectile->velocity = Vector4{ deprecated_speed, 0.0f, 0.0f, 0.0f };
        }
    }
    else
    {
        SERIALIZE(ar, velocity, projectile->velocity);
    }
}

---

8. Verification & Unit Testing Framework

8.1 Engine-Level Test Runner (Juliet/src/UnitTest/)

Unit testing lives inside the Juliet engine layer (Juliet/src/UnitTest/serialization_test.cpp) and executes during engine startup in debug builds via UnitTest::RunUnitTests() in RunUnitTests.cpp.

8.2 Test Coverage Matrix

Test Function	Target Feature	Validation Criteria
test_parser_tokenization	Zero-copy text parser	Validates handling of # and // comments, whitespace trimming, mixed CRLF/LF, and fast CRC property lookups.
test_default_value_retention	Partial schema loading	Validates that missing properties in partial files preserve existing struct default values without corruption.
test_deprecation_migration	Stack-based schema migration	Loads a v1 asset containing obsolete ; speed, verifies version = 1 is received, and validates migration into modern fields.
test_class_inheritance	Runtime IsA queries	Validates polymorphic inheritance checks across base and derived Class instances.
test_string_and_vector4	Primitives & text quoting	Verifies parsing and quote handling of String and multi-component Vector4.

---

9. Deliverables & File Summary

File	Responsibilities
Juliet/include/Core/Common/serialization.h	ArchivePropertyNode, ParsedArchive, Archive struct, serialize template, and SERIALIZE macros.
Juliet/src/Core/Common/serialization.cpp	tokenize_archive, find_property, audit_unconsumed_properties, read_prop, read, and write primitives.
Juliet/include/Engine/Class.h	Class struct, MakeClass, DECLARE_CLASS(), DEFINE_CLASS_VERSIONED, and IsA declarations.
Juliet/src/Engine/class.cpp	Universal serialize(Archive&, NonNullPtr<Class>, void*) and runtime IsA traversal.
Juliet/src/UnitTest/serialization_test.cpp	Exhaustive unit tests for tokenization, defaults retention, version migration, and type queries.
Game/Entity/Entity.h	DECLARE_ENTITY(), DEFINE_ENTITY_VERSIONED, and Entity struct definition.
Game/Entity/Entity.cpp	serialize_entity registration and two-tier serialize(Archive&, NonNullPtr<Entity>) composition.