43 KiB
43 KiB
Juliet Game Engine: Serialization Core & .jasset Text Archive
Technical Specification & Implementation Plan
Document ID: SPEC-001-SERIALIZATION-CORE
Component: Juliet Engine Core / Asset Pipeline
Target Architecture: Juliet Game Engine (C++20, x64, D3D12)
File Location: Game/Plans/01_Serialization_And_Text_Archive.md
1. Executive Summary & Architecture Goals
1.1 Context & Current Limitations
The Juliet game engine previously utilized packed binary records for level and entity serialization (Assets/world.bin, WorldFileHeader, WorldEntityDiskRecord). While packed binary formats are compact, they present severe architectural roadblocks during collaborative development:
- Merge Incompatibility: Binary assets cannot be merged or diffed across version control systems (Git / Perforce). Concurrent edits by multiple designers or engineers result in unresolvable binary conflicts and data loss.
- Schema Rigidity: Adding, removing, or reordering a single struct field invalidates all previously serialized binary blobs unless complex, manual byte-offset mapping tables are maintained.
- Opacity: Programmers and technical artists cannot inspect, debug, or patch asset properties in a standard text editor.
1.2 Architectural Goals
The .jasset text archive framework is engineered to replace legacy binary blobs with a robust, human-readable, diff-friendly property serialization pipeline while strictly adhering to Juliet's performance and memory constraints:
- Zero Dynamic Heap Allocations: All parsing, tokenization, formatting, and buffer transformations execute entirely within Juliet memory arenas (
Arena,TempArena,scratch_begin/scratch_end). Standard library containers (std::string,std::vector,std::map) and raw heap allocators (malloc,new) are forbidden. - Zero-Copy In-Memory Tokenization: Files are loaded into arena memory once via
LoadFile. The tokenizer parses properties into lightweight slices represented by Juliet'sString(char* Str; size_t Size;), referencing existing file buffer bytes without string replication or null-termination overhead. O(1)Property Lookup via Compile-Time & Runtime CRC32: Property keys in text files are hashed once during tokenization into 32-bit CRC values. In code, lookups utilize compile-time hashed literals (operator""_crc32). Property resolution reduces to a single 32-bit integer comparison, avoiding string comparison overhead in serialization loops.- Symmetric Single-Function Serialization: A single
Serializeimplementation per entity or data structure handles both Save and Load paths, guaranteeing that write and read schemas never diverge. - Graceful Forward/Backward Compatibility: Missing keys during load automatically preserve struct default values. Unknown keys present in newer asset files are safely ignored without parse failure.
- Two-Tier Decoupled Versioning: Core engine entity properties (
kEntityBaseVersion) and derived gameplay class properties (Class::Version) are versioned independently. Engine-level updates never bump derived entity class versions. - Post-Serialization In-Place Migration: Deprecated fields can be read through specialized primitives (
ReadDeprecated*) to migrate legacy data structures in-place upon loading, producing cleaned, modern schemas on subsequent saves. - Zero Exceptions & Total Warning Cleanliness: Fully conforming to Juliet coding guidelines: strict assertions (
Assert),[[nodiscard]],auto*/auto&, mandatory braces, andstatic_cast/reinterpret_cast.
2. The .jasset Text Format Specification
2.1 Grammar & Structural Rules
The .jasset format uses a line-oriented, key-value property hierarchy designed for visual clarity and clean Git diffs.
AssetFile ::= { CommentLine | EmptyLine | PropertyDeclaration } ;
CommentLine ::= ( "#" | "//" ) { Character } LineEnding ;
EmptyLine ::= { Whitespace } LineEnding ;
PropertyDeclaration ::= KeyHeader LineEnding ValueBlock ;
KeyHeader ::= ";" { Whitespace } Identifier ;
ValueBlock ::= { ValueLine LineEnding } ;
ValueLine ::= { Whitespace } ValueString { Whitespace } ;
LineEnding ::= "\r\n" | "\n" ;
Identifier ::= [a-zA-Z_][a-zA-Z0-9_]* ;
- Property Key Header (
;\n):- Every property declaration begins with a semicolon
;as the first non-whitespace character. - The semicolon is followed by optional whitespace and a CamelCase identifier:
; PropertyName. - Keys are case-sensitive and must be valid C++ identifier tokens.
- Every property declaration begins with a semicolon
- Value Block:
- The line immediately following the property header contains the property's serialized payload.
- For multi-line values or arrays, lines continue until the next property header
;or end-of-file.
- Comments & Ignored Tokens:
- Any line starting with
#or//(after optional leading whitespace) is treated as a comment. - Empty lines and lines containing only whitespace are ignored.
- Any line starting with
- Line Termination Handling:
- The parser natively accepts both Windows (
\r\n) and POSIX (\n) line endings. - Trailing carriage returns (
\r) are automatically stripped during tokenization.
- The parser natively accepts both Windows (
- Whitespace Tolerance:
- Leading and trailing spaces and horizontal tabs (
\t) on both key and value lines are stripped during tokenization.
- Leading and trailing spaces and horizontal tabs (
2.2 Formatting Specifications & Examples
Scalar Types
- Floating Point (
float,float32): Formatted via%f(default 6 decimal digits) or%.9gfor full single-precision round-trip fidelity.; Health 100.000000 ; Mass 14.250000 - 32-Bit Signed Integer (
int32): Formatted as decimal integer%d.; AmmoCount 45 ; TeamIndex -1 - 64-Bit Unsigned Integer (
uint64,EntityID): Formatted as a 16-character padded hexadecimal integer prefixed with0x. Hexadecimal ensures 64-bit handle readability and exact bit-pattern preservation.; EntityID 0x00000000DEADBEEF ; LayerMask 0x0000000000000001 - Boolean (
bool): Formatted astrueorfalse. For backwards tolerance, the parser also accepts1and0.; IsStatic true ; CastShadows false
Vector Types
- 3D Vector (
Vector3/float x, y, z): Formatted as three space-delimited floating-point values on a single line.; Position 10.500000 0.000000 -25.250000 ; Scale 1.000000 1.000000 1.000000 - 2D Vector (
Vector2/float x, y): Formatted as two space-delimited floating-point values.; UVOffset 0.000000 0.500000
String Types
- String (
String/String8):- Strings without spaces can be serialized as raw text tokens.
- Strings containing spaces or symbols are enclosed in double quotes
"...".
; AssetName Character_Mesh_Hero ; DisplayName "Grand Citadel Knight"
Comprehensive Entity Asset File Example (Entity_01.jasset)
# Juliet Game Engine Asset File
# Generated automatically by AssetPipeline. Do not manually corrupt keys.
; AssetType
Entity
; BaseVersion
1
; EntityID
0x0000000000000042
; Kind
Inert
; Position
100.000000 25.500000 -50.000000
; ClassVersion
2
; MeshInstance
4
; MaterialOverride
"Materials/M_Granite_Polished"
; IsVisible
true
3. The In-Memory Zero-Copy Parser
3.1 Memory Layout & Property Nodes
To eliminate heap fragmentation and per-object allocation overhead during level loading, the parser loads the entire .jasset file into contiguous arena memory via LoadFile. The tokenizer then constructs a flat array of TextPropertyNode structures on a TempArena.
struct TextPropertyNode
{
uint32 KeyCRC;
String Value;
bool Consumed;
};
KeyCRC: The 32-bit CRC hash of the trimmed property name.Value: AString(String8) struct containing achar* Strpointer directly into the file buffer andsize_t Size. No new string allocations are performed.Consumed: A boolean flag initialized tofalse. When a property is queried and read viaSerializeProp,Consumedis set totrue.
File Buffer in Arena:
+-------------------------------------------------------------------------------+
| ; Position\n10.0 20.0 30.0\n; Health\n100.0\n |
+-------------------------------------------------------------------------------+
^ ^ ^ ^
| | | |
TextPropertyNode[0]: | | |
KeyCRC: CRC("Position") | |
Value: { Str --------+ , Size: 14 } |
Consumed: true |
|
TextPropertyNode[1]: |
KeyCRC: CRC("Health") |
Value: { Str -------------------------------+ , Size: 5 }
Consumed: true
3.2 Dual-Mode CRC32: Compile-Time & Runtime
Juliet's CRC32.h defines consteval uint32 crc32(const char* str, size_t length). However, consteval guarantees compilation failure when called on runtime input (such as keys extracted from a text file during parsing).
The engine requires a unified Crc32 implementation that functions at runtime for parsed text tokens while retaining constexpr / consteval capability for compile-time string literals.
// Core/Common/CRC32.h additions
[[nodiscard]] constexpr uint32 Crc32(const char* str, size_t length)
{
Assert(str != nullptr || length == 0);
const char* p = str;
uint32 crc = ~0U;
while (length--)
{
crc = details::crc32_tab[(crc ^ static_cast<uint8>(*p++)) & 0xFF] ^ (crc >> 8);
}
return crc ^ ~0U;
}
[[nodiscard]] constexpr uint32 Crc32(String str)
{
return Crc32(str.Str, str.Size);
}
[[nodiscard]] consteval uint32 operator""_crc32(const char* str, size_t length)
{
return Crc32(str, length);
}
3.3 Zero-Copy Tokenization Algorithm
The tokenization algorithm scans the memory buffer in a single pass. It first counts property keys to allocate the exact array size on the arena, then populates the TextPropertyNode array.
[[nodiscard]] inline String TrimWhitespace(String str)
{
while (str.Size > 0 && (*str.Str == ' ' || *str.Str == '\t' || *str.Str == '\r' || *str.Str == '\n'))
{
str.Str++;
str.Size--;
}
while (str.Size > 0 && (str.Str[str.Size - 1] == ' ' || str.Str[str.Size - 1] == '\t' ||
str.Str[str.Size - 1] == '\r' || str.Str[str.Size - 1] == '\n'))
{
str.Size--;
}
return str;
}
struct ParsedTextArchive
{
TextPropertyNode* Nodes = nullptr;
uint32 PropertyCount = 0;
};
[[nodiscard]] ParsedTextArchive TokenizeTextArchive(NonNullPtr<Arena> arena, ByteBuffer fileBuffer)
{
Assert(fileBuffer.Data != nullptr);
char* cursor = reinterpret_cast<char*>(fileBuffer.Data);
char* end = cursor + fileBuffer.Size;
// Pass 1: Count properties to allocate exactly on arena
uint32 propertyCount = 0;
char* scan = cursor;
while (scan < end)
{
if (*scan == ';')
{
if (scan == cursor || *(scan - 1) == '\n')
{
propertyCount++;
}
}
scan++;
}
if (propertyCount == 0)
{
return { nullptr, 0 };
}
auto* nodes = ArenaPushArray<TextPropertyNode>(arena, propertyCount);
Assert(nodes != nullptr);
// Pass 2: Extract keys and value slices
uint32 nodeIndex = 0;
scan = cursor;
while (scan < end && nodeIndex < propertyCount)
{
// Skip leading whitespace / empty lines / comments
while (scan < end && (*scan == '\r' || *scan == '\n' || *scan == ' ' || *scan == '\t'))
{
scan++;
}
if (scan >= end)
{
break;
}
// Check for comment line
if (*scan == '#' || (*scan == '/' && scan + 1 < end && *(scan + 1) == '/'))
{
while (scan < end && *scan != '\n')
{
scan++;
}
continue;
}
// Check for property declaration ';'
if (*scan == ';')
{
scan++; // skip ';'
// Extract key name
char* keyStart = scan;
while (scan < end && *scan != '\r' && *scan != '\n')
{
scan++;
}
String rawKey = { .Str = keyStart, .Size = static_cast<size_t>(scan - keyStart) };
String key = TrimWhitespace(rawKey);
// Skip newline after key
while (scan < end && (*scan == '\r' || *scan == '\n'))
{
scan++;
}
// Extract value block (everything until next ';' at start of line or EOF)
char* valStart = scan;
char* valEnd = scan;
while (scan < end)
{
if (*scan == ';' && (scan == cursor || *(scan - 1) == '\n'))
{
break;
}
scan++;
valEnd = scan;
}
String rawVal = { .Str = valStart, .Size = static_cast<size_t>(valEnd - valStart) };
String val = TrimWhitespace(rawVal);
nodes[nodeIndex].KeyCRC = Crc32(key);
nodes[nodeIndex].Value = val;
nodes[nodeIndex].Consumed = false;
nodeIndex++;
}
else
{
// Advance unexpected character
scan++;
}
}
return { nodes, nodeIndex };
}
3.4 Key Lookup & Unconsumed Key Audit
Lookup performs a fast linear scan over the contiguous TextPropertyNode array. Because typical game entities possess between 5 and 50 properties, a cache-coherent linear scan over contiguous 16-byte structs executes in single-digit nanoseconds, comfortably fitting within CPU L1/L2 data cache.
[[nodiscard]] inline TextPropertyNode* FindProperty(TextPropertyNode* nodes, uint32 count, uint32 keyCRC)
{
Assert(nodes != nullptr || count == 0);
for (uint32 index = 0; index < count; ++index)
{
if (nodes[index].KeyCRC == keyCRC)
{
nodes[index].Consumed = true;
return &nodes[index];
}
}
return nullptr;
}
#if JULIET_DEBUG
inline void AuditUnconsumedProperties(const TextPropertyNode* nodes, uint32 count, const char* contextName)
{
Assert(nodes != nullptr || count == 0);
for (uint32 index = 0; index < count; ++index)
{
if (!nodes[index].Consumed)
{
LogWarning(LogCategory::Core,
"[%s] Unconsumed or obsolete property detected: CRC 0x%08X (Value: '%.*s')",
contextName, nodes[index].KeyCRC,
static_cast<int>(nodes[index].Value.Size), nodes[index].Value.Str);
}
}
}
#endif
4. The archive Struct & Mode Handling
4.1 Struct Definition & Mode Flags
Juliet's existing archive struct in Core/Common/serialization.h is restricted to binary offsets and raw arena pointers. We upgrade archive into a unified serialization context supporting both text .jasset and binary streams.
enum class ArchiveMode : uint8
{
SavingText,
LoadingText,
SavingBinary,
LoadingBinary
};
struct TextPropertyNode;
struct archive
{
Arena* ArenaInstance = nullptr;
IOStream* Stream = nullptr;
TextPropertyNode* Properties = nullptr;
uint32 PropertyCount = 0;
ArchiveMode Mode = ArchiveMode::LoadingText;
uint32 BaseVersion = 0;
uint16 ClassVersion = 0;
// Legacy binary support fields
void* BasePtr = nullptr;
index_t Offset = 0;
[[nodiscard]] bool IsLoading() const
{
return Mode == ArchiveMode::LoadingText || Mode == ArchiveMode::LoadingBinary;
}
[[nodiscard]] bool IsSaving() const
{
return Mode == ArchiveMode::SavingText || Mode == ArchiveMode::SavingBinary;
}
[[nodiscard]] bool IsText() const
{
return Mode == ArchiveMode::LoadingText || Mode == ArchiveMode::SavingText;
}
[[nodiscard]] bool IsBinary() const
{
return Mode == ArchiveMode::LoadingBinary || Mode == ArchiveMode::SavingBinary;
}
};
4.2 Output Stream Formatting via IOPrintf
When saving in ArchiveMode::SavingText, archive outputs directly to an open IOStream using IOPrintf.
inline void WritePropertyHeader(archive& ar, const char* keyName)
{
Assert(ar.IsSaving());
Assert(ar.Stream != nullptr);
Assert(keyName != nullptr);
IOPrintf(ar.Stream, "; %s\n", keyName);
}
This design provides:
- Direct stream output with zero heap buffer allocations.
- Canonical spacing and formatting across all entity serializers.
- Formatted outputs immediately flushed or buffered according to
IOStreamInterfaceconfiguration.
5. SerializeProp API & Implementation
5.1 Unified Serialization Idiom
The SerializeProp function family encapsulates both loading and saving behind a single call. If an asset file lacks a given property (e.g. an older asset file loaded by newer code), SerializeProp returns false during loading, and the destination variable retains its existing default value.
#define SERIALIZE_PROP(ar, var) SerializeProp((ar), #var, #var##_crc32, (var))
5.2 Primitive Type Helpers
Float (float)
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, float& value)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
IOPrintf(ar.Stream, "%f\n\n", value);
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
// Fast float conversion from String slice
char buffer[64];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
char* endPtr = nullptr;
float parsed = strtof(buffer, &endPtr);
if (endPtr != buffer)
{
value = parsed;
return true;
}
return false;
}
32-Bit Signed Integer (int32)
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, int32& value)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
IOPrintf(ar.Stream, "%d\n\n", value);
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
char buffer[32];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
char* endPtr = nullptr;
int32 parsed = static_cast<int32>(strtol(buffer, &endPtr, 10));
if (endPtr != buffer)
{
value = parsed;
return true;
}
return false;
}
64-Bit Unsigned Integer (uint64, EntityID)
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, uint64& value)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
IOPrintf(ar.Stream, "0x%016llX\n\n", value);
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
char buffer[32];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
char* endPtr = nullptr;
int base = (buffer[0] == '0' && (buffer[1] == 'x' || buffer[1] == 'X')) ? 16 : 10;
uint64 parsed = strtoull(buffer, &endPtr, base);
if (endPtr != buffer)
{
value = parsed;
return true;
}
return false;
}
Boolean (bool)
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, bool& value)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
IOPrintf(ar.Stream, "%s\n\n", value ? "true" : "false");
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
if (StringCompare(node->Value, WrapString("true")) == 0 ||
StringCompare(node->Value, WrapString("1")) == 0)
{
value = true;
return true;
}
if (StringCompare(node->Value, WrapString("false")) == 0 ||
StringCompare(node->Value, WrapString("0")) == 0)
{
value = false;
return true;
}
return false;
}
5.3 Vector Helpers (float x, y, z)
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, float& x, float& y, float& z)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
IOPrintf(ar.Stream, "%f %f %f\n\n", x, y, z);
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
char buffer[128];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
float parsedX = 0.0f;
float parsedY = 0.0f;
float parsedZ = 0.0f;
int matches = sscanf_s(buffer, "%f %f %f", &parsedX, &parsedY, &parsedZ);
if (matches == 3)
{
x = parsedX;
y = parsedY;
z = parsedZ;
return true;
}
return false;
}
5.4 String Helpers
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, String& value)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
bool hasSpace = ContainsChar(value, ' ');
if (hasSpace)
{
IOPrintf(ar.Stream, "\"%.*s\"\n\n", static_cast<int>(value.Size), value.Str);
}
else
{
IOPrintf(ar.Stream, "%.*s\n\n", static_cast<int>(value.Size), value.Str);
}
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
String parsed = node->Value;
// Strip optional surrounding double quotes
if (parsed.Size >= 2 && parsed.Str[0] == '"' && parsed.Str[parsed.Size - 1] == '"')
{
parsed.Str++;
parsed.Size -= 2;
}
// Allocate persistent string copy on the archive arena
Assert(ar.ArenaInstance != nullptr);
value = StringCopy(ar.ArenaInstance, parsed);
return true;
}
6. Two-Tier Versioning Architecture
6.1 Architectural Rationale: Base vs Derived Decoupling
In entity-component systems or object-oriented engine hierarchies, entities consist of two distinct domains:
- Core Engine Identity (Base Entity): Position, Rotation, Scale, EntityID, Class Kind, Render Flags, Layer Masks. Managed by engine architects.
- Gameplay Specialization (Derived Class): Ammo, Health, AI State, Mesh Instance ID, Patrol Paths. Managed by gameplay programmers.
The Fragility of Monolithic Versioning
In naive serialization architectures, a single uint32 Version governs the entire file. When an engine programmer updates the base Entity struct (e.g. adding a uint32 LayerMask), bumping the global version invalidates or forces schema changes across every single derived entity type in the project.
The Two-Tier Solution
Juliet decouples versioning into two independent tiers:
- Tier 1: Base Engine Version (
kEntityBaseVersion): Declared centrally inEntity.h. GovernsEntitybase fields. - Tier 2: Derived Class Version (
Class::Version): Declared per-entity class inClass.hand initialized inDEFINE_ENTITY_VERSIONED.
========================================================================
.jasset Text Archive
========================================================================
; AssetType
Entity
; BaseVersion ------> Governed by kEntityBaseVersion in Entity.h
1
; EntityID
0x0000000000000001
; Kind
Inert
; Position
0.0 0.0 0.0
------------------------------------------------------------------------
; ClassVersion ------> Governed by Class::Version in Class.h
2
; MeshInstance
42
========================================================================
When an engine programmer bumps kEntityBaseVersion from 1 to 2 to add LayerMask, no derived game classes (Inert, Monster, Vehicle) need version increments or code modifications.
6.2 Implementation Details
Engine Base Version (Game/Entity/Entity.h)
constexpr uint32 kEntityBaseVersion = 1;
void SerializeEntityBase(archive& ar, NonNullPtr<Entity> entity);
Derived Class Version (Juliet/include/Engine/Class.h)
struct Class
{
uint32 CRC;
uint8 kind;
uint16 Version; // Added derived class version
serialize_fct_type serialize_fct;
size_t size_of;
size_t alignment;
#if JULIET_DEBUG
String Name;
#endif
};
consteval Class MakeClass(String name, uint8 kind, uint16 version, size_t size, size_t align, serialize_fct_type fct)
{
Class cls = {};
cls.CRC = crc32(name.Str, name.Size);
cls.kind = kind;
cls.Version = version;
cls.size_of = size;
cls.alignment = align;
cls.serialize_fct = fct;
#if JULIET_DEBUG
cls.Name = name;
#endif
return cls;
}
Class Registration Macros (Game/Entity/Entity.h)
#define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct) \
Class entityKind##entity = \
MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version), sizeof(entity), alignof(entity), serialize_fct); \
Class* entity::Kind = &entityKind##entity;
6.3 Execution Flow in SerializeEntity
void Serialize(archive& ar, NonNullPtr<Entity> entity)
{
Assert(entity.Get() != nullptr);
// --- Tier 1: Base Entity Serialization ---
if (ar.IsSaving())
{
uint32 baseVer = kEntityBaseVersion;
SERIALIZE_PROP(ar, baseVer);
SERIALIZE_PROP(ar, entity->ID);
String kindStr = WrapString(kEntity_type_names[entity->Kind->kind]);
SerializeProp(ar, "Kind", "Kind"_crc32, kindStr);
SerializeProp(ar, "Position", "Position"_crc32, entity->X, entity->Y, entity->Z);
}
else
{
uint32 baseVer = 0;
if (!SerializeProp(ar, "BaseVersion", "BaseVersion"_crc32, baseVer))
{
baseVer = 1; // Default to initial schema if absent
}
ar.BaseVersion = baseVer;
SERIALIZE_PROP(ar, entity->ID);
String kindStr = {};
if (SerializeProp(ar, "Kind", "Kind"_crc32, kindStr))
{
// Resolve class pointer from name
for (uint8 i = 0; i < ToUnderlying(Entity_Type::Count); ++i)
{
if (StringCompare(kindStr, WrapString(kEntity_type_names[i])) == 0)
{
entity->Kind = kEntity_type_class_ptr[i];
break;
}
}
}
Assert(entity->Kind != nullptr);
SerializeProp(ar, "Position", "Position"_crc32, entity->X, entity->Y, entity->Z);
}
// --- Tier 2: Derived Entity Serialization ---
if (entity->Kind->serialize_fct != nullptr && entity->Derived != nullptr)
{
if (ar.IsSaving())
{
uint32 classVer = entity->Kind->Version;
SerializeProp(ar, "ClassVersion", "ClassVersion"_crc32, classVer);
}
else
{
uint32 classVer = 0;
if (!SerializeProp(ar, "ClassVersion", "ClassVersion"_crc32, classVer))
{
classVer = entity->Kind->Version;
}
ar.ClassVersion = static_cast<uint16>(classVer);
}
entity->Kind->serialize_fct(&ar, entity->Derived);
}
}
7. Post-Serialization Deprecation & Migration
7.1 Schema Evolution Challenge
Over the lifecycle of a game, gameplay mechanics evolve:
- A scalar float
Speedis replaced by a directional 2D vectorVelocity. - A single texture index
TextureIDis replaced by an asset path stringDiffuseTexture. - Obsolete properties are deleted entirely.
Retaining deprecated members in active C++ structs creates code clutter, wastes memory, and invites bugs.
7.2 Deprecation Primitives (ReadDeprecated*)
Deprecation primitives allow serializers to ingest obsolete properties exclusively during loading without polluting modern structs or writing deprecated keys back to disk during saving.
bool ReadDeprecated(archive& ar, const char* keyName, uint32 keyCRC, float& outVal)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
return false; // Deprecated fields are never saved
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
char buffer[64];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
char* endPtr = nullptr;
float parsed = strtof(buffer, &endPtr);
if (endPtr != buffer)
{
outVal = parsed;
return true;
}
return false;
}
bool ReadDeprecatedVec2(archive& ar, const char* keyName, uint32 keyCRC, float& outX, float& outY)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
return false;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
char buffer[128];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
float x = 0.0f;
float y = 0.0f;
if (sscanf_s(buffer, "%f %f", &x, &y) == 2)
{
outX = x;
outY = y;
return true;
}
return false;
}
bool ReadDeprecatedString(archive& ar, const char* keyName, uint32 keyCRC, NonNullPtr<Arena> arena, String& outVal)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
return false;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
String parsed = node->Value;
if (parsed.Size >= 2 && parsed.Str[0] == '"' && parsed.Str[parsed.Size - 1] == '"')
{
parsed.Str++;
parsed.Size -= 2;
}
outVal = StringCopy(arena, parsed);
return true;
}
7.3 In-Place Migration Pattern
When an asset file with an older ClassVersion is loaded, the derived serializer detects ar.ClassVersion < N, calls ReadDeprecated* to read obsolete fields, maps the legacy data into the modern struct, and continues. On the subsequent save, the asset file is emitted using the modern schema without deprecated keys.
struct Projectile
{
DECLARE_ENTITY()
// Modern Schema (v2)
float VelocityX = 0.0f;
float VelocityY = 0.0f;
float Damage = 50.0f;
};
void SerializeProjectile(archive* arPtr, void* payload)
{
Assert(arPtr != nullptr);
Assert(payload != nullptr);
auto& ar = *arPtr;
auto* projectile = static_cast<Projectile*>(payload);
if (ar.IsSaving())
{
SERIALIZE_PROP(ar, projectile->VelocityX);
SERIALIZE_PROP(ar, projectile->VelocityY);
SERIALIZE_PROP(ar, projectile->Damage);
}
else
{
SERIALIZE_PROP(ar, projectile->Damage);
if (ar.ClassVersion < 2)
{
// Migration from v1: scalar 'Speed' converted to 'VelocityX'
float legacySpeed = 0.0f;
if (ReadDeprecated(ar, "Speed", "Speed"_crc32, legacySpeed))
{
projectile->VelocityX = legacySpeed;
projectile->VelocityY = 0.0f;
}
}
else
{
SERIALIZE_PROP(ar, projectile->VelocityX);
SERIALIZE_PROP(ar, projectile->VelocityY);
}
}
}
8. Step-by-Step Implementation Roadmap & Unit Testing Plan
8.1 Implementation Roadmap
+-------------------------------------------------------------------------+
| Phase 1: Core Utilities & Dual-Mode CRC32 |
| - Update CRC32.h with constexpr Crc32(String) and Crc32(char*, len) |
| - Extend struct Class in Class.h with uint16 Version |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Phase 2: In-Memory Zero-Copy Parser |
| - Implement TextArchiveParser.h / .cpp |
| - TokenizeTextArchive, TrimWhitespace, FindProperty |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Phase 3: Archive Struct & SerializeProp Helpers |
| - Upgrade archive in serialization.h with ArchiveMode |
| - Implement primitive, vector, and string SerializeProp helpers |
| - Implement ReadDeprecated* primitives |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Phase 4: Entity & World Integration |
| - Update Entity.h / Entity.cpp with two-tier serialization |
| - Refactor World.cpp to load/save .jasset files |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Phase 5: Verification & Unit Testing Suite |
| - Build and run SerializationUnitTest.cpp |
+-------------------------------------------------------------------------+
Phase 1: Core Utilities & Dual-Mode CRC32
- Target:
Juliet/include/Core/Common/CRC32.h- Add
constexpr uint32 Crc32(const char* str, size_t length)andconstexpr uint32 Crc32(String str). - Ensure existing
operator""_crc32callsCrc32.
- Add
- Target:
Juliet/include/Engine/Class.h- Add
uint16 Versiontostruct Class. - Update
MakeClassto acceptuint16 version = 1.
- Add
Phase 2: In-Memory Zero-Copy Parser
- Target:
Juliet/include/Core/Common/TextArchiveParser.h(andsrc/Core/Common/TextArchiveParser.cpp)- Define
struct TextPropertyNode { uint32 KeyCRC; String Value; bool Consumed; }. - Implement
TokenizeTextArchive(NonNullPtr<Arena> arena, ByteBuffer buffer). - Implement
FindPropertyandAuditUnconsumedProperties.
- Define
Phase 3: Archive Struct & SerializeProp Helpers
- Target:
Juliet/include/Core/Common/serialization.h- Introduce
enum class ArchiveMode : uint8. - Upgrade
struct archivewith stream pointer, property array, mode, and versions. - Implement overloaded
SerializePropforfloat,int32,uint64,bool, vectors, andString. - Implement
ReadDeprecated,ReadDeprecatedVec2,ReadDeprecatedString.
- Introduce
Phase 4: Entity & World Integration
- Target:
Game/Entity/Entity.handGame/Entity/Entity.cpp- Define
constexpr uint32 kEntityBaseVersion = 1;. - Update
DEFINE_ENTITYand addDEFINE_ENTITY_VERSIONED. - Implement
Serialize(archive& ar, NonNullPtr<Entity> entity)supporting.jassettext format.
- Define
- Target:
Game/Data/World.handGame/Data/World.cpp- Implement text-based world saving and loading using
.jassetformatting.
- Implement text-based world saving and loading using
Phase 5: Comprehensive Unit Testing
- Target:
Game/UnitTest/SerializationUnitTest.handGame/UnitTest/SerializationUnitTest.cpp- Add unit tests verifying parsing, round-trip serialization, defaults preservation, versioning, and deprecation.
8.2 Comprehensive Unit Testing Plan (SerializationUnitTest.cpp)
The unit test suite validates all architectural requirements without modifying engine framework code for test-specific cases.
// Game/UnitTest/SerializationUnitTest.h
#pragma once
#include <Juliet.h>
#if JULIET_DEBUG
namespace UnitTest
{
void RunSerializationUnitTests();
}
#endif
// Game/UnitTest/SerializationUnitTest.cpp
#include <UnitTest/SerializationUnitTest.h>
#if JULIET_DEBUG
#include <Core/Common/CRC32.h>
#include <Core/Common/String.h>
#include <Core/Common/serialization.h>
#include <Core/HAL/IO/IOStream.h>
#include <Core/Logging/LogManager.h>
#include <Core/Logging/LogTypes.h>
#include <Core/Memory/MemoryArena.h>
#include <Core/Thread/ThreadContext.h>
#include <Entity/Entity.h>
namespace UnitTest
{
// Test Struct for Derived Entity Testing
struct DummyVehicle
{
DECLARE_ENTITY()
float MaxSpeed = 120.0f;
int32 GearCount = 6;
uint64 ChassisUUID = 0xABCDEF0123456789ULL;
bool Turbo = true;
float PosX = 10.0f;
float PosY = 20.0f;
float PosZ = 30.0f;
};
void SerializeDummyVehicle(archive* arPtr, void* payload)
{
Assert(arPtr != nullptr);
Assert(payload != nullptr);
auto& ar = *arPtr;
auto* vehicle = static_cast<DummyVehicle*>(payload);
SERIALIZE_PROP(ar, vehicle->MaxSpeed);
SERIALIZE_PROP(ar, vehicle->GearCount);
SERIALIZE_PROP(ar, vehicle->ChassisUUID);
SERIALIZE_PROP(ar, vehicle->Turbo);
SerializeProp(ar, "Position", "Position"_crc32, vehicle->PosX, vehicle->PosY, vehicle->PosZ);
}
DEFINE_ENTITY_VERSIONED(DummyVehicle, 1, SerializeDummyVehicle);
// Test 1: Parser tokenization with whitespace, comments, and mixed line endings
static void TestParserTokenization()
{
TempArena temp = scratch_begin(nullptr, 0);
const char* testContent =
"# Header Comment\r\n"
"// Secondary comment\n"
"\n"
"; Health\r\n"
" 100.500000 \r\n"
"\n"
"; Name\n"
"\"Paladin Hero\"\n"
"\n"
"; Position\r\n"
"1.0 2.0 3.0\r\n";
ByteBuffer buffer = {
.Data = reinterpret_cast<Byte*>(const_cast<char*>(testContent)),
.Size = strlen(testContent)
};
ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer);
Assert(parsed.PropertyCount == 3);
auto* healthNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Health"_crc32);
Assert(healthNode != nullptr);
Assert(StringCompare(healthNode->Value, WrapString("100.500000")) == 0);
auto* nameNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Name"_crc32);
Assert(nameNode != nullptr);
Assert(StringCompare(nameNode->Value, WrapString("\"Paladin Hero\"")) == 0);
auto* posNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Position"_crc32);
Assert(posNode != nullptr);
Assert(StringCompare(posNode->Value, WrapString("1.0 2.0 3.0")) == 0);
scratch_end(temp);
LogMessage(LogCategory::Core, "TestParserTokenization passed.");
}
// Test 2: Missing properties retain default struct values
static void TestDefaultValueRetention()
{
TempArena temp = scratch_begin(nullptr, 0);
const char* incompleteContent =
"; MaxSpeed\n"
"180.0\n";
ByteBuffer buffer = {
.Data = reinterpret_cast<Byte*>(const_cast<char*>(incompleteContent)),
.Size = strlen(incompleteContent)
};
ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer);
archive ar = {};
ar.ArenaInstance = temp.Arena;
ar.Mode = ArchiveMode::LoadingText;
ar.Properties = parsed.Nodes;
ar.PropertyCount = parsed.PropertyCount;
DummyVehicle vehicle;
// Defaults: MaxSpeed=120, GearCount=6, Turbo=true
SerializeDummyVehicle(&ar, &vehicle);
Assert(vehicle.MaxSpeed == 180.0f); // Overwritten by archive
Assert(vehicle.GearCount == 6); // Preserved default
Assert(vehicle.Turbo == true); // Preserved default
Assert(vehicle.PosX == 10.0f); // Preserved default
scratch_end(temp);
LogMessage(LogCategory::Core, "TestDefaultValueRetention passed.");
}
// Test 3: Deprecation migration from v1 to v2
struct LegacyWeapon
{
DECLARE_ENTITY()
float VelocityX = 0.0f;
float VelocityY = 0.0f;
};
void SerializeLegacyWeapon(archive* arPtr, void* payload)
{
Assert(arPtr != nullptr);
Assert(payload != nullptr);
auto& ar = *arPtr;
auto* weapon = static_cast<LegacyWeapon*>(payload);
if (ar.IsSaving())
{
SERIALIZE_PROP(ar, weapon->VelocityX);
SERIALIZE_PROP(ar, weapon->VelocityY);
}
else
{
if (ar.ClassVersion < 2)
{
float oldSpeed = 0.0f;
if (ReadDeprecated(ar, "Speed", "Speed"_crc32, oldSpeed))
{
weapon->VelocityX = oldSpeed;
weapon->VelocityY = 0.0f;
}
}
else
{
SERIALIZE_PROP(ar, weapon->VelocityX);
SERIALIZE_PROP(ar, weapon->VelocityY);
}
}
}
DEFINE_ENTITY_VERSIONED(LegacyWeapon, 2, SerializeLegacyWeapon);
static void TestDeprecationMigration()
{
TempArena temp = scratch_begin(nullptr, 0);
// Simulated v1 file containing obsolete 'Speed'
const char* v1Content =
"; ClassVersion\n"
"1\n"
"; Speed\n"
"75.500000\n";
ByteBuffer buffer = {
.Data = reinterpret_cast<Byte*>(const_cast<char*>(v1Content)),
.Size = strlen(v1Content)
};
ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer);
archive ar = {};
ar.ArenaInstance = temp.Arena;
ar.Mode = ArchiveMode::LoadingText;
ar.Properties = parsed.Nodes;
ar.PropertyCount = parsed.PropertyCount;
ar.ClassVersion = 1;
LegacyWeapon weapon;
SerializeLegacyWeapon(&ar, &weapon);
Assert(weapon.VelocityX == 75.5f);
Assert(weapon.VelocityY == 0.0f);
scratch_end(temp);
LogMessage(LogCategory::Core, "TestDeprecationMigration passed.");
}
void RunSerializationUnitTests()
{
LogMessage(LogCategory::Core, "=== Running Serialization & .jasset Unit Tests ===");
TestParserTokenization();
TestDefaultValueRetention();
TestDeprecationMigration();
LogMessage(LogCategory::Core, "=== All Serialization Unit Tests Passed Successfully ===");
}
} // namespace UnitTest
#endif