44 KiB
44 KiB
Juliet Game Engine: Serialization Core & .jasset Text Archive
Technical Specification & Implementation Plan
Document ID: SPEC-001-SERIALIZATION-CORE
Component: Juliet Engine Core / Asset Pipeline
Target Architecture: Juliet Game Engine (C++20, x64, D3D12)
File Location: Game/Plans/01_Serialization_And_Text_Archive.md
1. Executive Summary & Architecture Goals
1.1 Context & Current Limitations
The Juliet game engine previously utilized packed binary records for level and entity serialization (Assets/world.bin, WorldFileHeader, WorldEntityDiskRecord). While packed binary formats are compact, they present severe architectural roadblocks during collaborative development:
- Merge Incompatibility: Binary assets cannot be merged or diffed across version control systems (Git / Perforce). Concurrent edits by multiple designers or engineers result in unresolvable binary conflicts and data loss.
- Schema Rigidity: Adding, removing, or reordering a single struct field invalidates all previously serialized binary blobs unless complex, manual byte-offset mapping tables are maintained.
- Opacity: Programmers and technical artists cannot inspect, debug, or patch asset properties in a standard text editor.
1.2 Architectural Goals
The .jasset text archive framework is engineered to replace legacy binary blobs with a robust, human-readable, diff-friendly property serialization pipeline while strictly adhering to Juliet's performance and memory constraints:
- Zero Dynamic Heap Allocations: All parsing, tokenization, formatting, and buffer transformations execute entirely within Juliet memory arenas (
Arena,TempArena,scratch_begin/scratch_end). Standard library containers (std::string,std::vector,std::map) and raw heap allocators (malloc,new) are forbidden. - Zero-Copy In-Memory Tokenization: Files are loaded into arena memory once via
LoadFile. The tokenizer parses properties into lightweight slices represented by Juliet'sString(char* Str; size_t Size;), referencing existing file buffer bytes without string replication or null-termination overhead. O(1)Property Lookup via Compile-Time & Runtime CRC32: Property keys in text files are hashed once during tokenization into 32-bit CRC values. In code, lookups utilize compile-time hashed literals (operator""_crc32). Property resolution reduces to a single 32-bit integer comparison, avoiding string comparison overhead in serialization loops.- Symmetric Single-Function Serialization: A single
Serializeimplementation per entity or data structure handles both Save and Load paths, guaranteeing that write and read schemas never diverge. - Graceful Forward/Backward Compatibility: Missing keys during load automatically preserve struct default values. Unknown keys present in newer asset files are safely ignored without parse failure.
- Two-Tier Decoupled Versioning: Core engine entity properties (
kEntityBaseVersion) and derived gameplay class properties (Class::Version) are versioned independently. Engine-level updates never bump derived entity class versions. - In-Place Schema Migration: Deprecated properties no longer present in modern structs can be read during loading into local stack variables using standard serialization (
serialize/SERIALIZE) guarded by version checks (if (ar.loading && ar.class_version < N)), seamlessly transforming legacy values without struct pollution or persisting obsolete keys on subsequent saves. - Zero Exceptions & Total Warning Cleanliness: Fully conforming to Juliet coding guidelines: strict assertions (
Assert),[[nodiscard]],auto*/auto&, mandatory braces, andstatic_cast/reinterpret_cast.
2. The .jasset Text Format Specification
2.1 Grammar & Structural Rules
The .jasset format uses a line-oriented, key-value property hierarchy designed for visual clarity and clean Git diffs.
AssetFile ::= { CommentLine | EmptyLine | PropertyDeclaration } ;
CommentLine ::= ( "#" | "//" ) { Character } LineEnding ;
EmptyLine ::= { Whitespace } LineEnding ;
PropertyDeclaration ::= KeyHeader LineEnding ValueBlock ;
KeyHeader ::= ";" { Whitespace } Identifier ;
ValueBlock ::= { ValueLine LineEnding } ;
ValueLine ::= { Whitespace } ValueString { Whitespace } ;
LineEnding ::= "\r\n" | "\n" ;
Identifier ::= [a-zA-Z_][a-zA-Z0-9_]* ;
- Property Key Header (
;\n):- Every property declaration begins with a semicolon
;as the first non-whitespace character. - The semicolon is followed by optional whitespace and a CamelCase identifier:
; PropertyName. - Keys are case-sensitive and must be valid C++ identifier tokens.
- Every property declaration begins with a semicolon
- Value Block:
- The line immediately following the property header contains the property's serialized payload.
- For multi-line values or arrays, lines continue until the next property header
;or end-of-file.
- Comments & Ignored Tokens:
- Any line starting with
#or//(after optional leading whitespace) is treated as a comment. - Empty lines and lines containing only whitespace are ignored.
- Any line starting with
- Line Termination Handling:
- The parser natively accepts both Windows (
\r\n) and POSIX (\n) line endings. - Trailing carriage returns (
\r) are automatically stripped during tokenization.
- The parser natively accepts both Windows (
- Whitespace Tolerance:
- Leading and trailing spaces and horizontal tabs (
\t) on both key and value lines are stripped during tokenization.
- Leading and trailing spaces and horizontal tabs (
2.2 Formatting Specifications & Examples
Scalar Types
- Floating Point (
float,float32): Formatted via%f(default 6 decimal digits) or%.9gfor full single-precision round-trip fidelity.; Health 100.000000 ; Mass 14.250000 - 32-Bit Signed Integer (
int32): Formatted as decimal integer%d.; AmmoCount 45 ; TeamIndex -1 - 64-Bit Unsigned Integer (
uint64,EntityID): Formatted as a 16-character padded hexadecimal integer prefixed with0x. Hexadecimal ensures 64-bit handle readability and exact bit-pattern preservation.; EntityID 0x00000000DEADBEEF ; LayerMask 0x0000000000000001 - Boolean (
bool): Formatted astrueorfalse. For backwards tolerance, the parser also accepts1and0.; IsStatic true ; CastShadows false
Vector Types
- 3D Vector (
Vector3/float x, y, z): Formatted as three space-delimited floating-point values on a single line.; Position 10.500000 0.000000 -25.250000 ; Scale 1.000000 1.000000 1.000000 - 2D Vector (
Vector2/float x, y): Formatted as two space-delimited floating-point values.; UVOffset 0.000000 0.500000
String Types
- String (
String/String8):- Strings without spaces can be serialized as raw text tokens.
- Strings containing spaces or symbols are enclosed in double quotes
"...".
; AssetName Character_Mesh_Hero ; DisplayName "Grand Citadel Knight"
Comprehensive Entity Asset File Example (Entity_01.jasset)
# Juliet Game Engine Asset File
# Generated automatically by AssetPipeline. Do not manually corrupt keys.
; AssetType
Entity
; Version
1
; EntityID
0x0000000000000042
; Kind
Inert
; Position
100.000000 25.500000 -50.000000
; ClassVersion
2
; MeshInstance
4
; MaterialOverride
"Materials/M_Granite_Polished"
; IsVisible
true
3. The In-Memory Zero-Copy Parser
3.1 Memory Layout & Property Nodes
To eliminate heap fragmentation and per-object allocation overhead during level loading, the parser loads the entire .jasset file into contiguous arena memory via LoadFile. The tokenizer then constructs a flat array of TextPropertyNode structures on a TempArena.
struct TextPropertyNode
{
uint32 KeyCRC;
String Value;
bool Consumed;
};
KeyCRC: The 32-bit CRC hash of the trimmed property name.Value: AString(String8) struct containing achar* Strpointer directly into the file buffer andsize_t Size. No new string allocations are performed.Consumed: A boolean flag initialized tofalse. When a property is queried and read viaSerializeProp,Consumedis set totrue.
File Buffer in Arena:
+-------------------------------------------------------------------------------+
| ; Position\n10.0 20.0 30.0\n; Health\n100.0\n |
+-------------------------------------------------------------------------------+
^ ^ ^ ^
| | | |
TextPropertyNode[0]: | | |
KeyCRC: CRC("Position") | |
Value: { Str --------+ , Size: 14 } |
Consumed: true |
|
TextPropertyNode[1]: |
KeyCRC: CRC("Health") |
Value: { Str -------------------------------+ , Size: 5 }
Consumed: true
3.2 Dual-Mode CRC32: Compile-Time & Runtime
Juliet's CRC32.h defines consteval uint32 crc32(const char* str, size_t length). However, consteval guarantees compilation failure when called on runtime input (such as keys extracted from a text file during parsing).
The engine requires a unified Crc32 implementation that functions at runtime for parsed text tokens while retaining constexpr / consteval capability for compile-time string literals.
// Core/Common/CRC32.h additions
[[nodiscard]] constexpr uint32 Crc32(const char* str, size_t length)
{
Assert(str != nullptr || length == 0);
const char* p = str;
uint32 crc = ~0U;
while (length--)
{
crc = details::crc32_tab[(crc ^ static_cast<uint8>(*p++)) & 0xFF] ^ (crc >> 8);
}
return crc ^ ~0U;
}
[[nodiscard]] constexpr uint32 Crc32(String str)
{
return Crc32(str.Str, str.Size);
}
[[nodiscard]] consteval uint32 operator""_crc32(const char* str, size_t length)
{
return Crc32(str, length);
}
3.3 Zero-Copy Tokenization Algorithm
The tokenization algorithm scans the memory buffer in a single pass. It first counts property keys to allocate the exact array size on the arena, then populates the TextPropertyNode array.
[[nodiscard]] inline String TrimWhitespace(String str)
{
while (str.Size > 0 && (*str.Str == ' ' || *str.Str == '\t' || *str.Str == '\r' || *str.Str == '\n'))
{
str.Str++;
str.Size--;
}
while (str.Size > 0 && (str.Str[str.Size - 1] == ' ' || str.Str[str.Size - 1] == '\t' ||
str.Str[str.Size - 1] == '\r' || str.Str[str.Size - 1] == '\n'))
{
str.Size--;
}
return str;
}
struct ParsedTextArchive
{
TextPropertyNode* Nodes = nullptr;
uint32 PropertyCount = 0;
};
[[nodiscard]] ParsedTextArchive TokenizeTextArchive(NonNullPtr<Arena> arena, ByteBuffer fileBuffer)
{
Assert(fileBuffer.Data != nullptr);
char* cursor = reinterpret_cast<char*>(fileBuffer.Data);
char* end = cursor + fileBuffer.Size;
// Pass 1: Count properties to allocate exactly on arena
uint32 propertyCount = 0;
char* scan = cursor;
while (scan < end)
{
if (*scan == ';')
{
if (scan == cursor || *(scan - 1) == '\n')
{
propertyCount++;
}
}
scan++;
}
if (propertyCount == 0)
{
return { nullptr, 0 };
}
auto* nodes = ArenaPushArray<TextPropertyNode>(arena, propertyCount);
Assert(nodes != nullptr);
// Pass 2: Extract keys and value slices
uint32 nodeIndex = 0;
scan = cursor;
while (scan < end && nodeIndex < propertyCount)
{
// Skip leading whitespace / empty lines / comments
while (scan < end && (*scan == '\r' || *scan == '\n' || *scan == ' ' || *scan == '\t'))
{
scan++;
}
if (scan >= end)
{
break;
}
// Check for comment line
if (*scan == '#' || (*scan == '/' && scan + 1 < end && *(scan + 1) == '/'))
{
while (scan < end && *scan != '\n')
{
scan++;
}
continue;
}
// Check for property declaration ';'
if (*scan == ';')
{
scan++; // skip ';'
// Extract key name
char* keyStart = scan;
while (scan < end && *scan != '\r' && *scan != '\n')
{
scan++;
}
String rawKey = { .Str = keyStart, .Size = static_cast<size_t>(scan - keyStart) };
String key = TrimWhitespace(rawKey);
// Skip newline after key
while (scan < end && (*scan == '\r' || *scan == '\n'))
{
scan++;
}
// Extract value block (everything until next ';' at start of line or EOF)
char* valStart = scan;
char* valEnd = scan;
while (scan < end)
{
if (*scan == ';' && (scan == cursor || *(scan - 1) == '\n'))
{
break;
}
scan++;
valEnd = scan;
}
String rawVal = { .Str = valStart, .Size = static_cast<size_t>(valEnd - valStart) };
String val = TrimWhitespace(rawVal);
nodes[nodeIndex].KeyCRC = Crc32(key);
nodes[nodeIndex].Value = val;
nodes[nodeIndex].Consumed = false;
nodeIndex++;
}
else
{
// Advance unexpected character
scan++;
}
}
return { nodes, nodeIndex };
}
3.4 Key Lookup & Unconsumed Key Audit
Lookup performs a fast linear scan over the contiguous TextPropertyNode array. Because typical game entities possess between 5 and 50 properties, a cache-coherent linear scan over contiguous 16-byte structs executes in single-digit nanoseconds, comfortably fitting within CPU L1/L2 data cache.
[[nodiscard]] inline TextPropertyNode* FindProperty(TextPropertyNode* nodes, uint32 count, uint32 keyCRC)
{
Assert(nodes != nullptr || count == 0);
for (uint32 index = 0; index < count; ++index)
{
if (nodes[index].KeyCRC == keyCRC)
{
nodes[index].Consumed = true;
return &nodes[index];
}
}
return nullptr;
}
#if JULIET_DEBUG
inline void AuditUnconsumedProperties(const TextPropertyNode* nodes, uint32 count, const char* contextName)
{
Assert(nodes != nullptr || count == 0);
for (uint32 index = 0; index < count; ++index)
{
if (!nodes[index].Consumed)
{
LogWarning(LogCategory::Core,
"[%s] Unconsumed or obsolete property detected: CRC 0x%08X (Value: '%.*s')",
contextName, nodes[index].KeyCRC,
static_cast<int>(nodes[index].Value.Size), nodes[index].Value.Str);
}
}
}
#endif
4. The archive Struct & Mode Handling
4.1 Struct Definition & Mode Flags
Juliet's existing archive struct in Core/Common/serialization.h is restricted to binary offsets and raw arena pointers. We upgrade archive into a unified serialization context supporting both text .jasset and binary streams.
enum class ArchiveMode : uint8
{
SavingText,
LoadingText,
SavingBinary,
LoadingBinary
};
struct TextPropertyNode;
struct archive
{
Arena* ArenaInstance = nullptr;
IOStream* Stream = nullptr;
TextPropertyNode* Properties = nullptr;
uint32 PropertyCount = 0;
ArchiveMode Mode = ArchiveMode::LoadingText;
uint32 BaseVersion = 0;
uint16 ClassVersion = 0;
// Legacy binary support fields
void* BasePtr = nullptr;
index_t Offset = 0;
[[nodiscard]] bool IsLoading() const
{
return Mode == ArchiveMode::LoadingText || Mode == ArchiveMode::LoadingBinary;
}
[[nodiscard]] bool IsSaving() const
{
return Mode == ArchiveMode::SavingText || Mode == ArchiveMode::SavingBinary;
}
[[nodiscard]] bool IsText() const
{
return Mode == ArchiveMode::LoadingText || Mode == ArchiveMode::SavingText;
}
[[nodiscard]] bool IsBinary() const
{
return Mode == ArchiveMode::LoadingBinary || Mode == ArchiveMode::SavingBinary;
}
};
4.2 Output Stream Formatting via IOPrintf
When saving in ArchiveMode::SavingText, archive outputs directly to an open IOStream using IOPrintf.
inline void WritePropertyHeader(archive& ar, const char* keyName)
{
Assert(ar.IsSaving());
Assert(ar.Stream != nullptr);
Assert(keyName != nullptr);
IOPrintf(ar.Stream, "; %s\n", keyName);
}
This design provides:
- Direct stream output with zero heap buffer allocations.
- Canonical spacing and formatting across all entity serializers.
- Formatted outputs immediately flushed or buffered according to
IOStreamInterfaceconfiguration.
5. SerializeProp API & Implementation
5.1 Unified Serialization Idiom
The SerializeProp function family encapsulates both loading and saving behind a single call. If an asset file lacks a given property (e.g. an older asset file loaded by newer code), SerializeProp returns false during loading, and the destination variable retains its existing default value.
#define SERIALIZE_PROP(ar, var) SerializeProp((ar), #var, #var##_crc32, (var))
5.2 Primitive Type Helpers
Float (float)
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, float& value)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
IOPrintf(ar.Stream, "%f\n\n", value);
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
// Fast float conversion from String slice
char buffer[64];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
char* endPtr = nullptr;
float parsed = strtof(buffer, &endPtr);
if (endPtr != buffer)
{
value = parsed;
return true;
}
return false;
}
32-Bit Signed Integer (int32)
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, int32& value)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
IOPrintf(ar.Stream, "%d\n\n", value);
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
char buffer[32];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
char* endPtr = nullptr;
int32 parsed = static_cast<int32>(strtol(buffer, &endPtr, 10));
if (endPtr != buffer)
{
value = parsed;
return true;
}
return false;
}
64-Bit Unsigned Integer (uint64, EntityID)
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, uint64& value)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
IOPrintf(ar.Stream, "0x%016llX\n\n", value);
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
char buffer[32];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
char* endPtr = nullptr;
int base = (buffer[0] == '0' && (buffer[1] == 'x' || buffer[1] == 'X')) ? 16 : 10;
uint64 parsed = strtoull(buffer, &endPtr, base);
if (endPtr != buffer)
{
value = parsed;
return true;
}
return false;
}
Boolean (bool)
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, bool& value)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
IOPrintf(ar.Stream, "%s\n\n", value ? "true" : "false");
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
if (StringCompare(node->Value, WrapString("true")) == 0 ||
StringCompare(node->Value, WrapString("1")) == 0)
{
value = true;
return true;
}
if (StringCompare(node->Value, WrapString("false")) == 0 ||
StringCompare(node->Value, WrapString("0")) == 0)
{
value = false;
return true;
}
return false;
}
5.3 Vector Helpers (float x, y, z)
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, float& x, float& y, float& z)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
IOPrintf(ar.Stream, "%f %f %f\n\n", x, y, z);
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
char buffer[128];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
float parsedX = 0.0f;
float parsedY = 0.0f;
float parsedZ = 0.0f;
int matches = sscanf_s(buffer, "%f %f %f", &parsedX, &parsedY, &parsedZ);
if (matches == 3)
{
x = parsedX;
y = parsedY;
z = parsedZ;
return true;
}
return false;
}
5.4 String Helpers
bool SerializeProp(archive& ar, const char* keyName, uint32 keyCRC, String& value)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
WritePropertyHeader(ar, keyName);
bool hasSpace = ContainsChar(value, ' ');
if (hasSpace)
{
IOPrintf(ar.Stream, "\"%.*s\"\n\n", static_cast<int>(value.Size), value.Str);
}
else
{
IOPrintf(ar.Stream, "%.*s\n\n", static_cast<int>(value.Size), value.Str);
}
return true;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
String parsed = node->Value;
// Strip optional surrounding double quotes
if (parsed.Size >= 2 && parsed.Str[0] == '"' && parsed.Str[parsed.Size - 1] == '"')
{
parsed.Str++;
parsed.Size -= 2;
}
// Allocate persistent string copy on the archive arena
Assert(ar.ArenaInstance != nullptr);
value = StringCopy(ar.ArenaInstance, parsed);
return true;
}
6. Two-Tier Versioning Architecture
6.1 Architectural Rationale: Base vs Derived Decoupling
In entity-component systems or object-oriented engine hierarchies, entities consist of two distinct domains:
- Core Engine Identity (Base Entity): Position, Rotation, Scale, EntityID, Class Kind, Render Flags, Layer Masks. Managed by engine architects.
- Gameplay Specialization (Derived Class): Ammo, Health, AI State, Mesh Instance ID, Patrol Paths. Managed by gameplay programmers.
The Fragility of Monolithic Versioning
In naive serialization architectures, a single uint32 Version governs the entire file. When an engine programmer updates the base Entity struct (e.g. adding a uint32 LayerMask), bumping the global version invalidates or forces schema changes across every single derived entity type in the project.
The Two-Tier Solution: Universal ; Version + Optional ; ClassVersion
To keep .jasset files clean and unified across all asset types:
- All
.jassetfiles declare a universal; Versiontag:- In simple standalone assets (e.g.,
WorldSettings.jasset,Material.jasset),; Versionis the single asset schema version. - In entity assets,
; Versionmaps to the Base Engine Version (kEntityBaseVersioninEntity.h), governing core engine fields (Position,Rotation,Scale, etc.).
- In simple standalone assets (e.g.,
- Entity classes with derived versioning add an optional
; ClassVersion:- Governed by
Class::VersioninClass.h. - Only written/read for entity classes that define custom versions. If omitted in the file, it defaults to
1. - Non-entity assets never see or use
; ClassVersion.
- Governed by
========================================================================
.jasset Text Archive
========================================================================
; AssetType
Entity
; Version ------> Universal asset version (kEntityBaseVersion for Entity)
1
; EntityID
0x0000000000000001
; Kind
Inert
; Position
0.0 0.0 0.0
------------------------------------------------------------------------
; ClassVersion ------> Optional derived version (Class::Version in Class.h)
2
; MeshInstance
42
========================================================================
When an engine programmer bumps kEntityBaseVersion from 1 to 2 to add LayerMask, no derived game classes (Inert, Monster, Vehicle) need version increments or code modifications. Conversely, non-entity assets (like WorldSettings.jasset) simply use ; Version \n 1 without carrying entity-specific terminology.
6.2 Implementation Details: The Generalized Class Model
In this architecture, every serializable struct in Juliet has a Class descriptor:
Generalized Class Struct (Juliet/include/Engine/Class.h)
using serialize_fct_type = void (*)(archive& ar, void* payload, uint16 version);
struct Class
{
uint32 CRC;
uint8 kind;
uint16 Version; // Struct schema version
Class* BaseClass; // Pointer to parent class (or nullptr if root)
serialize_fct_type serialize_fct; // Type-specific serialization callback
size_t size_of;
size_t alignment;
#if JULIET_DEBUG
String Name;
#endif
};
consteval Class MakeClass(String name, uint8 kind, uint16 version, Class* baseClass,
size_t size, size_t align, serialize_fct_type fct)
{
Class cls = {};
cls.CRC = crc32(name.Str, name.Size);
cls.kind = kind;
cls.Version = version;
cls.BaseClass = baseClass;
cls.size_of = size;
cls.alignment = align;
cls.serialize_fct = fct;
#if JULIET_DEBUG
cls.Name = name;
#endif
return cls;
}
Registration Macros
// General class registration (e.g. for Entity, WorldSettings, Materials)
#define DEFINE_CLASS_VERSIONED(cls, version, base_class, serialize_fct) \
Class classKind##cls = \
MakeClass(ConstString(#cls), 0, (version), (base_class), sizeof(cls), alignof(cls), (serialize_fct)); \
Class* cls::Kind = &classKind##cls;
// Entity derived class registration (automatically sets BaseClass = Entity::Kind)
#define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct) \
Class entityKind##entity = \
MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version), Entity::Kind, sizeof(entity), \
alignof(entity), (serialize_fct)); \
Class* entity::Kind = &entityKind##entity;
Universal Class Instance Serializer
Because every type is a Class, serializing any struct in the engine is unified into a single function:
void SerializeClassInstance(archive& ar, Class* cls, void* instance)
{
Assert(cls != nullptr);
Assert(instance != nullptr);
uint16 version = cls->Version;
if (ar.IsSaving())
{
// If this is a derived entity class, write as class_version; otherwise write universal version
if (cls->BaseClass != nullptr)
{
SerializeProp(ar, "class_version", "class_version"_crc32, version);
}
else
{
SerializeProp(ar, "version", "version"_crc32, version);
}
}
else
{
if (cls->BaseClass != nullptr)
{
(void)SerializeProp(ar, "class_version", "class_version"_crc32, version);
}
else
{
(void)SerializeProp(ar, "version", "version"_crc32, version);
}
}
if (cls->serialize_fct != nullptr)
{
cls->serialize_fct(ar, instance, version);
}
}
Hierarchical Type Queries via BaseClass
[[nodiscard]] inline bool IsA(const Class* queryClass, const Class* targetClass)
{
const Class* current = queryClass;
while (current != nullptr)
{
if (current == targetClass)
{
return true;
}
current = current->BaseClass;
}
return false;
}
6.3 Entity Serialization with Generalized Classes
An entity instance is cleanly composed of its base Entity class and its derived DerivedKind class:
void Serialize(archive& ar, NonNullPtr<Entity> entity)
{
Assert(entity.Get() != nullptr);
// 1. Serialize Base Entity using Entity's own Class descriptor (Entity::Kind)
SerializeClassInstance(ar, Entity::Kind, entity.Get());
// 2. Serialize Derived Entity using its Class descriptor (entity->DerivedKind)
if (entity->DerivedKind != nullptr && entity->Derived != nullptr)
{
SerializeClassInstance(ar, entity->DerivedKind, entity->Derived);
}
}
7. Post-Serialization Deprecation & Migration
7.1 Schema Evolution Challenge
Over the lifecycle of a game, gameplay mechanics evolve:
- A scalar float
Speedis replaced by a directional 2D vectorVelocity. - A single texture index
TextureIDis replaced by an asset path stringDiffuseTexture. - Obsolete properties are deleted entirely.
Retaining deprecated members in active C++ structs creates code clutter, wastes memory, and invites bugs.
7.2 Stack-Allocated Migration via Standard Serialization
Rather than maintaining dedicated deprecation primitives or polluting C++ structs with obsolete members, deprecated properties are migrated using the standard serialize / SERIALIZE function with temporary local variables allocated on the stack.
When an asset property is deprecated, restructured, or renamed:
- The obsolete field is completely removed from the modern C++ struct definition.
- In the entity/component serializer, an
if (ar.loading && ar.class_version < N)block is added. - A local variable of the legacy type is declared on the stack.
- The standard
SERIALIZE(ar, OldPropName, deprecated_val)(orserialize(ar, ConstString("OldPropName"), "OldPropName"_crc32, deprecated_val)) is called. If the property exists in the loaded asset, it parses intodeprecated_valand returnstrue. - The serializer performs whatever mapping or transformation is required into the modern struct fields.
- Because the migration block is strictly guarded by
ar.loading, it never executes during save operations (ar.loading == false). The serializer writes only modern struct fields, automatically purging obsolete keys on subsequent saves without requiring dedicated cleanup routines.
7.3 In-Place Migration Pattern
When an asset file with an older class_version is loaded, the serializer detects ar.class_version < N, reads obsolete fields into stack variables using standard serialization, maps the legacy data into the modern struct, and completes loading. On the subsequent save, the asset file is emitted using the modern schema without deprecated keys.
struct Projectile
{
DECLARE_ENTITY()
// Modern Schema (v2)
float VelocityX = 0.0f;
float VelocityY = 0.0f;
float Damage = 50.0f;
};
void SerializeProjectile(Archive* arPtr, void* payload)
{
Assert(arPtr != nullptr);
Assert(payload != nullptr);
auto& ar = *arPtr;
auto* projectile = static_cast<Projectile*>(payload);
if (ar.loading)
{
SERIALIZE(ar, Damage, projectile->Damage);
if (ar.class_version < 2)
{
// Migration from v1: scalar 'Speed' converted to 'VelocityX' on the stack
float deprecated_speed = 0.0f;
if (SERIALIZE(ar, Speed, deprecated_speed))
{
projectile->VelocityX = deprecated_speed;
projectile->VelocityY = 0.0f;
}
}
else
{
SERIALIZE(ar, VelocityX, projectile->VelocityX);
SERIALIZE(ar, VelocityY, projectile->VelocityY);
}
}
else
{
// Save modern schema
SERIALIZE(ar, VelocityX, projectile->VelocityX);
SERIALIZE(ar, VelocityY, projectile->VelocityY);
SERIALIZE(ar, Damage, projectile->Damage);
}
}
8. Step-by-Step Implementation Roadmap & Unit Testing Plan
8.1 Implementation Roadmap
+-------------------------------------------------------------------------+
| Phase 1: Core Utilities & Dual-Mode CRC32 |
| - Update CRC32.h with constexpr Crc32(String) and Crc32(char*, len) |
| - Extend struct Class in Class.h with uint16 Version |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Phase 2: In-Memory Zero-Copy Parser |
| - Implement TextArchiveParser.h / .cpp |
| - TokenizeTextArchive, TrimWhitespace, FindProperty |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Phase 3: Archive Struct & Serialize Helpers |
| - Upgrade Archive in serialization.h with loading / version state |
| - Implement primitive, vector, and string serialize helpers |
| - Support stack-allocated schema migration |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Phase 4: Entity & World Integration |
| - Update Entity.h / Entity.cpp with two-tier serialization |
| - Refactor World.cpp to load/save .jasset files |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Phase 5: Verification & Unit Testing Suite |
| - Build and run SerializationUnitTest.cpp |
+-------------------------------------------------------------------------+
Phase 1: Core Utilities & Dual-Mode CRC32
- Target:
Juliet/include/Core/Common/CRC32.h- Add
constexpr uint32 Crc32(const char* str, size_t length)andconstexpr uint32 Crc32(String str). - Ensure existing
operator""_crc32callsCrc32.
- Add
- Target:
Juliet/include/Engine/Class.h- Add
uint16 Versiontostruct Class. - Update
MakeClassto acceptuint16 version = 1.
- Add
Phase 2: In-Memory Zero-Copy Parser
- Target:
Juliet/include/Core/Common/TextArchiveParser.h(andsrc/Core/Common/TextArchiveParser.cpp)- Define
struct TextPropertyNode { uint32 KeyCRC; String Value; bool Consumed; }. - Implement
TokenizeTextArchive(NonNullPtr<Arena> arena, ByteBuffer buffer). - Implement
FindPropertyandAuditUnconsumedProperties.
- Define
Phase 3: Archive Struct & serialize Helpers
- Target:
Juliet/include/Core/Common/serialization.h- Upgrade
struct Archivewith stream pointer, property array,loadingflag, and versions (base_version,class_version). - Implement overloaded
serialize,read_prop, andwriteforfloat,int32,uint64,bool, vectors, andString. - Support deprecation migration using standard
serialize/SERIALIZEwith local stack variables underif (ar.loading && ar.class_version < N).
- Upgrade
Phase 4: Entity & World Integration
- Target:
Game/Entity/Entity.handGame/Entity/Entity.cpp- Define
constexpr uint32 kEntityBaseVersion = 1;. - Update
DEFINE_ENTITYand addDEFINE_ENTITY_VERSIONED. - Implement
Serialize(archive& ar, NonNullPtr<Entity> entity)supporting.jassettext format.
- Define
- Target:
Game/Data/World.handGame/Data/World.cpp- Implement text-based world saving and loading using
.jassetformatting.
- Implement text-based world saving and loading using
Phase 5: Comprehensive Unit Testing
- Target:
Game/UnitTest/SerializationUnitTest.handGame/UnitTest/SerializationUnitTest.cpp- Add unit tests verifying parsing, round-trip serialization, defaults preservation, versioning, and deprecation.
8.2 Comprehensive Unit Testing Plan (SerializationUnitTest.cpp)
The unit test suite validates all architectural requirements without modifying engine framework code for test-specific cases.
// Game/UnitTest/SerializationUnitTest.h
#pragma once
#include <Juliet.h>
#if JULIET_DEBUG
namespace UnitTest
{
void RunSerializationUnitTests();
}
#endif
// Game/UnitTest/SerializationUnitTest.cpp
#include <UnitTest/SerializationUnitTest.h>
#if JULIET_DEBUG
#include <Core/Common/CRC32.h>
#include <Core/Common/String.h>
#include <Core/Common/serialization.h>
#include <Core/HAL/IO/IOStream.h>
#include <Core/Logging/LogManager.h>
#include <Core/Logging/LogTypes.h>
#include <Core/Memory/MemoryArena.h>
#include <Core/Thread/ThreadContext.h>
#include <Entity/Entity.h>
namespace UnitTest
{
// Test Struct for Derived Entity Testing
struct DummyVehicle
{
DECLARE_ENTITY()
float MaxSpeed = 120.0f;
int32 GearCount = 6;
uint64 ChassisUUID = 0xABCDEF0123456789ULL;
bool Turbo = true;
float PosX = 10.0f;
float PosY = 20.0f;
float PosZ = 30.0f;
};
void SerializeDummyVehicle(archive* arPtr, void* payload)
{
Assert(arPtr != nullptr);
Assert(payload != nullptr);
auto& ar = *arPtr;
auto* vehicle = static_cast<DummyVehicle*>(payload);
SERIALIZE_PROP(ar, vehicle->MaxSpeed);
SERIALIZE_PROP(ar, vehicle->GearCount);
SERIALIZE_PROP(ar, vehicle->ChassisUUID);
SERIALIZE_PROP(ar, vehicle->Turbo);
SerializeProp(ar, "Position", "Position"_crc32, vehicle->PosX, vehicle->PosY, vehicle->PosZ);
}
DEFINE_ENTITY_VERSIONED(DummyVehicle, 1, SerializeDummyVehicle);
// Test 1: Parser tokenization with whitespace, comments, and mixed line endings
static void TestParserTokenization()
{
TempArena temp = scratch_begin(nullptr, 0);
const char* testContent =
"# Header Comment\r\n"
"// Secondary comment\n"
"\n"
"; Health\r\n"
" 100.500000 \r\n"
"\n"
"; Name\n"
"\"Paladin Hero\"\n"
"\n"
"; Position\r\n"
"1.0 2.0 3.0\r\n";
ByteBuffer buffer = {
.Data = reinterpret_cast<Byte*>(const_cast<char*>(testContent)),
.Size = strlen(testContent)
};
ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer);
Assert(parsed.PropertyCount == 3);
auto* healthNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Health"_crc32);
Assert(healthNode != nullptr);
Assert(StringCompare(healthNode->Value, WrapString("100.500000")) == 0);
auto* nameNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Name"_crc32);
Assert(nameNode != nullptr);
Assert(StringCompare(nameNode->Value, WrapString("\"Paladin Hero\"")) == 0);
auto* posNode = FindProperty(parsed.Nodes, parsed.PropertyCount, "Position"_crc32);
Assert(posNode != nullptr);
Assert(StringCompare(posNode->Value, WrapString("1.0 2.0 3.0")) == 0);
scratch_end(temp);
LogMessage(LogCategory::Core, "TestParserTokenization passed.");
}
// Test 2: Missing properties retain default struct values
static void TestDefaultValueRetention()
{
TempArena temp = scratch_begin(nullptr, 0);
const char* incompleteContent =
"; MaxSpeed\n"
"180.0\n";
ByteBuffer buffer = {
.Data = reinterpret_cast<Byte*>(const_cast<char*>(incompleteContent)),
.Size = strlen(incompleteContent)
};
ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer);
archive ar = {};
ar.ArenaInstance = temp.Arena;
ar.Mode = ArchiveMode::LoadingText;
ar.Properties = parsed.Nodes;
ar.PropertyCount = parsed.PropertyCount;
DummyVehicle vehicle;
// Defaults: MaxSpeed=120, GearCount=6, Turbo=true
SerializeDummyVehicle(&ar, &vehicle);
Assert(vehicle.MaxSpeed == 180.0f); // Overwritten by archive
Assert(vehicle.GearCount == 6); // Preserved default
Assert(vehicle.Turbo == true); // Preserved default
Assert(vehicle.PosX == 10.0f); // Preserved default
scratch_end(temp);
LogMessage(LogCategory::Core, "TestDefaultValueRetention passed.");
}
// Test 3: Deprecation migration from v1 to v2
struct LegacyWeapon
{
DECLARE_ENTITY()
float VelocityX = 0.0f;
float VelocityY = 0.0f;
};
void SerializeLegacyWeapon(Archive* arPtr, void* payload)
{
Assert(arPtr != nullptr);
Assert(payload != nullptr);
auto& ar = *arPtr;
auto* weapon = static_cast<LegacyWeapon*>(payload);
if (ar.loading)
{
if (ar.class_version < 2)
{
float deprecated_speed = 0.0f;
if (SERIALIZE(ar, Speed, deprecated_speed))
{
weapon->VelocityX = deprecated_speed;
weapon->VelocityY = 0.0f;
}
}
else
{
SERIALIZE(ar, VelocityX, weapon->VelocityX);
SERIALIZE(ar, VelocityY, weapon->VelocityY);
}
}
else
{
SERIALIZE(ar, VelocityX, weapon->VelocityX);
SERIALIZE(ar, VelocityY, weapon->VelocityY);
}
}
DEFINE_ENTITY_VERSIONED(LegacyWeapon, 2, SerializeLegacyWeapon);
static void TestDeprecationMigration()
{
TempArena temp = scratch_begin(nullptr, 0);
// Simulated v1 file containing obsolete 'Speed'
const char* v1Content =
"; class_version\n"
"1\n"
"; Speed\n"
"75.500000\n";
ByteBuffer buffer = {
.Data = reinterpret_cast<Byte*>(const_cast<char*>(v1Content)),
.Size = strlen(v1Content)
};
ParsedArchive parsed = tokenize_archive(temp.Arena, buffer);
Archive ar = {};
ar.arena = temp.Arena;
ar.loading = true;
ar.base = parsed;
ar.class_version = 1;
LegacyWeapon weapon;
SerializeLegacyWeapon(&ar, &weapon);
Assert(weapon.VelocityX == 75.5f);
Assert(weapon.VelocityY == 0.0f);
scratch_end(temp);
LogMessage(LogCategory::Core, "TestDeprecationMigration passed.");
}
void RunSerializationUnitTests()
{
LogMessage(LogCategory::Core, "=== Running Serialization & .jasset Unit Tests ===");
TestParserTokenization();
TestDefaultValueRetention();
TestDeprecationMigration();
LogMessage(LogCategory::Core, "=== All Serialization Unit Tests Passed Successfully ===");
}
} // namespace UnitTest
#endif