Patedam/Juliet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ongoing refactor of serialization and various cleanup

Browse Source
This commit is contained in:
Patedam
2026-09-07 16:58:57 -04:00
parent 1a045c3578
commit a462575af4
43 changed files with 952 additions and 483 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Game/Plans/01_Serialization_And_Text_Archive.md
+157 -215
View File
@@ -24,7 +24,7 @@ The `.jasset` text archive framework is engineered to replace legacy binary blob
- **Symmetric Single-Function Serialization**: A single `Serialize` implementation per entity or data structure handles both Save and Load paths, guaranteeing that write and read schemas never diverge.
- **Graceful Forward/Backward Compatibility**: Missing keys during load automatically preserve struct default values. Unknown keys present in newer asset files are safely ignored without parse failure.
- **Two-Tier Decoupled Versioning**: Core engine entity properties (`kEntityBaseVersion`) and derived gameplay class properties (`Class::Version`) are versioned independently. Engine-level updates never bump derived entity class versions.
- **Post-Serialization In-Place Migration**: Deprecated fields can be read through specialized primitives (`ReadDeprecated*`) to migrate legacy data structures in-place upon loading, producing cleaned, modern schemas on subsequent saves.
- **In-Place Schema Migration**: Deprecated properties no longer present in modern structs can be read during loading into local stack variables using standard serialization (`serialize` / `SERIALIZE`) guarded by version checks (`if (ar.loading && ar.class_version < N)`), seamlessly transforming legacy values without struct pollution or persisting obsolete keys on subsequent saves.
- **Zero Exceptions & Total Warning Cleanliness**: Fully conforming to Juliet coding guidelines: strict assertions (`Assert`), `[[nodiscard]]`, `auto*`/`auto&`, mandatory braces, and `static_cast`/`reinterpret_cast`.
---
@@ -133,7 +133,7 @@ Identifier ::= [a-zA-Z_][a-zA-Z0-9_]* ;
; AssetType
Entity
; BaseVersion
; Version
1
; EntityID
@@ -712,10 +712,15 @@ In entity-component systems or object-oriented engine hierarchies, entities cons
#### The Fragility of Monolithic Versioning
In naive serialization architectures, a single `uint32 Version` governs the entire file. When an engine programmer updates the base `Entity` struct (e.g. adding a `uint32 LayerMask`), bumping the global version invalidates or forces schema changes across every single derived entity type in the project.
#### The Two-Tier Solution
Juliet decouples versioning into two independent tiers:
- **Tier 1: Base Engine Version (`kEntityBaseVersion`)**: Declared centrally in `Entity.h`. Governs `Entity` base fields.
- **Tier 2: Derived Class Version (`Class::Version`)**: Declared per-entity class in `Class.h` and initialized in `DEFINE_ENTITY_VERSIONED`.
#### The Two-Tier Solution: Universal `; Version` + Optional `; ClassVersion`
To keep `.jasset` files clean and unified across all asset types:
- **All `.jasset` files declare a universal `; Version` tag**:
- In simple standalone assets (e.g., `WorldSettings.jasset`, `Material.jasset`), `; Version` is the single asset schema version.
- In entity assets, `; Version` maps to the Base Engine Version (`kEntityBaseVersion` in `Entity.h`), governing core engine fields (`Position`, `Rotation`, `Scale`, etc.).
- **Entity classes with derived versioning add an optional `; ClassVersion`**:
- Governed by `Class::Version` in `Class.h`.
- Only written/read for entity classes that define custom versions. If omitted in the file, it defaults to `1`.
- Non-entity assets never see or use `; ClassVersion`.
```
========================================================================
@@ -723,7 +728,7 @@ Juliet decouples versioning into two independent tiers:
========================================================================
; AssetType
Entity
; BaseVersion ------> Governed by kEntityBaseVersion in Entity.h
; Version ------> Universal asset version (kEntityBaseVersion for Entity)
1
; EntityID
0x0000000000000001
@@ -732,32 +737,30 @@ Juliet decouples versioning into two independent tiers:
; Position
0.0 0.0 0.0
------------------------------------------------------------------------
; ClassVersion ------> Governed by Class::Version in Class.h
; ClassVersion ------> Optional derived version (Class::Version in Class.h)
2
; MeshInstance
42
========================================================================
```
When an engine programmer bumps `kEntityBaseVersion` from `1` to `2` to add `LayerMask`, no derived game classes (`Inert`, `Monster`, `Vehicle`) need version increments or code modifications.
When an engine programmer bumps `kEntityBaseVersion` from `1` to `2` to add `LayerMask`, no derived game classes (`Inert`, `Monster`, `Vehicle`) need version increments or code modifications. Conversely, non-entity assets (like `WorldSettings.jasset`) simply use `; Version \n 1` without carrying entity-specific terminology.
### 6.2 Implementation Details
### 6.2 Implementation Details: The Generalized `Class` Model
#### Engine Base Version (`Game/Entity/Entity.h`)
In this architecture, **every serializable struct in Juliet has a `Class` descriptor**:
#### Generalized `Class` Struct (`Juliet/include/Engine/Class.h`)
```cpp
constexpr uint32 kEntityBaseVersion = 1;
using serialize_fct_type = void (*)(archive& ar, void* payload, uint16 version);
void SerializeEntityBase(archive& ar, NonNullPtr<Entity> entity);
```
#### Derived Class Version (`Juliet/include/Engine/Class.h`)
```cpp
struct Class
{
uint32 CRC;
uint8 kind;
uint16 Version; // Added derived class version
serialize_fct_type serialize_fct;
uint16 Version; // Struct schema version
Class* BaseClass; // Pointer to parent class (or nullptr if root)
serialize_fct_type serialize_fct; // Type-specific serialization callback
size_t size_of;
size_t alignment;
@@ -766,12 +769,14 @@ struct Class
#endif
};
consteval Class MakeClass(String name, uint8 kind, uint16 version, size_t size, size_t align, serialize_fct_type fct)
consteval Class MakeClass(String name, uint8 kind, uint16 version, Class* baseClass,
size_t size, size_t align, serialize_fct_type fct)
{
Class cls = {};
cls.CRC = crc32(name.Str, name.Size);
cls.kind = kind;
cls.Version = version;
cls.BaseClass = baseClass;
cls.size_of = size;
cls.alignment = align;
cls.serialize_fct = fct;
@@ -784,80 +789,95 @@ consteval Class MakeClass(String name, uint8 kind, uint16 version, size_t size,
}
```
#### Class Registration Macros (`Game/Entity/Entity.h`)
#### Registration Macros
```cpp
// General class registration (e.g. for Entity, WorldSettings, Materials)
#define DEFINE_CLASS_VERSIONED(cls, version, base_class, serialize_fct) \
Class classKind##cls = \
MakeClass(ConstString(#cls), 0, (version), (base_class), sizeof(cls), alignof(cls), (serialize_fct)); \
Class* cls::Kind = &classKind##cls;
// Entity derived class registration (automatically sets BaseClass = Entity::Kind)
#define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct) \
Class entityKind##entity = \
MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version), sizeof(entity), alignof(entity), serialize_fct); \
MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version), Entity::Kind, sizeof(entity), \
alignof(entity), (serialize_fct)); \
Class* entity::Kind = &entityKind##entity;
```
### 6.3 Execution Flow in `SerializeEntity`
#### Universal Class Instance Serializer
Because every type is a `Class`, serializing **any** struct in the engine is unified into a single function:
```cpp
void SerializeClassInstance(archive& ar, Class* cls, void* instance)
{
Assert(cls != nullptr);
Assert(instance != nullptr);
uint16 version = cls->Version;
if (ar.IsSaving())
{
// If this is a derived entity class, write as class_version; otherwise write universal version
if (cls->BaseClass != nullptr)
{
SerializeProp(ar, "class_version", "class_version"_crc32, version);
}
else
{
SerializeProp(ar, "version", "version"_crc32, version);
}
}
else
{
if (cls->BaseClass != nullptr)
{
(void)SerializeProp(ar, "class_version", "class_version"_crc32, version);
}
else
{
(void)SerializeProp(ar, "version", "version"_crc32, version);
}
}
if (cls->serialize_fct != nullptr)
{
cls->serialize_fct(ar, instance, version);
}
}
```
#### Hierarchical Type Queries via `BaseClass`
```cpp
[[nodiscard]] inline bool IsA(const Class* queryClass, const Class* targetClass)
{
const Class* current = queryClass;
while (current != nullptr)
{
if (current == targetClass)
{
return true;
}
current = current->BaseClass;
}
return false;
}
```
### 6.3 Entity Serialization with Generalized Classes
An entity instance is cleanly composed of its base `Entity` class and its derived `DerivedKind` class:
```cpp
void Serialize(archive& ar, NonNullPtr<Entity> entity)
{
Assert(entity.Get() != nullptr);
// --- Tier 1: Base Entity Serialization ---
if (ar.IsSaving())
// 1. Serialize Base Entity using Entity's own Class descriptor (Entity::Kind)
SerializeClassInstance(ar, Entity::Kind, entity.Get());
// 2. Serialize Derived Entity using its Class descriptor (entity->DerivedKind)
if (entity->DerivedKind != nullptr && entity->Derived != nullptr)
{
uint32 baseVer = kEntityBaseVersion;
SERIALIZE_PROP(ar, baseVer);
SERIALIZE_PROP(ar, entity->ID);
String kindStr = WrapString(kEntity_type_names[entity->Kind->kind]);
SerializeProp(ar, "Kind", "Kind"_crc32, kindStr);
SerializeProp(ar, "Position", "Position"_crc32, entity->X, entity->Y, entity->Z);
}
else
{
uint32 baseVer = 0;
if (!SerializeProp(ar, "BaseVersion", "BaseVersion"_crc32, baseVer))
{
baseVer = 1; // Default to initial schema if absent
}
ar.BaseVersion = baseVer;
SERIALIZE_PROP(ar, entity->ID);
String kindStr = {};
if (SerializeProp(ar, "Kind", "Kind"_crc32, kindStr))
{
// Resolve class pointer from name
for (uint8 i = 0; i < ToUnderlying(Entity_Type::Count); ++i)
{
if (StringCompare(kindStr, WrapString(kEntity_type_names[i])) == 0)
{
entity->Kind = kEntity_type_class_ptr[i];
break;
}
}
}
Assert(entity->Kind != nullptr);
SerializeProp(ar, "Position", "Position"_crc32, entity->X, entity->Y, entity->Z);
}
// --- Tier 2: Derived Entity Serialization ---
if (entity->Kind->serialize_fct != nullptr && entity->Derived != nullptr)
{
if (ar.IsSaving())
{
uint32 classVer = entity->Kind->Version;
SerializeProp(ar, "ClassVersion", "ClassVersion"_crc32, classVer);
}
else
{
uint32 classVer = 0;
if (!SerializeProp(ar, "ClassVersion", "ClassVersion"_crc32, classVer))
{
classVer = entity->Kind->Version;
}
ar.ClassVersion = static_cast<uint16>(classVer);
}
entity->Kind->serialize_fct(&ar, entity->Derived);
SerializeClassInstance(ar, entity->DerivedKind, entity->Derived);
}
}
```
@@ -874,96 +894,19 @@ Over the lifecycle of a game, gameplay mechanics evolve:
Retaining deprecated members in active C++ structs creates code clutter, wastes memory, and invites bugs.
### 7.2 Deprecation Primitives (`ReadDeprecated*`)
Deprecation primitives allow serializers to ingest obsolete properties exclusively during loading without polluting modern structs or writing deprecated keys back to disk during saving.
### 7.2 Stack-Allocated Migration via Standard Serialization
Rather than maintaining dedicated deprecation primitives or polluting C++ structs with obsolete members, deprecated properties are migrated using the standard `serialize` / `SERIALIZE` function with temporary local variables allocated on the stack.
```cpp
bool ReadDeprecated(archive& ar, const char* keyName, uint32 keyCRC, float& outVal)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
return false; // Deprecated fields are never saved
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
char buffer[64];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
char* endPtr = nullptr;
float parsed = strtof(buffer, &endPtr);
if (endPtr != buffer)
{
outVal = parsed;
return true;
}
return false;
}
bool ReadDeprecatedVec2(archive& ar, const char* keyName, uint32 keyCRC, float& outX, float& outY)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
return false;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
char buffer[128];
size_t copySize = Min(node->Value.Size, sizeof(buffer) - 1);
MemCopy(buffer, node->Value.Str, copySize);
buffer[copySize] = '\0';
float x = 0.0f;
float y = 0.0f;
if (sscanf_s(buffer, "%f %f", &x, &y) == 2)
{
outX = x;
outY = y;
return true;
}
return false;
}
bool ReadDeprecatedString(archive& ar, const char* keyName, uint32 keyCRC, NonNullPtr<Arena> arena, String& outVal)
{
Assert(keyName != nullptr);
if (ar.IsSaving())
{
return false;
}
auto* node = FindProperty(ar.Properties, ar.PropertyCount, keyCRC);
if (node == nullptr)
{
return false;
}
String parsed = node->Value;
if (parsed.Size >= 2 && parsed.Str[0] == '"' && parsed.Str[parsed.Size - 1] == '"')
{
parsed.Str++;
parsed.Size -= 2;
}
outVal = StringCopy(arena, parsed);
return true;
}
```
When an asset property is deprecated, restructured, or renamed:
1. The obsolete field is completely removed from the modern C++ struct definition.
2. In the entity/component serializer, an `if (ar.loading && ar.class_version < N)` block is added.
3. A local variable of the legacy type is declared on the stack.
4. The standard `SERIALIZE(ar, OldPropName, deprecated_val)` (or `serialize(ar, ConstString("OldPropName"), "OldPropName"_crc32, deprecated_val)`) is called. If the property exists in the loaded asset, it parses into `deprecated_val` and returns `true`.
5. The serializer performs whatever mapping or transformation is required into the modern struct fields.
6. Because the migration block is strictly guarded by `ar.loading`, it never executes during save operations (`ar.loading == false`). The serializer writes only modern struct fields, automatically purging obsolete keys on subsequent saves without requiring dedicated cleanup routines.
### 7.3 In-Place Migration Pattern
When an asset file with an older `ClassVersion` is loaded, the derived serializer detects `ar.ClassVersion < N`, calls `ReadDeprecated*` to read obsolete fields, maps the legacy data into the modern struct, and continues. On the subsequent save, the asset file is emitted using the modern schema without deprecated keys.
When an asset file with an older `class_version` is loaded, the serializer detects `ar.class_version < N`, reads obsolete fields into stack variables using standard serialization, maps the legacy data into the modern struct, and completes loading. On the subsequent save, the asset file is emitted using the modern schema without deprecated keys.
```cpp
struct Projectile
@@ -976,39 +919,40 @@ struct Projectile
float Damage = 50.0f;
};
void SerializeProjectile(archive* arPtr, void* payload)
void SerializeProjectile(Archive* arPtr, void* payload)
{
Assert(arPtr != nullptr);
Assert(payload != nullptr);
auto& ar = *arPtr;
auto* projectile = static_cast<Projectile*>(payload);
if (ar.IsSaving())
if (ar.loading)
{
SERIALIZE_PROP(ar, projectile->VelocityX);
SERIALIZE_PROP(ar, projectile->VelocityY);
SERIALIZE_PROP(ar, projectile->Damage);
}
else
{
SERIALIZE_PROP(ar, projectile->Damage);
SERIALIZE(ar, Damage, projectile->Damage);
if (ar.ClassVersion < 2)
if (ar.class_version < 2)
{
// Migration from v1: scalar 'Speed' converted to 'VelocityX'
float legacySpeed = 0.0f;
if (ReadDeprecated(ar, "Speed", "Speed"_crc32, legacySpeed))
// Migration from v1: scalar 'Speed' converted to 'VelocityX' on the stack
float deprecated_speed = 0.0f;
if (SERIALIZE(ar, Speed, deprecated_speed))
{
projectile->VelocityX = legacySpeed;
projectile->VelocityX = deprecated_speed;
projectile->VelocityY = 0.0f;
}
}
else
{
SERIALIZE_PROP(ar, projectile->VelocityX);
SERIALIZE_PROP(ar, projectile->VelocityY);
SERIALIZE(ar, VelocityX, projectile->VelocityX);
SERIALIZE(ar, VelocityY, projectile->VelocityY);
}
}
else
{
// Save modern schema
SERIALIZE(ar, VelocityX, projectile->VelocityX);
SERIALIZE(ar, VelocityY, projectile->VelocityY);
SERIALIZE(ar, Damage, projectile->Damage);
}
}
```
@@ -1034,10 +978,10 @@ void SerializeProjectile(archive* arPtr, void* payload)
|
v
+-------------------------------------------------------------------------+
| Phase 3: Archive Struct & SerializeProp Helpers |
| - Upgrade archive in serialization.h with ArchiveMode |
| - Implement primitive, vector, and string SerializeProp helpers |
| - Implement ReadDeprecated* primitives |
| Phase 3: Archive Struct & Serialize Helpers |
| - Upgrade Archive in serialization.h with loading / version state |
| - Implement primitive, vector, and string serialize helpers |
| - Support stack-allocated schema migration |
+-------------------------------------------------------------------------+
|
v
@@ -1068,12 +1012,11 @@ void SerializeProjectile(archive* arPtr, void* payload)
- Implement `TokenizeTextArchive(NonNullPtr<Arena> arena, ByteBuffer buffer)`.
- Implement `FindProperty` and `AuditUnconsumedProperties`.
#### Phase 3: Archive Struct & `SerializeProp` Helpers
#### Phase 3: Archive Struct & `serialize` Helpers
1. **Target**: `Juliet/include/Core/Common/serialization.h`
- Introduce `enum class ArchiveMode : uint8`.
- Upgrade `struct archive` with stream pointer, property array, mode, and versions.
- Implement overloaded `SerializeProp` for `float`, `int32`, `uint64`, `bool`, vectors, and `String`.
- Implement `ReadDeprecated`, `ReadDeprecatedVec2`, `ReadDeprecatedString`.
- Upgrade `struct Archive` with stream pointer, property array, `loading` flag, and versions (`base_version`, `class_version`).
- Implement overloaded `serialize`, `read_prop`, and `write` for `float`, `int32`, `uint64`, `bool`, vectors, and `String`.
- Support deprecation migration using standard `serialize` / `SERIALIZE` with local stack variables under `if (ar.loading && ar.class_version < N)`.
#### Phase 4: Entity & World Integration
1. **Target**: `Game/Entity/Entity.h` and `Game/Entity/Entity.cpp`
@@ -1240,35 +1183,35 @@ namespace UnitTest
float VelocityY = 0.0f;
};
void SerializeLegacyWeapon(archive* arPtr, void* payload)
void SerializeLegacyWeapon(Archive* arPtr, void* payload)
{
Assert(arPtr != nullptr);
Assert(payload != nullptr);
auto& ar = *arPtr;
auto* weapon = static_cast<LegacyWeapon*>(payload);
if (ar.IsSaving())
if (ar.loading)
{
SERIALIZE_PROP(ar, weapon->VelocityX);
SERIALIZE_PROP(ar, weapon->VelocityY);
}
else
{
if (ar.ClassVersion < 2)
if (ar.class_version < 2)
{
float oldSpeed = 0.0f;
if (ReadDeprecated(ar, "Speed", "Speed"_crc32, oldSpeed))
float deprecated_speed = 0.0f;
if (SERIALIZE(ar, Speed, deprecated_speed))
{
weapon->VelocityX = oldSpeed;
weapon->VelocityX = deprecated_speed;
weapon->VelocityY = 0.0f;
}
}
else
{
SERIALIZE_PROP(ar, weapon->VelocityX);
SERIALIZE_PROP(ar, weapon->VelocityY);
SERIALIZE(ar, VelocityX, weapon->VelocityX);
SERIALIZE(ar, VelocityY, weapon->VelocityY);
}
}
else
{
SERIALIZE(ar, VelocityX, weapon->VelocityX);
SERIALIZE(ar, VelocityY, weapon->VelocityY);
}
}
DEFINE_ENTITY_VERSIONED(LegacyWeapon, 2, SerializeLegacyWeapon);
@@ -1279,7 +1222,7 @@ namespace UnitTest
// Simulated v1 file containing obsolete 'Speed'
const char* v1Content =
"; ClassVersion\n"
"; class_version\n"
"1\n"
"; Speed\n"
"75.500000\n";
@@ -1289,14 +1232,13 @@ namespace UnitTest
.Size = strlen(v1Content)
};
ParsedTextArchive parsed = TokenizeTextArchive(temp.Arena, buffer);
ParsedArchive parsed = tokenize_archive(temp.Arena, buffer);
archive ar = {};
ar.ArenaInstance = temp.Arena;
ar.Mode = ArchiveMode::LoadingText;
ar.Properties = parsed.Nodes;
ar.PropertyCount = parsed.PropertyCount;
ar.ClassVersion = 1;
Archive ar = {};
ar.arena = temp.Arena;
ar.loading = true;
ar.base = parsed;
ar.class_version = 1;
LegacyWeapon weapon;
SerializeLegacyWeapon(&ar, &weapon);