Patedam/Juliet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating serialization tdd with latest changes

Browse Source
This commit is contained in:
Patedam
2026-09-07 20:05:17 -04:00
parent b1de6ccc49
commit 3123d2f2e6
4 changed files with 454 additions and 1378 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Game/Plans/01_Serialization_And_Text_Archive.md
+269 -1123
View File
File diff suppressed because it is too large Load Diff
Game/Plans/02_Entity_Allocation_And_Lifecycle.md
+120 -154
View File
@@ -82,10 +82,10 @@ This is the classic **chicken-and-egg memory problem**:
### 1.4 Architectural Objectives
This specification establishes a robust in-place lifecycle pipeline that completely eliminates stack temporaries and dynamic heap allocations:
1. **Direct In-Place Allocation:** Introduce `AllocateEntity(EntityManager& manager, Class* classPtr)` which allocates both the base `Entity` and the derived struct directly within their respective engine memory arenas.
2. **Bidirectional Pointer Integrity:** Wire mutual pointers (`base->Derived` and `derived->Base`) at allocation time before any field deserialization begins.
2. **Bidirectional Pointer Integrity:** Wire mutual pointers (`base->derived` and `derived->base`) at allocation time before any field deserialization begins.
3. **In-Place Stream Deserialization:** Read class reflection metadata first, invoke `AllocateEntity`, and stream base and derived properties directly into arena-resident memory.
4. **Isolated Entity Assets (`.jasset`):** Transition from a monolithic `world.bin` to a modular one-file-per-entity architecture (`Assets/Entities/{ID}.jasset`).
5. **Dirty Tracking & Optimal Saves:** Introduce an `IsDirty` flag on `Entity` to avoid rewriting unchanged entity files, minimizing disk I/O and eliminating spurious Git repository modifications.
5. **Dirty Tracking & Optimal Saves:** Introduce an `is_dirty` flag on `Entity` to avoid rewriting unchanged entity files, minimizing disk I/O and eliminating spurious Git repository modifications.
6. **Robust Deletion Lifecycle:** Decouple in-memory removal (`RemoveAtFast` with pointer fixup) from disk synchronization using `World::PendingDeletions`.
---
@@ -129,8 +129,8 @@ VectorArena<Entity, 100'000> Entities;
```
- **Capacity:** Fixed reserve of 100,000 entities allocated from the `WorldArena`.
- **Memory Footprint:**
$$\text{sizeof(Entity)} = 8\text{ (ID)} + 8\text{ (Kind)} + 8\text{ (Derived)} + 12\text{ (X, Y, Z)} + 1\text{ (IsDirty)} + 3\text{ (Padding)} = 40\text{ bytes}$$
Total reserved space: $100{,}000 \times 40\text{ bytes} \approx 4.0\text{ MB}$.
$$\text{sizeof(Entity)} = 8\text{ (ID)} + 8\text{ (derived\_kind)} + 8\text{ (derived)} + 16\text{ (position)} + 1\text{ (is\_dirty)} + 7\text{ (Padding)} = 48\text{ bytes}$$
Total reserved space: $100{,}000 \times 48\text{ bytes} \approx 4.8\text{ MB}$.
- **Access Speed:** O(1) random access by index; sequential streaming utilizes L1/L2 hardware prefetchers with zero cache line waste.
### 2.3 `manager.by_type[kind].arena`: Typed Component Arenas
@@ -193,11 +193,11 @@ The canonical allocation function is defined in `EntityManager.h`:
- A new `Entity` record is appended to `manager.Entities`.
- A new typed block of `classPtr->size_of` bytes is allocated in `manager.by_type[classPtr->kind].arena`.
- The derived memory is zeroed.
- `base->Derived` points to the derived struct.
- `derived->Base` points to the base `Entity`.
- `base->Kind` is assigned to `classPtr`.
- `base->derived` points to the derived struct.
- `derived->base` points to the base `Entity`.
- `base->derived_kind` is assigned to `class_ptr`.
- `base->ID` is assigned the next unique `EntityManager::ID`.
- `base->IsDirty` is initialized to `true`.
- `base->is_dirty` is initialized to `true`.
- `typed_entity_array::count` is incremented.
- `typed_entity_array::array` is initialized if this is the first entity of this type.
@@ -214,13 +214,11 @@ The implementation replaces the flawed `RegisterEntity` routine in [`Game/Entity
// 1. Allocate uninitialized Base Entity in the contiguous VectorArena
Entity baseTemplate{};
baseTemplate.ID = EntityManager::ID++;
baseTemplate.DerivedKind = derivedClassPtr;
baseTemplate.Derived = nullptr;
baseTemplate.X = 0.0f;
baseTemplate.Y = 0.0f;
baseTemplate.Z = 0.0f;
baseTemplate.IsDirty = true;
baseTemplate.ID = EntityManager::ID++;
baseTemplate.derived_kind = derivedClassPtr;
baseTemplate.derived = nullptr;
baseTemplate.position = {};
baseTemplate.is_dirty = true;
manager.Entities.PushBack(baseTemplate);
Entity* basePtr = manager.Entities.Back();
@@ -343,16 +341,15 @@ Inert
#### Important: No Header Structs for Derived Types
- **Derived types NEVER require their own file header**: You do **not** write an `InertHeader`, `DoorHeader`, or `PlayerHeader`. Derived types only serialize their own member variables.
- **Universal `; version` + optional `; class_version`**: Every `.jasset` file has a universal `; version` tag. For entity assets, `; version` governs base entity properties (`kEntityBaseVersion`), while an optional `; class_version` governs derived class properties (`Class::Version`). Non-entity assets like `WorldSettings.jasset` only have `; version`.
- **No binary `EntityFileHeader` struct is needed**: Under the `; variable_name\nvalues` text format, there is no packed binary C-struct header at all. The common properties (`; id`, `; class`, `; version`, `; class_version`, `; position`) are standard text Key-Value nodes read by the exact same `archive` parser.
- **No binary `EntityFileHeader` struct is needed**: Under the `; variable_name\nvalues` text format, there is no packed binary C-struct header at all. The common properties (`; id`, `; class`, `; version`, `; class_version`, `; position`) are standard text Key-Value nodes read by the exact same `Archive` parser.
### 4.2 Eliminating Intermediate Stack Allocations
Under the new pipeline:
1. The `.jasset` text file is read into memory onto a `TempArena` via `LoadFile`.
2. The lines are tokenized into a `TextPropertyNode` scratch table.
3. The engine reads `; class` (e.g. `"Inert"`) and resolves `Class* classPtr = FindClassByName(className)`.
4. `AllocateEntity(manager, classPtr)` is called immediately. Memory for both base and derived components is allocated in-place in their permanent engine arenas.
5. Base properties (`id`, `position`, etc.) are read directly into `*basePtr` via `SerializeEntityBase`.
6. `classPtr->serialize_fct(ar, basePtr->Derived, classVersion)` is called. Derived fields stream **directly into the typed arena** without temporary staging buffers or stack copies.
2. The property nodes are parsed into a `ParsedArchive` via `tokenize_archive(tempArena.Arena, fileBuffer, &ar.base)`.
3. The engine reads `; class` (e.g. `"Inert"`) and resolves `Class* class_ptr = find_class_by_name(class_name)`.
4. `AllocateEntity(manager, class_ptr)` is called immediately. Memory for both base and derived components is allocated in-place in their permanent engine arenas.
5. `serialize(ar, NonNullPtr<Entity>(base_ptr))` is called. Base (`Entity::kind`) and derived (`base_ptr->derived_kind`) fields stream **directly into their permanent memory arenas** without temporary staging buffers or stack copies.
### 4.3 Runtime Class Resolution
To ensure fast and safe type lookup during file deserialization:
@@ -387,73 +384,58 @@ To ensure fast and safe type lookup during file deserialization:
| In-Place Deserialization Flowchart |
+---------------------------------------------------------------------------------------+
| |
| 1. Read Header from IOStream/ByteBuffer |
| 1. LoadFile(scratch.Arena, filepath) into ByteBuffer |
| | |
| v |
| 2. Validate Magic ('JAST') and Version (1) |
| 2. tokenize_archive(scratch.Arena, file_buffer, &ar.base) |
| | |
| v |
| 3. Resolve Class* from header.Kind & header.ClassCRC |
| 3. Read "; class" & Resolve Class* via find_class_by_name(class_name) |
| | |
| v |
| 4. basePtr = AllocateEntity(manager, classPtr) |
| 4. base_ptr = AllocateEntity(manager, class_ptr) |
| | |
| +--> [manager.Entities]: Allocates base Entity |
| +--> [manager.by_type[kind].arena]: Allocates derived struct |
| +--> Mutual Back-Pointers Wired In-Place |
| | |
| v |
| 5. Direct Copy Base Properties (ID, X, Y, Z) |
| 5. serialize(ar, NonNullPtr<Entity>(base_ptr)) |
| | |
| +--> Streams Base Entity (Entity::kind, version, ID, position) |
| +--> Streams Derived Component (base_ptr->derived_kind, class_version) |
| | |
| v |
| 6. Does classPtr->serialize_fct exist? |
| | | |
| Yes No |
| | | |
| v | |
| Invoke: | |
| serialize_fct(&ar, | |
| Derived) | |
| | | |
| +---------------------+ |
| | |
| v |
| 7. Clear Dirty Flag: basePtr->IsDirty = false |
| 6. Clear Dirty Flag: base_ptr->is_dirty = false |
| |
+---------------------------------------------------------------------------------------+
```
```cpp
[[nodiscard]] Entity* DeserializeEntityInPlace(EntityManager& manager, archive& ar)
[[nodiscard]] Entity* deserialize_entity_in_place(EntityManager& manager, Archive& ar)
{
Assert(ar.loading);
// 1. Read class name and resolve Class*
String className = {};
SerializeProp(ar, "class", className, ar.arena);
Class* classPtr = FindClassByName(className);
if (!classPtr)
String class_name = {};
SERIALIZE(ar, class, class_name);
Class* class_ptr = find_class_by_name(class_name);
if (!class_ptr)
{
return nullptr;
}
// 2. Allocate persistent memory for base and derived components in their respective arenas
Entity* basePtr = AllocateEntity(manager, classPtr);
Assert(basePtr != nullptr);
Entity* base_ptr = AllocateEntity(manager, class_ptr);
Assert(base_ptr != nullptr);
// 3. Serialize Base Entity in-place using Entity::Kind
SerializeClassInstance(ar, Entity::Kind, basePtr);
// 4. Stream derived properties in-place using basePtr->DerivedKind
if (basePtr->DerivedKind != nullptr && basePtr->Derived != nullptr)
{
SerializeClassInstance(ar, basePtr->DerivedKind, basePtr->Derived);
}
// 3. Serialize Base Entity and Derived in-place
serialize(ar, NonNullPtr<Entity>(base_ptr));
// Freshly loaded entity matches disk state exactly
basePtr->IsDirty = false;
base_ptr->is_dirty = false;
return basePtr;
return base_ptr;
}
```
@@ -653,26 +635,24 @@ In a level containing $10{,}000$ entities:
- **Monolithic `world.bin` Save:** Modifying a single entity's $X$ coordinate requires re-serializing all $10{,}000$ entities and overwriting a multi-megabyte binary file. This introduces a huge Git diff and constant merge conflicts.
- **Full-Directory `.jasset` Save:** Iterating through all $10{,}000$ entities and unconditionally writing $10{,}000$ `.jasset` files incurs massive OS file-system overhead (directory table locks, I/O bandwidth) and changes the file timestamps of every asset. Git reports thousands of modified files even when only one entity changed!
### 6.2 The `IsDirty` Flag on `Entity`
### 6.2 The `is_dirty` Flag on `Entity`
To solve this, `Entity` in [`Game/Entity/Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h#L28-L36) is augmented with an explicit dirty flag:
```cpp
struct Entity final
{
DECLARE_ENTITY() // static Class* Kind; (Entity's own Class descriptor)
DECLARE_CLASS() // static Class* kind; (Entity's own Class descriptor)
EntityID ID = 0;
Class* DerivedKind = nullptr; // Pointer to derived class descriptor (e.g. Inert::Kind)
DerivedType Derived = nullptr; // Pointer to derived component memory
float X = 0.0f;
float Y = 0.0f;
float Z = 0.0f;
bool IsDirty = false;
EntityID ID = 0;
Class* derived_kind = nullptr; // Pointer to derived class descriptor (e.g. Inert::kind)
DerivedType derived = nullptr; // Pointer to derived component memory
Vector4 position = {};
bool is_dirty = false;
};
```
### 6.3 Granular State Transitions
The `IsDirty` flag obeys a strict lifecycle state machine:
The `is_dirty` flag obeys a strict lifecycle state machine:
```
+-----------------------------------+
@@ -682,25 +662,25 @@ The `IsDirty` flag obeys a strict lifecycle state machine:
|
v
+---------------+
+------->| IsDirty: TRUE |<-------+
+------->|is_dirty: TRUE |<-------+
| +-------+-------+ |
| | |
Entity Mutated | SaveWorld
(Position, Component) | Completed
| v |
| +---------------+ |
+--------+ IsDirty: FALSE+--------+
+--------+is_dirty: FALSE+--------+
+-------+-------+
^
|
Deserialization
Deserialization
(LoadWorld / Asset)
```
1. **Entity Creation:** Newly spawned entities in the editor have `IsDirty = true`.
2. **Property Mutation:** Any modification to `X, Y, Z` or derived component payload sets `entity->IsDirty = true`.
3. **Successful Deserialization:** Entities loaded from disk initialize with `IsDirty = false`.
4. **Successful Save:** Upon successfully writing an entity to its `.jasset` file, the engine resets `entity->IsDirty = false`.
1. **Entity Creation:** Newly spawned entities in the editor have `is_dirty = true`.
2. **Property Mutation:** Any modification to `position` or derived component payload sets `entity->is_dirty = true`.
3. **Successful Deserialization:** Entities loaded from disk initialize with `is_dirty = false`.
4. **Successful Save:** Upon successfully writing an entity to its `.jasset` file, the engine resets `entity->is_dirty = false`.
### 6.4 Version Control Benefits (Git Friendly Assets)
By coupling the one-file-per-entity `.jasset` format with dirty tracking:
@@ -712,13 +692,13 @@ By coupling the one-file-per-entity `.jasset` format with dirty tracking:
In [`Game/Data/World.cpp`](file:///w:/Classified/Juliet/Game/Data/World.cpp#L291-L311), editor UI widgets automatically set the dirty flag upon receiving user input:
```cpp
float pos[3] = { ent.X, ent.Y, ent.Z };
float pos[4] = { ent.position.x, ent.position.y, ent.position.z, ent.position.w };
if (ImGui::DragFloat3("Position", pos, 0.1f))
{
ent.X = pos[0];
ent.Y = pos[1];
ent.Z = pos[2];
ent.IsDirty = true; // Mark dirty for persistence
ent.position.x = pos[0];
ent.position.y = pos[1];
ent.position.z = pos[2];
ent.is_dirty = true; // Mark dirty for persistence
UpdateWorld(world);
}
```
@@ -729,13 +709,12 @@ if (ImGui::DragFloat3("Position", pos, 0.1f))
### Phase 1: Data Structures & Header Definitions
1. **Update `Entity.h`:**
- Add `bool IsDirty = false;` to `struct Entity`.
- Add `bool is_dirty = false;` to `struct Entity`.
- Update `MakeEntity<EntityType>` to delegate to `AllocateEntity`.
- Define `EntityFileHeader`, `kEntityAssetMagic`, and `kEntityAssetVersion`.
2. **Update `EntityManager.h`:**
- Declare `[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr);`.
- Declare `[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* class_ptr);`.
- Declare `void DestroyEntity(EntityManager& manager, EntityID id);`.
- Declare `void RemoveDerivedComponent(EntityManager& manager, Class* classPtr, DerivedType derivedPtr);`.
- Declare `void RemoveDerivedComponent(EntityManager& manager, Class* class_ptr, DerivedType derived_ptr);`.
3. **Update `World.h`:**
- Add `VectorArena<EntityID, 1024> PendingDeletions;` to `struct World`.
- Update `SaveWorld` and `LoadWorld` signatures to take directory paths.
@@ -745,7 +724,7 @@ if (ImGui::DragFloat3("Position", pos, 0.1f))
- Enforce parameter assertions.
- Push to `manager.Entities`.
- Allocate zeroed block in `manager.by_type[kind].arena`.
- Wire mutual pointers (`base->Derived` and `derived->Base`).
- Wire mutual pointers (`base->derived` and `derived->base`).
- Increment `typedArray.count` and initialize `typedArray.array`.
2. Implement `DestroyEntity` & `RemoveDerivedComponent`:
- Implement swap-and-pop for derived components with base pointer update.
@@ -753,27 +732,25 @@ if (ImGui::DragFloat3("Position", pos, 0.1f))
### Phase 3: In-Place Deserialization & Serialization Pipeline
1. In `Entity.cpp`:
- Implement `EntitySerialize(archive& ar, Entity* entity)`.
- Fix the existing reversed `if (ar.loading)` branch.
- `serialize(Archive& ar, NonNullPtr<Entity> entity)` handles both base and derived class serialization.
2. In `World.cpp`:
- Implement `SerializeEntityAsset(archive& ar, Entity* entity, String filepath)`.
- Implement `DeserializeEntityAsset(EntityManager& manager, archive& ar, String filepath)`.
- Stream derived properties directly using `classPtr->serialize_fct(&ar, entity->Derived)`.
- Implement `serialize_entity_asset(Archive& ar, NonNullPtr<Entity> entity, String filepath)`.
- Implement `deserialize_entity_asset(EntityManager& manager, Archive& ar, String filepath)`.
### Phase 4: World Save/Load Pipeline & Disk Deletion
1. In `World.cpp`:
- Implement `ProcessPendingDeletions(World& world, NonNullPtr<Arena> scratchArena)`.
- Implement `SaveWorld(World& world, String worldDirectory)`:
- Process pending deletions.
- Iterate `manager.Entities`, skipping entities where `!entity.IsDirty`.
- Write dirty entities to `.jasset` files and clear `IsDirty`.
- Iterate `manager.Entities`, skipping entities where `!entity.is_dirty`.
- Write dirty entities to `.jasset` files and clear `is_dirty`.
- Implement `LoadWorld(World& world, String worldDirectory)`:
- Enumerate `.jasset` files in directory.
- Call `DeserializeEntityAsset` for each file.
- Call `deserialize_entity_asset` for each file.
### Phase 5: Editor Integration
1. In `RenderWorldEditorUI`:
- Hook `ImGui::DragFloat3` and property inspectors to set `IsDirty = true`.
- Hook `ImGui::DragFloat3` and property inspectors to set `is_dirty = true`.
- Hook "Add Entity" button to call `MakeEntity<Inert>(*world.EntityManager, 0.0f, 0.0f, 0.0f)`.
- Hook "Delete Entity" button to call `RemoveWorldEntity(world, selectedEntityId)`.
@@ -824,26 +801,26 @@ namespace UnitTest
EntityManager& manager = *testWorld.EntityManager;
// 1. Allocate Inert Entity via AllocateEntity
Entity* baseEntity = AllocateEntity(manager, Inert::Kind);
Assert(baseEntity != nullptr);
Assert(baseEntity->ID > 0);
Assert(baseEntity->Kind == Inert::Kind);
Assert(baseEntity->Derived != nullptr);
Assert(baseEntity->IsDirty == true);
Entity* base_entity = AllocateEntity(manager, Inert::kind);
Assert(base_entity != nullptr);
Assert(base_entity->ID > 0);
Assert(base_entity->derived_kind == Inert::kind);
Assert(base_entity->derived != nullptr);
Assert(base_entity->is_dirty == true);
// 2. Validate mutual back-pointer wiring
auto* derived = reinterpret_cast<entity_template*>(baseEntity->Derived);
Assert(derived->Base == baseEntity);
auto* derived = reinterpret_cast<entity_template*>(base_entity->derived);
Assert(derived->base == base_entity);
// 3. DownCast verification
Inert* inert = DownCast<Inert>(baseEntity);
Inert* inert = DownCast<Inert>(base_entity);
Assert(inert != nullptr);
Assert(inert->Base == baseEntity);
Assert(inert->base == base_entity);
// 4. Validate typed array tracking
typed_entity_array& inertArray = manager.by_type[ENTITY(Inert)];
Assert(inertArray.count == 1);
Assert(inertArray.array == derived);
typed_entity_array& inert_array = manager.by_type[ENTITY(Inert)];
Assert(inert_array.count == 1);
Assert(inert_array.array == derived);
ShutdownEntityManager();
ShutdownWorld(&testWorld);
@@ -864,50 +841,39 @@ namespace UnitTest
EntityManager& manager = *testWorld.EntityManager;
// 1. Create and populate entity
Inert* createdInert = MakeEntity<Inert>(manager, 12.5f, -44.0f, 108.2f);
Assert(createdInert != nullptr);
createdInert->MeshInstance = 42;
Inert* created_inert = MakeEntity<Inert>(manager, 12.5f, -44.0f, 108.2f);
Assert(created_inert != nullptr);
created_inert->MeshInstance = 42;
Entity* originalBase = createdInert->Base;
EntityID originalID = originalBase->ID;
Entity* original_base = created_inert->base;
EntityID original_id = original_base->ID;
// 2. Serialize to memory archive
archive saveAr{ .arena = tempArena.Arena, .base_ptr = nullptr, .offset = 0, .loading = false };
saveAr.base_ptr = ArenaPushArray<uint8>(tempArena.Arena, Kilobytes(16));
EntityFileHeader header{
.Magic = kEntityAssetMagic,
.Version = kEntityAssetVersion,
.ClassCRC = originalBase->Kind->CRC,
.Kind = originalBase->Kind->kind,
.EntityID = originalBase->ID,
.PositionX = originalBase->X,
.PositionY = originalBase->Y,
.PositionZ = originalBase->Z,
.PayloadSize = sizeof(index_t)
};
serialize_elem(saveAr, header);
serialize_elem(saveAr, createdInert->MeshInstance);
// 2. Serialize to text archive memory stream
MemoryStream mem_stream = MakeMemoryStream(tempArena.Arena);
Archive save_ar{ .arena = tempArena.Arena, .loading = false, .stream = &mem_stream };
serialize(save_ar, NonNullPtr<Entity>(original_base));
// 3. Clear manager to simulate fresh load
ShutdownEntityManager();
InitEntityManager(&testWorld);
EntityManager& freshManager = *testWorld.EntityManager;
EntityManager& fresh_manager = *testWorld.EntityManager;
// 4. Deserialize in-place
archive loadAr{ .arena = tempArena.Arena, .base_ptr = saveAr.base_ptr, .offset = 0, .loading = true };
Entity* loadedBase = DeserializeEntityInPlace(freshManager, loadAr);
// 4. Tokenize and deserialize in-place
Archive load_ar{ .arena = tempArena.Arena, .loading = true };
tokenize_archive(tempArena.Arena, mem_stream.buffer, &load_ar.base);
Entity* loaded_base = deserialize_entity_in_place(fresh_manager, load_ar);
Assert(loadedBase != nullptr);
Assert(loadedBase->ID == originalID);
Assert(loadedBase->X == 12.5f);
Assert(loadedBase->Y == -44.0f);
Assert(loadedBase->Z == 108.2f);
Assert(loadedBase->IsDirty == false);
Assert(loaded_base != nullptr);
Assert(loaded_base->ID == original_id);
Assert(loaded_base->position.x == 12.5f);
Assert(loaded_base->position.y == -44.0f);
Assert(loaded_base->position.z == 108.2f);
Assert(loaded_base->is_dirty == false);
Inert* loadedInert = DownCast<Inert>(loadedBase);
Assert(loadedInert != nullptr);
Assert(loadedInert->Base == loadedBase);
Inert* loaded_inert = DownCast<Inert>(loaded_base);
Assert(loaded_inert != nullptr);
Assert(loaded_inert->base == loaded_base);
Assert(loaded_inert->MeshInstance == 42);
ShutdownEntityManager();
ShutdownWorld(&testWorld);
@@ -932,9 +898,9 @@ namespace UnitTest
Inert* e1 = MakeEntity<Inert>(manager, 2.0f, 0.0f, 0.0f);
Inert* e2 = MakeEntity<Inert>(manager, 3.0f, 0.0f, 0.0f);
EntityID id0 = e0->Base->ID;
EntityID id1 = e1->Base->ID;
EntityID id2 = e2->Base->ID;
EntityID id0 = e0->base->ID;
EntityID id1 = e1->base->ID;
EntityID id2 = e2->base->ID;
Assert(manager.Entities.Size() == 3);
@@ -950,11 +916,11 @@ namespace UnitTest
Assert(remaining0->ID == id0);
Assert(remaining1->ID == id2);
auto* derived0 = reinterpret_cast<entity_template*>(remaining0->Derived);
auto* derived1 = reinterpret_cast<entity_template*>(remaining1->Derived);
auto* derived0 = reinterpret_cast<entity_template*>(remaining0->derived);
auto* derived1 = reinterpret_cast<entity_template*>(remaining1->derived);
Assert(derived0->Base == remaining0);
Assert(derived1->Base == remaining1);
Assert(derived0->base == remaining0);
Assert(derived1->base == remaining1);
ShutdownEntityManager();
ShutdownWorld(&testWorld);
@@ -974,17 +940,17 @@ namespace UnitTest
InitEntityManager(&testWorld);
EntityManager& manager = *testWorld.EntityManager;
Entity* entity = AllocateEntity(manager, Inert::Kind);
Assert(entity->IsDirty == true);
Entity* entity = AllocateEntity(manager, Inert::kind);
Assert(entity->is_dirty == true);
// Simulate save
entity->IsDirty = false;
Assert(entity->IsDirty == false);
entity->is_dirty = false;
Assert(entity->is_dirty == false);
// Simulate mutation
entity->X += 1.0f;
entity->IsDirty = true;
Assert(entity->IsDirty == true);
entity->position.x += 1.0f;
entity->is_dirty = true;
Assert(entity->is_dirty == true);
ShutdownEntityManager();
ShutdownWorld(&testWorld);
Game/Plans/03_Entity_ID_And_World_Directory.md
+29 -66
View File
@@ -654,10 +654,8 @@ To guarantee robust loading regardless of traversal order:
#### Phase 1: Allocation, Deserialization, and ID Registration
1. For each `.jasset` file found in `Entities/`:
- Read the file buffer via `LoadFile`.
- Validate `EntityFileHeader` (Magic `kEntityMagic`, Version `kEntityVersion`).
- Push an `Entity` record into `EntityManager::Entities`.
- Allocate the type-specific memory block in `EntityManager::by_type[Kind].arena`.
- Copy base transform values (`X, Y, Z`) and invoke `Class::serialize_fct`.
- Tokenize text properties via `tokenize_archive`.
- Call `deserialize_entity_in_place(entityManager, ar)` which reads `; class`, invokes `AllocateEntity`, and streams base and derived fields in-place.
- All entity references are left as raw `EntityID` values (or embedded within components).
- Register the entity into a temporary lookup table: `EntityID -> Entity*`.
- Call `ObserveEntityIDForCounterContinuity` to maintain the high-water mark.
@@ -680,8 +678,8 @@ struct EntityLookupEntry
struct WorldLoadContext
{
World* TargetWorld;
Arena* TempMapArena;
World* TargetWorld;
Arena* TempMapArena;
VectorArena<EntityLookupEntry, 65'536> LookupTable;
};
@@ -699,72 +697,38 @@ void EntityFileDiscoveryCallback(String filename, String fullPath, bool isDirect
TempArena scratch = scratch_begin(0, 0);
ByteBuffer fileBuffer = LoadFile(scratch.Arena, fullPath);
if (fileBuffer.Size < sizeof(EntityFileHeader))
if (fileBuffer.Size == 0)
{
Log(LogLevel::Error, LogCategory::Game, "LoadWorld: Entity file '%s' corrupted (too small)", CStr(filename));
Log(LogLevel::Error, LogCategory::Game, "LoadWorld: Entity file '%s' is empty or missing", CStr(filename));
scratch_end(scratch);
return;
}
const auto* header = reinterpret_cast<const EntityFileHeader*>(fileBuffer.Data);
if (header->Magic != 0x544E454A || header->Version != 1)
{
Log(LogLevel::Error, LogCategory::Game, "LoadWorld: File '%s' has invalid header", CStr(filename));
scratch_end(scratch);
return;
}
// Phase 1: Allocate & deserialize
Entity baseEntity = {};
baseEntity.ID = header->ID;
baseEntity.Kind = kEntity_type_class_ptr[header->Kind];
Assert(baseEntity.Kind != nullptr);
archive ar = {
.arena = scratch.Arena,
.base_ptr = fileBuffer.Data + sizeof(EntityFileHeader),
.offset = 0,
.loading = true
Archive ar = {
.arena = scratch.Arena,
.loading = true
};
// Deserialize base entity properties
serialize_elem(ar, baseEntity.X);
serialize_elem(ar, baseEntity.Y);
serialize_elem(ar, baseEntity.Z);
// Register into EntityManager
entityManager.Entities.PushBack(baseEntity);
Entity* registeredBase = entityManager.Entities.Back();
// Allocate derived type payload
auto* derivedPtr = static_cast<entity_template*>(ArenaPushSize(
entityManager.ByType[header->Kind].arena,
baseEntity.Kind->size_of,
baseEntity.Kind->alignment,
false JULIET_DEBUG_PARAM(kEntity_type_names[header->Kind])
));
Assert(derivedPtr != nullptr);
// Deserialize derived data if class registered a serialize function
if (baseEntity.Kind->serialize_fct != nullptr)
if (!tokenize_archive(scratch.Arena, fileBuffer, &ar.base))
{
baseEntity.Kind->serialize_fct(ar, derivedPtr);
Log(LogLevel::Error, LogCategory::Game, "LoadWorld: Entity file '%s' failed to tokenize", CStr(filename));
scratch_end(scratch);
return;
}
derivedPtr->Base = registeredBase;
registeredBase->Derived = derivedPtr;
if (entityManager.ByType[header->Kind].array == nullptr)
// Phase 1: Allocate & deserialize in-place
Entity* registeredBase = deserialize_entity_in_place(entityManager, ar);
if (registeredBase == nullptr)
{
entityManager.ByType[header->Kind].array = derivedPtr;
Log(LogLevel::Error, LogCategory::Game, "LoadWorld: Entity file '%s' failed to deserialize", CStr(filename));
scratch_end(scratch);
return;
}
entityManager.ByType[header->Kind].count += 1;
// Track in temporary lookup table for Phase 2
context->LookupTable.PushBack({ .ID = header->ID, .EntityPtr = registeredBase });
context->LookupTable.PushBack({ .ID = registeredBase->ID, .EntityPtr = registeredBase });
// Update session counter continuity
ObserveEntityIDForCounterContinuity(entityManager, header->ID);
ObserveEntityIDForCounterContinuity(entityManager, registeredBase->ID);
scratch_end(scratch);
}
@@ -838,15 +802,15 @@ void PostLoadWorld(World& world, const WorldLoadContext& context)
String settingsPath = { settingsPathBuf, settingsPathLen - 1 };
ByteBuffer settingsBuffer = LoadFile(scratch.Arena, settingsPath);
if (settingsBuffer.Size >= sizeof(WorldSettingsFileHeader))
if (settingsBuffer.Size > 0)
{
const auto* header = reinterpret_cast<const WorldSettingsFileHeader*>(settingsBuffer.Data);
if (header->Magic == 0x5453574A && header->Version == 1)
Archive settingsAr = {
.arena = scratch.Arena,
.loading = true
};
if (tokenize_archive(scratch.Arena, settingsBuffer, &settingsAr.base))
{
const auto* settings = reinterpret_cast<const WorldEnvironmentSettings*>(
settingsBuffer.Data + sizeof(WorldSettingsFileHeader)
);
world.Environment = *settings;
serialize(settingsAr, world.Environment);
Log(LogLevel::Message, LogCategory::Game, "LoadWorld: Loaded WorldSettings from %s", CStr(settingsPath));
}
}
@@ -912,9 +876,8 @@ graph TD
- `[NEW]` [EntityFileIO.h](file:///w:/Classified/Juliet/Game/Entity/EntityFileIO.h)
- `[NEW]` [EntityFileIO.cpp](file:///w:/Classified/Juliet/Game/Entity/EntityFileIO.cpp)
- **Deliverables**:
- Define `WorldSettingsFileHeader` ('JWST') and `EntityFileHeader` ('JENT').
- Implement `AtomicWriteEntityFile` using `.tmp` and Win32 `MoveFileExA`.
- Implement serialization routines for `WorldEnvironmentSettings`.
- Implement text serialization routines for `WorldEnvironmentSettings` and entity `.jasset` files.
### Phase 4: Two-Phase Load Pipeline & EntityManager Refactoring
- **Files**:
Game/Plans/04_Entity_Templates_And_Inheritance.md
+36 -35
View File
@@ -206,13 +206,13 @@ When `GetOrLoadTemplate` is invoked with a relative path:
6. **Default Base Setup**: Initialize a local `Entity defaultBase`:
```cpp
Entity defaultBase = {};
defaultBase.Kind = entityClass;
defaultBase.Derived = archetypeMem;
defaultBase.derived_kind = entityClass;
defaultBase.derived = archetypeMem;
```
7. **Back-Pointer Linking**: Set `entity_template::Base` in the archetype memory:
7. **Back-Pointer Linking**: Set `entity_template::base` in the archetype memory:
```cpp
auto* archetypeTemplate = reinterpret_cast<entity_template*>(archetypeMem);
archetypeTemplate->Base = &cachedEntry->DefaultBase;
archetypeTemplate->base = &cachedEntry->DefaultBase;
```
8. **KV Property Parsing**: Parse all key-value pairs in the template `.jasset` file and write their deserialized values directly into `archetypeMem` and `defaultBase`.
9. **Cache Insertion**: Store the fully baked `CachedTemplate` record in `cache->Templates`.
@@ -228,21 +228,22 @@ In Juliet, an entity is split into two tightly coupled structures:
```cpp
struct Entity final
{
EntityID ID = 0;
Class* Kind = nullptr;
DerivedType Derived = nullptr; // Points to the specialized struct
float X = 0.0f;
float Y = 0.0f;
float Z = 0.0f;
DECLARE_CLASS()
EntityID ID = 0;
Class* derived_kind = nullptr;
DerivedType derived = nullptr; // Points to the specialized struct
Vector4 position = {};
bool is_dirty = false;
};
```
2. **`Derived` (Specialized Type)**: e.g., `Inert`, `Collectible`, `Player`. The first member is always `DECLARE_ENTITY()`, which expands to:
```cpp
Entity* Base; // Back-pointer to the base Entity
static Class* Kind;
Entity* base; // Back-pointer to the base Entity
DECLARE_CLASS() // static Class* kind;
```
Because `DerivedType` stores a back-pointer (`Base`) to `Entity`, **a shallow memory copy of an archetype invalidates this pointer**! The loading pipeline must explicitly restore this invariant.
Because `DerivedType` stores a back-pointer (`base`) to `Entity`, **a shallow memory copy of an archetype invalidates this pointer**! The loading pipeline must explicitly restore this invariant.
### 4.2 The 5-Step Instantiation Pipeline
When an entity instance is spawned or deserialized from a `.jasset` file, the engine executes this strict 5-step sequence:
@@ -400,12 +401,12 @@ void ApplyKvDeltaOverrides(Entity* base, void* derived, Class* cls, String kvCon
}
// Check base properties first
if (StringCompare(pair.Key, ConstString("Position")) == 0)
if (StringCompare(pair.Key, ConstString("Position")) == 0 || StringCompare(pair.Key, ConstString("position")) == 0)
{
Vector3 pos = ParseVector3(pair.Value);
base->X = pos.X;
base->Y = pos.Y;
base->Z = pos.Z;
base->position.x = pos.x;
base->position.y = pos.y;
base->position.z = pos.z;
continue;
}
@@ -483,8 +484,8 @@ bool CreateTemplateFromEntity(World& world,
Assert(entityIndex < manager.Entities.Size());
Entity* sourceEntity = &manager.Entities[entityIndex];
Class* entityKind = sourceEntity->Kind;
void* derivedMem = sourceEntity->Derived;
Class* entityKind = sourceEntity->derived_kind;
void* derivedMem = sourceEntity->derived;
// Format destination template path
String templatePath = Format(scratchArena, "Assets/Templates/{}.jasset", CStr(templateName));
@@ -561,11 +562,11 @@ void RenderEntityPropertyInspector(Entity* entity, CachedTemplate* archetype)
}
// Iterate through properties
Class* cls = entity->Kind;
Class* cls = entity->derived_kind;
for (size_t i = 0; i < cls->PropertyCount; ++i)
{
const PropertyDescriptor& prop = cls->Properties[i];
void* instanceField = static_cast<uint8*>(entity->Derived) + prop.Offset;
void* instanceField = static_cast<uint8*>(entity->derived) + prop.Offset;
void* templateField = isTemplated ? (static_cast<uint8*>(archetype->DefaultDerivedMemory) + prop.Offset) : nullptr;
const bool isOverridden = isTemplated && (MemCompare(instanceField, templateField, GetPropertySize(prop.Type)) != 0);
@@ -605,10 +606,10 @@ Reverting an entire entity to its template archetype restores all properties whi
```cpp
void RevertEntityToTemplate(NonNullPtr<Entity> instance, NonNullPtr<const CachedTemplate> archetype)
{
Assert(instance->Kind == archetype->EntityKind);
Assert(instance->derived_kind == archetype->EntityKind);
Assert(archetype->DefaultDerivedMemory != nullptr);
void* derivedMem = instance->Derived;
void* derivedMem = instance->derived;
const size_t derivedSize = archetype->EntityKind->size_of;
// Preserve the current Base pointer
@@ -619,16 +620,16 @@ void RevertEntityToTemplate(NonNullPtr<Entity> instance, NonNullPtr<const Cached
// 2. Re-establish the Base back-pointer
auto* templateDerived = reinterpret_cast<entity_template*>(derivedMem);
templateDerived->Base = basePtr;
templateDerived->base = basePtr;
// 3. Mark visual / physics state as updated
if (instance->Kind->kind == ENTITY(Inert))
if (instance->derived_kind->kind == ENTITY(Inert))
{
auto* inert = reinterpret_cast<Inert*>(derivedMem);
if (inert->MeshInstance != indexMax)
{
SetMeshInstanceTransform(inert->MeshInstance,
MatrixTranslation(basePtr->X, basePtr->Y, basePtr->Z));
MatrixTranslation(basePtr->position.x, basePtr->position.y, basePtr->position.z));
}
}
}
@@ -844,17 +845,17 @@ namespace UnitTest
Assert(instance != nullptr);
// Verify Step 1 & 2: Base spatial delta applied, non-overridden derived property retained
Assert(instance->X == 100.0f);
Assert(instance->Y == 200.0f);
Assert(instance->Z == 300.0f);
Assert(instance->position.x == 100.0f);
Assert(instance->position.y == 200.0f);
Assert(instance->position.z == 300.0f);
auto* inertDerived = DownCast<Inert>(instance);
Assert(inertDerived != nullptr);
Assert(inertDerived->MeshInstance == 42); // Retained from template!
// Verify Step 3: CRITICAL back-pointer fixup check
Assert(inertDerived->Base == instance);
Assert(instance->Derived == inertDerived);
Assert(inertDerived->base == instance);
Assert(instance->derived == inertDerived);
ShutdownEntityManager();
ShutdownWorld(&world);
@@ -900,12 +901,12 @@ namespace UnitTest
Assert(inertDerived->MeshInstance == 100);
// Verify world position is preserved across revert
Assert(instance->X == 5.0f);
Assert(instance->Y == 5.0f);
Assert(instance->Z == 5.0f);
Assert(instance->position.x == 5.0f);
Assert(instance->position.y == 5.0f);
Assert(instance->position.z == 5.0f);
// Verify back-pointer invariant preserved after revert
Assert(inertDerived->Base == instance);
Assert(inertDerived->base == instance);
ShutdownEntityManager();
ShutdownWorld(&world);