# Juliet Game Engine Architecture Specification ## Document: 02 - Entity Allocation & In-Place Lifecycle **Document ID:** JULIET-SPEC-002 **Status:** Approved for Implementation **Author:** Senior Engine Architect **Subsystems:** `Game/Entity`, `Game/Data`, `Game/UnitTest` **Target Files:** - [`Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h) - [`Entity.cpp`](file:///w:/Classified/Juliet/Game/Entity/Entity.cpp) - [`EntityManager.h`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.h) - [`EntityManager.cpp`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.cpp) - [`World.h`](file:///w:/Classified/Juliet/Game/Data/World.h) - [`World.cpp`](file:///w:/Classified/Juliet/Game/Data/World.cpp) - [`WorldUnitTest.h`](file:///w:/Classified/Juliet/Game/UnitTest/WorldUnitTest.h) - [`WorldUnitTest.cpp`](file:///w:/Classified/Juliet/Game/UnitTest/WorldUnitTest.cpp) --- ## 1. Executive Summary & Problem Statement ### 1.1 Background & Context The Juliet game engine organizes game entities using a hybrid data-oriented architecture: 1. A flat array of **Base Entities** (`Entity`) encapsulating universal properties: unique 64-bit ID, runtime type reflection pointer (`Class* Kind`), spatial coordinates (`X, Y, Z`), and an opaque pointer to the derived payload (`DerivedType Derived`). 2. Type-segregated contiguous arrays of **Derived Entities** (`Inert`, etc.) stored in dedicated per-type memory arenas (`typed_entity_array`). This design is intended to provide maximum cache efficiency during spatial and general-purpose entity processing, while retaining dense SIMD-friendly streaming for type-specific systems (e.g., transform updates on `Inert` mesh instances). ### 1.2 Root-Cause Analysis of `Game/Data/World.cpp` Loading Flaws In the initial implementation of entity serialization in [`Game/Data/World.cpp`](file:///w:/Classified/Juliet/Game/Data/World.cpp#L40-L68), loading was fundamentally broken and incomplete: ```cpp // Existing flawed deserialization in World.cpp for (typed_entity_array& type : entityManager.by_type) { serialize_elem(ar, type.count); if (type.count > 0) { // Unserialize the base entity to get informations Entity entity; serialize(ar, &entity); RegisterBaseEntity(entityManager, entity); } } ``` This implementation suffers from several fatal defects: 1. **Single-Element Iteration Bug:** It uses `if (type.count > 0)` instead of a loop `for (size_t i = 0; i < type.count; ++i)`, deserializing at most one single entity per type bucket, leaving all subsequent entities in the stream unread and corrupting the archive read offset. 2. **Missing Derived Allocation:** It invokes `RegisterBaseEntity(entityManager, entity)`, which merely pushes the stack-allocated `Entity` into `manager.Entities`. The derived payload arena (`type.arena`) is completely untouched: `type.array` remains null, `type.count` in the manager is desynchronized, and `entity.Derived` remains unassigned or points to an invalid address. 3. **Invalid Pointer in `RegisterEntity`:** In [`Game/Entity/EntityManager.cpp`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.cpp#L46-L66), `RegisterEntity` assigns `base->Derived = entity` before pushing `*base` into `manager.Entities`. The parameter `entity` is a pointer to caller-provided memory (often stack-allocated in helper functions like `MakeEntity`). When `ArenaPushSize` later allocates the persistent derived memory block, `base->Derived` stored inside `manager.Entities.Back()` is **never updated**—it remains dangling, pointing to the transient caller stack! ### 1.3 The "Chicken-and-Egg" Stack Allocation Dilemma The existing registration API requires a pre-existing derived instance: ```cpp entity_template* RegisterEntity(EntityManager& manager, Entity* base, DerivedType entity); ``` During runtime programmatic creation via `MakeEntity()`, a temporary instance of `T` is created on the stack and passed by pointer: ```cpp template EntityType* MakeEntity(EntityManager& manager, float x, float y, float z) { EntityType result; // Stack allocation Entity base; // Stack allocation base.X = x; base.Y = y; base.Z = z; base.Kind = EntityType::Kind; return (EntityType*)RegisterEntity(manager, &base, &result); } ``` When loading an entity from a stream or disk file, **the type is not known at compile time**. The engine reads a runtime type tag (`uint8 kind` or `uint32 CRC`), looks up the reflection metadata (`Class*`), and must instantiate the entity dynamically. Because C++ does not permit allocating a dynamic struct of unknown type on the stack, and because Juliet strictly forbids heap allocations (`malloc`, `new`, `std::vector`), deserialization cannot construct a temporary instance on the stack to pass to `RegisterEntity`. This is the classic **chicken-and-egg memory problem**: - `RegisterEntity` requires an existing instance in memory to copy from. - Deserialization requires an allocated memory buffer to deserialize into. ### 1.4 Architectural Objectives This specification establishes a robust in-place lifecycle pipeline that completely eliminates stack temporaries and dynamic heap allocations: 1. **Direct In-Place Allocation:** Introduce `AllocateEntity(EntityManager& manager, Class* classPtr)` which allocates both the base `Entity` and the derived struct directly within their respective engine memory arenas. 2. **Bidirectional Pointer Integrity:** Wire mutual pointers (`base->derived` and `derived->base`) at allocation time before any field deserialization begins. 3. **In-Place Stream Deserialization:** Read class reflection metadata first, invoke `AllocateEntity`, and stream base and derived properties directly into arena-resident memory. 4. **Isolated Entity Assets (`.jasset`):** Transition from a monolithic `world.bin` to a modular one-file-per-entity architecture (`Assets/Entities/{ID}.jasset`). 5. **Dirty Tracking & Optimal Saves:** Introduce an `is_dirty` flag on `Entity` to avoid rewriting unchanged entity files, minimizing disk I/O and eliminating spurious Git repository modifications. 6. **Robust Deletion Lifecycle:** Decouple in-memory removal (`RemoveAtFast` with pointer fixup) from disk synchronization using `World::PendingDeletions`. --- ## 2. The Dual Arena Memory Model in `EntityManager` ### 2.1 The Need for Dual Storage Game engines execute systems with vastly different cache locality profiles: - **Spatial / Frustum Culling / Transform Sync:** Iterates every entity in the world, needing only `X, Y, Z`, bounding spheres, and base status flags. - **Specialized Logic / Render Updates:** Iterates only entities possessing specific components (e.g., `Inert` static meshes requiring instance transform updates to the GPU bindless descriptor table). Storing large monolithic polymorphic structs in a single array causes severe cache line pollution during spatial passes. Conversely, storing entities in fragmented individual allocations introduces cache misses and pointer-chasing overhead. Juliet resolves this with a **Dual Arena Memory Model**: ``` +---------------------------------------------------------------------------------------------+ | EntityManager | +---------------------------------------------------------------------------------------------+ | | | manager.Entities (VectorArena) | | +------------------------------------+------------------------------------+ | | | Entity 0 (ID=1001, X, Y, Z) | Entity 1 (ID=1002, X, Y, Z) | ... | | | Derived ------------------------+ | Derived ---------------------+ | | | +---------------------------------|--+------------------------------|-----+ | | | | | | v v | | manager.by_type[ENTITY(Inert)].arena | | +------------------------------------+------------------------------------+ | | | Inert 0 (MeshInstance=4) | Inert 1 (MeshInstance=12) | ... | | | Base ---------------------------+ | Base ------------------------+ | | | +---------------------------------|--+------------------------------|-----+ | | +---------------------------------+ | +---------------------------------------------------------------------------------------------+ ``` ### 2.2 `manager.Entities`: Cache-Friendly Base Entity Vector Base entities reside in a pre-reserved contiguous array: ```cpp VectorArena Entities; ``` - **Capacity:** Fixed reserve of 100,000 entities allocated from the `WorldArena`. - **Memory Footprint:** $$\text{sizeof(Entity)} = 8\text{ (ID)} + 8\text{ (derived\_kind)} + 8\text{ (derived)} + 16\text{ (position)} + 1\text{ (is\_dirty)} + 7\text{ (Padding)} = 48\text{ bytes}$$ Total reserved space: $100{,}000 \times 48\text{ bytes} \approx 4.8\text{ MB}$. - **Access Speed:** O(1) random access by index; sequential streaming utilizes L1/L2 hardware prefetchers with zero cache line waste. ### 2.3 `manager.by_type[kind].arena`: Typed Component Arenas Derived structs reside in per-type contiguous memory arenas: ```cpp struct typed_entity_array { Arena* arena; entity_template* array; size_t count; }; ``` - Each `Entity_Type` index owns an isolated `Arena*` allocated during `InitEntityManager`. - Allocations are packed linearly with alignment specified by `classPtr->alignment`. - `array` points directly to the first element in the arena, permitting typed array indexing: ```cpp Inert* inertArray = reinterpret_cast(manager.by_type[ENTITY(Inert)].array); ``` ### 2.4 Mutual Back-Pointer Architecture & Invariants Every entity instance consists of two mutually linked allocations: 1. `basePtr->Derived`: Points from `Entity` in `manager.Entities` to the derived struct in `by_type[kind].arena`. 2. `derivedPtr->Base`: Points from the derived struct (via `DECLARE_ENTITY()`) back to `Entity` in `manager.Entities`. #### Invariant Rules: 1. **Non-Null Invariant:** For any active entity, `basePtr->Derived != nullptr` and `reinterpret_cast(basePtr->Derived)->Base == basePtr`. 2. **Type Coherence Invariant:** `basePtr->Kind->kind == derivedTypeId`. 3. **Array Count Invariant:** $$\sum_{k=0}^{\text{ENTITY(Count)}-1} \text{manager.by\_type}[k].\text{count} == \text{manager.Entities.Size()}$$ ### 2.5 Pointer Stability in `VectorArena` `VectorArena::Create` executes `Reserve(ReserveSize)` on creation: ```cpp newManager->Entities.Create(world->WorldArena JULIET_DEBUG_PARAM("Entities")); ``` Because capacity ($100{,}000$) is fully reserved upfront in virtual address space, `VectorArena::PushBack` **never reallocates or moves existing memory**. Therefore: - Pointers to `Entity` elements in `manager.Entities` remain absolutely stable across allocations. - Derived struct `Base` pointers remain valid indefinitely unless an element is deleted. - Element removal via swap-and-pop alters memory positions, requiring systematic pointer fixups (addressed in Section 5.2). --- ## 3. The Solution: `AllocateEntity(EntityManager& manager, Class* classPtr)` ### 3.1 Function Signature & Contract The canonical allocation function is defined in `EntityManager.h`: ```cpp [[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr); ``` #### Preconditions: - `classPtr != nullptr`. - `classPtr->kind < ENTITY(Count)`. - `classPtr->size_of >= sizeof(entity_template)`. - `manager.Entities.Size() < manager.Entities.Capacity`. - `manager.by_type[classPtr->kind].arena != nullptr`. #### Postconditions: - A new `Entity` record is appended to `manager.Entities`. - A new typed block of `classPtr->size_of` bytes is allocated in `manager.by_type[classPtr->kind].arena`. - The derived memory is initialized with C++ struct defaults via `classPtr->default_init_fct` (or zeroed via `MemZero` if null). - `base->derived` points to the derived struct. - `derived->base` points to the base `Entity`. - `base->derived_kind` is assigned to `class_ptr`. - `base->ID` is initialized to `0` (unassigned; populated by `MakeEntity` or deserialization). - `base->is_dirty` is initialized to `true`. - `typed_entity_array::count` is incremented. - `typed_entity_array::array` is initialized if this is the first entity of this type. ### 3.2 Detailed Step-by-Step Implementation The implementation replaces the flawed `RegisterEntity` routine in [`Game/Entity/EntityManager.cpp`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.cpp): ```cpp [[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* derivedClassPtr) { Assert(derivedClassPtr != nullptr); Assert(derivedClassPtr->kind < ENTITY(Count)); Assert(derivedClassPtr->size_of >= sizeof(entity_template)); Assert(derivedClassPtr->alignment > 0); // 1. Allocate Base Entity in the contiguous VectorArena (ID is 0 until assigned by MakeEntity or deserialization) Entity baseTemplate{}; baseTemplate.ID = 0; baseTemplate.derived_kind = derivedClassPtr; baseTemplate.derived = nullptr; baseTemplate.position = {}; baseTemplate.is_dirty = true; manager.Entities.PushBack(baseTemplate); Entity* basePtr = manager.Entities.Back(); Assert(basePtr != nullptr); // 2. Allocate derived component memory in the typed arena typed_entity_array& typedArray = manager.by_type[derivedClassPtr->kind]; Assert(typedArray.arena != nullptr); void* rawMemory = ArenaPushSize( typedArray.arena, derivedClassPtr->size_of, derivedClassPtr->alignment, false JULIET_DEBUG_PARAM(kEntity_type_names[derivedClassPtr->kind])); Assert(rawMemory != nullptr); // 3. Initialize derived component defaults via Class reflection stub if (derivedClassPtr->default_init_fct != nullptr) { derivedClassPtr->default_init_fct(rawMemory); } else { MemZero(rawMemory, derivedClassPtr->size_of); } auto* derivedTemplate = reinterpret_cast(rawMemory); // 4. Establish mutual back-pointers basePtr->derived = rawMemory; derivedTemplate->base = basePtr; // 5. Update typed array tracking if (typedArray.array == nullptr) { typedArray.array = derivedTemplate; } typedArray.count += 1; return basePtr; } ``` ### 3.3 Default Struct Initialization via `Class::default_init_fct` #### The Problem with Zero-Only Initialization If allocation only zeroes memory (`MemZero` / `0x00`), any C++ member variables with non-zero defaults (such as `index_t MeshInstance = indexMax;` or `float Density = 1.0f;`) are populated with `0`. During deserialization, if a `.jasset` file lacks that property (e.g. an older file version or an optional field), `SERIALIZE` leaves the field untouched, meaning it incorrectly remains `0` rather than its intended default sentinel value! #### The Solution: Compile-Time Default Stub in `DEFINE_ENTITY_VERSIONED` Every entity descriptor `Class` includes a function pointer: ```cpp using default_init_fct_type = void (*)(void* payload); struct Class { ... default_init_fct_type default_init_fct = nullptr; }; ``` When registering an entity type with `DEFINE_ENTITY_VERSIONED`, the macro automatically defines a tiny type-safe stub that aggregate value-initializes the struct: ```cpp #define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct) \ inline void default_init_##entity(void* payload) \ { \ *static_cast(payload) = entity{}; \ } \ constexpr Class entityKind##entity = MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version), \ &classKindEntity, sizeof(entity), alignof(entity), \ (serialize_fct), default_init_##entity); \ Class* entity::kind = const_cast(&entityKind##entity); ``` #### Architectural Advantages: 1. **Zero Boilerplate**: Developers write member initializers once in the struct definition (`index_t MeshInstance = indexMax;`). 2. **Type-Agnostic Core**: `AllocateEntity` does not need to know any C++ struct types; it unconditionally calls `derivedClassPtr->default_init_fct(rawMemory)`. 3. **Robust Deserialization**: In `deserialize_entity_in_place`, newly allocated entities already hold their canonical C++ defaults. Any properties absent in the `.jasset` file naturally retain their correct initial values. 4. **No Dynamic Heap / Placement-New**: `*static_cast(payload) = entity{}` is pure aggregate value-assignment without `` headers or exceptions. ### 3.4 Bidirectional Pointer Wiring Notice the sequence: 1. `manager.Entities.PushBack(baseTemplate)` places the struct at its final, fixed arena address. 2. `base_ptr = manager.Entities.Back()` retrieves the persistent memory pointer. 3. `derived_class_ptr->default_init_fct(raw_memory)` initializes canonical struct defaults. 4. `derived_template->base = base_ptr` wires the derived back-pointer directly to this permanent location. 5. `base_ptr->derived = raw_memory` wires the base forward-pointer to the arena-allocated derived struct. No stack copying occurs. Neither pointer is ever left dangling. ### 3.5 Updating Type Counts and Array Cache `typed_entity_array` maintains: - `typedArray.count`: The exact count of active entities of this type. - `typedArray.array`: Pointer to the first element in the arena. When the first entity of a given type is allocated, `typedArray.array` is set to `derived_template`. Because the arena allocates sequentially, `typedArray.array[i]` can be indexed directly with stride `class_ptr->size_of` as long as memory remains contiguous. ### 3.6 Refactoring `MakeEntity` Template With `AllocateEntity` handling memory allocation, default member initialization, and mutual pointer wiring, `MakeEntity` in [`Game/Entity/Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h) becomes clean, safe, and stack-free: ```cpp template requires EntityConcept [[nodiscard]] EntityType* MakeEntity(EntityManager& manager, float x, float y, float z) { Entity* base_ptr = AllocateEntity(manager, EntityType::kind); Assert(base_ptr != nullptr); base_ptr->ID = EntityManager::ID++; base_ptr->position.x = x; base_ptr->position.y = y; base_ptr->position.z = z; base_ptr->position.w = 1.0f; return static_cast(base_ptr->derived); } ``` --- ## 4. In-Place Deserialization Pipeline ### 4.1 Asset Format Specification (`.jasset`) To achieve robust version control and modular streaming, Juliet adopts a human-readable, Git-diffable **one-file-per-entity** disk format with extension `.jasset`. ```ini ; asset_type entity_instance ; id 0x0100000000000042 ; class Inert ; version 1 ; class_version 1 ; position 0.43 0.32 1.56 ; mesh_instance 12 ``` #### Important: No Header Structs for Derived Types - **Derived types NEVER require their own file header**: You do **not** write an `InertHeader`, `DoorHeader`, or `PlayerHeader`. Derived types only serialize their own member variables. - **Universal `; version` + optional `; class_version`**: Every `.jasset` file has a universal `; version` tag. For entity assets, `; version` governs base entity properties (`kEntityBaseVersion`), while an optional `; class_version` governs derived class properties (`Class::Version`). Non-entity assets like `WorldSettings.jasset` only have `; version`. - **No binary `EntityFileHeader` struct is needed**: Under the `; variable_name\nvalues` text format, there is no packed binary C-struct header at all. The common properties (`; id`, `; class`, `; version`, `; class_version`, `; position`) are standard text Key-Value nodes read by the exact same `Archive` parser. ### 4.2 Eliminating Intermediate Stack Allocations Under the new pipeline: 1. The `.jasset` text file is read into memory onto a `TempArena` via `LoadFile`. 2. The property nodes are parsed into a `ParsedArchive` via `tokenize_archive(tempArena.Arena, fileBuffer, &ar.base)`. 3. The engine reads `; class` (e.g. `"Inert"`) and resolves `Class* class_ptr = find_class_by_name(class_name)`. 4. `AllocateEntity(manager, class_ptr)` is called immediately. Memory for both base and derived components is allocated in-place in their permanent engine arenas. 5. `serialize(ar, NonNullPtr(base_ptr))` is called. Base (`Entity::kind`) and derived (`base_ptr->derived_kind`) fields stream **directly into their permanent memory arenas** without temporary staging buffers or stack copies. ### 4.3 Runtime Class Resolution To ensure fast and safe type lookup during file deserialization, Juliet provides two tiers of type resolution: 1. **Low-Level Indexed Validation (`ResolveEntityClass`)**: An $O(1)$ array lookup into `kEntity_type_class_ptr[kind]` that validates the class pointer and verifies schema compatibility against `classPtr->CRC`. This is the core validation primitive used by binary streaming, network replication, and internal lookups. 2. **High-Level Name Resolution (`find_class_by_name`)**: Bridges the human-readable text archive format (`; class\nInert`) to the entity class registry. It computes the `crc32` of the parsed class name and queries `ResolveEntityClass` across registered entity kinds. ```cpp [[nodiscard]] const Class* ResolveEntityClass(uint8 kind, uint32 crc) { if (kind >= ENTITY(Count)) { return nullptr; } const Class* class_ptr = kEntity_type_class_ptr[kind]; if (class_ptr == nullptr) { return nullptr; } if (class_ptr->CRC != crc) { return nullptr; } return class_ptr; } [[nodiscard]] Class* find_class_by_name(String name) { if (!IsValid(name)) { return nullptr; } const uint32 name_crc = crc32(name.Str, name.Size); for (uint8 kind = 0; kind < ENTITY(Count); ++kind) { const Class* class_ptr = ResolveEntityClass(kind, name_crc); if (class_ptr != nullptr) { return const_cast(class_ptr); } } return nullptr; } ``` ### 4.4 In-Place Deserialization Algorithm ``` +---------------------------------------------------------------------------------------+ | In-Place Deserialization Flowchart | +---------------------------------------------------------------------------------------+ | | | 1. LoadFile(scratch.Arena, filepath) into ByteBuffer | | | | | v | | 2. tokenize_archive(scratch.Arena, file_buffer, &ar.base) | | | | | v | | 3. Read "; class" & Resolve Class* via find_class_by_name(class_name) | | | | | v | | 4. base_ptr = AllocateEntity(manager, class_ptr) | | | | | +--> [manager.Entities]: Allocates base Entity | | +--> [manager.by_type[kind].arena]: Allocates derived struct | | +--> Mutual Back-Pointers Wired In-Place | | | | | v | | 5. serialize(ar, NonNullPtr(base_ptr)) | | | | | +--> Streams Base Entity (Entity::kind, version, ID, position) | | +--> Streams Derived Component (base_ptr->derived_kind, class_version) | | | | | v | | 6. Clear Dirty Flag: base_ptr->is_dirty = false | | | +---------------------------------------------------------------------------------------+ ``` ```cpp [[nodiscard]] Entity* deserialize_entity_in_place(EntityManager& manager, Archive& ar) { Assert(ar.loading); // 1. Read class name and resolve Class* String class_name = {}; SERIALIZE(ar, class, class_name); Class* class_ptr = find_class_by_name(class_name); if (!class_ptr) { return nullptr; } // 2. Allocate persistent memory for base and derived components in their respective arenas Entity* base_ptr = AllocateEntity(manager, class_ptr); Assert(base_ptr != nullptr); // 3. Serialize Base Entity and Derived in-place (loads ID and properties from disk) serialize(ar, NonNullPtr(base_ptr)); // 4. Advance generator counter to avoid collisions with loaded IDs if (base_ptr->ID >= EntityManager::ID) { EntityManager::ID = base_ptr->ID + 1; } // Freshly loaded entity matches disk state exactly base_ptr->is_dirty = false; return base_ptr; } ``` --- ## 5. Entity Deletion Lifecycle & Disk Synchronization (Extracted for Rework) > [!WARNING] > **Status: Extracted for Rework** > The original swap-and-pop in-memory removal logic (`DestroyEntity`, `RemoveDerivedComponent`, and mutual back-pointer fixups) was determined to be overly complex and has been extracted to [`Game/Plans/Entity_Removal_Brainstorm.md`](file:///w:/Classified/Juliet/Game/Plans/Entity_Removal_Brainstorm.md) for further brainstorming and redesign. > > Simpler alternative architectures under consideration include: > - **Active / Tombstone Flag (`is_active` bool):** Retaining entities in-place without moving memory during frame simulation, eliminating pointer invalidation entirely. > - **Intrusive Free List:** Linking inactive slots via an intrusive linked list to find the first free slot in $O(1)$ without memory shifting. > - **Generational Handles / Slot Map:** Enabling safe, non-dangling entity references across systems. > - **Deferred Compaction:** Batch-compacting memory during level loads or scene transitions rather than per-frame swap-and-pop. --- ## 6. Dirty Tracking for Optimal Saves ### 6.1 The Cost of Naive Monolithic & Full-Directory Writes In a level containing $10{,}000$ entities: - **Monolithic `world.bin` Save:** Modifying a single entity's $X$ coordinate requires re-serializing all $10{,}000$ entities and overwriting a multi-megabyte binary file. This introduces a huge Git diff and constant merge conflicts. - **Full-Directory `.jasset` Save:** Iterating through all $10{,}000$ entities and unconditionally writing $10{,}000$ `.jasset` files incurs massive OS file-system overhead (directory table locks, I/O bandwidth) and changes the file timestamps of every asset. Git reports thousands of modified files even when only one entity changed! ### 6.2 The `is_dirty` Flag on `Entity` To solve this, `Entity` in [`Game/Entity/Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h#L28-L36) is augmented with an explicit dirty flag: ```cpp struct Entity final { DECLARE_CLASS() // static Class* kind; (Entity's own Class descriptor) EntityID ID = 0; Class* derived_kind = nullptr; // Pointer to derived class descriptor (e.g. Inert::kind) DerivedType derived = nullptr; // Pointer to derived component memory Vector4 position = {}; bool is_dirty = false; }; ``` ### 6.3 Granular State Transitions The `is_dirty` flag obeys a strict lifecycle state machine: ``` +-----------------------------------+ | Entity Created | | (AllocateEntity / Editor) | +-----------------+-----------------+ | v +---------------+ +------->|is_dirty: TRUE |<-------+ | +-------+-------+ | | | | Entity Mutated | SaveWorld (Position, Component) | Completed | v | | +---------------+ | +--------+is_dirty: FALSE+--------+ +-------+-------+ ^ | Deserialization (LoadWorld / Asset) ``` 1. **Entity Creation:** Newly spawned entities in the editor have `is_dirty = true`. 2. **Property Mutation:** Any modification to `position` or derived component payload sets `entity->is_dirty = true`. 3. **Successful Deserialization:** Entities loaded from disk initialize with `is_dirty = false`. 4. **Successful Save:** Upon successfully writing an entity to its `.jasset` file, the engine resets `entity->is_dirty = false`. ### 6.4 Version Control Benefits (Git Friendly Assets) By coupling the one-file-per-entity `.jasset` format with dirty tracking: - Only modified entities are touched on disk. - Git status displays only the exact `.jasset` files that were altered by the designer. - Team members can work concurrently in the same game scene without encountering binary merge conflicts. ### 6.5 Editor Integration (`RenderWorldEditorUI` Hooks) In [`Game/Data/World.cpp`](file:///w:/Classified/Juliet/Game/Data/World.cpp#L291-L311), editor UI widgets automatically set the dirty flag upon receiving user input: ```cpp float pos[4] = { ent.position.x, ent.position.y, ent.position.z, ent.position.w }; if (ImGui::DragFloat3("Position", pos, 0.1f)) { ent.position.x = pos[0]; ent.position.y = pos[1]; ent.position.z = pos[2]; ent.is_dirty = true; // Mark dirty for persistence UpdateWorld(world); } ``` --- ## 7. Step-by-Step Implementation Roadmap ### Phase 1: Data Structures & Header Definitions 1. **Update `Class.h` & `Entity.h`:** - Add `using default_init_fct_type = void (*)(void* payload);` and `default_init_fct` to `struct Class` and `MakeClass`. - Update `DEFINE_ENTITY_VERSIONED` and `DEFINE_CLASS_VERSIONED` to define `default_init_##entity` and pass it to `MakeClass`. - Add `bool is_dirty = false;` to `struct Entity`. - Update `MakeEntity` to assign `base_ptr->ID = EntityManager::ID++;` and delegate allocation and defaults cleanly to `AllocateEntity`. - Declare `[[nodiscard]] const Class* ResolveEntityClass(uint8 kind, uint32 crc);` and `[[nodiscard]] Class* find_class_by_name(String name);` in `Entity.h`. 2. **Update `EntityManager.h`:** - Declare `[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* class_ptr);`. - *(Note: `DestroyEntity` and `RemoveDerivedComponent` deferred to `Entity_Removal_Brainstorm.md`)* 3. **Update `World.h`:** - Add `VectorArena PendingDeletions;` to `struct World`. - Update `SaveWorld` and `LoadWorld` signatures to take directory paths. ### Phase 2: Core Memory Allocation & Wiring in `EntityManager.cpp` 1. Implement `AllocateEntity`: - Enforce parameter assertions. - Push to `manager.Entities` with `baseTemplate.ID = 0` (unassigned). - Allocate block in `manager.by_type[kind].arena`. - Call `derivedClassPtr->default_init_fct(rawMemory)` (or `MemZero` if null) to initialize struct defaults. - Wire mutual pointers (`base->derived` and `derived->base`). - Increment `typedArray.count` and initialize `typedArray.array`. 2. *(Deferred for rework)* `DestroyEntity` & `RemoveDerivedComponent` (See [`Game/Plans/Entity_Removal_Brainstorm.md`](file:///w:/Classified/Juliet/Game/Plans/Entity_Removal_Brainstorm.md)). ### Phase 3: In-Place Deserialization & Serialization Pipeline 1. In `Entity.cpp`: - Implement `ResolveEntityClass` and `find_class_by_name`. - `serialize(Archive& ar, NonNullPtr entity)` handles both base and derived class serialization. 2. In `World.cpp`: - Implement `serialize_entity_asset(Archive& ar, NonNullPtr entity, String filepath)`. - Implement `deserialize_entity_asset(EntityManager& manager, Archive& ar, String filepath)`. - In `deserialize_entity_in_place`, advance `EntityManager::ID` past `base_ptr->ID` to prevent ID collisions. ### Phase 4: World Save/Load Pipeline & Disk Deletion 1. In `World.cpp`: - Implement `ProcessPendingDeletions(World& world, NonNullPtr scratchArena)`. - Implement `SaveWorld(World& world, String worldDirectory)`: - Process pending deletions. - Iterate `manager.Entities`, skipping entities where `!entity.is_dirty`. - Write dirty entities to `.jasset` files and clear `is_dirty`. - Implement `LoadWorld(World& world, String worldDirectory)`: - Enumerate `.jasset` files in directory. - Call `deserialize_entity_asset` for each file. ### Phase 5: Editor Integration 1. In `RenderWorldEditorUI`: - Hook `ImGui::DragFloat3` and property inspectors to set `is_dirty = true`. - Hook "Add Entity" button to call `MakeEntity(*world.EntityManager, 0.0f, 0.0f, 0.0f)`. - Hook "Delete Entity" button to call `RemoveWorldEntity(world, selectedEntityId)`. --- ## 8. Unit Testing & Verification Plan ### 8.1 Test Philosophy & Constraints Following Juliet coding guidelines: > "When creating a new system framework, make a unit test. To make the unit test we should not modify the framework code for special unit test case." Testing is isolated in [`Game/UnitTest/WorldUnitTest.cpp`](file:///w:/Classified/Juliet/Game/UnitTest/WorldUnitTest.cpp) and executed during engine initialization in debug builds. ### 8.2 Comprehensive Test Suite (`WorldUnitTest.cpp`) The test suite validates every guarantee made in this specification: ```cpp #include #if JULIET_DEBUG #include #include #include #include #include #include #include #include #include #include #include namespace UnitTest { namespace { void TestEntityAllocationAndWiring() { Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestEntityAllocationAndWiring..."); TempArena tempArena = scratch_begin(nullptr, 0); World testWorld{}; InitWorld(&testWorld, tempArena.Arena); InitEntityManager(&testWorld); EntityManager& manager = *testWorld.EntityManager; // 1. Allocate Inert Entity via AllocateEntity Entity* base_entity = AllocateEntity(manager, Inert::kind); Assert(base_entity != nullptr); Assert(base_entity->ID == 0); // Pure memory allocator leaves ID unassigned (0) until MakeEntity or deserialization base_entity->ID = EntityManager::ID++; Assert(base_entity->ID > 0); Assert(base_entity->derived_kind == Inert::kind); Assert(base_entity->derived != nullptr); Assert(base_entity->is_dirty == true); // 2. Validate mutual back-pointer wiring auto* derived = reinterpret_cast(base_entity->derived); Assert(derived->base == base_entity); // 3. DownCast verification Inert* inert = DownCast(base_entity); Assert(inert != nullptr); Assert(inert->base == base_entity); // 4. Validate typed array tracking typed_entity_array& inert_array = manager.by_type[ENTITY(Inert)]; Assert(inert_array.count == 1); Assert(inert_array.array == derived); ShutdownEntityManager(); ShutdownWorld(&testWorld); scratch_end(tempArena); Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestEntityAllocationAndWiring"); } void TestInPlaceDeserialization() { Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestInPlaceDeserialization..."); TempArena tempArena = scratch_begin(nullptr, 0); World testWorld{}; InitWorld(&testWorld, tempArena.Arena); InitEntityManager(&testWorld); EntityManager& manager = *testWorld.EntityManager; // 1. Create and populate entity Inert* created_inert = MakeEntity(manager, 12.5f, -44.0f, 108.2f); Assert(created_inert != nullptr); created_inert->MeshInstance = 42; Entity* original_base = created_inert->base; EntityID original_id = original_base->ID; // 2. Serialize to text archive memory stream MemoryStream mem_stream = MakeMemoryStream(tempArena.Arena); Archive save_ar{ .arena = tempArena.Arena, .loading = false, .stream = &mem_stream }; serialize(save_ar, NonNullPtr(original_base)); // 3. Clear manager to simulate fresh load ShutdownEntityManager(); InitEntityManager(&testWorld); EntityManager& fresh_manager = *testWorld.EntityManager; // 4. Tokenize and deserialize in-place Archive load_ar{ .arena = tempArena.Arena, .loading = true }; tokenize_archive(tempArena.Arena, mem_stream.buffer, &load_ar.base); Entity* loaded_base = deserialize_entity_in_place(fresh_manager, load_ar); Assert(loaded_base != nullptr); Assert(loaded_base->ID == original_id); Assert(EntityManager::ID > original_id); // Counter was advanced past loaded ID to prevent collisions Assert(loaded_base->position.x == 12.5f); Assert(loaded_base->position.y == -44.0f); Assert(loaded_base->position.z == 108.2f); Assert(loaded_base->is_dirty == false); Inert* loaded_inert = DownCast(loaded_base); Assert(loaded_inert != nullptr); Assert(loaded_inert->base == loaded_base); Assert(loaded_inert->MeshInstance == 42); ShutdownEntityManager(); ShutdownWorld(&testWorld); scratch_end(tempArena); Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestInPlaceDeserialization"); } void TestDirtyTrackingLifecycle() { Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestDirtyTrackingLifecycle..."); TempArena tempArena = scratch_begin(nullptr, 0); World testWorld{}; InitWorld(&testWorld, tempArena.Arena); InitEntityManager(&testWorld); EntityManager& manager = *testWorld.EntityManager; Entity* entity = AllocateEntity(manager, Inert::kind); Assert(entity->is_dirty == true); // Simulate save entity->is_dirty = false; Assert(entity->is_dirty == false); // Simulate mutation entity->position.x += 1.0f; entity->is_dirty = true; Assert(entity->is_dirty == true); ShutdownEntityManager(); ShutdownWorld(&testWorld); scratch_end(tempArena); Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestDirtyTrackingLifecycle"); } } // namespace void WorldUnitTest() { Log(LogLevel::Message, LogCategory::Game, "=================================================="); Log(LogLevel::Message, LogCategory::Game, "Starting Entity Allocation & Lifecycle Unit Tests"); Log(LogLevel::Message, LogCategory::Game, "=================================================="); TestEntityAllocationAndWiring(); TestInPlaceDeserialization(); // TestSwapAndPopPointerFixup(); // Deferred to Entity_Removal_Brainstorm.md TestDirtyTrackingLifecycle(); Log(LogLevel::Message, LogCategory::Game, "=================================================="); Log(LogLevel::Message, LogCategory::Game, "All Entity Lifecycle Unit Tests PASSED Successfully"); Log(LogLevel::Message, LogCategory::Game, "=================================================="); } } // namespace UnitTest #endif