Juliet/Game/Plans/02_Entity_Allocation_And_Lifecycle.md

Juliet Game Engine Architecture Specification

Document: 02 - Entity Allocation & In-Place Lifecycle

Document ID: JULIET-SPEC-002
Status: Approved for Implementation
Author: Senior Engine Architect
Subsystems: Game/Entity, Game/Data, Game/UnitTest
Target Files:

• Entity.h
• Entity.cpp
• EntityManager.h
• EntityManager.cpp
• World.h
• World.cpp
• WorldUnitTest.h
• WorldUnitTest.cpp

---

1. Executive Summary & Problem Statement

1.1 Background & Context

The Juliet game engine organizes game entities using a hybrid data-oriented architecture:

1. A flat array of Base Entities (Entity) encapsulating universal properties: unique 64-bit ID, runtime type reflection pointer (Class* Kind), spatial coordinates (X, Y, Z), and an opaque pointer to the derived payload (DerivedType Derived).
2. Type-segregated contiguous arrays of Derived Entities (Inert, etc.) stored in dedicated per-type memory arenas (typed_entity_array).

This design is intended to provide maximum cache efficiency during spatial and general-purpose entity processing, while retaining dense SIMD-friendly streaming for type-specific systems (e.g., transform updates on Inert mesh instances).

1.2 Root-Cause Analysis of Game/Data/World.cpp Loading Flaws

In the initial implementation of entity serialization in Game/Data/World.cpp, loading was fundamentally broken and incomplete:

// Existing flawed deserialization in World.cpp
for (typed_entity_array& type : entityManager.by_type)
{
    serialize_elem(ar, type.count);

    if (type.count > 0)
    {
        // Unserialize the base entity to get informations
        Entity entity;
        serialize(ar, &entity);

        RegisterBaseEntity(entityManager, entity);
    }
}

This implementation suffers from several fatal defects:

1. Single-Element Iteration Bug: It uses if (type.count > 0) instead of a loop for (size_t i = 0; i < type.count; ++i), deserializing at most one single entity per type bucket, leaving all subsequent entities in the stream unread and corrupting the archive read offset.
2. Missing Derived Allocation: It invokes RegisterBaseEntity(entityManager, entity), which merely pushes the stack-allocated Entity into manager.Entities. The derived payload arena (type.arena) is completely untouched: type.array remains null, type.count in the manager is desynchronized, and entity.Derived remains unassigned or points to an invalid address.
3. Invalid Pointer in RegisterEntity: In Game/Entity/EntityManager.cpp, RegisterEntity assigns base->Derived = entity before pushing *base into manager.Entities. The parameter entity is a pointer to caller-provided memory (often stack-allocated in helper functions like MakeEntity). When ArenaPushSize later allocates the persistent derived memory block, base->Derived stored inside manager.Entities.Back() is never updated—it remains dangling, pointing to the transient caller stack!

1.3 The "Chicken-and-Egg" Stack Allocation Dilemma

The existing registration API requires a pre-existing derived instance:

entity_template* RegisterEntity(EntityManager& manager, Entity* base, DerivedType entity);

During runtime programmatic creation via MakeEntity<T>(), a temporary instance of T is created on the stack and passed by pointer:

template <typename EntityType>
EntityType* MakeEntity(EntityManager& manager, float x, float y, float z)
{
    EntityType result; // Stack allocation
    Entity     base;   // Stack allocation
    base.X    = x;
    base.Y    = y;
    base.Z    = z;
    base.Kind = EntityType::Kind;

    return (EntityType*)RegisterEntity(manager, &base, &result);
}

When loading an entity from a stream or disk file, the type is not known at compile time. The engine reads a runtime type tag (uint8 kind or uint32 CRC), looks up the reflection metadata (Class*), and must instantiate the entity dynamically.

Because C++ does not permit allocating a dynamic struct of unknown type on the stack, and because Juliet strictly forbids heap allocations (malloc, new, std::vector), deserialization cannot construct a temporary instance on the stack to pass to RegisterEntity.

This is the classic chicken-and-egg memory problem:

• RegisterEntity requires an existing instance in memory to copy from.
• Deserialization requires an allocated memory buffer to deserialize into.

1.4 Architectural Objectives

This specification establishes a robust in-place lifecycle pipeline that completely eliminates stack temporaries and dynamic heap allocations:

1. Direct In-Place Allocation: Introduce AllocateEntity(EntityManager& manager, Class* classPtr) which allocates both the base Entity and the derived struct directly within their respective engine memory arenas.
2. Bidirectional Pointer Integrity: Wire mutual pointers (base->derived and derived->base) at allocation time before any field deserialization begins.
3. In-Place Stream Deserialization: Read class reflection metadata first, invoke AllocateEntity, and stream base and derived properties directly into arena-resident memory.
4. Isolated Entity Assets (.jasset): Transition from a monolithic world.bin to a modular one-file-per-entity architecture (Assets/Entities/{ID}.jasset).
5. Dirty Tracking & Optimal Saves: Introduce an is_dirty flag on Entity to avoid rewriting unchanged entity files, minimizing disk I/O and eliminating spurious Git repository modifications.
6. Robust Deletion Lifecycle: Decouple in-memory removal (RemoveAtFast with pointer fixup) from disk synchronization using World::PendingDeletions.

---

2. The Dual Arena Memory Model in EntityManager

2.1 The Need for Dual Storage

Game engines execute systems with vastly different cache locality profiles:

• Spatial / Frustum Culling / Transform Sync: Iterates every entity in the world, needing only X, Y, Z, bounding spheres, and base status flags.
• Specialized Logic / Render Updates: Iterates only entities possessing specific components (e.g., Inert static meshes requiring instance transform updates to the GPU bindless descriptor table).

Storing large monolithic polymorphic structs in a single array causes severe cache line pollution during spatial passes. Conversely, storing entities in fragmented individual allocations introduces cache misses and pointer-chasing overhead.

Juliet resolves this with a Dual Arena Memory Model:

+---------------------------------------------------------------------------------------------+
|                                    EntityManager                                            |
+---------------------------------------------------------------------------------------------+
|                                                                                             |
|   manager.Entities (VectorArena<Entity, 100'000>)                                          |
|   +------------------------------------+------------------------------------+           |
|   | Entity 0 (ID=1001, X, Y, Z)        | Entity 1 (ID=1002, X, Y, Z)        | ...       |
|   | Derived ------------------------+  | Derived ---------------------+     |           |
|   +---------------------------------|--+------------------------------|-----+           |
|                                     |                                 |                     |
|                                     v                                 v                     |
|   manager.by_type[ENTITY(Inert)].arena                                                      |
|   +------------------------------------+------------------------------------+           |
|   | Inert 0 (MeshInstance=4)           | Inert 1 (MeshInstance=12)          | ...       |
|   | Base ---------------------------+  | Base ------------------------+     |           |
|   +---------------------------------|--+------------------------------|-----+           |
|                                     +---------------------------------+                     |
+---------------------------------------------------------------------------------------------+

2.2 manager.Entities: Cache-Friendly Base Entity Vector

Base entities reside in a pre-reserved contiguous array:

VectorArena<Entity, 100'000> Entities;

• Capacity: Fixed reserve of 100,000 entities allocated from the WorldArena.
• Memory Footprint: \text{sizeof(Entity)} = 8\text{ (ID)} + 8\text{ (derived\_kind)} + 8\text{ (derived)} + 16\text{ (position)} + 1\text{ (is\_dirty)} + 7\text{ (Padding)} = 48\text{ bytes} Total reserved space: 100{,}000 \times 48\text{ bytes} \approx 4.8\text{ MB}.
• Access Speed: O(1) random access by index; sequential streaming utilizes L1/L2 hardware prefetchers with zero cache line waste.

2.3 manager.by_type[kind].arena: Typed Component Arenas

Derived structs reside in per-type contiguous memory arenas:

struct typed_entity_array
{
    Arena*           arena;
    entity_template* array;
    size_t           count;
};

• Each Entity_Type index owns an isolated Arena* allocated during InitEntityManager.
• Allocations are packed linearly with alignment specified by classPtr->alignment.
• array points directly to the first element in the arena, permitting typed array indexing:

  Inert* inertArray = reinterpret_cast<Inert*>(manager.by_type[ENTITY(Inert)].array);
  

2.4 Mutual Back-Pointer Architecture & Invariants

Every entity instance consists of two mutually linked allocations:

1. basePtr->Derived: Points from Entity in manager.Entities to the derived struct in by_type[kind].arena.
2. derivedPtr->Base: Points from the derived struct (via DECLARE_ENTITY()) back to Entity in manager.Entities.

Invariant Rules:

1. Non-Null Invariant: For any active entity, basePtr->Derived != nullptr and reinterpret_cast<entity_template*>(basePtr->Derived)->Base == basePtr.
2. Type Coherence Invariant: basePtr->Kind->kind == derivedTypeId.
3. Array Count Invariant: \sum_{k=0}^{\text{ENTITY(Count)}-1} \text{manager.by\_type}[k].\text{count} == \text{manager.Entities.Size()}

2.5 Pointer Stability in VectorArena

VectorArena::Create executes Reserve(ReserveSize) on creation:

newManager->Entities.Create(world->WorldArena JULIET_DEBUG_PARAM("Entities"));

Because capacity (100{,}000) is fully reserved upfront in virtual address space, VectorArena::PushBack never reallocates or moves existing memory. Therefore:

• Pointers to Entity elements in manager.Entities remain absolutely stable across allocations.
• Derived struct Base pointers remain valid indefinitely unless an element is deleted.
• Element removal via swap-and-pop alters memory positions, requiring systematic pointer fixups (addressed in Section 5.2).

---

3. The Solution: AllocateEntity(EntityManager& manager, Class* classPtr)

3.1 Function Signature & Contract

The canonical allocation function is defined in EntityManager.h:

[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr);

Preconditions:

• classPtr != nullptr.
• classPtr->kind < ENTITY(Count).
• classPtr->size_of >= sizeof(entity_template).
• manager.Entities.Size() < manager.Entities.Capacity.
• manager.by_type[classPtr->kind].arena != nullptr.

Postconditions:

• A new Entity record is appended to manager.Entities.
• A new typed block of classPtr->size_of bytes is allocated in manager.by_type[classPtr->kind].arena.
• The derived memory is initialized with C++ struct defaults via classPtr->default_init_fct (or zeroed via MemZero if null).
• base->derived points to the derived struct.
• derived->base points to the base Entity.
• base->derived_kind is assigned to class_ptr.
• base->ID is initialized to 0 (unassigned; populated by MakeEntity or deserialization).
• base->is_dirty is initialized to true.
• typed_entity_array::count is incremented.
• typed_entity_array::array is initialized if this is the first entity of this type.

3.2 Detailed Step-by-Step Implementation

The implementation replaces the flawed RegisterEntity routine in Game/Entity/EntityManager.cpp:

[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* derivedClassPtr)
{
    Assert(derivedClassPtr != nullptr);
    Assert(derivedClassPtr->kind < ENTITY(Count));
    Assert(derivedClassPtr->size_of >= sizeof(entity_template));
    Assert(derivedClassPtr->alignment > 0);

    // 1. Allocate Base Entity in the contiguous VectorArena (ID is 0 until assigned by MakeEntity or deserialization)
    Entity baseTemplate{};
    baseTemplate.ID           = 0;
    baseTemplate.derived_kind = derivedClassPtr;
    baseTemplate.derived      = nullptr;
    baseTemplate.position     = {};
    baseTemplate.is_dirty     = true;

    manager.Entities.PushBack(baseTemplate);
    Entity* basePtr = manager.Entities.Back();
    Assert(basePtr != nullptr);

    // 2. Allocate derived component memory in the typed arena
    typed_entity_array& typedArray = manager.by_type[derivedClassPtr->kind];
    Assert(typedArray.arena != nullptr);

    void* rawMemory = ArenaPushSize(
        typedArray.arena, 
        derivedClassPtr->size_of, 
        derivedClassPtr->alignment, 
        false JULIET_DEBUG_PARAM(kEntity_type_names[derivedClassPtr->kind]));
    Assert(rawMemory != nullptr);

    // 3. Initialize derived component defaults via Class reflection stub
    if (derivedClassPtr->default_init_fct != nullptr)
    {
        derivedClassPtr->default_init_fct(rawMemory);
    }
    else
    {
        MemZero(rawMemory, derivedClassPtr->size_of);
    }

    auto* derivedTemplate = reinterpret_cast<entity_template*>(rawMemory);

    // 4. Establish mutual back-pointers
    basePtr->derived      = rawMemory;
    derivedTemplate->base = basePtr;

    // 5. Update typed array tracking
    if (typedArray.array == nullptr)
    {
        typedArray.array = derivedTemplate;
    }
    typedArray.count += 1;

    return basePtr;
}

3.3 Default Struct Initialization via Class::default_init_fct

The Problem with Zero-Only Initialization

If allocation only zeroes memory (MemZero / 0x00), any C++ member variables with non-zero defaults (such as index_t MeshInstance = indexMax; or float Density = 1.0f;) are populated with 0. During deserialization, if a .jasset file lacks that property (e.g. an older file version or an optional field), SERIALIZE leaves the field untouched, meaning it incorrectly remains 0 rather than its intended default sentinel value!

The Solution: Compile-Time Default Stub in DEFINE_ENTITY_VERSIONED

Every entity descriptor Class includes a function pointer:

using default_init_fct_type = void (*)(void* payload);

struct Class
{
    ...
    default_init_fct_type default_init_fct = nullptr;
};

When registering an entity type with DEFINE_ENTITY_VERSIONED, the macro automatically defines a tiny type-safe stub that aggregate value-initializes the struct:

#define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct)                                                         \
    inline void default_init_##entity(void* payload)                                                                    \
    {                                                                                                                   \
        *static_cast<entity*>(payload) = entity{};                                                                      \
    }                                                                                                                   \
    constexpr Class entityKind##entity = MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version),         \
                                                   &classKindEntity, sizeof(entity), alignof(entity),                   \
                                                   (serialize_fct), default_init_##entity);                             \
    Class*          entity::kind       = const_cast<Class*>(&entityKind##entity);

Architectural Advantages:

1. Zero Boilerplate: Developers write member initializers once in the struct definition (index_t MeshInstance = indexMax;).
2. Type-Agnostic Core: AllocateEntity does not need to know any C++ struct types; it unconditionally calls derivedClassPtr->default_init_fct(rawMemory).
3. Robust Deserialization: In deserialize_entity_in_place, newly allocated entities already hold their canonical C++ defaults. Any properties absent in the .jasset file naturally retain their correct initial values.
4. No Dynamic Heap / Placement-New: *static_cast<entity*>(payload) = entity{} is pure aggregate value-assignment without <new> headers or exceptions.

3.4 Bidirectional Pointer Wiring

Notice the sequence:

1. manager.Entities.PushBack(baseTemplate) places the struct at its final, fixed arena address.
2. base_ptr = manager.Entities.Back() retrieves the persistent memory pointer.
3. derived_class_ptr->default_init_fct(raw_memory) initializes canonical struct defaults.
4. derived_template->base = base_ptr wires the derived back-pointer directly to this permanent location.
5. base_ptr->derived = raw_memory wires the base forward-pointer to the arena-allocated derived struct.

No stack copying occurs. Neither pointer is ever left dangling.

3.5 Updating Type Counts and Array Cache

typed_entity_array maintains:

• typedArray.count: The exact count of active entities of this type.
• typedArray.array: Pointer to the first element in the arena.

When the first entity of a given type is allocated, typedArray.array is set to derived_template. Because the arena allocates sequentially, typedArray.array[i] can be indexed directly with stride class_ptr->size_of as long as memory remains contiguous.

3.6 Refactoring MakeEntity<EntityType> Template

With AllocateEntity handling memory allocation, default member initialization, and mutual pointer wiring, MakeEntity in Game/Entity/Entity.h becomes clean, safe, and stack-free:

template <typename EntityType>
    requires EntityConcept<EntityType>
[[nodiscard]] EntityType* MakeEntity(EntityManager& manager, float x, float y, float z)
{
    Entity* base_ptr = AllocateEntity(manager, EntityType::kind);
    Assert(base_ptr != nullptr);

    base_ptr->ID         = EntityManager::ID++;
    base_ptr->position.x = x;
    base_ptr->position.y = y;
    base_ptr->position.z = z;
    base_ptr->position.w = 1.0f;

    return static_cast<EntityType*>(base_ptr->derived);
}

---

4. In-Place Deserialization Pipeline

4.1 Asset Format Specification (.jasset)

To achieve robust version control and modular streaming, Juliet adopts a human-readable, Git-diffable one-file-per-entity disk format with extension .jasset.

; asset_type
entity_instance
; id
0x0100000000000042
; class
Inert
; version
1
; class_version
1
; position
0.43 0.32 1.56
; mesh_instance
12

Important: No Header Structs for Derived Types

• Derived types NEVER require their own file header: You do not write an InertHeader, DoorHeader, or PlayerHeader. Derived types only serialize their own member variables.
• Universal ; version + optional ; class_version: Every .jasset file has a universal ; version tag. For entity assets, ; version governs base entity properties (kEntityBaseVersion), while an optional ; class_version governs derived class properties (Class::Version). Non-entity assets like WorldSettings.jasset only have ; version.
• No binary EntityFileHeader struct is needed: Under the ; variable_name\nvalues text format, there is no packed binary C-struct header at all. The common properties (; id, ; class, ; version, ; class_version, ; position) are standard text Key-Value nodes read by the exact same Archive parser.

4.2 Eliminating Intermediate Stack Allocations

Under the new pipeline:

1. The .jasset text file is read into memory onto a TempArena via LoadFile.
2. The property nodes are parsed into a ParsedArchive via tokenize_archive(tempArena.Arena, fileBuffer, &ar.base).
3. The engine reads ; class (e.g. "Inert") and resolves Class* class_ptr = find_class_by_name(class_name).
4. AllocateEntity(manager, class_ptr) is called immediately. Memory for both base and derived components is allocated in-place in their permanent engine arenas.
5. serialize(ar, NonNullPtr<Entity>(base_ptr)) is called. Base (Entity::kind) and derived (base_ptr->derived_kind) fields stream directly into their permanent memory arenas without temporary staging buffers or stack copies.

4.3 Runtime Class Resolution

To ensure fast and safe type lookup during file deserialization, Juliet provides two tiers of type resolution:

1. Low-Level Indexed Validation (ResolveEntityClass): An O(1) array lookup into kEntity_type_class_ptr[kind] that validates the class pointer and verifies schema compatibility against classPtr->CRC. This is the core validation primitive used by binary streaming, network replication, and internal lookups.
2. High-Level Name Resolution (find_class_by_name): Bridges the human-readable text archive format (; class\nInert) to the entity class registry. It computes the crc32 of the parsed class name and queries ResolveEntityClass across registered entity kinds.

[[nodiscard]] const Class* ResolveEntityClass(uint8 kind, uint32 crc)
{
    if (kind >= ENTITY(Count))
    {
        return nullptr;
    }

    const Class* class_ptr = kEntity_type_class_ptr[kind];
    if (class_ptr == nullptr)
    {
        return nullptr;
    }

    if (class_ptr->CRC != crc)
    {
        return nullptr;
    }

    return class_ptr;
}

[[nodiscard]] Class* find_class_by_name(String name)
{
    if (!IsValid(name))
    {
        return nullptr;
    }

    const uint32 name_crc = crc32(name.Str, name.Size);

    for (uint8 kind = 0; kind < ENTITY(Count); ++kind)
    {
        const Class* class_ptr = ResolveEntityClass(kind, name_crc);
        if (class_ptr != nullptr)
        {
            return const_cast<Class*>(class_ptr);
        }
    }

    return nullptr;
}

4.4 In-Place Deserialization Algorithm

+---------------------------------------------------------------------------------------+
|                       In-Place Deserialization Flowchart                              |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   1. LoadFile(scratch.Arena, filepath) into ByteBuffer                                |
|              |                                                                        |
|              v                                                                        |
|   2. tokenize_archive(scratch.Arena, file_buffer, &ar.base)                            |
|              |                                                                        |
|              v                                                                        |
|   3. Read "; class" & Resolve Class* via find_class_by_name(class_name)               |
|              |                                                                        |
|              v                                                                        |
|   4. base_ptr = AllocateEntity(manager, class_ptr)                                    |
|              |                                                                        |
|              +--> [manager.Entities]: Allocates base Entity                           |
|              +--> [manager.by_type[kind].arena]: Allocates derived struct             |
|              +--> Mutual Back-Pointers Wired In-Place                                 |
|              |                                                                        |
|              v                                                                        |
|   5. serialize(ar, NonNullPtr<Entity>(base_ptr))                                      |
|              |                                                                        |
|              +--> Streams Base Entity (Entity::kind, version, ID, position)           |
|              +--> Streams Derived Component (base_ptr->derived_kind, class_version)   |
|              |                                                                        |
|              v                                                                        |
|   6. Clear Dirty Flag: base_ptr->is_dirty = false                                     |
|                                                                                       |
+---------------------------------------------------------------------------------------+

[[nodiscard]] Entity* deserialize_entity_in_place(EntityManager& manager, Archive& ar)
{
    Assert(ar.loading);

    // 1. Read class name and resolve Class*
    String class_name = {};
    SERIALIZE(ar, class, class_name);
    Class* class_ptr = find_class_by_name(class_name);
    if (!class_ptr)
    {
        return nullptr;
    }

    // 2. Allocate persistent memory for base and derived components in their respective arenas
    Entity* base_ptr = AllocateEntity(manager, class_ptr);
    Assert(base_ptr != nullptr);

    // 3. Serialize Base Entity and Derived in-place (loads ID and properties from disk)
    serialize(ar, NonNullPtr<Entity>(base_ptr));

    // 4. Advance generator counter to avoid collisions with loaded IDs
    if (base_ptr->ID >= EntityManager::ID)
    {
        EntityManager::ID = base_ptr->ID + 1;
    }

    // Freshly loaded entity matches disk state exactly
    base_ptr->is_dirty = false;

    return base_ptr;
}

---

5. Entity Deletion Lifecycle & Disk Synchronization (Extracted for Rework)

Warning

Status: Extracted for Rework The original swap-and-pop in-memory removal logic (DestroyEntity, RemoveDerivedComponent, and mutual back-pointer fixups) was determined to be overly complex and has been extracted to Game/Plans/Entity_Removal_Brainstorm.md for further brainstorming and redesign.

Simpler alternative architectures under consideration include:

• Active / Tombstone Flag (is_active bool): Retaining entities in-place without moving memory during frame simulation, eliminating pointer invalidation entirely.
• Intrusive Free List: Linking inactive slots via an intrusive linked list to find the first free slot in O(1) without memory shifting.
• Generational Handles / Slot Map: Enabling safe, non-dangling entity references across systems.
• Deferred Compaction: Batch-compacting memory during level loads or scene transitions rather than per-frame swap-and-pop.

---

6. Dirty Tracking for Optimal Saves

6.1 The Cost of Naive Monolithic & Full-Directory Writes

In a level containing 10{,}000 entities:

• Monolithic world.bin Save: Modifying a single entity's X coordinate requires re-serializing all 10{,}000 entities and overwriting a multi-megabyte binary file. This introduces a huge Git diff and constant merge conflicts.
• Full-Directory .jasset Save: Iterating through all 10{,}000 entities and unconditionally writing 10{,}000 .jasset files incurs massive OS file-system overhead (directory table locks, I/O bandwidth) and changes the file timestamps of every asset. Git reports thousands of modified files even when only one entity changed!

6.2 The is_dirty Flag on Entity

To solve this, Entity in Game/Entity/Entity.h is augmented with an explicit dirty flag:

struct Entity final
{
    DECLARE_CLASS() // static Class* kind; (Entity's own Class descriptor)

    EntityID    ID           = 0;
    Class*      derived_kind = nullptr; // Pointer to derived class descriptor (e.g. Inert::kind)
    DerivedType derived      = nullptr; // Pointer to derived component memory
    Vector4     position     = {};
    bool        is_dirty     = false;
};

6.3 Granular State Transitions

The is_dirty flag obeys a strict lifecycle state machine:

                  +-----------------------------------+
                  |          Entity Created           |
                  |     (AllocateEntity / Editor)     |
                  +-----------------+-----------------+
                                    |
                                    v
                            +---------------+
                   +------->|is_dirty: TRUE |<-------+
                   |        +-------+-------+        |
                   |                |                |
        Entity Mutated              |            SaveWorld
    (Position, Component)           |            Completed
                   |                v                |
                   |        +---------------+        |
                   +--------+is_dirty: FALSE+--------+
                            +-------+-------+
                                    ^
                                    |
                             Deserialization
                            (LoadWorld / Asset)

1. Entity Creation: Newly spawned entities in the editor have is_dirty = true.
2. Property Mutation: Any modification to position or derived component payload sets entity->is_dirty = true.
3. Successful Deserialization: Entities loaded from disk initialize with is_dirty = false.
4. Successful Save: Upon successfully writing an entity to its .jasset file, the engine resets entity->is_dirty = false.

6.4 Version Control Benefits (Git Friendly Assets)

By coupling the one-file-per-entity .jasset format with dirty tracking:

• Only modified entities are touched on disk.
• Git status displays only the exact .jasset files that were altered by the designer.
• Team members can work concurrently in the same game scene without encountering binary merge conflicts.

6.5 Editor Integration (RenderWorldEditorUI Hooks)

In Game/Data/World.cpp, editor UI widgets automatically set the dirty flag upon receiving user input:

float pos[4] = { ent.position.x, ent.position.y, ent.position.z, ent.position.w };
if (ImGui::DragFloat3("Position", pos, 0.1f))
{
    ent.position.x = pos[0];
    ent.position.y = pos[1];
    ent.position.z = pos[2];
    ent.is_dirty   = true; // Mark dirty for persistence
    UpdateWorld(world);
}

---

7. Step-by-Step Implementation Roadmap

Phase 1: Data Structures & Header Definitions

1. Update Class.h & Entity.h:
   • Add using default_init_fct_type = void (*)(void* payload); and default_init_fct to struct Class and MakeClass.
   • Update DEFINE_ENTITY_VERSIONED and DEFINE_CLASS_VERSIONED to define default_init_##entity and pass it to MakeClass.
   • Add bool is_dirty = false; to struct Entity.
   • Update MakeEntity<EntityType> to assign base_ptr->ID = EntityManager::ID++; and delegate allocation and defaults cleanly to AllocateEntity.
   • Declare [[nodiscard]] const Class* ResolveEntityClass(uint8 kind, uint32 crc); and [[nodiscard]] Class* find_class_by_name(String name); in Entity.h.
2. Update EntityManager.h:
   • Declare [[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* class_ptr);.
   • (Note: DestroyEntity and RemoveDerivedComponent deferred to Entity_Removal_Brainstorm.md)
3. Update World.h:
   • Add VectorArena<EntityID, 1024> PendingDeletions; to struct World.
   • Update SaveWorld and LoadWorld signatures to take directory paths.

Phase 2: Core Memory Allocation & Wiring in EntityManager.cpp

1. Implement AllocateEntity:
   • Enforce parameter assertions.
   • Push to manager.Entities with baseTemplate.ID = 0 (unassigned).
   • Allocate block in manager.by_type[kind].arena.
   • Call derivedClassPtr->default_init_fct(rawMemory) (or MemZero if null) to initialize struct defaults.
   • Wire mutual pointers (base->derived and derived->base).
   • Increment typedArray.count and initialize typedArray.array.
2. (Deferred for rework) DestroyEntity & RemoveDerivedComponent (See Game/Plans/Entity_Removal_Brainstorm.md).

Phase 3: In-Place Deserialization & Serialization Pipeline

1. In Entity.cpp:
   • Implement ResolveEntityClass and find_class_by_name.
   • serialize(Archive& ar, NonNullPtr<Entity> entity) handles both base and derived class serialization.
2. In World.cpp:
   • Implement serialize_entity_asset(Archive& ar, NonNullPtr<Entity> entity, String filepath).
   • Implement deserialize_entity_asset(EntityManager& manager, Archive& ar, String filepath).
   • In deserialize_entity_in_place, advance EntityManager::ID past base_ptr->ID to prevent ID collisions.

Phase 4: World Save/Load Pipeline & Disk Deletion

1. In World.cpp:
   • Implement ProcessPendingDeletions(World& world, NonNullPtr<Arena> scratchArena).
   • Implement SaveWorld(World& world, String worldDirectory):
     • Process pending deletions.
     • Iterate manager.Entities, skipping entities where !entity.is_dirty.
     • Write dirty entities to .jasset files and clear is_dirty.
   • Implement LoadWorld(World& world, String worldDirectory):
     • Enumerate .jasset files in directory.
     • Call deserialize_entity_asset for each file.

Phase 5: Editor Integration

1. In RenderWorldEditorUI:
   • Hook ImGui::DragFloat3 and property inspectors to set is_dirty = true.
   • Hook "Add Entity" button to call MakeEntity<Inert>(*world.EntityManager, 0.0f, 0.0f, 0.0f).
   • Hook "Delete Entity" button to call RemoveWorldEntity(world, selectedEntityId).

---

8. Unit Testing & Verification Plan

8.1 Test Philosophy & Constraints

Following Juliet coding guidelines:

"When creating a new system framework, make a unit test. To make the unit test we should not modify the framework code for special unit test case."

Testing is isolated in Game/UnitTest/WorldUnitTest.cpp and executed during engine initialization in debug builds.

8.2 Comprehensive Test Suite (WorldUnitTest.cpp)

The test suite validates every guarantee made in this specification:

#include <UnitTest/WorldUnitTest.h>

#if JULIET_DEBUG

#include <Core/Common/CoreUtils.h>
#include <Core/Common/serialization.h>
#include <Core/HAL/Filesystem/Filesystem.h>
#include <Core/HAL/IO/IOStream.h>
#include <Core/Logging/LogManager.h>
#include <Core/Logging/LogTypes.h>
#include <Core/Memory/MemoryArena.h>
#include <Core/Thread/ThreadContext.h>
#include <Data/World.h>
#include <Entity/Entity.h>
#include <Entity/EntityManager.h>

namespace UnitTest
{
    namespace
    {
        void TestEntityAllocationAndWiring()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestEntityAllocationAndWiring...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);

            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // 1. Allocate Inert Entity via AllocateEntity
            Entity* base_entity = AllocateEntity(manager, Inert::kind);
            Assert(base_entity != nullptr);
            Assert(base_entity->ID == 0); // Pure memory allocator leaves ID unassigned (0) until MakeEntity or deserialization
            base_entity->ID = EntityManager::ID++;
            Assert(base_entity->ID > 0);
            Assert(base_entity->derived_kind == Inert::kind);
            Assert(base_entity->derived != nullptr);
            Assert(base_entity->is_dirty == true);

            // 2. Validate mutual back-pointer wiring
            auto* derived = reinterpret_cast<entity_template*>(base_entity->derived);
            Assert(derived->base == base_entity);

            // 3. DownCast verification
            Inert* inert = DownCast<Inert>(base_entity);
            Assert(inert != nullptr);
            Assert(inert->base == base_entity);

            // 4. Validate typed array tracking
            typed_entity_array& inert_array = manager.by_type[ENTITY(Inert)];
            Assert(inert_array.count == 1);
            Assert(inert_array.array == derived);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestEntityAllocationAndWiring");
        }

        void TestInPlaceDeserialization()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestInPlaceDeserialization...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // 1. Create and populate entity
            Inert* created_inert = MakeEntity<Inert>(manager, 12.5f, -44.0f, 108.2f);
            Assert(created_inert != nullptr);
            created_inert->MeshInstance = 42;

            Entity*  original_base = created_inert->base;
            EntityID original_id   = original_base->ID;

            // 2. Serialize to text archive memory stream
            MemoryStream mem_stream = MakeMemoryStream(tempArena.Arena);
            Archive      save_ar{ .arena = tempArena.Arena, .loading = false, .stream = &mem_stream };
            serialize(save_ar, NonNullPtr<Entity>(original_base));

            // 3. Clear manager to simulate fresh load
            ShutdownEntityManager();
            InitEntityManager(&testWorld);
            EntityManager& fresh_manager = *testWorld.EntityManager;

            // 4. Tokenize and deserialize in-place
            Archive load_ar{ .arena = tempArena.Arena, .loading = true };
            tokenize_archive(tempArena.Arena, mem_stream.buffer, &load_ar.base);
            Entity* loaded_base = deserialize_entity_in_place(fresh_manager, load_ar);

            Assert(loaded_base != nullptr);
            Assert(loaded_base->ID == original_id);
            Assert(EntityManager::ID > original_id); // Counter was advanced past loaded ID to prevent collisions
            Assert(loaded_base->position.x == 12.5f);
            Assert(loaded_base->position.y == -44.0f);
            Assert(loaded_base->position.z == 108.2f);
            Assert(loaded_base->is_dirty == false);

            Inert* loaded_inert = DownCast<Inert>(loaded_base);
            Assert(loaded_inert != nullptr);
            Assert(loaded_inert->base == loaded_base);
            Assert(loaded_inert->MeshInstance == 42);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestInPlaceDeserialization");
        }

        void TestDirtyTrackingLifecycle()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestDirtyTrackingLifecycle...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            Entity* entity = AllocateEntity(manager, Inert::kind);
            Assert(entity->is_dirty == true);

            // Simulate save
            entity->is_dirty = false;
            Assert(entity->is_dirty == false);

            // Simulate mutation
            entity->position.x += 1.0f;
            entity->is_dirty = true;
            Assert(entity->is_dirty == true);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestDirtyTrackingLifecycle");
        }
    } // namespace

    void WorldUnitTest()
    {
        Log(LogLevel::Message, LogCategory::Game, "==================================================");
        Log(LogLevel::Message, LogCategory::Game, "Starting Entity Allocation & Lifecycle Unit Tests");
        Log(LogLevel::Message, LogCategory::Game, "==================================================");

        TestEntityAllocationAndWiring();
        TestInPlaceDeserialization();
        // TestSwapAndPopPointerFixup(); // Deferred to Entity_Removal_Brainstorm.md
        TestDirtyTrackingLifecycle();

        Log(LogLevel::Message, LogCategory::Game, "==================================================");
        Log(LogLevel::Message, LogCategory::Game, "All Entity Lifecycle Unit Tests PASSED Successfully");
        Log(LogLevel::Message, LogCategory::Game, "==================================================");
    }
} // namespace UnitTest

#endif