Juliet/Game/Plans/02_Entity_Allocation_And_Lifecycle.md

Juliet Game Engine Architecture Specification

Document: 02 - Entity Allocation & In-Place Lifecycle

Document ID: JULIET-SPEC-002
Status: Approved for Implementation
Author: Senior Engine Architect
Subsystems: Game/Entity, Game/Data, Game/UnitTest
Target Files:

• Entity.h
• Entity.cpp
• EntityManager.h
• EntityManager.cpp
• World.h
• World.cpp
• WorldUnitTest.h
• WorldUnitTest.cpp

---

1. Executive Summary & Problem Statement

1.1 Background & Context

The Juliet game engine organizes game entities using a hybrid data-oriented architecture:

1. A flat array of Base Entities (Entity) encapsulating universal properties: unique 64-bit ID, runtime type reflection pointer (Class* Kind), spatial coordinates (X, Y, Z), and an opaque pointer to the derived payload (DerivedType Derived).
2. Type-segregated contiguous arrays of Derived Entities (Inert, etc.) stored in dedicated per-type memory arenas (typed_entity_array).

This design is intended to provide maximum cache efficiency during spatial and general-purpose entity processing, while retaining dense SIMD-friendly streaming for type-specific systems (e.g., transform updates on Inert mesh instances).

1.2 Root-Cause Analysis of Game/Data/World.cpp Loading Flaws

In the initial implementation of entity serialization in Game/Data/World.cpp, loading was fundamentally broken and incomplete:

// Existing flawed deserialization in World.cpp
for (typed_entity_array& type : entityManager.by_type)
{
    serialize_elem(ar, type.count);

    if (type.count > 0)
    {
        // Unserialize the base entity to get informations
        Entity entity;
        serialize(ar, &entity);

        RegisterBaseEntity(entityManager, entity);
    }
}

This implementation suffers from several fatal defects:

1. Single-Element Iteration Bug: It uses if (type.count > 0) instead of a loop for (size_t i = 0; i < type.count; ++i), deserializing at most one single entity per type bucket, leaving all subsequent entities in the stream unread and corrupting the archive read offset.
2. Missing Derived Allocation: It invokes RegisterBaseEntity(entityManager, entity), which merely pushes the stack-allocated Entity into manager.Entities. The derived payload arena (type.arena) is completely untouched: type.array remains null, type.count in the manager is desynchronized, and entity.Derived remains unassigned or points to an invalid address.
3. Invalid Pointer in RegisterEntity: In Game/Entity/EntityManager.cpp, RegisterEntity assigns base->Derived = entity before pushing *base into manager.Entities. The parameter entity is a pointer to caller-provided memory (often stack-allocated in helper functions like MakeEntity). When ArenaPushSize later allocates the persistent derived memory block, base->Derived stored inside manager.Entities.Back() is never updated—it remains dangling, pointing to the transient caller stack!

1.3 The "Chicken-and-Egg" Stack Allocation Dilemma

The existing registration API requires a pre-existing derived instance:

entity_template* RegisterEntity(EntityManager& manager, Entity* base, DerivedType entity);

During runtime programmatic creation via MakeEntity<T>(), a temporary instance of T is created on the stack and passed by pointer:

template <typename EntityType>
EntityType* MakeEntity(EntityManager& manager, float x, float y, float z)
{
    EntityType result; // Stack allocation
    Entity     base;   // Stack allocation
    base.X    = x;
    base.Y    = y;
    base.Z    = z;
    base.Kind = EntityType::Kind;

    return (EntityType*)RegisterEntity(manager, &base, &result);
}

When loading an entity from a stream or disk file, the type is not known at compile time. The engine reads a runtime type tag (uint8 kind or uint32 CRC), looks up the reflection metadata (Class*), and must instantiate the entity dynamically.

Because C++ does not permit allocating a dynamic struct of unknown type on the stack, and because Juliet strictly forbids heap allocations (malloc, new, std::vector), deserialization cannot construct a temporary instance on the stack to pass to RegisterEntity.

This is the classic chicken-and-egg memory problem:

• RegisterEntity requires an existing instance in memory to copy from.
• Deserialization requires an allocated memory buffer to deserialize into.

1.4 Architectural Objectives

This specification establishes a robust in-place lifecycle pipeline that completely eliminates stack temporaries and dynamic heap allocations:

1. Direct In-Place Allocation: Introduce AllocateEntity(EntityManager& manager, Class* classPtr) which allocates both the base Entity and the derived struct directly within their respective engine memory arenas.
2. Bidirectional Pointer Integrity: Wire mutual pointers (base->Derived and derived->Base) at allocation time before any field deserialization begins.
3. In-Place Stream Deserialization: Read class reflection metadata first, invoke AllocateEntity, and stream base and derived properties directly into arena-resident memory.
4. Isolated Entity Assets (.jasset): Transition from a monolithic world.bin to a modular one-file-per-entity architecture (Assets/Entities/{ID}.jasset).
5. Dirty Tracking & Optimal Saves: Introduce an IsDirty flag on Entity to avoid rewriting unchanged entity files, minimizing disk I/O and eliminating spurious Git repository modifications.
6. Robust Deletion Lifecycle: Decouple in-memory removal (RemoveAtFast with pointer fixup) from disk synchronization using World::PendingDeletions.

---

2. The Dual Arena Memory Model in EntityManager

2.1 The Need for Dual Storage

Game engines execute systems with vastly different cache locality profiles:

• Spatial / Frustum Culling / Transform Sync: Iterates every entity in the world, needing only X, Y, Z, bounding spheres, and base status flags.
• Specialized Logic / Render Updates: Iterates only entities possessing specific components (e.g., Inert static meshes requiring instance transform updates to the GPU bindless descriptor table).

Storing large monolithic polymorphic structs in a single array causes severe cache line pollution during spatial passes. Conversely, storing entities in fragmented individual allocations introduces cache misses and pointer-chasing overhead.

Juliet resolves this with a Dual Arena Memory Model:

+---------------------------------------------------------------------------------------------+
|                                    EntityManager                                            |
+---------------------------------------------------------------------------------------------+
|                                                                                             |
|   manager.Entities (VectorArena<Entity, 100'000>)                                          |
|   +------------------------------------+------------------------------------+           |
|   | Entity 0 (ID=1001, X, Y, Z)        | Entity 1 (ID=1002, X, Y, Z)        | ...       |
|   | Derived ------------------------+  | Derived ---------------------+     |           |
|   +---------------------------------|--+------------------------------|-----+           |
|                                     |                                 |                     |
|                                     v                                 v                     |
|   manager.by_type[ENTITY(Inert)].arena                                                      |
|   +------------------------------------+------------------------------------+           |
|   | Inert 0 (MeshInstance=4)           | Inert 1 (MeshInstance=12)          | ...       |
|   | Base ---------------------------+  | Base ------------------------+     |           |
|   +---------------------------------|--+------------------------------|-----+           |
|                                     +---------------------------------+                     |
+---------------------------------------------------------------------------------------------+

2.2 manager.Entities: Cache-Friendly Base Entity Vector

Base entities reside in a pre-reserved contiguous array:

VectorArena<Entity, 100'000> Entities;

• Capacity: Fixed reserve of 100,000 entities allocated from the WorldArena.
• Memory Footprint: \text{sizeof(Entity)} = 8\text{ (ID)} + 8\text{ (Kind)} + 8\text{ (Derived)} + 12\text{ (X, Y, Z)} + 1\text{ (IsDirty)} + 3\text{ (Padding)} = 40\text{ bytes} Total reserved space: 100{,}000 \times 40\text{ bytes} \approx 4.0\text{ MB}.
• Access Speed: O(1) random access by index; sequential streaming utilizes L1/L2 hardware prefetchers with zero cache line waste.

2.3 manager.by_type[kind].arena: Typed Component Arenas

Derived structs reside in per-type contiguous memory arenas:

struct typed_entity_array
{
    Arena*           arena;
    entity_template* array;
    size_t           count;
};

• Each Entity_Type index owns an isolated Arena* allocated during InitEntityManager.
• Allocations are packed linearly with alignment specified by classPtr->alignment.
• array points directly to the first element in the arena, permitting typed array indexing:

  Inert* inertArray = reinterpret_cast<Inert*>(manager.by_type[ENTITY(Inert)].array);
  

2.4 Mutual Back-Pointer Architecture & Invariants

Every entity instance consists of two mutually linked allocations:

1. basePtr->Derived: Points from Entity in manager.Entities to the derived struct in by_type[kind].arena.
2. derivedPtr->Base: Points from the derived struct (via DECLARE_ENTITY()) back to Entity in manager.Entities.

Invariant Rules:

1. Non-Null Invariant: For any active entity, basePtr->Derived != nullptr and reinterpret_cast<entity_template*>(basePtr->Derived)->Base == basePtr.
2. Type Coherence Invariant: basePtr->Kind->kind == derivedTypeId.
3. Array Count Invariant: \sum_{k=0}^{\text{ENTITY(Count)}-1} \text{manager.by\_type}[k].\text{count} == \text{manager.Entities.Size()}

2.5 Pointer Stability in VectorArena

VectorArena::Create executes Reserve(ReserveSize) on creation:

newManager->Entities.Create(world->WorldArena JULIET_DEBUG_PARAM("Entities"));

Because capacity (100{,}000) is fully reserved upfront in virtual address space, VectorArena::PushBack never reallocates or moves existing memory. Therefore:

• Pointers to Entity elements in manager.Entities remain absolutely stable across allocations.
• Derived struct Base pointers remain valid indefinitely unless an element is deleted.
• Element removal via swap-and-pop alters memory positions, requiring systematic pointer fixups (addressed in Section 5.2).

---

3. The Solution: AllocateEntity(EntityManager& manager, Class* classPtr)

3.1 Function Signature & Contract

The canonical allocation function is defined in EntityManager.h:

[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr);

Preconditions:

• classPtr != nullptr.
• classPtr->kind < ENTITY(Count).
• classPtr->size_of >= sizeof(entity_template).
• manager.Entities.Size() < manager.Entities.Capacity.
• manager.by_type[classPtr->kind].arena != nullptr.

Postconditions:

• A new Entity record is appended to manager.Entities.
• A new typed block of classPtr->size_of bytes is allocated in manager.by_type[classPtr->kind].arena.
• The derived memory is zeroed.
• base->Derived points to the derived struct.
• derived->Base points to the base Entity.
• base->Kind is assigned to classPtr.
• base->ID is assigned the next unique EntityManager::ID.
• base->IsDirty is initialized to true.
• typed_entity_array::count is incremented.
• typed_entity_array::array is initialized if this is the first entity of this type.

3.2 Detailed Step-by-Step Implementation

The implementation replaces the flawed RegisterEntity routine in Game/Entity/EntityManager.cpp:

[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr)
{
    Assert(classPtr != nullptr);
    Assert(classPtr->kind < ENTITY(Count));
    Assert(classPtr->size_of >= sizeof(entity_template));
    Assert(classPtr->alignment > 0);

    // 1. Allocate uninitialized Base Entity in the contiguous VectorArena
    Entity baseTemplate{};
    baseTemplate.ID      = EntityManager::ID++;
    baseTemplate.Kind    = classPtr;
    baseTemplate.Derived = nullptr;
    baseTemplate.X       = 0.0f;
    baseTemplate.Y       = 0.0f;
    baseTemplate.Z       = 0.0f;
    baseTemplate.IsDirty = true;

    manager.Entities.PushBack(baseTemplate);
    Entity* basePtr = manager.Entities.Back();
    Assert(basePtr != nullptr);

    // 2. Allocate zeroed derived component memory in the typed arena
    typed_entity_array& typedArray = manager.by_type[classPtr->kind];
    Assert(typedArray.arena != nullptr);

    void* rawMemory = ArenaPushSize(
        typedArray.arena, 
        classPtr->size_of, 
        classPtr->alignment, 
        true JULIET_DEBUG_PARAM(kEntity_type_names[classPtr->kind]));
    Assert(rawMemory != nullptr);

    auto* derivedTemplate = reinterpret_cast<entity_template*>(rawMemory);

    // 3. Establish mutual back-pointers
    basePtr->Derived      = rawMemory;
    derivedTemplate->Base = basePtr;

    // 4. Update typed array tracking
    if (typedArray.array == nullptr)
    {
        typedArray.array = derivedTemplate;
    }
    typedArray.count += 1;

    return basePtr;
}

3.3 Pure C-Style Zeroing & Initialization

ArenaPushSize is called with shouldBeZeroed = true (or followed by MemZero). This guarantees that:

• All bytes are zeroed (0x00).
• Pointers inside derived structs default to nullptr.
• Numerical fields default to 0.

For structs that require specific non-zero sentinel values (such as Inert::MeshInstance = indexMax), Juliet follows a pure C approach without C++ constructor or placement-new machinery:

1. Direct assignment or C-style init function:

inline void InitInert(Inert* inert)
{
    Assert(inert != nullptr);
    inert->MeshInstance = indexMax;
}

2. Or simple struct literal assignment:

*inert = Inert{ .MeshInstance = indexMax };

No C++ placement new, <new> headers, or hidden constructor/destructor calls are ever used. Data structures remain pure C-style aggregates.

3.4 Bidirectional Pointer Wiring

Notice the sequence:

1. manager.Entities.PushBack(baseTemplate) places the struct at its final, fixed arena address.
2. basePtr = manager.Entities.Back() retrieves the persistent memory pointer.
3. derivedTemplate->Base = basePtr wires the derived back-pointer directly to this permanent location.
4. basePtr->Derived = rawMemory wires the base forward-pointer to the arena-allocated derived struct.

No stack copying occurs. Neither pointer is ever left dangling.

3.5 Updating Type Counts and Array Cache

typed_entity_array maintains:

• typedArray.count: The exact count of active entities of this type.
• typedArray.array: Pointer to the first element in the arena.

When the first entity of a given type is allocated, typedArray.array is set to derivedTemplate. Because the arena allocates sequentially, typedArray.array[i] can be indexed directly with stride classPtr->size_of as long as memory remains contiguous.

3.6 Refactoring MakeEntity<EntityType> Template

With AllocateEntity operational, MakeEntity in Game/Entity/Entity.h becomes clean, safe, and stack-free:

template <typename EntityType>
    requires EntityConcept<EntityType>
[[nodiscard]] EntityType* MakeEntity(EntityManager& manager, float x, float y, float z)
{
    Entity* basePtr = AllocateEntity(manager, EntityType::Kind);
    Assert(basePtr != nullptr);

    basePtr->X = x;
    basePtr->Y = y;
    basePtr->Z = z;

    auto* derivedPtr = static_cast<EntityType*>(basePtr->Derived);
    ConstructDerivedDefaults<EntityType>(derivedPtr);
    
    // Re-establish Base pointer after placement-new
    derivedPtr->Base = basePtr;

    return derivedPtr;
}

---

4. In-Place Deserialization Pipeline

4.1 Asset Format Specification (.jasset)

To achieve robust version control and modular streaming, Juliet adopts a human-readable, Git-diffable one-file-per-entity disk format with extension .jasset.

; asset_type
entity_instance
; id
0x0100000000000042
; class
Inert
; base_version
1
; derived_version
1
; position
0.43 0.32 1.56
; mesh_instance
12

Important: No Header Structs for Derived Types

• Derived types NEVER require their own file header: You do not write an InertHeader, DoorHeader, or PlayerHeader. Derived types only serialize their own member variables.
• No binary EntityFileHeader struct is needed: Under the ; variable_name\nvalues text format, there is no packed binary C-struct header at all. The common properties (; id, ; class, ; base_version, ; derived_version, ; position) are standard text Key-Value nodes read by the exact same archive parser.

4.2 Eliminating Intermediate Stack Allocations

Under the new pipeline:

1. The .jasset text file is read into memory onto a TempArena via LoadFile.
2. The lines are tokenized into a TextPropertyNode scratch table.
3. The engine reads ; class (e.g. "Inert") and resolves Class* classPtr = FindClassByName(className).
4. AllocateEntity(manager, classPtr) is called immediately. Memory for both base and derived components is allocated in-place in their permanent engine arenas.
5. Base properties (id, position, etc.) are read directly into *basePtr via SerializeEntityBase.
6. classPtr->serialize_fct(ar, basePtr->Derived, derivedVersion) is called. Derived fields stream directly into the typed arena without temporary staging buffers or stack copies.

4.3 Runtime Class Resolution

To ensure fast and safe type lookup during file deserialization:

[[nodiscard]] Class* ResolveEntityClass(uint8 kind, uint32 crc)
{
    if (kind >= ENTITY(Count))
    {
        return nullptr;
    }

    Class* classPtr = kEntity_type_class_ptr[kind];
    if (!classPtr)
    {
        return nullptr;
    }

    if (classPtr->CRC != crc)
    {
        return nullptr;
    }

    return classPtr;
}

4.4 In-Place Deserialization Algorithm

+---------------------------------------------------------------------------------------+
|                       In-Place Deserialization Flowchart                              |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   1. Read Header from IOStream/ByteBuffer                                             |
|              |                                                                        |
|              v                                                                        |
|   2. Validate Magic ('JAST') and Version (1)                                          |
|              |                                                                        |
|              v                                                                        |
|   3. Resolve Class* from header.Kind & header.ClassCRC                                |
|              |                                                                        |
|              v                                                                        |
|   4. basePtr = AllocateEntity(manager, classPtr)                                      |
|              |                                                                        |
|              +--> [manager.Entities]: Allocates base Entity                           |
|              +--> [manager.by_type[kind].arena]: Allocates derived struct             |
|              +--> Mutual Back-Pointers Wired In-Place                                 |
|              |                                                                        |
|              v                                                                        |
|   5. Direct Copy Base Properties (ID, X, Y, Z)                                        |
|              |                                                                        |
|              v                                                                        |
|   6. Does classPtr->serialize_fct exist?                                              |
|         |                     |                                                       |
|        Yes                    No                                                      |
|         |                     |                                                       |
|         v                     |                                                       |
|      Invoke:                  |                                                       |
|      serialize_fct(&ar,       |                                                       |
|                    Derived)   |                                                       |
|         |                     |                                                       |
|         +---------------------+                                                       |
|              |                                                                        |
|              v                                                                        |
|   7. Clear Dirty Flag: basePtr->IsDirty = false                                       |
|                                                                                       |
+---------------------------------------------------------------------------------------+

[[nodiscard]] Entity* DeserializeEntityInPlace(EntityManager& manager, archive& ar)
{
    Assert(ar.loading);

    // 1. Read class name and resolve Class*
    String className = {};
    SerializeProp(ar, "class", className, ar.arena);
    Class* classPtr = FindClassByName(className);
    if (!classPtr)
    {
        return nullptr;
    }

    // 2. Allocate persistent memory for base and derived components in their respective arenas
    Entity* basePtr = AllocateEntity(manager, classPtr);
    Assert(basePtr != nullptr);

    // 3. Read base versions and properties in-place
    uint16 baseVersion    = 1;
    uint16 derivedVersion = 1;
    SerializeProp(ar, "base_version", baseVersion);
    SerializeProp(ar, "derived_version", derivedVersion);

    SerializeEntityBase(ar, *basePtr, baseVersion);

    // 4. Stream derived properties in-place directly into the typed arena
    if (classPtr->serialize_fct != nullptr)
    {
        classPtr->serialize_fct(ar, basePtr->Derived, derivedVersion);
    }

    // Freshly loaded entity matches disk state exactly
    basePtr->IsDirty = false;

    return basePtr;
}

---

5. Entity Deletion Lifecycle & Disk Synchronization

5.1 In-Memory Removal vs. Immediate Disk Deletion Hazards

In game development, deleting an entity in the editor or during gameplay must never synchronously invoke disk deletion:

1. Frame Rate Stutters: Blocking on synchronous OS filesystem APIs (DeleteFileA) introduces multisecond frame freezes.
2. Transactional Safety: If the editor crashes or the user exits without saving, disk modifications cannot be rolled back.
3. Undo/Redo Support: An editor action stack must allow recovering deleted entities before changes are permanently committed to disk.

Therefore, Juliet enforces a strict separation:

• Immediate in-memory destruction: Releases the entity from active simulation and registers its identifier in World::PendingDeletions.
• Deferred disk deletion: Executed strictly during explicit SaveWorld operations.

5.2 Fast In-Memory Removal (RemoveAtFast) & Mutual Pointer Fixup

VectorArena::RemoveAtFast utilizes swap-and-pop: the element at the target index is replaced by the last element in the vector, and Count is decremented.

void RemoveAtFast(index_t index)
{
    Assert(Arena);
    Assert(index < Count);
    Assert(Count > 0);

    Type* elementAdr = DataFirst + index;

    if (DataLast != elementAdr)
    {
        Swap(DataLast, elementAdr);
    }

    --DataLast;
    --Count;
}

The Pointer Invalidation Problem:

When Entity A (at index) is swapped with Entity Z (at DataLast), the physical address of Entity Z changes from DataLast to elementAdr. If Entity Z has a derived struct derivedZ, derivedZ->Base previously pointed to DataLast. After RemoveAtFast, derivedZ->Base points to garbage or the freed slot!

The Pointer Fixup Protocol:

To preserve the mutual back-pointer invariant, DestroyEntity explicitly fixes up the swapped entity's derived back-pointer:

void DestroyEntity(EntityManager& manager, EntityID id)
{
    Entity* baseArray = manager.Entities.DataPtr();
    size_t  count     = manager.Entities.Size();

    size_t targetIndex = indexMax;
    for (size_t i = 0; i < count; ++i)
    {
        if (baseArray[i].ID == id)
        {
            targetIndex = i;
            break;
        }
    }

    if (targetIndex == indexMax)
    {
        return;
    }

    Entity* targetEntity = &baseArray[targetIndex];
    Class*  classPtr     = targetEntity->Kind;
    Assert(classPtr != nullptr);

    // 1. Remove derived component from typed array via swap-and-pop
    RemoveDerivedComponent(manager, classPtr, targetEntity->Derived);

    // 2. Remove base entity via swap-and-pop in VectorArena
    bool wasLast = (targetIndex == count - 1);
    manager.Entities.RemoveAtFast(targetIndex);

    // 3. Pointer fixup: If an element was swapped into targetIndex, fix its back-pointer!
    if (!wasLast && targetIndex < manager.Entities.Size())
    {
        Entity* movedEntity = &manager.Entities[targetIndex];
        auto*   derivedTemp = reinterpret_cast<entity_template*>(movedEntity->Derived);
        Assert(derivedTemp != nullptr);
        derivedTemp->Base = movedEntity;
    }
}

5.3 O(1) Component Removal in typed_entity_array via Swap-and-Pop

To keep derived components packed contiguously for SIMD/cache iteration:

1. Locate the component's index within by_type[kind].arena. Because components have uniform stride classPtr->size_of: \text{componentIndex} = \frac{\text{reinterpret\_cast<uint8*>(derivedPtr)} - \text{reinterpret\_cast<uint8*>(typedArray.array)}}{\text{classPtr->size\_of}}
2. If the component is not the last one in the typed arena:
   • Copy the last component into the slot occupied by the deleted component.
   • Update the moved component's Base->Derived pointer to point to its new slot.
3. Decrement typedArray.count.
4. Pop the arena allocation if it was the top of the stack, or decrement count to mark slot reclamation.

void RemoveDerivedComponent(EntityManager& manager, Class* classPtr, DerivedType derivedPtr)
{
    Assert(classPtr != nullptr);
    Assert(derivedPtr != nullptr);

    typed_entity_array& typedArray = manager.by_type[classPtr->kind];
    Assert(typedArray.count > 0);
    Assert(typedArray.array != nullptr);

    size_t stride     = classPtr->size_of;
    auto*  targetByte = reinterpret_cast<uint8*>(derivedPtr);
    auto*  firstByte  = reinterpret_cast<uint8*>(typedArray.array);

    size_t componentIndex = static_cast<size_t>(targetByte - firstByte) / stride;
    Assert(componentIndex < typedArray.count);

    size_t lastIndex = typedArray.count - 1;
    if (componentIndex != lastIndex)
    {
        uint8* lastByte = firstByte + (lastIndex * stride);
        
        // Copy last component data into target slot
        MemCopy(targetByte, lastByte, stride);

        // Fixup the base pointer of the moved component
        auto* movedDerived = reinterpret_cast<entity_template*>(targetByte);
        Assert(movedDerived->Base != nullptr);
        movedDerived->Base->Derived = targetByte;
    }

    typedArray.count -= 1;
    if (typedArray.count == 0)
    {
        typedArray.array = nullptr;
    }
}

5.4 Tracking Deletions in World::PendingDeletions

In World.h, the World struct is extended with a pending deletions container:

struct World
{
    Arena*                       WorldArena    = nullptr;
    EntityManager*               EntityManager = nullptr;
    VectorArena<EntityID, 1024>  PendingDeletions;
};

When an entity is deleted in the world:

void RemoveWorldEntity(World& world, EntityID id)
{
    Assert(world.EntityManager != nullptr);

    // Record pending disk deletion
    world.PendingDeletions.PushBack(id);

    // Destroy in memory immediately
    DestroyEntity(*world.EntityManager, id);
}

5.5 Disk File Cleanup during SaveWorld

During SaveWorld, before saving modified entities, the engine iterates over world.PendingDeletions and removes their associated .jasset files:

void ProcessPendingDeletions(World& world, NonNullPtr<Arena> scratchArena)
{
    for (size_t i = 0; i < world.PendingDeletions.Size(); ++i)
    {
        EntityID id = world.PendingDeletions[i];

        // Format relative asset path: Assets/Entities/{ID}.jasset
        char filenameBuffer[64];
        juliet_snprintf(filenameBuffer, sizeof(filenameBuffer), "Entities/%llu.jasset", id);
        
        String assetPath = GetAssetPath(scratchArena, WrapString(filenameBuffer));

        if (PlatformDeleteFile(assetPath))
        {
            Log(LogLevel::Message, LogCategory::Game, "Deleted entity asset: %s", CStr(assetPath));
        }
    }

    world.PendingDeletions.Clear();
}

---

6. Dirty Tracking for Optimal Saves

6.1 The Cost of Naive Monolithic & Full-Directory Writes

In a level containing 10{,}000 entities:

• Monolithic world.bin Save: Modifying a single entity's X coordinate requires re-serializing all 10{,}000 entities and overwriting a multi-megabyte binary file. This introduces a huge Git diff and constant merge conflicts.
• Full-Directory .jasset Save: Iterating through all 10{,}000 entities and unconditionally writing 10{,}000 .jasset files incurs massive OS file-system overhead (directory table locks, I/O bandwidth) and changes the file timestamps of every asset. Git reports thousands of modified files even when only one entity changed!

6.2 The IsDirty Flag on Entity

To solve this, Entity in Game/Entity/Entity.h is augmented with an explicit dirty flag:

struct Entity final
{
    EntityID    ID      = 0;
    Class*      Kind    = nullptr;
    DerivedType Derived = nullptr;
    float       X       = 0.0f;
    float       Y       = 0.0f;
    float       Z       = 0.0f;
    bool        IsDirty = false;
};

6.3 Granular State Transitions

The IsDirty flag obeys a strict lifecycle state machine:

                  +-----------------------------------+
                  |          Entity Created           |
                  |     (AllocateEntity / Editor)     |
                  +-----------------+-----------------+
                                    |
                                    v
                            +---------------+
                   +------->| IsDirty: TRUE |<-------+
                   |        +-------+-------+        |
                   |                |                |
        Entity Mutated              |            SaveWorld
    (Position, Component)           |            Completed
                   |                v                |
                   |        +---------------+        |
                   +--------+ IsDirty: FALSE+--------+
                            +-------+-------+
                                    ^
                                    |
                            Deserialization
                            (LoadWorld / Asset)

1. Entity Creation: Newly spawned entities in the editor have IsDirty = true.
2. Property Mutation: Any modification to X, Y, Z or derived component payload sets entity->IsDirty = true.
3. Successful Deserialization: Entities loaded from disk initialize with IsDirty = false.
4. Successful Save: Upon successfully writing an entity to its .jasset file, the engine resets entity->IsDirty = false.

6.4 Version Control Benefits (Git Friendly Assets)

By coupling the one-file-per-entity .jasset format with dirty tracking:

• Only modified entities are touched on disk.
• Git status displays only the exact .jasset files that were altered by the designer.
• Team members can work concurrently in the same game scene without encountering binary merge conflicts.

6.5 Editor Integration (RenderWorldEditorUI Hooks)

In Game/Data/World.cpp, editor UI widgets automatically set the dirty flag upon receiving user input:

float pos[3] = { ent.X, ent.Y, ent.Z };
if (ImGui::DragFloat3("Position", pos, 0.1f))
{
    ent.X       = pos[0];
    ent.Y       = pos[1];
    ent.Z       = pos[2];
    ent.IsDirty = true; // Mark dirty for persistence
    UpdateWorld(world);
}

---

7. Step-by-Step Implementation Roadmap

Phase 1: Data Structures & Header Definitions

1. Update Entity.h:
   • Add bool IsDirty = false; to struct Entity.
   • Update MakeEntity<EntityType> to delegate to AllocateEntity.
   • Define EntityFileHeader, kEntityAssetMagic, and kEntityAssetVersion.
2. Update EntityManager.h:
   • Declare [[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr);.
   • Declare void DestroyEntity(EntityManager& manager, EntityID id);.
   • Declare void RemoveDerivedComponent(EntityManager& manager, Class* classPtr, DerivedType derivedPtr);.
3. Update World.h:
   • Add VectorArena<EntityID, 1024> PendingDeletions; to struct World.
   • Update SaveWorld and LoadWorld signatures to take directory paths.

Phase 2: Core Memory Allocation & Wiring in EntityManager.cpp

1. Implement AllocateEntity:
   • Enforce parameter assertions.
   • Push to manager.Entities.
   • Allocate zeroed block in manager.by_type[kind].arena.
   • Wire mutual pointers (base->Derived and derived->Base).
   • Increment typedArray.count and initialize typedArray.array.
2. Implement DestroyEntity & RemoveDerivedComponent:
   • Implement swap-and-pop for derived components with base pointer update.
   • Implement manager.Entities.RemoveAtFast with mutual back-pointer fixup.

Phase 3: In-Place Deserialization & Serialization Pipeline

1. In Entity.cpp:
   • Implement EntitySerialize(archive& ar, Entity* entity).
   • Fix the existing reversed if (ar.loading) branch.
2. In World.cpp:
   • Implement SerializeEntityAsset(archive& ar, Entity* entity, String filepath).
   • Implement DeserializeEntityAsset(EntityManager& manager, archive& ar, String filepath).
   • Stream derived properties directly using classPtr->serialize_fct(&ar, entity->Derived).

Phase 4: World Save/Load Pipeline & Disk Deletion

1. In World.cpp:
   • Implement ProcessPendingDeletions(World& world, NonNullPtr<Arena> scratchArena).
   • Implement SaveWorld(World& world, String worldDirectory):
     • Process pending deletions.
     • Iterate manager.Entities, skipping entities where !entity.IsDirty.
     • Write dirty entities to .jasset files and clear IsDirty.
   • Implement LoadWorld(World& world, String worldDirectory):
     • Enumerate .jasset files in directory.
     • Call DeserializeEntityAsset for each file.

Phase 5: Editor Integration

1. In RenderWorldEditorUI:
   • Hook ImGui::DragFloat3 and property inspectors to set IsDirty = true.
   • Hook "Add Entity" button to call MakeEntity<Inert>(*world.EntityManager, 0.0f, 0.0f, 0.0f).
   • Hook "Delete Entity" button to call RemoveWorldEntity(world, selectedEntityId).

---

8. Unit Testing & Verification Plan

8.1 Test Philosophy & Constraints

Following Juliet coding guidelines:

"When creating a new system framework, make a unit test. To make the unit test we should not modify the framework code for special unit test case."

Testing is isolated in Game/UnitTest/WorldUnitTest.cpp and executed during engine initialization in debug builds.

8.2 Comprehensive Test Suite (WorldUnitTest.cpp)

The test suite validates every guarantee made in this specification:

#include <UnitTest/WorldUnitTest.h>

#if JULIET_DEBUG

#include <Core/Common/CoreUtils.h>
#include <Core/Common/serialization.h>
#include <Core/HAL/Filesystem/Filesystem.h>
#include <Core/HAL/IO/IOStream.h>
#include <Core/Logging/LogManager.h>
#include <Core/Logging/LogTypes.h>
#include <Core/Memory/MemoryArena.h>
#include <Core/Thread/ThreadContext.h>
#include <Data/World.h>
#include <Entity/Entity.h>
#include <Entity/EntityManager.h>

namespace UnitTest
{
    namespace
    {
        void TestEntityAllocationAndWiring()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestEntityAllocationAndWiring...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);

            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // 1. Allocate Inert Entity via AllocateEntity
            Entity* baseEntity = AllocateEntity(manager, Inert::Kind);
            Assert(baseEntity != nullptr);
            Assert(baseEntity->ID > 0);
            Assert(baseEntity->Kind == Inert::Kind);
            Assert(baseEntity->Derived != nullptr);
            Assert(baseEntity->IsDirty == true);

            // 2. Validate mutual back-pointer wiring
            auto* derived = reinterpret_cast<entity_template*>(baseEntity->Derived);
            Assert(derived->Base == baseEntity);

            // 3. DownCast verification
            Inert* inert = DownCast<Inert>(baseEntity);
            Assert(inert != nullptr);
            Assert(inert->Base == baseEntity);

            // 4. Validate typed array tracking
            typed_entity_array& inertArray = manager.by_type[ENTITY(Inert)];
            Assert(inertArray.count == 1);
            Assert(inertArray.array == derived);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestEntityAllocationAndWiring");
        }

        void TestInPlaceDeserialization()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestInPlaceDeserialization...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // 1. Create and populate entity
            Inert* createdInert = MakeEntity<Inert>(manager, 12.5f, -44.0f, 108.2f);
            Assert(createdInert != nullptr);
            createdInert->MeshInstance = 42;

            Entity* originalBase = createdInert->Base;
            EntityID originalID  = originalBase->ID;

            // 2. Serialize to memory archive
            archive saveAr{ .arena = tempArena.Arena, .base_ptr = nullptr, .offset = 0, .loading = false };
            saveAr.base_ptr = ArenaPushArray<uint8>(tempArena.Arena, Kilobytes(16));

            EntityFileHeader header{
                .Magic       = kEntityAssetMagic,
                .Version     = kEntityAssetVersion,
                .ClassCRC    = originalBase->Kind->CRC,
                .Kind        = originalBase->Kind->kind,
                .EntityID    = originalBase->ID,
                .PositionX   = originalBase->X,
                .PositionY   = originalBase->Y,
                .PositionZ   = originalBase->Z,
                .PayloadSize = sizeof(index_t)
            };
            serialize_elem(saveAr, header);
            serialize_elem(saveAr, createdInert->MeshInstance);

            // 3. Clear manager to simulate fresh load
            ShutdownEntityManager();
            InitEntityManager(&testWorld);
            EntityManager& freshManager = *testWorld.EntityManager;

            // 4. Deserialize in-place
            archive loadAr{ .arena = tempArena.Arena, .base_ptr = saveAr.base_ptr, .offset = 0, .loading = true };
            Entity* loadedBase = DeserializeEntityInPlace(freshManager, loadAr);

            Assert(loadedBase != nullptr);
            Assert(loadedBase->ID == originalID);
            Assert(loadedBase->X == 12.5f);
            Assert(loadedBase->Y == -44.0f);
            Assert(loadedBase->Z == 108.2f);
            Assert(loadedBase->IsDirty == false);

            Inert* loadedInert = DownCast<Inert>(loadedBase);
            Assert(loadedInert != nullptr);
            Assert(loadedInert->Base == loadedBase);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestInPlaceDeserialization");
        }

        void TestSwapAndPopPointerFixup()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestSwapAndPopPointerFixup...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // Allocate 3 entities: E0, E1, E2
            Inert* e0 = MakeEntity<Inert>(manager, 1.0f, 0.0f, 0.0f);
            Inert* e1 = MakeEntity<Inert>(manager, 2.0f, 0.0f, 0.0f);
            Inert* e2 = MakeEntity<Inert>(manager, 3.0f, 0.0f, 0.0f);

            EntityID id0 = e0->Base->ID;
            EntityID id1 = e1->Base->ID;
            EntityID id2 = e2->Base->ID;

            Assert(manager.Entities.Size() == 3);

            // Delete middle entity E1 (triggers swap with E2)
            DestroyEntity(manager, id1);

            Assert(manager.Entities.Size() == 2);

            // Verify E2's mutual back-pointers are still completely intact
            Entity* remaining0 = &manager.Entities[0];
            Entity* remaining1 = &manager.Entities[1];

            Assert(remaining0->ID == id0);
            Assert(remaining1->ID == id2);

            auto* derived0 = reinterpret_cast<entity_template*>(remaining0->Derived);
            auto* derived1 = reinterpret_cast<entity_template*>(remaining1->Derived);

            Assert(derived0->Base == remaining0);
            Assert(derived1->Base == remaining1);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestSwapAndPopPointerFixup");
        }

        void TestDirtyTrackingLifecycle()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestDirtyTrackingLifecycle...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            Entity* entity = AllocateEntity(manager, Inert::Kind);
            Assert(entity->IsDirty == true);

            // Simulate save
            entity->IsDirty = false;
            Assert(entity->IsDirty == false);

            // Simulate mutation
            entity->X += 1.0f;
            entity->IsDirty = true;
            Assert(entity->IsDirty == true);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestDirtyTrackingLifecycle");
        }
    } // namespace

    void WorldUnitTest()
    {
        Log(LogLevel::Message, LogCategory::Game, "==================================================");
        Log(LogLevel::Message, LogCategory::Game, "Starting Entity Allocation & Lifecycle Unit Tests");
        Log(LogLevel::Message, LogCategory::Game, "==================================================");

        TestEntityAllocationAndWiring();
        TestInPlaceDeserialization();
        TestSwapAndPopPointerFixup();
        TestDirtyTrackingLifecycle();

        Log(LogLevel::Message, LogCategory::Game, "==================================================");
        Log(LogLevel::Message, LogCategory::Game, "All Entity Lifecycle Unit Tests PASSED Successfully");
        Log(LogLevel::Message, LogCategory::Game, "==================================================");
    }
} // namespace UnitTest

#endif