Juliet/Game/Plans/02_Entity_Allocation_And_Lifecycle.md

Juliet Game Engine Architecture Specification

Document: 02 - Entity Allocation & In-Place Lifecycle

Document ID: JULIET-SPEC-002
Status: Approved for Implementation
Author: Senior Engine Architect
Subsystems: Game/Entity, Game/Data, Game/UnitTest
Target Files:

• Entity.h
• Entity.cpp
• EntityManager.h
• EntityManager.cpp
• World.h
• World.cpp
• WorldUnitTest.h
• WorldUnitTest.cpp

---

1. Executive Summary & Problem Statement

1.1 Background & Context

The Juliet game engine organizes game entities using a hybrid data-oriented architecture:

1. A flat array of Base Entities (Entity) encapsulating universal properties: unique 64-bit ID, runtime type reflection pointer (Class* Kind), spatial coordinates (X, Y, Z), and an opaque pointer to the derived payload (DerivedType Derived).
2. Type-segregated contiguous arrays of Derived Entities (Inert, etc.) stored in dedicated per-type memory arenas (typed_entity_array).

This design is intended to provide maximum cache efficiency during spatial and general-purpose entity processing, while retaining dense SIMD-friendly streaming for type-specific systems (e.g., transform updates on Inert mesh instances).

1.2 Root-Cause Analysis of Game/Data/World.cpp Loading Flaws

In the initial implementation of entity serialization in Game/Data/World.cpp, loading was fundamentally broken and incomplete:

// Existing flawed deserialization in World.cpp
for (typed_entity_array& type : entityManager.by_type)
{
    serialize_elem(ar, type.count);

    if (type.count > 0)
    {
        // Unserialize the base entity to get informations
        Entity entity;
        serialize(ar, &entity);

        RegisterBaseEntity(entityManager, entity);
    }
}

This implementation suffers from several fatal defects:

1. Single-Element Iteration Bug: It uses if (type.count > 0) instead of a loop for (size_t i = 0; i < type.count; ++i), deserializing at most one single entity per type bucket, leaving all subsequent entities in the stream unread and corrupting the archive read offset.
2. Missing Derived Allocation: It invokes RegisterBaseEntity(entityManager, entity), which merely pushes the stack-allocated Entity into manager.Entities. The derived payload arena (type.arena) is completely untouched: type.array remains null, type.count in the manager is desynchronized, and entity.Derived remains unassigned or points to an invalid address.
3. Invalid Pointer in RegisterEntity: In Game/Entity/EntityManager.cpp, RegisterEntity assigns base->Derived = entity before pushing *base into manager.Entities. The parameter entity is a pointer to caller-provided memory (often stack-allocated in helper functions like MakeEntity). When ArenaPushSize later allocates the persistent derived memory block, base->Derived stored inside manager.Entities.Back() is never updated—it remains dangling, pointing to the transient caller stack!

1.3 The "Chicken-and-Egg" Stack Allocation Dilemma

The existing registration API requires a pre-existing derived instance:

entity_template* RegisterEntity(EntityManager& manager, Entity* base, DerivedType entity);

During runtime programmatic creation via MakeEntity<T>(), a temporary instance of T is created on the stack and passed by pointer:

template <typename EntityType>
EntityType* MakeEntity(EntityManager& manager, float x, float y, float z)
{
    EntityType result; // Stack allocation
    Entity     base;   // Stack allocation
    base.X    = x;
    base.Y    = y;
    base.Z    = z;
    base.Kind = EntityType::Kind;

    return (EntityType*)RegisterEntity(manager, &base, &result);
}

When loading an entity from a stream or disk file, the type is not known at compile time. The engine reads a runtime type tag (uint8 kind or uint32 CRC), looks up the reflection metadata (Class*), and must instantiate the entity dynamically.

Because C++ does not permit allocating a dynamic struct of unknown type on the stack, and because Juliet strictly forbids heap allocations (malloc, new, std::vector), deserialization cannot construct a temporary instance on the stack to pass to RegisterEntity.

This is the classic chicken-and-egg memory problem:

• RegisterEntity requires an existing instance in memory to copy from.
• Deserialization requires an allocated memory buffer to deserialize into.

1.4 Architectural Objectives

This specification establishes a robust in-place lifecycle pipeline that completely eliminates stack temporaries and dynamic heap allocations:

1. Direct In-Place Allocation: Introduce AllocateEntity(EntityManager& manager, Class* classPtr) which allocates both the base Entity and the derived struct directly within their respective engine memory arenas.
2. Bidirectional Pointer Integrity: Wire mutual pointers (base->derived and derived->base) at allocation time before any field deserialization begins.
3. In-Place Stream Deserialization: Read class reflection metadata first, invoke AllocateEntity, and stream base and derived properties directly into arena-resident memory.
4. Isolated Entity Assets (.jasset): Transition from a monolithic world.bin to a modular one-file-per-entity architecture (Assets/Entities/{ID}.jasset).
5. Dirty Tracking & Optimal Saves: Introduce an is_dirty flag on Entity to avoid rewriting unchanged entity files, minimizing disk I/O and eliminating spurious Git repository modifications.
6. Robust Deletion Lifecycle: Decouple in-memory removal (RemoveAtFast with pointer fixup) from disk synchronization using World::PendingDeletions.

---

2. The Dual Arena Memory Model in EntityManager

2.1 The Need for Dual Storage

Game engines execute systems with vastly different cache locality profiles:

• Spatial / Frustum Culling / Transform Sync: Iterates every entity in the world, needing only X, Y, Z, bounding spheres, and base status flags.
• Specialized Logic / Render Updates: Iterates only entities possessing specific components (e.g., Inert static meshes requiring instance transform updates to the GPU bindless descriptor table).

Storing large monolithic polymorphic structs in a single array causes severe cache line pollution during spatial passes. Conversely, storing entities in fragmented individual allocations introduces cache misses and pointer-chasing overhead.

Juliet resolves this with a Dual Arena Memory Model:

+---------------------------------------------------------------------------------------------+
|                                    EntityManager                                            |
+---------------------------------------------------------------------------------------------+
|                                                                                             |
|   manager.Entities (VectorArena<Entity, 100'000>)                                          |
|   +------------------------------------+------------------------------------+           |
|   | Entity 0 (ID=1001, X, Y, Z)        | Entity 1 (ID=1002, X, Y, Z)        | ...       |
|   | Derived ------------------------+  | Derived ---------------------+     |           |
|   +---------------------------------|--+------------------------------|-----+           |
|                                     |                                 |                     |
|                                     v                                 v                     |
|   manager.by_type[ENTITY(Inert)].arena                                                      |
|   +------------------------------------+------------------------------------+           |
|   | Inert 0 (MeshInstance=4)           | Inert 1 (MeshInstance=12)          | ...       |
|   | Base ---------------------------+  | Base ------------------------+     |           |
|   +---------------------------------|--+------------------------------|-----+           |
|                                     +---------------------------------+                     |
+---------------------------------------------------------------------------------------------+

2.2 manager.Entities: Cache-Friendly Base Entity Vector

Base entities reside in a pre-reserved contiguous array:

VectorArena<Entity, 100'000> Entities;

• Capacity: Fixed reserve of 100,000 entities allocated from the WorldArena.
• Memory Footprint: \text{sizeof(Entity)} = 8\text{ (ID)} + 8\text{ (derived\_kind)} + 8\text{ (derived)} + 16\text{ (position)} + 1\text{ (is\_dirty)} + 7\text{ (Padding)} = 48\text{ bytes} Total reserved space: 100{,}000 \times 48\text{ bytes} \approx 4.8\text{ MB}.
• Access Speed: O(1) random access by index; sequential streaming utilizes L1/L2 hardware prefetchers with zero cache line waste.

2.3 manager.by_type[kind].arena: Typed Component Arenas

Derived structs reside in per-type contiguous memory arenas:

struct typed_entity_array
{
    Arena*           arena;
    entity_template* array;
    size_t           count;
};

• Each Entity_Type index owns an isolated Arena* allocated during InitEntityManager.
• Allocations are packed linearly with alignment specified by classPtr->alignment.
• array points directly to the first element in the arena, permitting typed array indexing:

  Inert* inertArray = reinterpret_cast<Inert*>(manager.by_type[ENTITY(Inert)].array);
  

2.4 Mutual Back-Pointer Architecture & Invariants

Every entity instance consists of two mutually linked allocations:

1. basePtr->Derived: Points from Entity in manager.Entities to the derived struct in by_type[kind].arena.
2. derivedPtr->Base: Points from the derived struct (via DECLARE_ENTITY()) back to Entity in manager.Entities.

Invariant Rules:

1. Non-Null Invariant: For any active entity, basePtr->Derived != nullptr and reinterpret_cast<entity_template*>(basePtr->Derived)->Base == basePtr.
2. Type Coherence Invariant: basePtr->Kind->kind == derivedTypeId.
3. Array Count Invariant: \sum_{k=0}^{\text{ENTITY(Count)}-1} \text{manager.by\_type}[k].\text{count} == \text{manager.Entities.Size()}

2.5 Pointer Stability in VectorArena

VectorArena::Create executes Reserve(ReserveSize) on creation:

newManager->Entities.Create(world->WorldArena JULIET_DEBUG_PARAM("Entities"));

Because capacity (100{,}000) is fully reserved upfront in virtual address space, VectorArena::PushBack never reallocates or moves existing memory. Therefore:

• Pointers to Entity elements in manager.Entities remain absolutely stable across allocations.
• Derived struct Base pointers remain valid indefinitely unless an element is deleted.
• Element removal via swap-and-pop alters memory positions, requiring systematic pointer fixups (addressed in Section 5.2).

---

3. The Solution: AllocateEntity(EntityManager& manager, Class* classPtr)

3.1 Function Signature & Contract

The canonical allocation function is defined in EntityManager.h:

[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr);

Preconditions:

• classPtr != nullptr.
• classPtr->kind < ENTITY(Count).
• classPtr->size_of >= sizeof(entity_template).
• manager.Entities.Size() < manager.Entities.Capacity.
• manager.by_type[classPtr->kind].arena != nullptr.

Postconditions:

• A new Entity record is appended to manager.Entities.
• A new typed block of classPtr->size_of bytes is allocated in manager.by_type[classPtr->kind].arena.
• The derived memory is initialized with C++ struct defaults via classPtr->default_init_fct (or zeroed via MemZero if null).
• base->derived points to the derived struct.
• derived->base points to the base Entity.
• base->derived_kind is assigned to class_ptr.
• base->ID is initialized to 0 (unassigned; populated by MakeEntity or deserialization).
• base->is_dirty is initialized to true.
• typed_entity_array::count is incremented.
• typed_entity_array::array is initialized if this is the first entity of this type.

3.2 Detailed Step-by-Step Implementation

The implementation replaces the flawed RegisterEntity routine in Game/Entity/EntityManager.cpp:

[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* derivedClassPtr)
{
    Assert(derivedClassPtr != nullptr);
    Assert(derivedClassPtr->kind < ENTITY(Count));
    Assert(derivedClassPtr->size_of >= sizeof(entity_template));
    Assert(derivedClassPtr->alignment > 0);

    // 1. Allocate Base Entity in the contiguous VectorArena (ID is 0 until assigned by MakeEntity or deserialization)
    Entity baseTemplate{};
    baseTemplate.ID           = 0;
    baseTemplate.derived_kind = derivedClassPtr;
    baseTemplate.derived      = nullptr;
    baseTemplate.position     = {};
    baseTemplate.is_dirty     = true;

    manager.Entities.PushBack(baseTemplate);
    Entity* basePtr = manager.Entities.Back();
    Assert(basePtr != nullptr);

    // 2. Allocate derived component memory in the typed arena
    typed_entity_array& typedArray = manager.by_type[derivedClassPtr->kind];
    Assert(typedArray.arena != nullptr);

    void* rawMemory = ArenaPushSize(
        typedArray.arena, 
        derivedClassPtr->size_of, 
        derivedClassPtr->alignment, 
        false JULIET_DEBUG_PARAM(kEntity_type_names[derivedClassPtr->kind]));
    Assert(rawMemory != nullptr);

    // 3. Initialize derived component defaults via Class reflection stub
    if (derivedClassPtr->default_init_fct != nullptr)
    {
        derivedClassPtr->default_init_fct(rawMemory);
    }
    else
    {
        MemZero(rawMemory, derivedClassPtr->size_of);
    }

    auto* derivedTemplate = reinterpret_cast<entity_template*>(rawMemory);

    // 4. Establish mutual back-pointers
    basePtr->derived      = rawMemory;
    derivedTemplate->base = basePtr;

    // 5. Update typed array tracking
    if (typedArray.array == nullptr)
    {
        typedArray.array = derivedTemplate;
    }
    typedArray.count += 1;

    return basePtr;
}

3.3 Default Struct Initialization via Class::default_init_fct

The Problem with Zero-Only Initialization

If allocation only zeroes memory (MemZero / 0x00), any C++ member variables with non-zero defaults (such as index_t MeshInstance = indexMax; or float Density = 1.0f;) are populated with 0. During deserialization, if a .jasset file lacks that property (e.g. an older file version or an optional field), SERIALIZE leaves the field untouched, meaning it incorrectly remains 0 rather than its intended default sentinel value!

The Solution: Compile-Time Default Stub in DEFINE_ENTITY_VERSIONED

Every entity descriptor Class includes a function pointer:

using default_init_fct_type = void (*)(void* payload);

struct Class
{
    ...
    default_init_fct_type default_init_fct = nullptr;
};

When registering an entity type with DEFINE_ENTITY_VERSIONED, the macro automatically defines a tiny type-safe stub that aggregate value-initializes the struct:

#define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct)                                                         \
    inline void default_init_##entity(void* payload)                                                                    \
    {                                                                                                                   \
        *static_cast<entity*>(payload) = entity{};                                                                      \
    }                                                                                                                   \
    constexpr Class entityKind##entity = MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version),         \
                                                   &classKindEntity, sizeof(entity), alignof(entity),                   \
                                                   (serialize_fct), default_init_##entity);                             \
    Class*          entity::kind       = const_cast<Class*>(&entityKind##entity);

Architectural Advantages:

1. Zero Boilerplate: Developers write member initializers once in the struct definition (index_t MeshInstance = indexMax;).
2. Type-Agnostic Core: AllocateEntity does not need to know any C++ struct types; it unconditionally calls derivedClassPtr->default_init_fct(rawMemory).
3. Robust Deserialization: In deserialize_entity_in_place, newly allocated entities already hold their canonical C++ defaults. Any properties absent in the .jasset file naturally retain their correct initial values.
4. No Dynamic Heap / Placement-New: *static_cast<entity*>(payload) = entity{} is pure aggregate value-assignment without <new> headers or exceptions.

3.4 Bidirectional Pointer Wiring

Notice the sequence:

1. manager.Entities.PushBack(baseTemplate) places the struct at its final, fixed arena address.
2. base_ptr = manager.Entities.Back() retrieves the persistent memory pointer.
3. derived_class_ptr->default_init_fct(raw_memory) initializes canonical struct defaults.
4. derived_template->base = base_ptr wires the derived back-pointer directly to this permanent location.
5. base_ptr->derived = raw_memory wires the base forward-pointer to the arena-allocated derived struct.

No stack copying occurs. Neither pointer is ever left dangling.

3.5 Updating Type Counts and Array Cache

typed_entity_array maintains:

• typedArray.count: The exact count of active entities of this type.
• typedArray.array: Pointer to the first element in the arena.

When the first entity of a given type is allocated, typedArray.array is set to derived_template. Because the arena allocates sequentially, typedArray.array[i] can be indexed directly with stride class_ptr->size_of as long as memory remains contiguous.

3.6 Refactoring MakeEntity<EntityType> Template

With AllocateEntity handling memory allocation, default member initialization, and mutual pointer wiring, MakeEntity in Game/Entity/Entity.h becomes clean, safe, and stack-free:

template <typename EntityType>
    requires EntityConcept<EntityType>
[[nodiscard]] EntityType* MakeEntity(EntityManager& manager, float x, float y, float z)
{
    Entity* base_ptr = AllocateEntity(manager, EntityType::kind);
    Assert(base_ptr != nullptr);

    base_ptr->ID         = EntityManager::ID++;
    base_ptr->position.x = x;
    base_ptr->position.y = y;
    base_ptr->position.z = z;
    base_ptr->position.w = 1.0f;

    return static_cast<EntityType*>(base_ptr->derived);
}

---

4. In-Place Deserialization Pipeline

4.1 Asset Format Specification (.jasset)

To achieve robust version control and modular streaming, Juliet adopts a human-readable, Git-diffable one-file-per-entity disk format with extension .jasset.

; asset_type
entity_instance
; id
0x0100000000000042
; class
Inert
; version
1
; class_version
1
; position
0.43 0.32 1.56
; mesh_instance
12

Important: No Header Structs for Derived Types

• Derived types NEVER require their own file header: You do not write an InertHeader, DoorHeader, or PlayerHeader. Derived types only serialize their own member variables.
• Universal ; version + optional ; class_version: Every .jasset file has a universal ; version tag. For entity assets, ; version governs base entity properties (kEntityBaseVersion), while an optional ; class_version governs derived class properties (Class::Version). Non-entity assets like WorldSettings.jasset only have ; version.
• No binary EntityFileHeader struct is needed: Under the ; variable_name\nvalues text format, there is no packed binary C-struct header at all. The common properties (; id, ; class, ; version, ; class_version, ; position) are standard text Key-Value nodes read by the exact same Archive parser.

4.2 Eliminating Intermediate Stack Allocations

Under the new pipeline:

1. The .jasset text file is read into memory onto a TempArena via LoadFile.
2. The property nodes are parsed into a ParsedArchive via tokenize_archive(tempArena.Arena, fileBuffer, &ar.base).
3. The engine reads ; class (e.g. "Inert") and resolves Class* class_ptr = find_class_by_name(class_name).
4. AllocateEntity(manager, class_ptr) is called immediately. Memory for both base and derived components is allocated in-place in their permanent engine arenas.
5. serialize(ar, NonNullPtr<Entity>(base_ptr)) is called. Base (Entity::kind) and derived (base_ptr->derived_kind) fields stream directly into their permanent memory arenas without temporary staging buffers or stack copies.

4.3 Runtime Class Resolution

To ensure fast and safe type lookup during file deserialization:

[[nodiscard]] Class* ResolveEntityClass(uint8 kind, uint32 crc)
{
    if (kind >= ENTITY(Count))
    {
        return nullptr;
    }

    Class* classPtr = kEntity_type_class_ptr[kind];
    if (!classPtr)
    {
        return nullptr;
    }

    if (classPtr->CRC != crc)
    {
        return nullptr;
    }

    return classPtr;
}

4.4 In-Place Deserialization Algorithm

+---------------------------------------------------------------------------------------+
|                       In-Place Deserialization Flowchart                              |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   1. LoadFile(scratch.Arena, filepath) into ByteBuffer                                |
|              |                                                                        |
|              v                                                                        |
|   2. tokenize_archive(scratch.Arena, file_buffer, &ar.base)                            |
|              |                                                                        |
|              v                                                                        |
|   3. Read "; class" & Resolve Class* via find_class_by_name(class_name)               |
|              |                                                                        |
|              v                                                                        |
|   4. base_ptr = AllocateEntity(manager, class_ptr)                                    |
|              |                                                                        |
|              +--> [manager.Entities]: Allocates base Entity                           |
|              +--> [manager.by_type[kind].arena]: Allocates derived struct             |
|              +--> Mutual Back-Pointers Wired In-Place                                 |
|              |                                                                        |
|              v                                                                        |
|   5. serialize(ar, NonNullPtr<Entity>(base_ptr))                                      |
|              |                                                                        |
|              +--> Streams Base Entity (Entity::kind, version, ID, position)           |
|              +--> Streams Derived Component (base_ptr->derived_kind, class_version)   |
|              |                                                                        |
|              v                                                                        |
|   6. Clear Dirty Flag: base_ptr->is_dirty = false                                     |
|                                                                                       |
+---------------------------------------------------------------------------------------+

[[nodiscard]] Entity* deserialize_entity_in_place(EntityManager& manager, Archive& ar)
{
    Assert(ar.loading);

    // 1. Read class name and resolve Class*
    String class_name = {};
    SERIALIZE(ar, class, class_name);
    Class* class_ptr = find_class_by_name(class_name);
    if (!class_ptr)
    {
        return nullptr;
    }

    // 2. Allocate persistent memory for base and derived components in their respective arenas
    Entity* base_ptr = AllocateEntity(manager, class_ptr);
    Assert(base_ptr != nullptr);

    // 3. Serialize Base Entity and Derived in-place (loads ID and properties from disk)
    serialize(ar, NonNullPtr<Entity>(base_ptr));

    // 4. Advance generator counter to avoid collisions with loaded IDs
    if (base_ptr->ID >= EntityManager::ID)
    {
        EntityManager::ID = base_ptr->ID + 1;
    }

    // Freshly loaded entity matches disk state exactly
    base_ptr->is_dirty = false;

    return base_ptr;
}

---

5. Entity Deletion Lifecycle & Disk Synchronization

5.1 In-Memory Removal vs. Immediate Disk Deletion Hazards

In game development, deleting an entity in the editor or during gameplay must never synchronously invoke disk deletion:

1. Frame Rate Stutters: Blocking on synchronous OS filesystem APIs (DeleteFileA) introduces multisecond frame freezes.
2. Transactional Safety: If the editor crashes or the user exits without saving, disk modifications cannot be rolled back.
3. Undo/Redo Support: An editor action stack must allow recovering deleted entities before changes are permanently committed to disk.

Therefore, Juliet enforces a strict separation:

• Immediate in-memory destruction: Releases the entity from active simulation and registers its identifier in World::PendingDeletions.
• Deferred disk deletion: Executed strictly during explicit SaveWorld operations.

5.2 Fast In-Memory Removal (RemoveAtFast) & Mutual Pointer Fixup

VectorArena::RemoveAtFast utilizes swap-and-pop: the element at the target index is replaced by the last element in the vector, and Count is decremented.

void RemoveAtFast(index_t index)
{
    Assert(Arena);
    Assert(index < Count);
    Assert(Count > 0);

    Type* elementAdr = DataFirst + index;

    if (DataLast != elementAdr)
    {
        Swap(DataLast, elementAdr);
    }

    --DataLast;
    --Count;
}

The Pointer Invalidation Problem:

When Entity A (at index) is swapped with Entity Z (at DataLast), the physical address of Entity Z changes from DataLast to elementAdr. If Entity Z has a derived struct derivedZ, derivedZ->Base previously pointed to DataLast. After RemoveAtFast, derivedZ->Base points to garbage or the freed slot!

The Pointer Fixup Protocol:

To preserve the mutual back-pointer invariant, DestroyEntity explicitly fixes up the swapped entity's derived back-pointer:

void DestroyEntity(EntityManager& manager, EntityID id)
{
    Entity* baseArray = manager.Entities.DataPtr();
    size_t  count     = manager.Entities.Size();

    size_t targetIndex = indexMax;
    for (size_t i = 0; i < count; ++i)
    {
        if (baseArray[i].ID == id)
        {
            targetIndex = i;
            break;
        }
    }

    if (targetIndex == indexMax)
    {
        return;
    }

    Entity* targetEntity = &baseArray[targetIndex];
    Class*  classPtr     = targetEntity->Kind;
    Assert(classPtr != nullptr);

    // 1. Remove derived component from typed array via swap-and-pop
    RemoveDerivedComponent(manager, classPtr, targetEntity->Derived);

    // 2. Remove base entity via swap-and-pop in VectorArena
    bool wasLast = (targetIndex == count - 1);
    manager.Entities.RemoveAtFast(targetIndex);

    // 3. Pointer fixup: If an element was swapped into targetIndex, fix its back-pointer!
    if (!wasLast && targetIndex < manager.Entities.Size())
    {
        Entity* movedEntity = &manager.Entities[targetIndex];
        auto*   derivedTemp = reinterpret_cast<entity_template*>(movedEntity->Derived);
        Assert(derivedTemp != nullptr);
        derivedTemp->Base = movedEntity;
    }
}

5.3 O(1) Component Removal in typed_entity_array via Swap-and-Pop

To keep derived components packed contiguously for SIMD/cache iteration:

1. Locate the component's index within by_type[kind].arena. Because components have uniform stride classPtr->size_of: \text{componentIndex} = \frac{\text{reinterpret\_cast<uint8*>(derivedPtr)} - \text{reinterpret\_cast<uint8*>(typedArray.array)}}{\text{classPtr->size\_of}}
2. If the component is not the last one in the typed arena:
   • Copy the last component into the slot occupied by the deleted component.
   • Update the moved component's Base->Derived pointer to point to its new slot.
3. Decrement typedArray.count.
4. Pop the arena allocation if it was the top of the stack, or decrement count to mark slot reclamation.

void RemoveDerivedComponent(EntityManager& manager, Class* classPtr, DerivedType derivedPtr)
{
    Assert(classPtr != nullptr);
    Assert(derivedPtr != nullptr);

    typed_entity_array& typedArray = manager.by_type[classPtr->kind];
    Assert(typedArray.count > 0);
    Assert(typedArray.array != nullptr);

    size_t stride     = classPtr->size_of;
    auto*  targetByte = reinterpret_cast<uint8*>(derivedPtr);
    auto*  firstByte  = reinterpret_cast<uint8*>(typedArray.array);

    size_t componentIndex = static_cast<size_t>(targetByte - firstByte) / stride;
    Assert(componentIndex < typedArray.count);

    size_t lastIndex = typedArray.count - 1;
    if (componentIndex != lastIndex)
    {
        uint8* lastByte = firstByte + (lastIndex * stride);
        
        // Copy last component data into target slot
        MemCopy(targetByte, lastByte, stride);

        // Fixup the base pointer of the moved component
        auto* movedDerived = reinterpret_cast<entity_template*>(targetByte);
        Assert(movedDerived->Base != nullptr);
        movedDerived->Base->Derived = targetByte;
    }

    typedArray.count -= 1;
    if (typedArray.count == 0)
    {
        typedArray.array = nullptr;
    }
}

5.4 Tracking Deletions in World::PendingDeletions

In World.h, the World struct is extended with a pending deletions container:

struct World
{
    Arena*                       WorldArena    = nullptr;
    EntityManager*               EntityManager = nullptr;
    VectorArena<EntityID, 1024>  PendingDeletions;
};

When an entity is deleted in the world:

void RemoveWorldEntity(World& world, EntityID id)
{
    Assert(world.EntityManager != nullptr);

    // Record pending disk deletion
    world.PendingDeletions.PushBack(id);

    // Destroy in memory immediately
    DestroyEntity(*world.EntityManager, id);
}

5.5 Disk File Cleanup during SaveWorld

During SaveWorld, before saving modified entities, the engine iterates over world.PendingDeletions and removes their associated .jasset files:

void ProcessPendingDeletions(World& world, NonNullPtr<Arena> scratchArena)
{
    for (size_t i = 0; i < world.PendingDeletions.Size(); ++i)
    {
        EntityID id = world.PendingDeletions[i];

        // Format relative asset path: Assets/Entities/{ID}.jasset
        char filenameBuffer[64];
        juliet_snprintf(filenameBuffer, sizeof(filenameBuffer), "Entities/%llu.jasset", id);
        
        String assetPath = GetAssetPath(scratchArena, WrapString(filenameBuffer));

        if (PlatformDeleteFile(assetPath))
        {
            Log(LogLevel::Message, LogCategory::Game, "Deleted entity asset: %s", CStr(assetPath));
        }
    }

    world.PendingDeletions.Clear();
}

---

6. Dirty Tracking for Optimal Saves

6.1 The Cost of Naive Monolithic & Full-Directory Writes

In a level containing 10{,}000 entities:

• Monolithic world.bin Save: Modifying a single entity's X coordinate requires re-serializing all 10{,}000 entities and overwriting a multi-megabyte binary file. This introduces a huge Git diff and constant merge conflicts.
• Full-Directory .jasset Save: Iterating through all 10{,}000 entities and unconditionally writing 10{,}000 .jasset files incurs massive OS file-system overhead (directory table locks, I/O bandwidth) and changes the file timestamps of every asset. Git reports thousands of modified files even when only one entity changed!

6.2 The is_dirty Flag on Entity

To solve this, Entity in Game/Entity/Entity.h is augmented with an explicit dirty flag:

struct Entity final
{
    DECLARE_CLASS() // static Class* kind; (Entity's own Class descriptor)

    EntityID    ID           = 0;
    Class*      derived_kind = nullptr; // Pointer to derived class descriptor (e.g. Inert::kind)
    DerivedType derived      = nullptr; // Pointer to derived component memory
    Vector4     position     = {};
    bool        is_dirty     = false;
};

6.3 Granular State Transitions

The is_dirty flag obeys a strict lifecycle state machine:

                  +-----------------------------------+
                  |          Entity Created           |
                  |     (AllocateEntity / Editor)     |
                  +-----------------+-----------------+
                                    |
                                    v
                            +---------------+
                   +------->|is_dirty: TRUE |<-------+
                   |        +-------+-------+        |
                   |                |                |
        Entity Mutated              |            SaveWorld
    (Position, Component)           |            Completed
                   |                v                |
                   |        +---------------+        |
                   +--------+is_dirty: FALSE+--------+
                            +-------+-------+
                                    ^
                                    |
                             Deserialization
                            (LoadWorld / Asset)

1. Entity Creation: Newly spawned entities in the editor have is_dirty = true.
2. Property Mutation: Any modification to position or derived component payload sets entity->is_dirty = true.
3. Successful Deserialization: Entities loaded from disk initialize with is_dirty = false.
4. Successful Save: Upon successfully writing an entity to its .jasset file, the engine resets entity->is_dirty = false.

6.4 Version Control Benefits (Git Friendly Assets)

By coupling the one-file-per-entity .jasset format with dirty tracking:

• Only modified entities are touched on disk.
• Git status displays only the exact .jasset files that were altered by the designer.
• Team members can work concurrently in the same game scene without encountering binary merge conflicts.

6.5 Editor Integration (RenderWorldEditorUI Hooks)

In Game/Data/World.cpp, editor UI widgets automatically set the dirty flag upon receiving user input:

float pos[4] = { ent.position.x, ent.position.y, ent.position.z, ent.position.w };
if (ImGui::DragFloat3("Position", pos, 0.1f))
{
    ent.position.x = pos[0];
    ent.position.y = pos[1];
    ent.position.z = pos[2];
    ent.is_dirty   = true; // Mark dirty for persistence
    UpdateWorld(world);
}

---

7. Step-by-Step Implementation Roadmap

Phase 1: Data Structures & Header Definitions

1. Update Class.h & Entity.h:
   • Add using default_init_fct_type = void (*)(void* payload); and default_init_fct to struct Class and MakeClass.
   • Update DEFINE_ENTITY_VERSIONED and DEFINE_CLASS_VERSIONED to define default_init_##entity and pass it to MakeClass.
   • Add bool is_dirty = false; to struct Entity.
   • Update MakeEntity<EntityType> to assign base_ptr->ID = EntityManager::ID++; and delegate allocation and defaults cleanly to AllocateEntity.
2. Update EntityManager.h:
   • Declare [[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* class_ptr);.
   • Declare void DestroyEntity(EntityManager& manager, EntityID id);.
   • Declare void RemoveDerivedComponent(EntityManager& manager, Class* class_ptr, DerivedType derived_ptr);.
3. Update World.h:
   • Add VectorArena<EntityID, 1024> PendingDeletions; to struct World.
   • Update SaveWorld and LoadWorld signatures to take directory paths.

Phase 2: Core Memory Allocation & Wiring in EntityManager.cpp

1. Implement AllocateEntity:
   • Enforce parameter assertions.
   • Push to manager.Entities with baseTemplate.ID = 0 (unassigned).
   • Allocate block in manager.by_type[kind].arena.
   • Call derivedClassPtr->default_init_fct(rawMemory) (or MemZero if null) to initialize struct defaults.
   • Wire mutual pointers (base->derived and derived->base).
   • Increment typedArray.count and initialize typedArray.array.
2. Implement DestroyEntity & RemoveDerivedComponent:
   • Implement swap-and-pop for derived components with base pointer update.
   • Implement manager.Entities.RemoveAtFast with mutual back-pointer fixup.

Phase 3: In-Place Deserialization & Serialization Pipeline

1. In Entity.cpp:
   • serialize(Archive& ar, NonNullPtr<Entity> entity) handles both base and derived class serialization.
2. In World.cpp:
   • Implement serialize_entity_asset(Archive& ar, NonNullPtr<Entity> entity, String filepath).
   • Implement deserialize_entity_asset(EntityManager& manager, Archive& ar, String filepath).
   • In deserialize_entity_in_place, advance EntityManager::ID past base_ptr->ID to prevent ID collisions.

Phase 4: World Save/Load Pipeline & Disk Deletion

1. In World.cpp:
   • Implement ProcessPendingDeletions(World& world, NonNullPtr<Arena> scratchArena).
   • Implement SaveWorld(World& world, String worldDirectory):
     • Process pending deletions.
     • Iterate manager.Entities, skipping entities where !entity.is_dirty.
     • Write dirty entities to .jasset files and clear is_dirty.
   • Implement LoadWorld(World& world, String worldDirectory):
     • Enumerate .jasset files in directory.
     • Call deserialize_entity_asset for each file.

Phase 5: Editor Integration

1. In RenderWorldEditorUI:
   • Hook ImGui::DragFloat3 and property inspectors to set is_dirty = true.
   • Hook "Add Entity" button to call MakeEntity<Inert>(*world.EntityManager, 0.0f, 0.0f, 0.0f).
   • Hook "Delete Entity" button to call RemoveWorldEntity(world, selectedEntityId).

---

8. Unit Testing & Verification Plan

8.1 Test Philosophy & Constraints

Following Juliet coding guidelines:

"When creating a new system framework, make a unit test. To make the unit test we should not modify the framework code for special unit test case."

Testing is isolated in Game/UnitTest/WorldUnitTest.cpp and executed during engine initialization in debug builds.

8.2 Comprehensive Test Suite (WorldUnitTest.cpp)

The test suite validates every guarantee made in this specification:

#include <UnitTest/WorldUnitTest.h>

#if JULIET_DEBUG

#include <Core/Common/CoreUtils.h>
#include <Core/Common/serialization.h>
#include <Core/HAL/Filesystem/Filesystem.h>
#include <Core/HAL/IO/IOStream.h>
#include <Core/Logging/LogManager.h>
#include <Core/Logging/LogTypes.h>
#include <Core/Memory/MemoryArena.h>
#include <Core/Thread/ThreadContext.h>
#include <Data/World.h>
#include <Entity/Entity.h>
#include <Entity/EntityManager.h>

namespace UnitTest
{
    namespace
    {
        void TestEntityAllocationAndWiring()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestEntityAllocationAndWiring...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);

            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // 1. Allocate Inert Entity via AllocateEntity
            Entity* base_entity = AllocateEntity(manager, Inert::kind);
            Assert(base_entity != nullptr);
            Assert(base_entity->ID == 0); // Pure memory allocator leaves ID unassigned (0) until MakeEntity or deserialization
            base_entity->ID = EntityManager::ID++;
            Assert(base_entity->ID > 0);
            Assert(base_entity->derived_kind == Inert::kind);
            Assert(base_entity->derived != nullptr);
            Assert(base_entity->is_dirty == true);

            // 2. Validate mutual back-pointer wiring
            auto* derived = reinterpret_cast<entity_template*>(base_entity->derived);
            Assert(derived->base == base_entity);

            // 3. DownCast verification
            Inert* inert = DownCast<Inert>(base_entity);
            Assert(inert != nullptr);
            Assert(inert->base == base_entity);

            // 4. Validate typed array tracking
            typed_entity_array& inert_array = manager.by_type[ENTITY(Inert)];
            Assert(inert_array.count == 1);
            Assert(inert_array.array == derived);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestEntityAllocationAndWiring");
        }

        void TestInPlaceDeserialization()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestInPlaceDeserialization...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // 1. Create and populate entity
            Inert* created_inert = MakeEntity<Inert>(manager, 12.5f, -44.0f, 108.2f);
            Assert(created_inert != nullptr);
            created_inert->MeshInstance = 42;

            Entity*  original_base = created_inert->base;
            EntityID original_id   = original_base->ID;

            // 2. Serialize to text archive memory stream
            MemoryStream mem_stream = MakeMemoryStream(tempArena.Arena);
            Archive      save_ar{ .arena = tempArena.Arena, .loading = false, .stream = &mem_stream };
            serialize(save_ar, NonNullPtr<Entity>(original_base));

            // 3. Clear manager to simulate fresh load
            ShutdownEntityManager();
            InitEntityManager(&testWorld);
            EntityManager& fresh_manager = *testWorld.EntityManager;

            // 4. Tokenize and deserialize in-place
            Archive load_ar{ .arena = tempArena.Arena, .loading = true };
            tokenize_archive(tempArena.Arena, mem_stream.buffer, &load_ar.base);
            Entity* loaded_base = deserialize_entity_in_place(fresh_manager, load_ar);

            Assert(loaded_base != nullptr);
            Assert(loaded_base->ID == original_id);
            Assert(EntityManager::ID > original_id); // Counter was advanced past loaded ID to prevent collisions
            Assert(loaded_base->position.x == 12.5f);
            Assert(loaded_base->position.y == -44.0f);
            Assert(loaded_base->position.z == 108.2f);
            Assert(loaded_base->is_dirty == false);

            Inert* loaded_inert = DownCast<Inert>(loaded_base);
            Assert(loaded_inert != nullptr);
            Assert(loaded_inert->base == loaded_base);
            Assert(loaded_inert->MeshInstance == 42);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestInPlaceDeserialization");
        }

        void TestSwapAndPopPointerFixup()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestSwapAndPopPointerFixup...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // Allocate 3 entities: E0, E1, E2
            Inert* e0 = MakeEntity<Inert>(manager, 1.0f, 0.0f, 0.0f);
            Inert* e1 = MakeEntity<Inert>(manager, 2.0f, 0.0f, 0.0f);
            Inert* e2 = MakeEntity<Inert>(manager, 3.0f, 0.0f, 0.0f);

            EntityID id0 = e0->base->ID;
            EntityID id1 = e1->base->ID;
            EntityID id2 = e2->base->ID;

            Assert(manager.Entities.Size() == 3);

            // Delete middle entity E1 (triggers swap with E2)
            DestroyEntity(manager, id1);

            Assert(manager.Entities.Size() == 2);

            // Verify E2's mutual back-pointers are still completely intact
            Entity* remaining0 = &manager.Entities[0];
            Entity* remaining1 = &manager.Entities[1];

            Assert(remaining0->ID == id0);
            Assert(remaining1->ID == id2);

            auto* derived0 = reinterpret_cast<entity_template*>(remaining0->derived);
            auto* derived1 = reinterpret_cast<entity_template*>(remaining1->derived);

            Assert(derived0->base == remaining0);
            Assert(derived1->base == remaining1);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestSwapAndPopPointerFixup");
        }

        void TestDirtyTrackingLifecycle()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestDirtyTrackingLifecycle...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            Entity* entity = AllocateEntity(manager, Inert::kind);
            Assert(entity->is_dirty == true);

            // Simulate save
            entity->is_dirty = false;
            Assert(entity->is_dirty == false);

            // Simulate mutation
            entity->position.x += 1.0f;
            entity->is_dirty = true;
            Assert(entity->is_dirty == true);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestDirtyTrackingLifecycle");
        }
    } // namespace

    void WorldUnitTest()
    {
        Log(LogLevel::Message, LogCategory::Game, "==================================================");
        Log(LogLevel::Message, LogCategory::Game, "Starting Entity Allocation & Lifecycle Unit Tests");
        Log(LogLevel::Message, LogCategory::Game, "==================================================");

        TestEntityAllocationAndWiring();
        TestInPlaceDeserialization();
        TestSwapAndPopPointerFixup();
        TestDirtyTrackingLifecycle();

        Log(LogLevel::Message, LogCategory::Game, "==================================================");
        Log(LogLevel::Message, LogCategory::Game, "All Entity Lifecycle Unit Tests PASSED Successfully");
        Log(LogLevel::Message, LogCategory::Game, "==================================================");
    }
} // namespace UnitTest

#endif