# Juliet Game Engine Architecture Specification
## Document: 02 - Entity Allocation & In-Place Lifecycle

**Document ID:** JULIET-SPEC-002  
**Status:** Approved for Implementation  
**Author:** Senior Engine Architect  
**Subsystems:** `Game/Entity`, `Game/Data`, `Game/UnitTest`  
**Target Files:**
- [`Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h)
- [`Entity.cpp`](file:///w:/Classified/Juliet/Game/Entity/Entity.cpp)
- [`EntityManager.h`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.h)
- [`EntityManager.cpp`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.cpp)
- [`World.h`](file:///w:/Classified/Juliet/Game/Data/World.h)
- [`World.cpp`](file:///w:/Classified/Juliet/Game/Data/World.cpp)
- [`WorldUnitTest.h`](file:///w:/Classified/Juliet/Game/UnitTest/WorldUnitTest.h)
- [`WorldUnitTest.cpp`](file:///w:/Classified/Juliet/Game/UnitTest/WorldUnitTest.cpp)

---

## 1. Executive Summary & Problem Statement

### 1.1 Background & Context
The Juliet game engine organizes game entities using a hybrid data-oriented architecture:
1. A flat array of **Base Entities** (`Entity`) encapsulating universal properties: unique 64-bit ID, runtime type reflection pointer (`Class* Kind`), spatial coordinates (`X, Y, Z`), and an opaque pointer to the derived payload (`DerivedType Derived`).
2. Type-segregated contiguous arrays of **Derived Entities** (`Inert`, etc.) stored in dedicated per-type memory arenas (`typed_entity_array`).

This design is intended to provide maximum cache efficiency during spatial and general-purpose entity processing, while retaining dense SIMD-friendly streaming for type-specific systems (e.g., transform updates on `Inert` mesh instances).

### 1.2 Root-Cause Analysis of `Game/Data/World.cpp` Loading Flaws
In the initial implementation of entity serialization in [`Game/Data/World.cpp`](file:///w:/Classified/Juliet/Game/Data/World.cpp#L40-L68), loading was fundamentally broken and incomplete:

```cpp
// Existing flawed deserialization in World.cpp
for (typed_entity_array& type : entityManager.by_type)
{
    serialize_elem(ar, type.count);

    if (type.count > 0)
    {
        // Unserialize the base entity to get informations
        Entity entity;
        serialize(ar, &entity);

        RegisterBaseEntity(entityManager, entity);
    }
}
```

This implementation suffers from several fatal defects:
1. **Single-Element Iteration Bug:** It uses `if (type.count > 0)` instead of a loop `for (size_t i = 0; i < type.count; ++i)`, deserializing at most one single entity per type bucket, leaving all subsequent entities in the stream unread and corrupting the archive read offset.
2. **Missing Derived Allocation:** It invokes `RegisterBaseEntity(entityManager, entity)`, which merely pushes the stack-allocated `Entity` into `manager.Entities`. The derived payload arena (`type.arena`) is completely untouched: `type.array` remains null, `type.count` in the manager is desynchronized, and `entity.Derived` remains unassigned or points to an invalid address.
3. **Invalid Pointer in `RegisterEntity`:** In [`Game/Entity/EntityManager.cpp`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.cpp#L46-L66), `RegisterEntity` assigns `base->Derived = entity` before pushing `*base` into `manager.Entities`. The parameter `entity` is a pointer to caller-provided memory (often stack-allocated in helper functions like `MakeEntity`). When `ArenaPushSize` later allocates the persistent derived memory block, `base->Derived` stored inside `manager.Entities.Back()` is **never updated**—it remains dangling, pointing to the transient caller stack!

### 1.3 The "Chicken-and-Egg" Stack Allocation Dilemma
The existing registration API requires a pre-existing derived instance:
```cpp
entity_template* RegisterEntity(EntityManager& manager, Entity* base, DerivedType entity);
```
During runtime programmatic creation via `MakeEntity<T>()`, a temporary instance of `T` is created on the stack and passed by pointer:
```cpp
template <typename EntityType>
EntityType* MakeEntity(EntityManager& manager, float x, float y, float z)
{
    EntityType result; // Stack allocation
    Entity     base;   // Stack allocation
    base.X    = x;
    base.Y    = y;
    base.Z    = z;
    base.Kind = EntityType::Kind;

    return (EntityType*)RegisterEntity(manager, &base, &result);
}
```
When loading an entity from a stream or disk file, **the type is not known at compile time**. The engine reads a runtime type tag (`uint8 kind` or `uint32 CRC`), looks up the reflection metadata (`Class*`), and must instantiate the entity dynamically. 

Because C++ does not permit allocating a dynamic struct of unknown type on the stack, and because Juliet strictly forbids heap allocations (`malloc`, `new`, `std::vector`), deserialization cannot construct a temporary instance on the stack to pass to `RegisterEntity`. 

This is the classic **chicken-and-egg memory problem**:
- `RegisterEntity` requires an existing instance in memory to copy from.
- Deserialization requires an allocated memory buffer to deserialize into.

### 1.4 Architectural Objectives
This specification establishes a robust in-place lifecycle pipeline that completely eliminates stack temporaries and dynamic heap allocations:
1. **Direct In-Place Allocation:** Introduce `AllocateEntity(EntityManager& manager, Class* classPtr)` which allocates both the base `Entity` and the derived struct directly within their respective engine memory arenas.
2. **Bidirectional Pointer Integrity:** Wire mutual pointers (`base->Derived` and `derived->Base`) at allocation time before any field deserialization begins.
3. **In-Place Stream Deserialization:** Read class reflection metadata first, invoke `AllocateEntity`, and stream base and derived properties directly into arena-resident memory.
4. **Isolated Entity Assets (`.jasset`):** Transition from a monolithic `world.bin` to a modular one-file-per-entity architecture (`Assets/Entities/{ID}.jasset`).
5. **Dirty Tracking & Optimal Saves:** Introduce an `IsDirty` flag on `Entity` to avoid rewriting unchanged entity files, minimizing disk I/O and eliminating spurious Git repository modifications.
6. **Robust Deletion Lifecycle:** Decouple in-memory removal (`RemoveAtFast` with pointer fixup) from disk synchronization using `World::PendingDeletions`.

---

## 2. The Dual Arena Memory Model in `EntityManager`

### 2.1 The Need for Dual Storage
Game engines execute systems with vastly different cache locality profiles:
- **Spatial / Frustum Culling / Transform Sync:** Iterates every entity in the world, needing only `X, Y, Z`, bounding spheres, and base status flags.
- **Specialized Logic / Render Updates:** Iterates only entities possessing specific components (e.g., `Inert` static meshes requiring instance transform updates to the GPU bindless descriptor table).

Storing large monolithic polymorphic structs in a single array causes severe cache line pollution during spatial passes. Conversely, storing entities in fragmented individual allocations introduces cache misses and pointer-chasing overhead. 

Juliet resolves this with a **Dual Arena Memory Model**:

```
+---------------------------------------------------------------------------------------------+
|                                    EntityManager                                            |
+---------------------------------------------------------------------------------------------+
|                                                                                             |
|   manager.Entities (VectorArena<Entity, 100'000>)                                          |
|   +------------------------------------+------------------------------------+           |
|   | Entity 0 (ID=1001, X, Y, Z)        | Entity 1 (ID=1002, X, Y, Z)        | ...       |
|   | Derived ------------------------+  | Derived ---------------------+     |           |
|   +---------------------------------|--+------------------------------|-----+           |
|                                     |                                 |                     |
|                                     v                                 v                     |
|   manager.by_type[ENTITY(Inert)].arena                                                      |
|   +------------------------------------+------------------------------------+           |
|   | Inert 0 (MeshInstance=4)           | Inert 1 (MeshInstance=12)          | ...       |
|   | Base ---------------------------+  | Base ------------------------+     |           |
|   +---------------------------------|--+------------------------------|-----+           |
|                                     +---------------------------------+                     |
+---------------------------------------------------------------------------------------------+
```

### 2.2 `manager.Entities`: Cache-Friendly Base Entity Vector
Base entities reside in a pre-reserved contiguous array:
```cpp
VectorArena<Entity, 100'000> Entities;
```
- **Capacity:** Fixed reserve of 100,000 entities allocated from the `WorldArena`.
- **Memory Footprint:** 
  $$\text{sizeof(Entity)} = 8\text{ (ID)} + 8\text{ (Kind)} + 8\text{ (Derived)} + 12\text{ (X, Y, Z)} + 1\text{ (IsDirty)} + 3\text{ (Padding)} = 40\text{ bytes}$$
  Total reserved space: $100{,}000 \times 40\text{ bytes} \approx 4.0\text{ MB}$.
- **Access Speed:** O(1) random access by index; sequential streaming utilizes L1/L2 hardware prefetchers with zero cache line waste.

### 2.3 `manager.by_type[kind].arena`: Typed Component Arenas
Derived structs reside in per-type contiguous memory arenas:
```cpp
struct typed_entity_array
{
    Arena*           arena;
    entity_template* array;
    size_t           count;
};
```
- Each `Entity_Type` index owns an isolated `Arena*` allocated during `InitEntityManager`.
- Allocations are packed linearly with alignment specified by `classPtr->alignment`.
- `array` points directly to the first element in the arena, permitting typed array indexing:
  ```cpp
  Inert* inertArray = reinterpret_cast<Inert*>(manager.by_type[ENTITY(Inert)].array);
  ```

### 2.4 Mutual Back-Pointer Architecture & Invariants
Every entity instance consists of two mutually linked allocations:
1. `basePtr->Derived`: Points from `Entity` in `manager.Entities` to the derived struct in `by_type[kind].arena`.
2. `derivedPtr->Base`: Points from the derived struct (via `DECLARE_ENTITY()`) back to `Entity` in `manager.Entities`.

#### Invariant Rules:
1. **Non-Null Invariant:** For any active entity, `basePtr->Derived != nullptr` and `reinterpret_cast<entity_template*>(basePtr->Derived)->Base == basePtr`.
2. **Type Coherence Invariant:** `basePtr->Kind->kind == derivedTypeId`.
3. **Array Count Invariant:** 
   $$\sum_{k=0}^{\text{ENTITY(Count)}-1} \text{manager.by\_type}[k].\text{count} == \text{manager.Entities.Size()}$$

### 2.5 Pointer Stability in `VectorArena`
`VectorArena::Create` executes `Reserve(ReserveSize)` on creation:
```cpp
newManager->Entities.Create(world->WorldArena JULIET_DEBUG_PARAM("Entities"));
```
Because capacity ($100{,}000$) is fully reserved upfront in virtual address space, `VectorArena::PushBack` **never reallocates or moves existing memory**. Therefore:
- Pointers to `Entity` elements in `manager.Entities` remain absolutely stable across allocations.
- Derived struct `Base` pointers remain valid indefinitely unless an element is deleted.
- Element removal via swap-and-pop alters memory positions, requiring systematic pointer fixups (addressed in Section 5.2).

---

## 3. The Solution: `AllocateEntity(EntityManager& manager, Class* classPtr)`

### 3.1 Function Signature & Contract
The canonical allocation function is defined in `EntityManager.h`:

```cpp
[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr);
```

#### Preconditions:
- `classPtr != nullptr`.
- `classPtr->kind < ENTITY(Count)`.
- `classPtr->size_of >= sizeof(entity_template)`.
- `manager.Entities.Size() < manager.Entities.Capacity`.
- `manager.by_type[classPtr->kind].arena != nullptr`.

#### Postconditions:
- A new `Entity` record is appended to `manager.Entities`.
- A new typed block of `classPtr->size_of` bytes is allocated in `manager.by_type[classPtr->kind].arena`.
- The derived memory is zeroed.
- `base->Derived` points to the derived struct.
- `derived->Base` points to the base `Entity`.
- `base->Kind` is assigned to `classPtr`.
- `base->ID` is assigned the next unique `EntityManager::ID`.
- `base->IsDirty` is initialized to `true`.
- `typed_entity_array::count` is incremented.
- `typed_entity_array::array` is initialized if this is the first entity of this type.

### 3.2 Detailed Step-by-Step Implementation
The implementation replaces the flawed `RegisterEntity` routine in [`Game/Entity/EntityManager.cpp`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.cpp):

```cpp
[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr)
{
    Assert(classPtr != nullptr);
    Assert(classPtr->kind < ENTITY(Count));
    Assert(classPtr->size_of >= sizeof(entity_template));
    Assert(classPtr->alignment > 0);

    // 1. Allocate uninitialized Base Entity in the contiguous VectorArena
    Entity baseTemplate{};
    baseTemplate.ID      = EntityManager::ID++;
    baseTemplate.Kind    = classPtr;
    baseTemplate.Derived = nullptr;
    baseTemplate.X       = 0.0f;
    baseTemplate.Y       = 0.0f;
    baseTemplate.Z       = 0.0f;
    baseTemplate.IsDirty = true;

    manager.Entities.PushBack(baseTemplate);
    Entity* basePtr = manager.Entities.Back();
    Assert(basePtr != nullptr);

    // 2. Allocate zeroed derived component memory in the typed arena
    typed_entity_array& typedArray = manager.by_type[classPtr->kind];
    Assert(typedArray.arena != nullptr);

    void* rawMemory = ArenaPushSize(
        typedArray.arena, 
        classPtr->size_of, 
        classPtr->alignment, 
        true JULIET_DEBUG_PARAM(kEntity_type_names[classPtr->kind]));
    Assert(rawMemory != nullptr);

    auto* derivedTemplate = reinterpret_cast<entity_template*>(rawMemory);

    // 3. Establish mutual back-pointers
    basePtr->Derived      = rawMemory;
    derivedTemplate->Base = basePtr;

    // 4. Update typed array tracking
    if (typedArray.array == nullptr)
    {
        typedArray.array = derivedTemplate;
    }
    typedArray.count += 1;

    return basePtr;
}
```

### 3.3 Pure C-Style Zeroing & Initialization
`ArenaPushSize` is called with `shouldBeZeroed = true` (or followed by `MemZero`). This guarantees that:
- All bytes are zeroed (`0x00`).
- Pointers inside derived structs default to `nullptr`.
- Numerical fields default to `0`.

For structs that require specific non-zero sentinel values (such as `Inert::MeshInstance = indexMax`), Juliet follows a pure C approach without C++ constructor or placement-new machinery:
1. **Direct assignment or C-style init function**:
```cpp
inline void InitInert(Inert* inert)
{
    Assert(inert != nullptr);
    inert->MeshInstance = indexMax;
}
```
2. **Or simple struct literal assignment**:
```cpp
*inert = Inert{ .MeshInstance = indexMax };
```
No C++ placement `new`, `<new>` headers, or hidden constructor/destructor calls are ever used. Data structures remain pure C-style aggregates.

### 3.4 Bidirectional Pointer Wiring
Notice the sequence:
1. `manager.Entities.PushBack(baseTemplate)` places the struct at its final, fixed arena address.
2. `basePtr = manager.Entities.Back()` retrieves the persistent memory pointer.
3. `derivedTemplate->Base = basePtr` wires the derived back-pointer directly to this permanent location.
4. `basePtr->Derived = rawMemory` wires the base forward-pointer to the arena-allocated derived struct.

No stack copying occurs. Neither pointer is ever left dangling.

### 3.5 Updating Type Counts and Array Cache
`typed_entity_array` maintains:
- `typedArray.count`: The exact count of active entities of this type.
- `typedArray.array`: Pointer to the first element in the arena.

When the first entity of a given type is allocated, `typedArray.array` is set to `derivedTemplate`. Because the arena allocates sequentially, `typedArray.array[i]` can be indexed directly with stride `classPtr->size_of` as long as memory remains contiguous.

### 3.6 Refactoring `MakeEntity<EntityType>` Template
With `AllocateEntity` operational, `MakeEntity` in [`Game/Entity/Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h#L95-L107) becomes clean, safe, and stack-free:

```cpp
template <typename EntityType>
    requires EntityConcept<EntityType>
[[nodiscard]] EntityType* MakeEntity(EntityManager& manager, float x, float y, float z)
{
    Entity* basePtr = AllocateEntity(manager, EntityType::Kind);
    Assert(basePtr != nullptr);

    basePtr->X = x;
    basePtr->Y = y;
    basePtr->Z = z;

    auto* derivedPtr = static_cast<EntityType*>(basePtr->Derived);
    ConstructDerivedDefaults<EntityType>(derivedPtr);
    
    // Re-establish Base pointer after placement-new
    derivedPtr->Base = basePtr;

    return derivedPtr;
}
```

---

## 4. In-Place Deserialization Pipeline

### 4.1 Asset Format Specification (`.jasset`)
To achieve robust version control and modular streaming, Juliet adopts a human-readable, Git-diffable **one-file-per-entity** disk format with extension `.jasset`.

```ini
; asset_type
entity_instance
; id
0x0100000000000042
; class
Inert
; base_version
1
; derived_version
1
; position
0.43 0.32 1.56
; mesh_instance
12
```

#### Important: No Header Structs for Derived Types
- **Derived types NEVER require their own file header**: You do **not** write an `InertHeader`, `DoorHeader`, or `PlayerHeader`. Derived types only serialize their own member variables.
- **No binary `EntityFileHeader` struct is needed**: Under the `; variable_name\nvalues` text format, there is no packed binary C-struct header at all. The common properties (`; id`, `; class`, `; base_version`, `; derived_version`, `; position`) are standard text Key-Value nodes read by the exact same `archive` parser.

### 4.2 Eliminating Intermediate Stack Allocations
Under the new pipeline:
1. The `.jasset` text file is read into memory onto a `TempArena` via `LoadFile`.
2. The lines are tokenized into a `TextPropertyNode` scratch table.
3. The engine reads `; class` (e.g. `"Inert"`) and resolves `Class* classPtr = FindClassByName(className)`.
4. `AllocateEntity(manager, classPtr)` is called immediately. Memory for both base and derived components is allocated in-place in their permanent engine arenas.
5. Base properties (`id`, `position`, etc.) are read directly into `*basePtr` via `SerializeEntityBase`.
6. `classPtr->serialize_fct(ar, basePtr->Derived, derivedVersion)` is called. Derived fields stream **directly into the typed arena** without temporary staging buffers or stack copies.

### 4.3 Runtime Class Resolution
To ensure fast and safe type lookup during file deserialization:

```cpp
[[nodiscard]] Class* ResolveEntityClass(uint8 kind, uint32 crc)
{
    if (kind >= ENTITY(Count))
    {
        return nullptr;
    }

    Class* classPtr = kEntity_type_class_ptr[kind];
    if (!classPtr)
    {
        return nullptr;
    }

    if (classPtr->CRC != crc)
    {
        return nullptr;
    }

    return classPtr;
}
```

### 4.4 In-Place Deserialization Algorithm

```
+---------------------------------------------------------------------------------------+
|                       In-Place Deserialization Flowchart                              |
+---------------------------------------------------------------------------------------+
|                                                                                       |
|   1. Read Header from IOStream/ByteBuffer                                             |
|              |                                                                        |
|              v                                                                        |
|   2. Validate Magic ('JAST') and Version (1)                                          |
|              |                                                                        |
|              v                                                                        |
|   3. Resolve Class* from header.Kind & header.ClassCRC                                |
|              |                                                                        |
|              v                                                                        |
|   4. basePtr = AllocateEntity(manager, classPtr)                                      |
|              |                                                                        |
|              +--> [manager.Entities]: Allocates base Entity                           |
|              +--> [manager.by_type[kind].arena]: Allocates derived struct             |
|              +--> Mutual Back-Pointers Wired In-Place                                 |
|              |                                                                        |
|              v                                                                        |
|   5. Direct Copy Base Properties (ID, X, Y, Z)                                        |
|              |                                                                        |
|              v                                                                        |
|   6. Does classPtr->serialize_fct exist?                                              |
|         |                     |                                                       |
|        Yes                    No                                                      |
|         |                     |                                                       |
|         v                     |                                                       |
|      Invoke:                  |                                                       |
|      serialize_fct(&ar,       |                                                       |
|                    Derived)   |                                                       |
|         |                     |                                                       |
|         +---------------------+                                                       |
|              |                                                                        |
|              v                                                                        |
|   7. Clear Dirty Flag: basePtr->IsDirty = false                                       |
|                                                                                       |
+---------------------------------------------------------------------------------------+
```

```cpp
[[nodiscard]] Entity* DeserializeEntityInPlace(EntityManager& manager, archive& ar)
{
    Assert(ar.loading);

    // 1. Read class name and resolve Class*
    String className = {};
    SerializeProp(ar, "class", className, ar.arena);
    Class* classPtr = FindClassByName(className);
    if (!classPtr)
    {
        return nullptr;
    }

    // 2. Allocate persistent memory for base and derived components in their respective arenas
    Entity* basePtr = AllocateEntity(manager, classPtr);
    Assert(basePtr != nullptr);

    // 3. Read base versions and properties in-place
    uint16 baseVersion    = 1;
    uint16 derivedVersion = 1;
    SerializeProp(ar, "base_version", baseVersion);
    SerializeProp(ar, "derived_version", derivedVersion);

    SerializeEntityBase(ar, *basePtr, baseVersion);

    // 4. Stream derived properties in-place directly into the typed arena
    if (classPtr->serialize_fct != nullptr)
    {
        classPtr->serialize_fct(ar, basePtr->Derived, derivedVersion);
    }

    // Freshly loaded entity matches disk state exactly
    basePtr->IsDirty = false;

    return basePtr;
}
```

---

## 5. Entity Deletion Lifecycle & Disk Synchronization

### 5.1 In-Memory Removal vs. Immediate Disk Deletion Hazards
In game development, deleting an entity in the editor or during gameplay must **never synchronously invoke disk deletion**:
1. **Frame Rate Stutters:** Blocking on synchronous OS filesystem APIs (`DeleteFileA`) introduces multisecond frame freezes.
2. **Transactional Safety:** If the editor crashes or the user exits without saving, disk modifications cannot be rolled back.
3. **Undo/Redo Support:** An editor action stack must allow recovering deleted entities before changes are permanently committed to disk.

Therefore, Juliet enforces a strict separation:
- **Immediate in-memory destruction:** Releases the entity from active simulation and registers its identifier in `World::PendingDeletions`.
- **Deferred disk deletion:** Executed strictly during explicit `SaveWorld` operations.

### 5.2 Fast In-Memory Removal (`RemoveAtFast`) & Mutual Pointer Fixup
`VectorArena::RemoveAtFast` utilizes swap-and-pop: the element at the target index is replaced by the last element in the vector, and `Count` is decremented.

```cpp
void RemoveAtFast(index_t index)
{
    Assert(Arena);
    Assert(index < Count);
    Assert(Count > 0);

    Type* elementAdr = DataFirst + index;

    if (DataLast != elementAdr)
    {
        Swap(DataLast, elementAdr);
    }

    --DataLast;
    --Count;
}
```

#### The Pointer Invalidation Problem:
When `Entity A` (at `index`) is swapped with `Entity Z` (at `DataLast`), the physical address of `Entity Z` changes from `DataLast` to `elementAdr`.
If `Entity Z` has a derived struct `derivedZ`, `derivedZ->Base` previously pointed to `DataLast`. After `RemoveAtFast`, `derivedZ->Base` points to garbage or the freed slot!

#### The Pointer Fixup Protocol:
To preserve the mutual back-pointer invariant, `DestroyEntity` explicitly fixes up the swapped entity's derived back-pointer:

```cpp
void DestroyEntity(EntityManager& manager, EntityID id)
{
    Entity* baseArray = manager.Entities.DataPtr();
    size_t  count     = manager.Entities.Size();

    size_t targetIndex = indexMax;
    for (size_t i = 0; i < count; ++i)
    {
        if (baseArray[i].ID == id)
        {
            targetIndex = i;
            break;
        }
    }

    if (targetIndex == indexMax)
    {
        return;
    }

    Entity* targetEntity = &baseArray[targetIndex];
    Class*  classPtr     = targetEntity->Kind;
    Assert(classPtr != nullptr);

    // 1. Remove derived component from typed array via swap-and-pop
    RemoveDerivedComponent(manager, classPtr, targetEntity->Derived);

    // 2. Remove base entity via swap-and-pop in VectorArena
    bool wasLast = (targetIndex == count - 1);
    manager.Entities.RemoveAtFast(targetIndex);

    // 3. Pointer fixup: If an element was swapped into targetIndex, fix its back-pointer!
    if (!wasLast && targetIndex < manager.Entities.Size())
    {
        Entity* movedEntity = &manager.Entities[targetIndex];
        auto*   derivedTemp = reinterpret_cast<entity_template*>(movedEntity->Derived);
        Assert(derivedTemp != nullptr);
        derivedTemp->Base = movedEntity;
    }
}
```

### 5.3 O(1) Component Removal in `typed_entity_array` via Swap-and-Pop
To keep derived components packed contiguously for SIMD/cache iteration:
1. Locate the component's index within `by_type[kind].arena`. Because components have uniform stride `classPtr->size_of`:
   $$\text{componentIndex} = \frac{\text{reinterpret\_cast<uint8*>(derivedPtr)} - \text{reinterpret\_cast<uint8*>(typedArray.array)}}{\text{classPtr->size\_of}}$$
2. If the component is not the last one in the typed arena:
   - Copy the last component into the slot occupied by the deleted component.
   - Update the moved component's `Base->Derived` pointer to point to its new slot.
3. Decrement `typedArray.count`.
4. Pop the arena allocation if it was the top of the stack, or decrement count to mark slot reclamation.

```cpp
void RemoveDerivedComponent(EntityManager& manager, Class* classPtr, DerivedType derivedPtr)
{
    Assert(classPtr != nullptr);
    Assert(derivedPtr != nullptr);

    typed_entity_array& typedArray = manager.by_type[classPtr->kind];
    Assert(typedArray.count > 0);
    Assert(typedArray.array != nullptr);

    size_t stride     = classPtr->size_of;
    auto*  targetByte = reinterpret_cast<uint8*>(derivedPtr);
    auto*  firstByte  = reinterpret_cast<uint8*>(typedArray.array);

    size_t componentIndex = static_cast<size_t>(targetByte - firstByte) / stride;
    Assert(componentIndex < typedArray.count);

    size_t lastIndex = typedArray.count - 1;
    if (componentIndex != lastIndex)
    {
        uint8* lastByte = firstByte + (lastIndex * stride);
        
        // Copy last component data into target slot
        MemCopy(targetByte, lastByte, stride);

        // Fixup the base pointer of the moved component
        auto* movedDerived = reinterpret_cast<entity_template*>(targetByte);
        Assert(movedDerived->Base != nullptr);
        movedDerived->Base->Derived = targetByte;
    }

    typedArray.count -= 1;
    if (typedArray.count == 0)
    {
        typedArray.array = nullptr;
    }
}
```

### 5.4 Tracking Deletions in `World::PendingDeletions`
In `World.h`, the `World` struct is extended with a pending deletions container:

```cpp
struct World
{
    Arena*                       WorldArena    = nullptr;
    EntityManager*               EntityManager = nullptr;
    VectorArena<EntityID, 1024>  PendingDeletions;
};
```

When an entity is deleted in the world:
```cpp
void RemoveWorldEntity(World& world, EntityID id)
{
    Assert(world.EntityManager != nullptr);

    // Record pending disk deletion
    world.PendingDeletions.PushBack(id);

    // Destroy in memory immediately
    DestroyEntity(*world.EntityManager, id);
}
```

### 5.5 Disk File Cleanup during `SaveWorld`
During `SaveWorld`, before saving modified entities, the engine iterates over `world.PendingDeletions` and removes their associated `.jasset` files:

```cpp
void ProcessPendingDeletions(World& world, NonNullPtr<Arena> scratchArena)
{
    for (size_t i = 0; i < world.PendingDeletions.Size(); ++i)
    {
        EntityID id = world.PendingDeletions[i];

        // Format relative asset path: Assets/Entities/{ID}.jasset
        char filenameBuffer[64];
        juliet_snprintf(filenameBuffer, sizeof(filenameBuffer), "Entities/%llu.jasset", id);
        
        String assetPath = GetAssetPath(scratchArena, WrapString(filenameBuffer));

        if (PlatformDeleteFile(assetPath))
        {
            Log(LogLevel::Message, LogCategory::Game, "Deleted entity asset: %s", CStr(assetPath));
        }
    }

    world.PendingDeletions.Clear();
}
```

---

## 6. Dirty Tracking for Optimal Saves

### 6.1 The Cost of Naive Monolithic & Full-Directory Writes
In a level containing $10{,}000$ entities:
- **Monolithic `world.bin` Save:** Modifying a single entity's $X$ coordinate requires re-serializing all $10{,}000$ entities and overwriting a multi-megabyte binary file. This introduces a huge Git diff and constant merge conflicts.
- **Full-Directory `.jasset` Save:** Iterating through all $10{,}000$ entities and unconditionally writing $10{,}000$ `.jasset` files incurs massive OS file-system overhead (directory table locks, I/O bandwidth) and changes the file timestamps of every asset. Git reports thousands of modified files even when only one entity changed!

### 6.2 The `IsDirty` Flag on `Entity`
To solve this, `Entity` in [`Game/Entity/Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h#L28-L36) is augmented with an explicit dirty flag:

```cpp
struct Entity final
{
    EntityID    ID      = 0;
    Class*      Kind    = nullptr;
    DerivedType Derived = nullptr;
    float       X       = 0.0f;
    float       Y       = 0.0f;
    float       Z       = 0.0f;
    bool        IsDirty = false;
};
```

### 6.3 Granular State Transitions
The `IsDirty` flag obeys a strict lifecycle state machine:

```
                  +-----------------------------------+
                  |          Entity Created           |
                  |     (AllocateEntity / Editor)     |
                  +-----------------+-----------------+
                                    |
                                    v
                            +---------------+
                   +------->| IsDirty: TRUE |<-------+
                   |        +-------+-------+        |
                   |                |                |
        Entity Mutated              |            SaveWorld
    (Position, Component)           |            Completed
                   |                v                |
                   |        +---------------+        |
                   +--------+ IsDirty: FALSE+--------+
                            +-------+-------+
                                    ^
                                    |
                            Deserialization
                            (LoadWorld / Asset)
```

1. **Entity Creation:** Newly spawned entities in the editor have `IsDirty = true`.
2. **Property Mutation:** Any modification to `X, Y, Z` or derived component payload sets `entity->IsDirty = true`.
3. **Successful Deserialization:** Entities loaded from disk initialize with `IsDirty = false`.
4. **Successful Save:** Upon successfully writing an entity to its `.jasset` file, the engine resets `entity->IsDirty = false`.

### 6.4 Version Control Benefits (Git Friendly Assets)
By coupling the one-file-per-entity `.jasset` format with dirty tracking:
- Only modified entities are touched on disk.
- Git status displays only the exact `.jasset` files that were altered by the designer.
- Team members can work concurrently in the same game scene without encountering binary merge conflicts.

### 6.5 Editor Integration (`RenderWorldEditorUI` Hooks)
In [`Game/Data/World.cpp`](file:///w:/Classified/Juliet/Game/Data/World.cpp#L291-L311), editor UI widgets automatically set the dirty flag upon receiving user input:

```cpp
float pos[3] = { ent.X, ent.Y, ent.Z };
if (ImGui::DragFloat3("Position", pos, 0.1f))
{
    ent.X       = pos[0];
    ent.Y       = pos[1];
    ent.Z       = pos[2];
    ent.IsDirty = true; // Mark dirty for persistence
    UpdateWorld(world);
}
```

---

## 7. Step-by-Step Implementation Roadmap

### Phase 1: Data Structures & Header Definitions
1. **Update `Entity.h`:**
   - Add `bool IsDirty = false;` to `struct Entity`.
   - Update `MakeEntity<EntityType>` to delegate to `AllocateEntity`.
   - Define `EntityFileHeader`, `kEntityAssetMagic`, and `kEntityAssetVersion`.
2. **Update `EntityManager.h`:**
   - Declare `[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr);`.
   - Declare `void DestroyEntity(EntityManager& manager, EntityID id);`.
   - Declare `void RemoveDerivedComponent(EntityManager& manager, Class* classPtr, DerivedType derivedPtr);`.
3. **Update `World.h`:**
   - Add `VectorArena<EntityID, 1024> PendingDeletions;` to `struct World`.
   - Update `SaveWorld` and `LoadWorld` signatures to take directory paths.

### Phase 2: Core Memory Allocation & Wiring in `EntityManager.cpp`
1. Implement `AllocateEntity`:
   - Enforce parameter assertions.
   - Push to `manager.Entities`.
   - Allocate zeroed block in `manager.by_type[kind].arena`.
   - Wire mutual pointers (`base->Derived` and `derived->Base`).
   - Increment `typedArray.count` and initialize `typedArray.array`.
2. Implement `DestroyEntity` & `RemoveDerivedComponent`:
   - Implement swap-and-pop for derived components with base pointer update.
   - Implement `manager.Entities.RemoveAtFast` with mutual back-pointer fixup.

### Phase 3: In-Place Deserialization & Serialization Pipeline
1. In `Entity.cpp`:
   - Implement `EntitySerialize(archive& ar, Entity* entity)`.
   - Fix the existing reversed `if (ar.loading)` branch.
2. In `World.cpp`:
   - Implement `SerializeEntityAsset(archive& ar, Entity* entity, String filepath)`.
   - Implement `DeserializeEntityAsset(EntityManager& manager, archive& ar, String filepath)`.
   - Stream derived properties directly using `classPtr->serialize_fct(&ar, entity->Derived)`.

### Phase 4: World Save/Load Pipeline & Disk Deletion
1. In `World.cpp`:
   - Implement `ProcessPendingDeletions(World& world, NonNullPtr<Arena> scratchArena)`.
   - Implement `SaveWorld(World& world, String worldDirectory)`:
     - Process pending deletions.
     - Iterate `manager.Entities`, skipping entities where `!entity.IsDirty`.
     - Write dirty entities to `.jasset` files and clear `IsDirty`.
   - Implement `LoadWorld(World& world, String worldDirectory)`:
     - Enumerate `.jasset` files in directory.
     - Call `DeserializeEntityAsset` for each file.

### Phase 5: Editor Integration
1. In `RenderWorldEditorUI`:
   - Hook `ImGui::DragFloat3` and property inspectors to set `IsDirty = true`.
   - Hook "Add Entity" button to call `MakeEntity<Inert>(*world.EntityManager, 0.0f, 0.0f, 0.0f)`.
   - Hook "Delete Entity" button to call `RemoveWorldEntity(world, selectedEntityId)`.

---

## 8. Unit Testing & Verification Plan

### 8.1 Test Philosophy & Constraints
Following Juliet coding guidelines:
> "When creating a new system framework, make a unit test. To make the unit test we should not modify the framework code for special unit test case."

Testing is isolated in [`Game/UnitTest/WorldUnitTest.cpp`](file:///w:/Classified/Juliet/Game/UnitTest/WorldUnitTest.cpp) and executed during engine initialization in debug builds.

### 8.2 Comprehensive Test Suite (`WorldUnitTest.cpp`)
The test suite validates every guarantee made in this specification:

```cpp
#include <UnitTest/WorldUnitTest.h>

#if JULIET_DEBUG

#include <Core/Common/CoreUtils.h>
#include <Core/Common/serialization.h>
#include <Core/HAL/Filesystem/Filesystem.h>
#include <Core/HAL/IO/IOStream.h>
#include <Core/Logging/LogManager.h>
#include <Core/Logging/LogTypes.h>
#include <Core/Memory/MemoryArena.h>
#include <Core/Thread/ThreadContext.h>
#include <Data/World.h>
#include <Entity/Entity.h>
#include <Entity/EntityManager.h>

namespace UnitTest
{
    namespace
    {
        void TestEntityAllocationAndWiring()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestEntityAllocationAndWiring...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);

            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // 1. Allocate Inert Entity via AllocateEntity
            Entity* baseEntity = AllocateEntity(manager, Inert::Kind);
            Assert(baseEntity != nullptr);
            Assert(baseEntity->ID > 0);
            Assert(baseEntity->Kind == Inert::Kind);
            Assert(baseEntity->Derived != nullptr);
            Assert(baseEntity->IsDirty == true);

            // 2. Validate mutual back-pointer wiring
            auto* derived = reinterpret_cast<entity_template*>(baseEntity->Derived);
            Assert(derived->Base == baseEntity);

            // 3. DownCast verification
            Inert* inert = DownCast<Inert>(baseEntity);
            Assert(inert != nullptr);
            Assert(inert->Base == baseEntity);

            // 4. Validate typed array tracking
            typed_entity_array& inertArray = manager.by_type[ENTITY(Inert)];
            Assert(inertArray.count == 1);
            Assert(inertArray.array == derived);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestEntityAllocationAndWiring");
        }

        void TestInPlaceDeserialization()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestInPlaceDeserialization...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // 1. Create and populate entity
            Inert* createdInert = MakeEntity<Inert>(manager, 12.5f, -44.0f, 108.2f);
            Assert(createdInert != nullptr);
            createdInert->MeshInstance = 42;

            Entity* originalBase = createdInert->Base;
            EntityID originalID  = originalBase->ID;

            // 2. Serialize to memory archive
            archive saveAr{ .arena = tempArena.Arena, .base_ptr = nullptr, .offset = 0, .loading = false };
            saveAr.base_ptr = ArenaPushArray<uint8>(tempArena.Arena, Kilobytes(16));

            EntityFileHeader header{
                .Magic       = kEntityAssetMagic,
                .Version     = kEntityAssetVersion,
                .ClassCRC    = originalBase->Kind->CRC,
                .Kind        = originalBase->Kind->kind,
                .EntityID    = originalBase->ID,
                .PositionX   = originalBase->X,
                .PositionY   = originalBase->Y,
                .PositionZ   = originalBase->Z,
                .PayloadSize = sizeof(index_t)
            };
            serialize_elem(saveAr, header);
            serialize_elem(saveAr, createdInert->MeshInstance);

            // 3. Clear manager to simulate fresh load
            ShutdownEntityManager();
            InitEntityManager(&testWorld);
            EntityManager& freshManager = *testWorld.EntityManager;

            // 4. Deserialize in-place
            archive loadAr{ .arena = tempArena.Arena, .base_ptr = saveAr.base_ptr, .offset = 0, .loading = true };
            Entity* loadedBase = DeserializeEntityInPlace(freshManager, loadAr);

            Assert(loadedBase != nullptr);
            Assert(loadedBase->ID == originalID);
            Assert(loadedBase->X == 12.5f);
            Assert(loadedBase->Y == -44.0f);
            Assert(loadedBase->Z == 108.2f);
            Assert(loadedBase->IsDirty == false);

            Inert* loadedInert = DownCast<Inert>(loadedBase);
            Assert(loadedInert != nullptr);
            Assert(loadedInert->Base == loadedBase);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestInPlaceDeserialization");
        }

        void TestSwapAndPopPointerFixup()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestSwapAndPopPointerFixup...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            // Allocate 3 entities: E0, E1, E2
            Inert* e0 = MakeEntity<Inert>(manager, 1.0f, 0.0f, 0.0f);
            Inert* e1 = MakeEntity<Inert>(manager, 2.0f, 0.0f, 0.0f);
            Inert* e2 = MakeEntity<Inert>(manager, 3.0f, 0.0f, 0.0f);

            EntityID id0 = e0->Base->ID;
            EntityID id1 = e1->Base->ID;
            EntityID id2 = e2->Base->ID;

            Assert(manager.Entities.Size() == 3);

            // Delete middle entity E1 (triggers swap with E2)
            DestroyEntity(manager, id1);

            Assert(manager.Entities.Size() == 2);

            // Verify E2's mutual back-pointers are still completely intact
            Entity* remaining0 = &manager.Entities[0];
            Entity* remaining1 = &manager.Entities[1];

            Assert(remaining0->ID == id0);
            Assert(remaining1->ID == id2);

            auto* derived0 = reinterpret_cast<entity_template*>(remaining0->Derived);
            auto* derived1 = reinterpret_cast<entity_template*>(remaining1->Derived);

            Assert(derived0->Base == remaining0);
            Assert(derived1->Base == remaining1);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestSwapAndPopPointerFixup");
        }

        void TestDirtyTrackingLifecycle()
        {
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestDirtyTrackingLifecycle...");

            TempArena tempArena = scratch_begin(nullptr, 0);

            World testWorld{};
            InitWorld(&testWorld, tempArena.Arena);
            InitEntityManager(&testWorld);
            EntityManager& manager = *testWorld.EntityManager;

            Entity* entity = AllocateEntity(manager, Inert::Kind);
            Assert(entity->IsDirty == true);

            // Simulate save
            entity->IsDirty = false;
            Assert(entity->IsDirty == false);

            // Simulate mutation
            entity->X += 1.0f;
            entity->IsDirty = true;
            Assert(entity->IsDirty == true);

            ShutdownEntityManager();
            ShutdownWorld(&testWorld);

            scratch_end(tempArena);
            Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestDirtyTrackingLifecycle");
        }
    } // namespace

    void WorldUnitTest()
    {
        Log(LogLevel::Message, LogCategory::Game, "==================================================");
        Log(LogLevel::Message, LogCategory::Game, "Starting Entity Allocation & Lifecycle Unit Tests");
        Log(LogLevel::Message, LogCategory::Game, "==================================================");

        TestEntityAllocationAndWiring();
        TestInPlaceDeserialization();
        TestSwapAndPopPointerFixup();
        TestDirtyTrackingLifecycle();

        Log(LogLevel::Message, LogCategory::Game, "==================================================");
        Log(LogLevel::Message, LogCategory::Game, "All Entity Lifecycle Unit Tests PASSED Successfully");
        Log(LogLevel::Message, LogCategory::Game, "==================================================");
    }
} // namespace UnitTest

#endif