Juliet/Game/Plans/Entity_Removal_Brainstorm.md

Entity Removal & Deletion Lifecycle (Brainstorm & Rework)

Warning

Status: Needs Rework The entity deletion and removal logic below was extracted as-is from Game/Plans/02_Entity_Allocation_And_Lifecycle.md for future brainstorming. The original swap-and-pop approach required complex mutual back-pointer fixups across both base and derived memory buffers.

---

1. Brainstorming Notes & Alternative Approaches

The original design relied on VectorArena::RemoveAtFast (swap-and-pop) for both Entities and by_type[kind].arena. While this maintained dense contiguous memory, swapping arbitrary elements in memory invalidated pointers in both directions, requiring runtime pointer patching (derived->Base = movedEntity; base->Derived = movedComponent).

Ideas for Simpler & More Robust Alternatives:

1. Active Flag / Tombstone (is_active bool):

   • Keep deleted entities in place rather than moving or swapping them.
   • Simply toggle entity->is_active = false;.
   • Iterators and simulation loops skip inactive entities (if (!entity->is_active) continue;).
   • Pointers to entities and derived components remain stable for their entire lifetime.

2. Intrusive Free List for Slot Recycling:

   • Instead of shifting memory on deletion, vacant slots are linked into an intrusive free list (first_free_index).
   • When an entity is destroyed:
     • entity->is_active = false;
     • Overwrite unused slot memory with next_free_index.
   • When allocating a new entity:
     • Check if first_free_index != indexMax.
     • Pop from the free list in O(1); otherwise push back to the end of the arena.
   • Zero pointer invalidation, zero memmove/memcpy overhead during deletion.

3. Stable Generational Handles / Slot Map:

   • If external systems need references to entities that might be deleted, combine the free list with a generational counter to detect stale lookups safely.

4. Deferred Compaction / Garbage Collection:

   • If contiguous packing is strictly necessary for SIMD/cache optimization, defer compaction to a designated level load or scene transition rather than doing it per-frame on individual deletes.

---

2. Extracted Original Removal Logic (As-Is from Plan 2)

2.1 In-Memory Removal vs. Immediate Disk Deletion Hazards

In game development, deleting an entity in the editor or during gameplay must never synchronously invoke disk deletion:

1. Frame Rate Stutters: Blocking on synchronous OS filesystem APIs (DeleteFileA) introduces multisecond frame freezes.
2. Transactional Safety: If the editor crashes or the user exits without saving, disk modifications cannot be rolled back.
3. Undo/Redo Support: An editor action stack must allow recovering deleted entities before changes are permanently committed to disk.

Therefore, Juliet enforces a strict separation:

• Immediate in-memory destruction: Releases the entity from active simulation and registers its identifier in World::PendingDeletions.
• Deferred disk deletion: Executed strictly during explicit SaveWorld operations.

2.2 Fast In-Memory Removal (RemoveAtFast) & Mutual Pointer Fixup

VectorArena::RemoveAtFast utilizes swap-and-pop: the element at the target index is replaced by the last element in the vector, and Count is decremented.

void RemoveAtFast(index_t index)
{
    Assert(Arena);
    Assert(index < Count);
    Assert(Count > 0);

    Type* elementAdr = DataFirst + index;

    if (DataLast != elementAdr)
    {
        Swap(DataLast, elementAdr);
    }

    --DataLast;
    --Count;
}

The Pointer Invalidation Problem:

When Entity A (at index) is swapped with Entity Z (at DataLast), the physical address of Entity Z changes from DataLast to elementAdr. If Entity Z has a derived struct derivedZ, derivedZ->Base previously pointed to DataLast. After RemoveAtFast, derivedZ->Base points to garbage or the freed slot!

The Pointer Fixup Protocol:

To preserve the mutual back-pointer invariant, DestroyEntity explicitly fixes up the swapped entity's derived back-pointer:

void DestroyEntity(EntityManager& manager, EntityID id)
{
    Entity* baseArray = manager.Entities.DataPtr();
    size_t  count     = manager.Entities.Size();

    size_t targetIndex = indexMax;
    for (size_t i = 0; i < count; ++i)
    {
        if (baseArray[i].ID == id)
        {
            targetIndex = i;
            break;
        }
    }

    if (targetIndex == indexMax)
    {
        return;
    }

    Entity* targetEntity = &baseArray[targetIndex];
    Class*  classPtr     = targetEntity->derived_kind;
    Assert(classPtr != nullptr);

    // 1. Remove derived component from typed array via swap-and-pop
    RemoveDerivedComponent(manager, classPtr, targetEntity->derived);

    // 2. Remove base entity via swap-and-pop in VectorArena
    bool wasLast = (targetIndex == count - 1);
    manager.Entities.RemoveAtFast(targetIndex);

    // 3. Pointer fixup: If an element was swapped into targetIndex, fix its back-pointer!
    if (!wasLast && targetIndex < manager.Entities.Size())
    {
        Entity* movedEntity = &manager.Entities[targetIndex];
        auto*   derivedTemp = reinterpret_cast<entity_template*>(movedEntity->derived);
        Assert(derivedTemp != nullptr);
        derivedTemp->base = movedEntity;
    }
}

2.3 O(1) Component Removal in typed_entity_array via Swap-and-Pop

To keep derived components packed contiguously for SIMD/cache iteration:

1. Locate the component's index within by_type[kind].arena. Because components have uniform stride classPtr->size_of: \text{componentIndex} = \frac{\text{reinterpret\_cast<uint8*>(derivedPtr)} - \text{reinterpret\_cast<uint8*>(typedArray.array)}}{\text{classPtr->size\_of}}
2. If the component is not the last one in the typed arena:
   • Copy the last component into the slot occupied by the deleted component.
   • Update the moved component's Base->Derived pointer to point to its new slot.
3. Decrement typedArray.count.
4. Pop the arena allocation if it was the top of the stack, or decrement count to mark slot reclamation.

void RemoveDerivedComponent(EntityManager& manager, Class* classPtr, DerivedType derivedPtr)
{
    Assert(classPtr != nullptr);
    Assert(derivedPtr != nullptr);

    typed_entity_array& typedArray = manager.by_type[classPtr->kind];
    Assert(typedArray.count > 0);
    Assert(typedArray.array != nullptr);

    size_t stride     = classPtr->size_of;
    auto*  targetByte = reinterpret_cast<uint8*>(derivedPtr);
    auto*  firstByte  = reinterpret_cast<uint8*>(typedArray.array);

    size_t componentIndex = static_cast<size_t>(targetByte - firstByte) / stride;
    Assert(componentIndex < typedArray.count);

    size_t lastIndex = typedArray.count - 1;
    if (componentIndex != lastIndex)
    {
        uint8* lastByte = firstByte + (lastIndex * stride);
        
        // Copy last component data into target slot
        MemCopy(targetByte, lastByte, stride);

        // Fixup the base pointer of the moved component
        auto* movedDerived = reinterpret_cast<entity_template*>(targetByte);
        Assert(movedDerived->base != nullptr);
        movedDerived->base->derived = targetByte;
    }

    typedArray.count -= 1;
    if (typedArray.count == 0)
    {
        typedArray.array = nullptr;
    }
}

2.4 Tracking Deletions in World::PendingDeletions

In World.h, the World struct is extended with a pending deletions container:

struct World
{
    Arena*                       WorldArena    = nullptr;
    EntityManager*               EntityManager = nullptr;
    VectorArena<EntityID, 1024>  PendingDeletions;
};

When an entity is deleted in the world:

void RemoveWorldEntity(World& world, EntityID id)
{
    Assert(world.EntityManager != nullptr);

    // Record pending disk deletion
    world.PendingDeletions.PushBack(id);

    // Destroy in memory immediately
    DestroyEntity(*world.EntityManager, id);
}

2.5 Disk File Cleanup during SaveWorld

During SaveWorld, before saving modified entities, the engine iterates over world.PendingDeletions and removes their associated .jasset files:

void ProcessPendingDeletions(World& world, NonNullPtr<Arena> scratchArena)
{
    for (size_t i = 0; i < world.PendingDeletions.Size(); ++i)
    {
        EntityID id = world.PendingDeletions[i];

        // Format relative asset path: Assets/Entities/{ID}.jasset
        char filenameBuffer[64];
        juliet_snprintf(filenameBuffer, sizeof(filenameBuffer), "Entities/%llu.jasset", id);
        
        String assetPath = GetAssetPath(scratchArena, WrapString(filenameBuffer));

        if (PlatformDeleteFile(assetPath))
        {
            Log(LogLevel::Message, LogCategory::Game, "Deleted entity asset: %s", CStr(assetPath));
        }
    }

    world.PendingDeletions.Clear();
}

2.6 Associated Swap-and-Pop Unit Test

void TestSwapAndPopPointerFixup()
{
    Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestSwapAndPopPointerFixup...");

    TempArena tempArena = scratch_begin(nullptr, 0);

    World testWorld{};
    InitWorld(&testWorld, tempArena.Arena);
    InitEntityManager(&testWorld);
    EntityManager& manager = *testWorld.EntityManager;

    // Allocate 3 entities: E0, E1, E2
    Inert* e0 = MakeEntity<Inert>(manager, 1.0f, 0.0f, 0.0f);
    Inert* e1 = MakeEntity<Inert>(manager, 2.0f, 0.0f, 0.0f);
    Inert* e2 = MakeEntity<Inert>(manager, 3.0f, 0.0f, 0.0f);

    EntityID id0 = e0->base->ID;
    EntityID id1 = e1->base->ID;
    EntityID id2 = e2->base->ID;

    Assert(manager.Entities.Size() == 3);

    // Delete middle entity E1 (triggers swap with E2)
    DestroyEntity(manager, id1);

    Assert(manager.Entities.Size() == 2);

    // Verify E2's mutual back-pointers are still completely intact
    Entity* remaining0 = &manager.Entities[0];
    Entity* remaining1 = &manager.Entities[1];

    Assert(remaining0->ID == id0);
    Assert(remaining1->ID == id2);

    auto* derived0 = reinterpret_cast<entity_template*>(remaining0->derived);
    auto* derived1 = reinterpret_cast<entity_template*>(remaining1->derived);

    Assert(derived0->base == remaining0);
    Assert(derived1->base == remaining1);

    ShutdownEntityManager();
    ShutdownWorld(&testWorld);

    scratch_end(tempArena);
    Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestSwapAndPopPointerFixup");
}