819 lines
41 KiB
Markdown
819 lines
41 KiB
Markdown
# Juliet Game Engine Architecture Specification
|
|
## Document: 02 - Entity Allocation & In-Place Lifecycle
|
|
|
|
**Document ID:** JULIET-SPEC-002
|
|
**Status:** Approved for Implementation
|
|
**Author:** Senior Engine Architect
|
|
**Subsystems:** `Game/Entity`, `Game/Data`, `Game/UnitTest`
|
|
**Target Files:**
|
|
- [`Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h)
|
|
- [`Entity.cpp`](file:///w:/Classified/Juliet/Game/Entity/Entity.cpp)
|
|
- [`EntityManager.h`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.h)
|
|
- [`EntityManager.cpp`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.cpp)
|
|
- [`World.h`](file:///w:/Classified/Juliet/Game/Data/World.h)
|
|
- [`World.cpp`](file:///w:/Classified/Juliet/Game/Data/World.cpp)
|
|
- [`WorldUnitTest.h`](file:///w:/Classified/Juliet/Game/UnitTest/WorldUnitTest.h)
|
|
- [`WorldUnitTest.cpp`](file:///w:/Classified/Juliet/Game/UnitTest/WorldUnitTest.cpp)
|
|
|
|
---
|
|
|
|
## 1. Executive Summary & Problem Statement
|
|
|
|
### 1.1 Background & Context
|
|
The Juliet game engine organizes game entities using a hybrid data-oriented architecture:
|
|
1. A flat array of **Base Entities** (`Entity`) encapsulating universal properties: unique 64-bit ID, runtime type reflection pointer (`Class* Kind`), spatial coordinates (`X, Y, Z`), and an opaque pointer to the derived payload (`DerivedType Derived`).
|
|
2. Type-segregated contiguous arrays of **Derived Entities** (`Inert`, etc.) stored in dedicated per-type memory arenas (`typed_entity_array`).
|
|
|
|
This design is intended to provide maximum cache efficiency during spatial and general-purpose entity processing, while retaining dense SIMD-friendly streaming for type-specific systems (e.g., transform updates on `Inert` mesh instances).
|
|
|
|
### 1.2 Root-Cause Analysis of `Game/Data/World.cpp` Loading Flaws
|
|
In the initial implementation of entity serialization in [`Game/Data/World.cpp`](file:///w:/Classified/Juliet/Game/Data/World.cpp#L40-L68), loading was fundamentally broken and incomplete:
|
|
|
|
```cpp
|
|
// Existing flawed deserialization in World.cpp
|
|
for (typed_entity_array& type : entityManager.by_type)
|
|
{
|
|
serialize_elem(ar, type.count);
|
|
|
|
if (type.count > 0)
|
|
{
|
|
// Unserialize the base entity to get informations
|
|
Entity entity;
|
|
serialize(ar, &entity);
|
|
|
|
RegisterBaseEntity(entityManager, entity);
|
|
}
|
|
}
|
|
```
|
|
|
|
This implementation suffers from several fatal defects:
|
|
1. **Single-Element Iteration Bug:** It uses `if (type.count > 0)` instead of a loop `for (size_t i = 0; i < type.count; ++i)`, deserializing at most one single entity per type bucket, leaving all subsequent entities in the stream unread and corrupting the archive read offset.
|
|
2. **Missing Derived Allocation:** It invokes `RegisterBaseEntity(entityManager, entity)`, which merely pushes the stack-allocated `Entity` into `manager.Entities`. The derived payload arena (`type.arena`) is completely untouched: `type.array` remains null, `type.count` in the manager is desynchronized, and `entity.Derived` remains unassigned or points to an invalid address.
|
|
3. **Invalid Pointer in `RegisterEntity`:** In [`Game/Entity/EntityManager.cpp`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.cpp#L46-L66), `RegisterEntity` assigns `base->Derived = entity` before pushing `*base` into `manager.Entities`. The parameter `entity` is a pointer to caller-provided memory (often stack-allocated in helper functions like `MakeEntity`). When `ArenaPushSize` later allocates the persistent derived memory block, `base->Derived` stored inside `manager.Entities.Back()` is **never updated**—it remains dangling, pointing to the transient caller stack!
|
|
|
|
### 1.3 The "Chicken-and-Egg" Stack Allocation Dilemma
|
|
The existing registration API requires a pre-existing derived instance:
|
|
```cpp
|
|
entity_template* RegisterEntity(EntityManager& manager, Entity* base, DerivedType entity);
|
|
```
|
|
During runtime programmatic creation via `MakeEntity<T>()`, a temporary instance of `T` is created on the stack and passed by pointer:
|
|
```cpp
|
|
template <typename EntityType>
|
|
EntityType* MakeEntity(EntityManager& manager, float x, float y, float z)
|
|
{
|
|
EntityType result; // Stack allocation
|
|
Entity base; // Stack allocation
|
|
base.X = x;
|
|
base.Y = y;
|
|
base.Z = z;
|
|
base.Kind = EntityType::Kind;
|
|
|
|
return (EntityType*)RegisterEntity(manager, &base, &result);
|
|
}
|
|
```
|
|
When loading an entity from a stream or disk file, **the type is not known at compile time**. The engine reads a runtime type tag (`uint8 kind` or `uint32 CRC`), looks up the reflection metadata (`Class*`), and must instantiate the entity dynamically.
|
|
|
|
Because C++ does not permit allocating a dynamic struct of unknown type on the stack, and because Juliet strictly forbids heap allocations (`malloc`, `new`, `std::vector`), deserialization cannot construct a temporary instance on the stack to pass to `RegisterEntity`.
|
|
|
|
This is the classic **chicken-and-egg memory problem**:
|
|
- `RegisterEntity` requires an existing instance in memory to copy from.
|
|
- Deserialization requires an allocated memory buffer to deserialize into.
|
|
|
|
### 1.4 Architectural Objectives
|
|
This specification establishes a robust in-place lifecycle pipeline that completely eliminates stack temporaries and dynamic heap allocations:
|
|
1. **Direct In-Place Allocation:** Introduce `AllocateEntity(EntityManager& manager, Class* classPtr)` which allocates both the base `Entity` and the derived struct directly within their respective engine memory arenas.
|
|
2. **Bidirectional Pointer Integrity:** Wire mutual pointers (`base->derived` and `derived->base`) at allocation time before any field deserialization begins.
|
|
3. **In-Place Stream Deserialization:** Read class reflection metadata first, invoke `AllocateEntity`, and stream base and derived properties directly into arena-resident memory.
|
|
4. **Isolated Entity Assets (`.jasset`):** Transition from a monolithic `world.bin` to a modular one-file-per-entity architecture (`Assets/Entities/{ID}.jasset`).
|
|
5. **Dirty Tracking & Optimal Saves:** Introduce an `is_dirty` flag on `Entity` to avoid rewriting unchanged entity files, minimizing disk I/O and eliminating spurious Git repository modifications.
|
|
6. **Robust Deletion Lifecycle:** Decouple in-memory removal (`RemoveAtFast` with pointer fixup) from disk synchronization using `World::PendingDeletions`.
|
|
|
|
---
|
|
|
|
## 2. The Dual Arena Memory Model in `EntityManager`
|
|
|
|
### 2.1 The Need for Dual Storage
|
|
Game engines execute systems with vastly different cache locality profiles:
|
|
- **Spatial / Frustum Culling / Transform Sync:** Iterates every entity in the world, needing only `X, Y, Z`, bounding spheres, and base status flags.
|
|
- **Specialized Logic / Render Updates:** Iterates only entities possessing specific components (e.g., `Inert` static meshes requiring instance transform updates to the GPU bindless descriptor table).
|
|
|
|
Storing large monolithic polymorphic structs in a single array causes severe cache line pollution during spatial passes. Conversely, storing entities in fragmented individual allocations introduces cache misses and pointer-chasing overhead.
|
|
|
|
Juliet resolves this with a **Dual Arena Memory Model**:
|
|
|
|
```
|
|
+---------------------------------------------------------------------------------------------+
|
|
| EntityManager |
|
|
+---------------------------------------------------------------------------------------------+
|
|
| |
|
|
| manager.Entities (VectorArena<Entity, 100'000>) |
|
|
| +------------------------------------+------------------------------------+ |
|
|
| | Entity 0 (ID=1001, X, Y, Z) | Entity 1 (ID=1002, X, Y, Z) | ... |
|
|
| | Derived ------------------------+ | Derived ---------------------+ | |
|
|
| +---------------------------------|--+------------------------------|-----+ |
|
|
| | | |
|
|
| v v |
|
|
| manager.by_type[ENTITY(Inert)].arena |
|
|
| +------------------------------------+------------------------------------+ |
|
|
| | Inert 0 (MeshInstance=4) | Inert 1 (MeshInstance=12) | ... |
|
|
| | Base ---------------------------+ | Base ------------------------+ | |
|
|
| +---------------------------------|--+------------------------------|-----+ |
|
|
| +---------------------------------+ |
|
|
+---------------------------------------------------------------------------------------------+
|
|
```
|
|
|
|
### 2.2 `manager.Entities`: Cache-Friendly Base Entity Vector
|
|
Base entities reside in a pre-reserved contiguous array:
|
|
```cpp
|
|
VectorArena<Entity, 100'000> Entities;
|
|
```
|
|
- **Capacity:** Fixed reserve of 100,000 entities allocated from the `WorldArena`.
|
|
- **Memory Footprint:**
|
|
$$\text{sizeof(Entity)} = 8\text{ (ID)} + 8\text{ (derived\_kind)} + 8\text{ (derived)} + 16\text{ (position)} + 1\text{ (is\_dirty)} + 7\text{ (Padding)} = 48\text{ bytes}$$
|
|
Total reserved space: $100{,}000 \times 48\text{ bytes} \approx 4.8\text{ MB}$.
|
|
- **Access Speed:** O(1) random access by index; sequential streaming utilizes L1/L2 hardware prefetchers with zero cache line waste.
|
|
|
|
### 2.3 `manager.by_type[kind].arena`: Typed Component Arenas
|
|
Derived structs reside in per-type contiguous memory arenas:
|
|
```cpp
|
|
struct typed_entity_array
|
|
{
|
|
Arena* arena;
|
|
entity_template* array;
|
|
size_t count;
|
|
};
|
|
```
|
|
- Each `Entity_Type` index owns an isolated `Arena*` allocated during `InitEntityManager`.
|
|
- Allocations are packed linearly with alignment specified by `classPtr->alignment`.
|
|
- `array` points directly to the first element in the arena, permitting typed array indexing:
|
|
```cpp
|
|
Inert* inertArray = reinterpret_cast<Inert*>(manager.by_type[ENTITY(Inert)].array);
|
|
```
|
|
|
|
### 2.4 Mutual Back-Pointer Architecture & Invariants
|
|
Every entity instance consists of two mutually linked allocations:
|
|
1. `basePtr->Derived`: Points from `Entity` in `manager.Entities` to the derived struct in `by_type[kind].arena`.
|
|
2. `derivedPtr->Base`: Points from the derived struct (via `DECLARE_ENTITY()`) back to `Entity` in `manager.Entities`.
|
|
|
|
#### Invariant Rules:
|
|
1. **Non-Null Invariant:** For any active entity, `basePtr->Derived != nullptr` and `reinterpret_cast<entity_template*>(basePtr->Derived)->Base == basePtr`.
|
|
2. **Type Coherence Invariant:** `basePtr->Kind->kind == derivedTypeId`.
|
|
3. **Array Count Invariant:**
|
|
$$\sum_{k=0}^{\text{ENTITY(Count)}-1} \text{manager.by\_type}[k].\text{count} == \text{manager.Entities.Size()}$$
|
|
|
|
### 2.5 Pointer Stability in `VectorArena`
|
|
`VectorArena::Create` executes `Reserve(ReserveSize)` on creation:
|
|
```cpp
|
|
newManager->Entities.Create(world->WorldArena JULIET_DEBUG_PARAM("Entities"));
|
|
```
|
|
Because capacity ($100{,}000$) is fully reserved upfront in virtual address space, `VectorArena::PushBack` **never reallocates or moves existing memory**. Therefore:
|
|
- Pointers to `Entity` elements in `manager.Entities` remain absolutely stable across allocations.
|
|
- Derived struct `Base` pointers remain valid indefinitely unless an element is deleted.
|
|
- Element removal via swap-and-pop alters memory positions, requiring systematic pointer fixups (addressed in Section 5.2).
|
|
|
|
---
|
|
|
|
## 3. The Solution: `AllocateEntity(EntityManager& manager, Class* classPtr)`
|
|
|
|
### 3.1 Function Signature & Contract
|
|
The canonical allocation function is defined in `EntityManager.h`:
|
|
|
|
```cpp
|
|
[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* classPtr);
|
|
```
|
|
|
|
#### Preconditions:
|
|
- `classPtr != nullptr`.
|
|
- `classPtr->kind < ENTITY(Count)`.
|
|
- `classPtr->size_of >= sizeof(entity_template)`.
|
|
- `manager.Entities.Size() < manager.Entities.Capacity`.
|
|
- `manager.by_type[classPtr->kind].arena != nullptr`.
|
|
|
|
#### Postconditions:
|
|
- A new `Entity` record is appended to `manager.Entities`.
|
|
- A new typed block of `classPtr->size_of` bytes is allocated in `manager.by_type[classPtr->kind].arena`.
|
|
- The derived memory is initialized with C++ struct defaults via `classPtr->default_init_fct` (or zeroed via `MemZero` if null).
|
|
- `base->derived` points to the derived struct.
|
|
- `derived->base` points to the base `Entity`.
|
|
- `base->derived_kind` is assigned to `class_ptr`.
|
|
- `base->ID` is initialized to `0` (unassigned; populated by `MakeEntity` or deserialization).
|
|
- `base->is_dirty` is initialized to `true`.
|
|
- `typed_entity_array::count` is incremented.
|
|
- `typed_entity_array::array` is initialized if this is the first entity of this type.
|
|
|
|
### 3.2 Detailed Step-by-Step Implementation
|
|
The implementation replaces the flawed `RegisterEntity` routine in [`Game/Entity/EntityManager.cpp`](file:///w:/Classified/Juliet/Game/Entity/EntityManager.cpp):
|
|
|
|
```cpp
|
|
[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* derivedClassPtr)
|
|
{
|
|
Assert(derivedClassPtr != nullptr);
|
|
Assert(derivedClassPtr->kind < ENTITY(Count));
|
|
Assert(derivedClassPtr->size_of >= sizeof(entity_template));
|
|
Assert(derivedClassPtr->alignment > 0);
|
|
|
|
// 1. Allocate Base Entity in the contiguous VectorArena (ID is 0 until assigned by MakeEntity or deserialization)
|
|
Entity baseTemplate{};
|
|
baseTemplate.ID = 0;
|
|
baseTemplate.derived_kind = derivedClassPtr;
|
|
baseTemplate.derived = nullptr;
|
|
baseTemplate.position = {};
|
|
baseTemplate.is_dirty = true;
|
|
|
|
manager.Entities.PushBack(baseTemplate);
|
|
Entity* basePtr = manager.Entities.Back();
|
|
Assert(basePtr != nullptr);
|
|
|
|
// 2. Allocate derived component memory in the typed arena
|
|
typed_entity_array& typedArray = manager.by_type[derivedClassPtr->kind];
|
|
Assert(typedArray.arena != nullptr);
|
|
|
|
void* rawMemory = ArenaPushSize(
|
|
typedArray.arena,
|
|
derivedClassPtr->size_of,
|
|
derivedClassPtr->alignment,
|
|
false JULIET_DEBUG_PARAM(kEntity_type_names[derivedClassPtr->kind]));
|
|
Assert(rawMemory != nullptr);
|
|
|
|
// 3. Initialize derived component defaults via Class reflection stub
|
|
if (derivedClassPtr->default_init_fct != nullptr)
|
|
{
|
|
derivedClassPtr->default_init_fct(rawMemory);
|
|
}
|
|
else
|
|
{
|
|
MemZero(rawMemory, derivedClassPtr->size_of);
|
|
}
|
|
|
|
auto* derivedTemplate = reinterpret_cast<entity_template*>(rawMemory);
|
|
|
|
// 4. Establish mutual back-pointers
|
|
basePtr->derived = rawMemory;
|
|
derivedTemplate->base = basePtr;
|
|
|
|
// 5. Update typed array tracking
|
|
if (typedArray.array == nullptr)
|
|
{
|
|
typedArray.array = derivedTemplate;
|
|
}
|
|
typedArray.count += 1;
|
|
|
|
return basePtr;
|
|
}
|
|
```
|
|
|
|
### 3.3 Default Struct Initialization via `Class::default_init_fct`
|
|
|
|
#### The Problem with Zero-Only Initialization
|
|
If allocation only zeroes memory (`MemZero` / `0x00`), any C++ member variables with non-zero defaults (such as `index_t MeshInstance = indexMax;` or `float Density = 1.0f;`) are populated with `0`. During deserialization, if a `.jasset` file lacks that property (e.g. an older file version or an optional field), `SERIALIZE` leaves the field untouched, meaning it incorrectly remains `0` rather than its intended default sentinel value!
|
|
|
|
#### The Solution: Compile-Time Default Stub in `DEFINE_ENTITY_VERSIONED`
|
|
Every entity descriptor `Class` includes a function pointer:
|
|
```cpp
|
|
using default_init_fct_type = void (*)(void* payload);
|
|
|
|
struct Class
|
|
{
|
|
...
|
|
default_init_fct_type default_init_fct = nullptr;
|
|
};
|
|
```
|
|
|
|
When registering an entity type with `DEFINE_ENTITY_VERSIONED`, the macro automatically defines a tiny type-safe stub that aggregate value-initializes the struct:
|
|
```cpp
|
|
#define DEFINE_ENTITY_VERSIONED(entity, version, serialize_fct) \
|
|
inline void default_init_##entity(void* payload) \
|
|
{ \
|
|
*static_cast<entity*>(payload) = entity{}; \
|
|
} \
|
|
constexpr Class entityKind##entity = MakeClass(ConstString(#entity), (uint8)Entity_Type::entity, (version), \
|
|
&classKindEntity, sizeof(entity), alignof(entity), \
|
|
(serialize_fct), default_init_##entity); \
|
|
Class* entity::kind = const_cast<Class*>(&entityKind##entity);
|
|
```
|
|
|
|
#### Architectural Advantages:
|
|
1. **Zero Boilerplate**: Developers write member initializers once in the struct definition (`index_t MeshInstance = indexMax;`).
|
|
2. **Type-Agnostic Core**: `AllocateEntity` does not need to know any C++ struct types; it unconditionally calls `derivedClassPtr->default_init_fct(rawMemory)`.
|
|
3. **Robust Deserialization**: In `deserialize_entity_in_place`, newly allocated entities already hold their canonical C++ defaults. Any properties absent in the `.jasset` file naturally retain their correct initial values.
|
|
4. **No Dynamic Heap / Placement-New**: `*static_cast<entity*>(payload) = entity{}` is pure aggregate value-assignment without `<new>` headers or exceptions.
|
|
|
|
### 3.4 Bidirectional Pointer Wiring
|
|
Notice the sequence:
|
|
1. `manager.Entities.PushBack(baseTemplate)` places the struct at its final, fixed arena address.
|
|
2. `base_ptr = manager.Entities.Back()` retrieves the persistent memory pointer.
|
|
3. `derived_class_ptr->default_init_fct(raw_memory)` initializes canonical struct defaults.
|
|
4. `derived_template->base = base_ptr` wires the derived back-pointer directly to this permanent location.
|
|
5. `base_ptr->derived = raw_memory` wires the base forward-pointer to the arena-allocated derived struct.
|
|
|
|
No stack copying occurs. Neither pointer is ever left dangling.
|
|
|
|
### 3.5 Updating Type Counts and Array Cache
|
|
`typed_entity_array` maintains:
|
|
- `typedArray.count`: The exact count of active entities of this type.
|
|
- `typedArray.array`: Pointer to the first element in the arena.
|
|
|
|
When the first entity of a given type is allocated, `typedArray.array` is set to `derived_template`. Because the arena allocates sequentially, `typedArray.array[i]` can be indexed directly with stride `class_ptr->size_of` as long as memory remains contiguous.
|
|
|
|
### 3.6 Refactoring `MakeEntity<EntityType>` Template
|
|
With `AllocateEntity` handling memory allocation, default member initialization, and mutual pointer wiring, `MakeEntity` in [`Game/Entity/Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h) becomes clean, safe, and stack-free:
|
|
|
|
```cpp
|
|
template <typename EntityType>
|
|
requires EntityConcept<EntityType>
|
|
[[nodiscard]] EntityType* MakeEntity(EntityManager& manager, float x, float y, float z)
|
|
{
|
|
Entity* base_ptr = AllocateEntity(manager, EntityType::kind);
|
|
Assert(base_ptr != nullptr);
|
|
|
|
base_ptr->ID = EntityManager::ID++;
|
|
base_ptr->position.x = x;
|
|
base_ptr->position.y = y;
|
|
base_ptr->position.z = z;
|
|
base_ptr->position.w = 1.0f;
|
|
|
|
return static_cast<EntityType*>(base_ptr->derived);
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 4. In-Place Deserialization Pipeline
|
|
|
|
### 4.1 Asset Format Specification (`.jasset`)
|
|
To achieve robust version control and modular streaming, Juliet adopts a human-readable, Git-diffable **one-file-per-entity** disk format with extension `.jasset`.
|
|
|
|
```ini
|
|
; asset_type
|
|
entity_instance
|
|
; id
|
|
0x0100000000000042
|
|
; class
|
|
Inert
|
|
; version
|
|
1
|
|
; class_version
|
|
1
|
|
; position
|
|
0.43 0.32 1.56
|
|
; mesh_instance
|
|
12
|
|
```
|
|
|
|
#### Important: No Header Structs for Derived Types
|
|
- **Derived types NEVER require their own file header**: You do **not** write an `InertHeader`, `DoorHeader`, or `PlayerHeader`. Derived types only serialize their own member variables.
|
|
- **Universal `; version` + optional `; class_version`**: Every `.jasset` file has a universal `; version` tag. For entity assets, `; version` governs base entity properties (`kEntityBaseVersion`), while an optional `; class_version` governs derived class properties (`Class::Version`). Non-entity assets like `WorldSettings.jasset` only have `; version`.
|
|
- **No binary `EntityFileHeader` struct is needed**: Under the `; variable_name\nvalues` text format, there is no packed binary C-struct header at all. The common properties (`; id`, `; class`, `; version`, `; class_version`, `; position`) are standard text Key-Value nodes read by the exact same `Archive` parser.
|
|
|
|
### 4.2 Eliminating Intermediate Stack Allocations
|
|
Under the new pipeline:
|
|
1. The `.jasset` text file is read into memory onto a `TempArena` via `LoadFile`.
|
|
2. The property nodes are parsed into a `ParsedArchive` via `tokenize_archive(tempArena.Arena, fileBuffer, &ar.base)`.
|
|
3. The engine reads `; class` (e.g. `"Inert"`) and resolves `Class* class_ptr = find_class_by_name(class_name)`.
|
|
4. `AllocateEntity(manager, class_ptr)` is called immediately. Memory for both base and derived components is allocated in-place in their permanent engine arenas.
|
|
5. `serialize(ar, NonNullPtr<Entity>(base_ptr))` is called. Base (`Entity::kind`) and derived (`base_ptr->derived_kind`) fields stream **directly into their permanent memory arenas** without temporary staging buffers or stack copies.
|
|
|
|
### 4.3 Runtime Class Resolution
|
|
To ensure fast and safe type lookup during file deserialization, Juliet provides two tiers of type resolution:
|
|
|
|
1. **Low-Level Indexed Validation (`ResolveEntityClass`)**: An $O(1)$ array lookup into `kEntity_type_class_ptr[kind]` that validates the class pointer and verifies schema compatibility against `classPtr->CRC`. This is the core validation primitive used by binary streaming, network replication, and internal lookups.
|
|
2. **High-Level Name Resolution (`find_class_by_name`)**: Bridges the human-readable text archive format (`; class\nInert`) to the entity class registry. It computes the `crc32` of the parsed class name and queries `ResolveEntityClass` across registered entity kinds.
|
|
|
|
```cpp
|
|
[[nodiscard]] const Class* ResolveEntityClass(uint8 kind, uint32 crc)
|
|
{
|
|
if (kind >= ENTITY(Count))
|
|
{
|
|
return nullptr;
|
|
}
|
|
|
|
const Class* class_ptr = kEntity_type_class_ptr[kind];
|
|
if (class_ptr == nullptr)
|
|
{
|
|
return nullptr;
|
|
}
|
|
|
|
if (class_ptr->CRC != crc)
|
|
{
|
|
return nullptr;
|
|
}
|
|
|
|
return class_ptr;
|
|
}
|
|
|
|
[[nodiscard]] Class* find_class_by_name(String name)
|
|
{
|
|
if (!IsValid(name))
|
|
{
|
|
return nullptr;
|
|
}
|
|
|
|
const uint32 name_crc = crc32(name.Str, name.Size);
|
|
|
|
for (uint8 kind = 0; kind < ENTITY(Count); ++kind)
|
|
{
|
|
const Class* class_ptr = ResolveEntityClass(kind, name_crc);
|
|
if (class_ptr != nullptr)
|
|
{
|
|
return const_cast<Class*>(class_ptr);
|
|
}
|
|
}
|
|
|
|
return nullptr;
|
|
}
|
|
```
|
|
|
|
### 4.4 In-Place Deserialization Algorithm
|
|
|
|
```
|
|
+---------------------------------------------------------------------------------------+
|
|
| In-Place Deserialization Flowchart |
|
|
+---------------------------------------------------------------------------------------+
|
|
| |
|
|
| 1. LoadFile(scratch.Arena, filepath) into ByteBuffer |
|
|
| | |
|
|
| v |
|
|
| 2. tokenize_archive(scratch.Arena, file_buffer, &ar.base) |
|
|
| | |
|
|
| v |
|
|
| 3. Read "; class" & Resolve Class* via find_class_by_name(class_name) |
|
|
| | |
|
|
| v |
|
|
| 4. base_ptr = AllocateEntity(manager, class_ptr) |
|
|
| | |
|
|
| +--> [manager.Entities]: Allocates base Entity |
|
|
| +--> [manager.by_type[kind].arena]: Allocates derived struct |
|
|
| +--> Mutual Back-Pointers Wired In-Place |
|
|
| | |
|
|
| v |
|
|
| 5. serialize(ar, NonNullPtr<Entity>(base_ptr)) |
|
|
| | |
|
|
| +--> Streams Base Entity (Entity::kind, version, ID, position) |
|
|
| +--> Streams Derived Component (base_ptr->derived_kind, class_version) |
|
|
| | |
|
|
| v |
|
|
| 6. Clear Dirty Flag: base_ptr->is_dirty = false |
|
|
| |
|
|
+---------------------------------------------------------------------------------------+
|
|
```
|
|
|
|
```cpp
|
|
[[nodiscard]] Entity* deserialize_entity_in_place(EntityManager& manager, Archive& ar)
|
|
{
|
|
Assert(ar.loading);
|
|
|
|
// 1. Read class name and resolve Class*
|
|
String class_name = {};
|
|
SERIALIZE(ar, class, class_name);
|
|
Class* class_ptr = find_class_by_name(class_name);
|
|
if (!class_ptr)
|
|
{
|
|
return nullptr;
|
|
}
|
|
|
|
// 2. Allocate persistent memory for base and derived components in their respective arenas
|
|
Entity* base_ptr = AllocateEntity(manager, class_ptr);
|
|
Assert(base_ptr != nullptr);
|
|
|
|
// 3. Serialize Base Entity and Derived in-place (loads ID and properties from disk)
|
|
serialize(ar, NonNullPtr<Entity>(base_ptr));
|
|
|
|
// 4. Advance generator counter to avoid collisions with loaded IDs
|
|
if (base_ptr->ID >= EntityManager::ID)
|
|
{
|
|
EntityManager::ID = base_ptr->ID + 1;
|
|
}
|
|
|
|
// Freshly loaded entity matches disk state exactly
|
|
base_ptr->is_dirty = false;
|
|
|
|
return base_ptr;
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Entity Deletion Lifecycle & Disk Synchronization (Extracted for Rework)
|
|
|
|
> [!WARNING]
|
|
> **Status: Extracted for Rework**
|
|
> The original swap-and-pop in-memory removal logic (`DestroyEntity`, `RemoveDerivedComponent`, and mutual back-pointer fixups) was determined to be overly complex and has been extracted to [`Game/Plans/Entity_Removal_Brainstorm.md`](file:///w:/Classified/Juliet/Game/Plans/Entity_Removal_Brainstorm.md) for further brainstorming and redesign.
|
|
>
|
|
> Simpler alternative architectures under consideration include:
|
|
> - **Active / Tombstone Flag (`is_active` bool):** Retaining entities in-place without moving memory during frame simulation, eliminating pointer invalidation entirely.
|
|
> - **Intrusive Free List:** Linking inactive slots via an intrusive linked list to find the first free slot in $O(1)$ without memory shifting.
|
|
> - **Generational Handles / Slot Map:** Enabling safe, non-dangling entity references across systems.
|
|
> - **Deferred Compaction:** Batch-compacting memory during level loads or scene transitions rather than per-frame swap-and-pop.
|
|
|
|
---
|
|
|
|
## 6. Dirty Tracking for Optimal Saves
|
|
|
|
### 6.1 The Cost of Naive Monolithic & Full-Directory Writes
|
|
In a level containing $10{,}000$ entities:
|
|
- **Monolithic `world.bin` Save:** Modifying a single entity's $X$ coordinate requires re-serializing all $10{,}000$ entities and overwriting a multi-megabyte binary file. This introduces a huge Git diff and constant merge conflicts.
|
|
- **Full-Directory `.jasset` Save:** Iterating through all $10{,}000$ entities and unconditionally writing $10{,}000$ `.jasset` files incurs massive OS file-system overhead (directory table locks, I/O bandwidth) and changes the file timestamps of every asset. Git reports thousands of modified files even when only one entity changed!
|
|
|
|
### 6.2 The `is_dirty` Flag on `Entity`
|
|
To solve this, `Entity` in [`Game/Entity/Entity.h`](file:///w:/Classified/Juliet/Game/Entity/Entity.h#L28-L36) is augmented with an explicit dirty flag:
|
|
|
|
```cpp
|
|
struct Entity final
|
|
{
|
|
DECLARE_CLASS() // static Class* kind; (Entity's own Class descriptor)
|
|
|
|
EntityID ID = 0;
|
|
Class* derived_kind = nullptr; // Pointer to derived class descriptor (e.g. Inert::kind)
|
|
DerivedType derived = nullptr; // Pointer to derived component memory
|
|
Vector4 position = {};
|
|
bool is_dirty = false;
|
|
};
|
|
```
|
|
|
|
### 6.3 Granular State Transitions
|
|
The `is_dirty` flag obeys a strict lifecycle state machine:
|
|
|
|
```
|
|
+-----------------------------------+
|
|
| Entity Created |
|
|
| (AllocateEntity / Editor) |
|
|
+-----------------+-----------------+
|
|
|
|
|
v
|
|
+---------------+
|
|
+------->|is_dirty: TRUE |<-------+
|
|
| +-------+-------+ |
|
|
| | |
|
|
Entity Mutated | SaveWorld
|
|
(Position, Component) | Completed
|
|
| v |
|
|
| +---------------+ |
|
|
+--------+is_dirty: FALSE+--------+
|
|
+-------+-------+
|
|
^
|
|
|
|
|
Deserialization
|
|
(LoadWorld / Asset)
|
|
```
|
|
|
|
1. **Entity Creation:** Newly spawned entities in the editor have `is_dirty = true`.
|
|
2. **Property Mutation:** Any modification to `position` or derived component payload sets `entity->is_dirty = true`.
|
|
3. **Successful Deserialization:** Entities loaded from disk initialize with `is_dirty = false`.
|
|
4. **Successful Save:** Upon successfully writing an entity to its `.jasset` file, the engine resets `entity->is_dirty = false`.
|
|
|
|
### 6.4 Version Control Benefits (Git Friendly Assets)
|
|
By coupling the one-file-per-entity `.jasset` format with dirty tracking:
|
|
- Only modified entities are touched on disk.
|
|
- Git status displays only the exact `.jasset` files that were altered by the designer.
|
|
- Team members can work concurrently in the same game scene without encountering binary merge conflicts.
|
|
|
|
### 6.5 Editor Integration (`RenderWorldEditorUI` Hooks)
|
|
In [`Game/Data/World.cpp`](file:///w:/Classified/Juliet/Game/Data/World.cpp#L291-L311), editor UI widgets automatically set the dirty flag upon receiving user input:
|
|
|
|
```cpp
|
|
float pos[4] = { ent.position.x, ent.position.y, ent.position.z, ent.position.w };
|
|
if (ImGui::DragFloat3("Position", pos, 0.1f))
|
|
{
|
|
ent.position.x = pos[0];
|
|
ent.position.y = pos[1];
|
|
ent.position.z = pos[2];
|
|
ent.is_dirty = true; // Mark dirty for persistence
|
|
UpdateWorld(world);
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Step-by-Step Implementation Roadmap
|
|
|
|
### Phase 1: Data Structures & Header Definitions
|
|
1. **Update `Class.h` & `Entity.h`:**
|
|
- Add `using default_init_fct_type = void (*)(void* payload);` and `default_init_fct` to `struct Class` and `MakeClass`.
|
|
- Update `DEFINE_ENTITY_VERSIONED` and `DEFINE_CLASS_VERSIONED` to define `default_init_##entity` and pass it to `MakeClass`.
|
|
- Add `bool is_dirty = false;` to `struct Entity`.
|
|
- Update `MakeEntity<EntityType>` to assign `base_ptr->ID = EntityManager::ID++;` and delegate allocation and defaults cleanly to `AllocateEntity`.
|
|
- Declare `[[nodiscard]] const Class* ResolveEntityClass(uint8 kind, uint32 crc);` and `[[nodiscard]] Class* find_class_by_name(String name);` in `Entity.h`.
|
|
2. **Update `EntityManager.h`:**
|
|
- Declare `[[nodiscard]] Entity* AllocateEntity(EntityManager& manager, Class* class_ptr);`.
|
|
- *(Note: `DestroyEntity` and `RemoveDerivedComponent` deferred to `Entity_Removal_Brainstorm.md`)*
|
|
3. **Update `World.h`:**
|
|
- Add `VectorArena<EntityID, 1024> PendingDeletions;` to `struct World`.
|
|
- Update `SaveWorld` and `LoadWorld` signatures to take directory paths.
|
|
|
|
### Phase 2: Core Memory Allocation & Wiring in `EntityManager.cpp`
|
|
1. Implement `AllocateEntity`:
|
|
- Enforce parameter assertions.
|
|
- Push to `manager.Entities` with `baseTemplate.ID = 0` (unassigned).
|
|
- Allocate block in `manager.by_type[kind].arena`.
|
|
- Call `derivedClassPtr->default_init_fct(rawMemory)` (or `MemZero` if null) to initialize struct defaults.
|
|
- Wire mutual pointers (`base->derived` and `derived->base`).
|
|
- Increment `typedArray.count` and initialize `typedArray.array`.
|
|
2. *(Deferred for rework)* `DestroyEntity` & `RemoveDerivedComponent` (See [`Game/Plans/Entity_Removal_Brainstorm.md`](file:///w:/Classified/Juliet/Game/Plans/Entity_Removal_Brainstorm.md)).
|
|
|
|
### Phase 3: In-Place Deserialization & Serialization Pipeline
|
|
1. In `Entity.cpp`:
|
|
- Implement `ResolveEntityClass` and `find_class_by_name`.
|
|
- `serialize(Archive& ar, NonNullPtr<Entity> entity)` handles both base and derived class serialization.
|
|
2. In `World.cpp`:
|
|
- Implement `serialize_entity_asset(Archive& ar, NonNullPtr<Entity> entity, String filepath)`.
|
|
- Implement `deserialize_entity_asset(EntityManager& manager, Archive& ar, String filepath)`.
|
|
- In `deserialize_entity_in_place`, advance `EntityManager::ID` past `base_ptr->ID` to prevent ID collisions.
|
|
|
|
### Phase 4: World Save/Load Pipeline & Disk Deletion
|
|
1. In `World.cpp`:
|
|
- Implement `ProcessPendingDeletions(World& world, NonNullPtr<Arena> scratchArena)`.
|
|
- Implement `SaveWorld(World& world, String worldDirectory)`:
|
|
- Process pending deletions.
|
|
- Iterate `manager.Entities`, skipping entities where `!entity.is_dirty`.
|
|
- Write dirty entities to `.jasset` files and clear `is_dirty`.
|
|
- Implement `LoadWorld(World& world, String worldDirectory)`:
|
|
- Enumerate `.jasset` files in directory.
|
|
- Call `deserialize_entity_asset` for each file.
|
|
|
|
### Phase 5: Editor Integration
|
|
1. In `RenderWorldEditorUI`:
|
|
- Hook `ImGui::DragFloat3` and property inspectors to set `is_dirty = true`.
|
|
- Hook "Add Entity" button to call `MakeEntity<Inert>(*world.EntityManager, 0.0f, 0.0f, 0.0f)`.
|
|
- Hook "Delete Entity" button to call `RemoveWorldEntity(world, selectedEntityId)`.
|
|
|
|
---
|
|
|
|
## 8. Unit Testing & Verification Plan
|
|
|
|
### 8.1 Test Philosophy & Constraints
|
|
Following Juliet coding guidelines:
|
|
> "When creating a new system framework, make a unit test. To make the unit test we should not modify the framework code for special unit test case."
|
|
|
|
Testing is isolated in [`Game/UnitTest/WorldUnitTest.cpp`](file:///w:/Classified/Juliet/Game/UnitTest/WorldUnitTest.cpp) and executed during engine initialization in debug builds.
|
|
|
|
### 8.2 Comprehensive Test Suite (`WorldUnitTest.cpp`)
|
|
The test suite validates every guarantee made in this specification:
|
|
|
|
```cpp
|
|
#include <UnitTest/WorldUnitTest.h>
|
|
|
|
#if JULIET_DEBUG
|
|
|
|
#include <Core/Common/CoreUtils.h>
|
|
#include <Core/Common/serialization.h>
|
|
#include <Core/HAL/Filesystem/Filesystem.h>
|
|
#include <Core/HAL/IO/IOStream.h>
|
|
#include <Core/Logging/LogManager.h>
|
|
#include <Core/Logging/LogTypes.h>
|
|
#include <Core/Memory/MemoryArena.h>
|
|
#include <Core/Thread/ThreadContext.h>
|
|
#include <Data/World.h>
|
|
#include <Entity/Entity.h>
|
|
#include <Entity/EntityManager.h>
|
|
|
|
namespace UnitTest
|
|
{
|
|
namespace
|
|
{
|
|
void TestEntityAllocationAndWiring()
|
|
{
|
|
Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestEntityAllocationAndWiring...");
|
|
|
|
TempArena tempArena = scratch_begin(nullptr, 0);
|
|
|
|
World testWorld{};
|
|
InitWorld(&testWorld, tempArena.Arena);
|
|
|
|
InitEntityManager(&testWorld);
|
|
EntityManager& manager = *testWorld.EntityManager;
|
|
|
|
// 1. Allocate Inert Entity via AllocateEntity
|
|
Entity* base_entity = AllocateEntity(manager, Inert::kind);
|
|
Assert(base_entity != nullptr);
|
|
Assert(base_entity->ID == 0); // Pure memory allocator leaves ID unassigned (0) until MakeEntity or deserialization
|
|
base_entity->ID = EntityManager::ID++;
|
|
Assert(base_entity->ID > 0);
|
|
Assert(base_entity->derived_kind == Inert::kind);
|
|
Assert(base_entity->derived != nullptr);
|
|
Assert(base_entity->is_dirty == true);
|
|
|
|
// 2. Validate mutual back-pointer wiring
|
|
auto* derived = reinterpret_cast<entity_template*>(base_entity->derived);
|
|
Assert(derived->base == base_entity);
|
|
|
|
// 3. DownCast verification
|
|
Inert* inert = DownCast<Inert>(base_entity);
|
|
Assert(inert != nullptr);
|
|
Assert(inert->base == base_entity);
|
|
|
|
// 4. Validate typed array tracking
|
|
typed_entity_array& inert_array = manager.by_type[ENTITY(Inert)];
|
|
Assert(inert_array.count == 1);
|
|
Assert(inert_array.array == derived);
|
|
|
|
ShutdownEntityManager();
|
|
ShutdownWorld(&testWorld);
|
|
|
|
scratch_end(tempArena);
|
|
Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestEntityAllocationAndWiring");
|
|
}
|
|
|
|
void TestInPlaceDeserialization()
|
|
{
|
|
Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestInPlaceDeserialization...");
|
|
|
|
TempArena tempArena = scratch_begin(nullptr, 0);
|
|
|
|
World testWorld{};
|
|
InitWorld(&testWorld, tempArena.Arena);
|
|
InitEntityManager(&testWorld);
|
|
EntityManager& manager = *testWorld.EntityManager;
|
|
|
|
// 1. Create and populate entity
|
|
Inert* created_inert = MakeEntity<Inert>(manager, 12.5f, -44.0f, 108.2f);
|
|
Assert(created_inert != nullptr);
|
|
created_inert->MeshInstance = 42;
|
|
|
|
Entity* original_base = created_inert->base;
|
|
EntityID original_id = original_base->ID;
|
|
|
|
// 2. Serialize to text archive memory stream
|
|
MemoryStream mem_stream = MakeMemoryStream(tempArena.Arena);
|
|
Archive save_ar{ .arena = tempArena.Arena, .loading = false, .stream = &mem_stream };
|
|
serialize(save_ar, NonNullPtr<Entity>(original_base));
|
|
|
|
// 3. Clear manager to simulate fresh load
|
|
ShutdownEntityManager();
|
|
InitEntityManager(&testWorld);
|
|
EntityManager& fresh_manager = *testWorld.EntityManager;
|
|
|
|
// 4. Tokenize and deserialize in-place
|
|
Archive load_ar{ .arena = tempArena.Arena, .loading = true };
|
|
tokenize_archive(tempArena.Arena, mem_stream.buffer, &load_ar.base);
|
|
Entity* loaded_base = deserialize_entity_in_place(fresh_manager, load_ar);
|
|
|
|
Assert(loaded_base != nullptr);
|
|
Assert(loaded_base->ID == original_id);
|
|
Assert(EntityManager::ID > original_id); // Counter was advanced past loaded ID to prevent collisions
|
|
Assert(loaded_base->position.x == 12.5f);
|
|
Assert(loaded_base->position.y == -44.0f);
|
|
Assert(loaded_base->position.z == 108.2f);
|
|
Assert(loaded_base->is_dirty == false);
|
|
|
|
Inert* loaded_inert = DownCast<Inert>(loaded_base);
|
|
Assert(loaded_inert != nullptr);
|
|
Assert(loaded_inert->base == loaded_base);
|
|
Assert(loaded_inert->MeshInstance == 42);
|
|
|
|
ShutdownEntityManager();
|
|
ShutdownWorld(&testWorld);
|
|
|
|
scratch_end(tempArena);
|
|
Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestInPlaceDeserialization");
|
|
}
|
|
|
|
void TestDirtyTrackingLifecycle()
|
|
{
|
|
Log(LogLevel::Message, LogCategory::Game, "[UnitTest] Running TestDirtyTrackingLifecycle...");
|
|
|
|
TempArena tempArena = scratch_begin(nullptr, 0);
|
|
|
|
World testWorld{};
|
|
InitWorld(&testWorld, tempArena.Arena);
|
|
InitEntityManager(&testWorld);
|
|
EntityManager& manager = *testWorld.EntityManager;
|
|
|
|
Entity* entity = AllocateEntity(manager, Inert::kind);
|
|
Assert(entity->is_dirty == true);
|
|
|
|
// Simulate save
|
|
entity->is_dirty = false;
|
|
Assert(entity->is_dirty == false);
|
|
|
|
// Simulate mutation
|
|
entity->position.x += 1.0f;
|
|
entity->is_dirty = true;
|
|
Assert(entity->is_dirty == true);
|
|
|
|
ShutdownEntityManager();
|
|
ShutdownWorld(&testWorld);
|
|
|
|
scratch_end(tempArena);
|
|
Log(LogLevel::Message, LogCategory::Game, "[UnitTest] PASSED: TestDirtyTrackingLifecycle");
|
|
}
|
|
} // namespace
|
|
|
|
void WorldUnitTest()
|
|
{
|
|
Log(LogLevel::Message, LogCategory::Game, "==================================================");
|
|
Log(LogLevel::Message, LogCategory::Game, "Starting Entity Allocation & Lifecycle Unit Tests");
|
|
Log(LogLevel::Message, LogCategory::Game, "==================================================");
|
|
|
|
TestEntityAllocationAndWiring();
|
|
TestInPlaceDeserialization();
|
|
// TestSwapAndPopPointerFixup(); // Deferred to Entity_Removal_Brainstorm.md
|
|
TestDirtyTrackingLifecycle();
|
|
|
|
Log(LogLevel::Message, LogCategory::Game, "==================================================");
|
|
Log(LogLevel::Message, LogCategory::Game, "All Entity Lifecycle Unit Tests PASSED Successfully");
|
|
Log(LogLevel::Message, LogCategory::Game, "==================================================");
|
|
}
|
|
} // namespace UnitTest
|
|
|
|
#endif
|