Little by little engine becomes usable after various changes (as usual it need to be completely rewritten after reading about some interesting idea :)). Last ideas were concurrency, memory debugging,fragmentation and reference counting. Anyway, it improves engine, makes it safer and usable.

Screen from “Deadly light” demo:

It’s very important to display debugging information about your level or models during developing game level. Previously, I used 3ds max to place all objects, including lights and it was not easy task. Tweak lighting in 3ds max, switch to engine, check lighting etc…
Now several debugging modes are supported in Glow engine - wire-frame mode, displaying normals in world space, depth of pixel, lights, physics, AI (triggers), navigation meshes.
All modes can be used same time (except normals, depth and diffuse color), button “layers” should be pressed and mode selected.
Room from “Deadly Light” demo, lighted with 6 lights:

Same room in wire-frame debug mode:

Normals mode:

Lights debugger:


Physics:
Depth:

www.glow3d.com is updated to reflect all recent changes.

Embedding of Intel Threading Building Blocks is finished, really good library with many abilities.
Development of a concurrent scene graph is started, scene graph (sectors - portals) was made templated, to hold different types of objects and it supports now read-only access from several threads (or tasks). Also several modules of engine have copy of the scene graph - renderer holds scene graph with visible objects, physics - with physical to optimize simulation.

deeper and deeper into concurrency and multi threading…
I have decision to use Intel Threading Building Blocks instead of experimental boost::thread_pool. All computations in the Glow engine are organized into tasks and theoretically would scale well when number of hardware threads (CPU cores) will be increased.
Previously, I used boost::thread_pool for task execution. Now, when I returned to concurrency, I realized that usage of boost::thread_pool is not scalable, because it heavily uses mutexes (spin-based). Later I will launch tests to measure scalability of both methods and post results here.

I am thinking about concurrent scene graph.
The are several modules in the engine, which needs queries to the scene graph, like Physics, Audio, Render. Moreover, in the future, several threads (tasks) inside of module will need concurrent access to the scene graph.
There several options how to do it:
1) copies of scene graph in every module (used also in OpenSceneGraph engine)- needs more memory, but should hold different types of objects - Physics module will hold only physics objects etc. Still it does not resolve problem with concurrent access in the future
2) concurrent containers in the scene graph - maybe the best solution, but currently I can not find best fit for it. Also looks like it hardware dependent and not robust enought. There is Intel Thread Building Blocks library with concurrent containers, but it’s not perfect and doesn’t support Android and iOS.
3) mutexes to isolate queries. Usage of mutex is not good thing for scalable multi threading, but maybe for some usage patterns it’s ok. In my case it’s SINGLE PRODUCER MULTIPLY CONSUMERS model.

Can anyone suggest robust cross platform concurrent containers (hash_map or vector at least) ?
excellent blog http://www.1024cores.net about concurrency and scalability.
UPD: I was wrong about Intel TBB, it’s cool and powerful library. It’s not clear now for me, how to use it under iOS or Android, but it’s Open Source and ports are available for different platforms.

High Dynamic Range lighting with eye adaptation is implemented in the Glow engine. Plus gamma-correct rendering. Plus Color Transformations (hue, saturation, brightness etc) were started.

Memory Management in the Glow engine

Memory management is very important thing in a console world. We can forget about it on a PC (at least for several hours without leaks, but consoles are restricted, so we should consider that after some time, allocation of memory will return NULL (or exception, depending on compiler). Also fragmentation is serious problem after allocation of many small objects. So with the plans to move on mobile devices, we decided to develop system for memory management.

New memory manager is implemented for Glow engine. Separate optimized singlethreaded heaps with pools for small objects for every module of the engine (physics, ai, navigation, sounds etc) are added instead of the single common C++ heap.

Special heap for transferring messages between modules, memory leak debugging, shared pointers (boost c++ library) for memory arrays.

Allocator for STL containers was implemented, so every memory allocation in the specific module goes from the module heap. STL containers and strings are evil force of a memory fragmentation, so process for removing every STL container from code is started. Some vectors are replaced by special Stack vector container, based on boost::array with restricted dynamic grow. Also major part of constant std::string was replaced with const char* equivalent (with memory allocated from constant string pool) and "copy on create if exists" usage.

Memory debugger UI, with rendering of memory fragmentation, information about allocated blocks - count, amount, overhead.

Remaining part for the memory manager implementation - garbage collector for Geometry.

January release date was very optimistic, so we moved release of alpha version to Spring, 2011. Major parts are ready, but there are many small task to polish editor and engine.

I am working currently on memory management for the engine, done a lot..later about it in details.. now about arrays. Everybody knows, that dynamic allocation is bad thing, especially in optimized applications. Fragmentations, locks, time to find free block etc. General advice is to avoid any kind of dynamic allocation in update loop. So I decided to remove it everywhere and move all dynamic data to the stack. There were many std::vectors, std::strings in the source, around 700 cases of using std::string, same for vector. Firstly I changed vector into C arrays, but it’s not convenient to use. After that I found boost::array, but it’s not dynamic. So my decision was to create mix of boost::array and std::vector, it’s based on sources of boost::array, but has ability to dynamically grew (one additional variable m_size). Useful and looks optimal. UPD. maybe not so optimal for complex objects with not simple constructors and also for big object, but optimal for pointers and simple types.

I visited SIGGRAPH Asia 2010, it was interesting. Several courses was attended, number of interesting people was met. Tessellation is still hot theme, it was discussed heavily in DirectX 11 and OpenGL 4.0 courses. Also Realtime Ray Tracing and stereoscopic rendering.