[PERFORMANCE] Memory allocation reduction

[Jorma Rebane]

Using MTuner data from a short run of the game, we can drastically improve 0 A.D. core engine performance, by reducing the number of allocations in the detected hotspots.

For more information, check the Memory Performance thread on the forums: http://www.wildfiregames.com/forum/index.php?showtopic=17310&st=20#entry270448

Sub-tickets: modeldef.cpp: #2006 parser.cpp: #2005 guimanager.cpp: #2007 textrenderer.cpp: #2027 shaderdefines.cpp: #2029

<progress>, <memblock>, <total memops>, <stacktrace>
(V),		size 4, 	count 61320, 	modeldef.cpp:300
(V),		size 8, 	count 61320, 	modeldef.cpp:301
(V),		size 48,	count 42028, 	parser.cpp:246
(V),		size 36,	count 40866, 	guimanager.cpp:273 & 265
(V),		size 8, 	count 32749, 	modeldef.cpp:241
(V),		size 32,	count 27638, 	textrenderer.cpp:136
(V),		size 8,		count 21952, 	cgui.cpp:986
(V),		size 16,	count 21952, 	textrenderer.cpp:77
(V),		size 16, 	count 21952, 	shaderdefines.cpp:135
(V),		size 48,	count 21952, 	cgui.cpp:954
(V),		size 12,	count 20688, 	modelrenderer.cpp:635
(V),		size 112,	count 17325, 	guirenderer.cpp:388
( ),		size 108,	count 16896,	patchrdata.cpp:173
(V),		size 8,		count 14852, 	shaderdefines.cpp:116
(x),		size 48,	count 14154, 	font.cpp:33
( ),		size 36,	count 13230, 	patchrdata.cpp:1025
(V),		size 48,	count 12008, 	parser.cpp:981
( ),		size 24, 	count 9783, 	patchrdata.cpp:1025
(V),		size 8,		count 9645, 	modelrenderer.cpp:433
(V),		size 8,		count 8935, 	modelrenderer.cpp:698
( ),		size 48,	count 8265, 	cgui.cpp:916
(V),		size 48,	count 8144, 	parser.cpp:933
( ),		size 93,	count 7806, 	guitext.cpp:266
( ),		size 132, 	count 7806, 	cgui.cpp:689
( ),		size 132,	count 7806, 	cgui.cpp:800 & 855
(V),		size 12,	count 7072, 	modelrenderer.cpp:636
(V),		,		,		textrenderer.cpp:77
(V),		,		,		textrenderer.cpp:179
( ),		,		,		patchrdata.cpp:1052
( ),		,		,		guitext.cpp:210
( ),		,		,		terraintextureentry.h:75
( ),		,		,		patchrdata.cpp:192
( ),		,		,		cgui.cpp:675
( ),		,		,		texturemanager.cpp:150
( ),		,		,		texturemanager.cpp:153
( ),		,		,		texturemanager.cpp:511
( ),		,		,		texturemanager.h:136
(V),		,		,		parser.cpp:639
( ),		,		,		componentmanager.cpp:620

[Jorma Rebane]

CModelDef fixes uploaded. This patch addresses two problems: 1) CModelDef loading performance

In order to make loading faster, had to change the format of cached binary model data. The new model cache files take a tiny bit more disk space, but in return they load several times faster and the loaded data takes less memory.

2) Complex dynamic objects in structs

One of the main perf hitters was a per-vertex (yes, per each vertex!) std::vector<float> for multiple UV's. This was replaced with 2x CVector2D UV0 & UV1. Another one was an std::string for model prop points, which was replaced with a fixed size buffer m_Name[20] and with an additional m_NameLength.

The result: much better memory performance and usage for model loading.

In order for this to work, I had to change the version of the PMD (binary model cache) format to 4 and change the way models are saved to PMD. This may invalidate any existing model cache. Backwards compatibility for versions 1, 2, 3 is there.

[Jorma Rebane]

CParser fix finished. Check ticket: #2005

[Jorma Rebane]

Updated: CTextRenderer patch added.

[Jorma Rebane]

CTextRenderer patch also fixes cgui.cpp:986

[Jorma Rebane]

Added ShaderDefines fix. Check ticket #2029

[wraitii]

Patch applied cleanly with svn patch. Compiled fine with Xcode for OS X (10.8.4) but for "graphics". I get the following errors:

-in ShaderManager.cpp, line 87, "Cannot pass object of non-POD type 'ShaderDefStr' through variadic method; call will abort at runtime"

-in ModelRenderer.cpp, there are several calls to "_alloca" (namely l501/506/510/580), my compiler expects "alloca"

Starting the game worked fine. I could load a map after recompiling Collada and clearing the cache, but I get:

"HWLightingModelRenderer.cpp(237): Assertion failed: "streamflags == (streamflags & (STREAM_POS | STREAM_UV0 | STREAM_UV1 | STREAM_NORMAL))" and a MEM_BAD_ACCESS error in the renderer. Any idea what might cause this? Mistake on my side?

[wraitii]

Latest patch fixed the compiler errors, but I get the same crash, here's the stacktrace. There's an assertion failure in AssertPointerBounds() in HWLightingModelRenderer.cpp at line 284 first.

* thread #1: tid = 0x1d07, 0x0000000107185c9d GLEngine`gleRunVertexSubmitImmediate + 5021, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)

  frame #0  : 0x0000000107185c9d GLEngine`gleRunVertexSubmitImmediate + 5021

  frame #1  : 0x0000000107178d67 GLEngine`gleDrawArraysOrElements_ExecCore + 1079

  frame #2  : 0x0000000107171b33 GLEngine`glDrawRangeElements_IMM_Exec + 350

  frame #3  : 0x00007fff8eec359f libGL.dylib`glDrawRangeElements + 45

  frame #4  : 0x000000010050a218 pyrogenesis`ShaderModelVertexRenderer::RenderModel  () + 376 at HWLightingModelRenderer.cpp:301

  frame #5  : 0x000000010050dd3c pyrogenesis`ShaderModelRenderer::Render(this=0x0000000101aa2e30, modifier=0x0000000101a78d80, context=0x0000000000000001, flags=1) + 3580 at ModelRenderer.cpp:634

  frame #6  : 0x0000000100532009 pyrogenesis`CRendererInternals::CallModelRenderers  (this=0x00000001015d4c00, context=0x00000001008a0438, flags=1) + 329 at Renderer.cpp:364

  frame #7  : 0x000000010052a6ef pyrogenesis`CRenderer::RenderShadowMap  (this=0x00000001015c8c00) + 815 at Renderer.cpp:931

  frame #8  : 0x000000010052e115 pyrogenesis`CRenderer::RenderSubmissions  (this=0x00000001015c8c00) + 1157 at Renderer.cpp:1506

  frame #9  : 0x000000010052fd96 pyrogenesis`CRenderer::RenderScene  (this=0x00000001015c8c00) + 134 at Renderer.cpp:1818

  frame #10  : 0x0000000100492af6 pyrogenesis`CGameView::Render() + 38 at GameView.cpp:503

  frame #11  : 0x000000010042a865 pyrogenesis`Render() + 421 at GameSetup.cpp:218

  frame #12  : 0x0000000100424fbe pyrogenesis`CGame::ReallyStartGame  () + 254 at Game.cpp:216

  frame #13  : 0x0000000100002766 pyrogenesis`SDL_main [inlined] ProgressiveLoad  () + 312 at main.cpp:251

  frame #14  : 0x000000010000262e pyrogenesis`SDL_main [inlined] Frame() + 1035 at main.cpp:349

  frame #15  : 0x0000000100002223 pyrogenesis`SDL_main [inlined] RunGameOrAtlas  () + 459 at main.cpp:510

  frame #16  : 0x0000000100002058 pyrogenesis`SDL_main() + 1288 at main.cpp:553

  frame #17  : 0x000000010029eaa1 pyrogenesis`-[SDLMain applicationDidFinishLaunching:] + 97

  frame #18  : 0x00007fff9006eeda CoreFoundation  `_CFXNotificationPost + 2554

  frame #19  : 0x00007fff8b6457b6 Foundation`-[NSNotificationCenter postNotificationName:object:userInfo  :] + 64

  frame #20  : 0x00007fff87e3052d AppKit  `-[NSApplication _postDidFinishNotification] + 292

  frame #21  : 0x00007fff87e30266 AppKit  `-[NSApplication _sendFinishLaunchingNotification] + 216

  frame #22  : 0x00007fff87e2d452 AppKit  `-[NSApplication(NSAppleEventHandling) _handleAEOpenEvent:] + 566

  frame #23  : 0x00007fff87e2d04c AppKit  `-[NSApplication(NSAppleEventHandling) _handleCoreEvent:withReplyEvent  :] + 351

  frame #24  : 0x00007fff8b65f07b Foundation`-[NSAppleEventManager dispatchRawAppleEvent:withRawReply:handlerRefCon  :] + 308

  frame #25  : 0x00007fff8b65eedd Foundation`_NSAppleEventManagerGenericHandler + 106

  frame #26  : 0x00007fff8ff30078 AE`aeDispatchAppleEvent(AEDesc const*, AEDesc*, unsigned int, unsigned char*) + 307

  frame #27  : 0x00007fff8ff2fed9 AE`dispatchEventAndSendReply(AEDesc const*, AEDesc*) + 37

  frame #28  : 0x00007fff8ff2fd99 AE`aeProcessAppleEvent + 318

  frame #29  : 0x00007fff89461709 HIToolbox`AEProcessAppleEvent + 100

  frame #30  : 0x00007fff87e29836 AppKit  `_DPSNextEvent + 1456

  frame #31  : 0x00007fff87e28df2 AppKit  `-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue  :] + 128

  frame #32  : 0x00007fff87e201a3 AppKit  `-[NSApplication run] + 517

  frame #33  : 0x000000010029e884 pyrogenesis`CustomApplicationMain   + 548

  frame #34  : 0x000000010029ee60 pyrogenesis`main + 272

  frame #35  : 0x00007fff86f3c7e1 libdyld.dylib`start + 1

[lsdh]

For me too the last patch solved the compiler (G++) errors: ​http://pastebin.com/QrGxmZjH

But game crashed at the end of map loading: ​http://pastebin.com/JiawfMKN

(both pastebin link available 1 week).

[Kieran P]

RedFox: Great work so far! But please don't commit this without thorough testing by many other parties. It's still considered WIP/unstable until all developers have tested this on their machines (compiles and plays multiplayer game through without issues).

[Jorma Rebane]

k776: That is only logical, k776. We will be doing a lot of testing on this before it is committed. :)

[Yves]

When running the tests I get this:

./test
Running 277 tests.......................................................................................utf8.cpp(84): Function call failed: return value was -100703 (Invalid UTF-8 sequence)
Function call failed: return value was -100703 (Invalid UTF-8 sequence)
Location: utf8.cpp:84 (RaiseError)

Call stack:

(0x8f470a) ./test() [0x8f470a]
(0x8a8201) ./test() [0x8a8201]
(0x8a83fd) ./test() [0x8a83fd]
(0x8a8dbb) ./test() [0x8a8dbb]
(0x8a6d41) ./test() [0x8a6d41]
(0x67810c) ./test() [0x67810c]
(0x49bb87) ./test() [0x49bb87]
(0x4329a4) ./test() [0x4329a4]
(0x43b748) ./test() [0x43b748]
(0x41c7cd) ./test() [0x41c7cd]
(0x7f456d58fea5) /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f456d58fea5]
(0x42d2e5) ./test() [0x42d2e5]

errno = 0 (No error reported here)
OS error = ?

And when starting the game it displays weird graphic fragments (nothing of the main menu is visible except the cursor).

When starting in autostart mode to avoid the buggy main-menu I get these assertion failures:

First:

Assertion failed: "streamflags == (streamflags & (STREAM_POS | STREAM_UV0 | STREAM_UV1 | STREAM_NORMAL))"
Location: HWLightingModelRenderer.cpp:237 (BeginPass)

If I click continue:

Assertion failed: "(m_StreamFlags & ~m_ValidStreams) == 0" Location: ShaderProgram.cpp:896 (AssertPointersBound)

I was using this commands:

./pyrogenesis -quickstart -autostart="Oasis 04" -autostart-ai=1:qbot -autostart-ai=2:qbot -autostart-ai=3:qbot -autostart-ai=4:qbot

EDIT: I figured out that the main menu displays correctly when I set preferglsl = "flase" and gentangents = "false" in local.cfg.

[historic_bruno]

With the latest patch there is a strange bug in Atlas. I placed an Iberian fortress (left) but as soon as the next placement preview appears (right), the textures become corrupted:

I can keep placing messed up looking buildings, and only after the preview entity is no longer visible do they return to normal. Sometimes the bug is more subtle, the texture appears darker than normal. It happens with other buildings too.

(Reproducible on my system with all graphics options enabled)

[Kieran P]

Got this when running the game. It then failed to draw anything on the screen.

ERROR: Failed to compile shader 'shaders/glsl/gui_text.vs':
ERROR: 0:2: '' :  #version must occur before any other statement in the program

ERROR: Failed to compile shader 'shaders/glsl/gui_basic.vs':
ERROR: 0:2: '' :  #version must occur before any other statement in the program

ERROR: Failed to compile shader 'shaders/glsl/gui_solid.vs':
ERROR: 0:2: '' :  #version must occur before any other statement in the program

Edit: Fixed. historic_bruno pointed out that both patches needed to be applied. I didn't apply the GLSL one, thought it was optional.

[lsdh]

Here is the log I have under Ubuntu 13.04 64bit: ​http://pastebin.com/KiZe59q5

A pyrogenesis bin is generated but crash directly.

[wraitii]

You need to apply the Glsl patch too, otherwise it crashes on startup.

[lsdh]

Replying to wraitii:

You need to apply the Glsl patch too, otherwise it crashes on startup.

I applied the megapatch on /source and the glsl on /binaries

Maybe I dit it wrong. I clean all and test again.

[edition] So I cleaned all and compiled again (last svn version at the moment: 13719).

glsl patch apply without any errors.
megapatch apply with one propriety error: ​http://pastebin.com/cCTMScz9

Make logs are a bit cleaner: ​http://pastebin.com/YTBVuxGE

But the game still crash. Here is the error:

ERROR: CCacheLoader failed to find archived or source file for: "shaders/fixed:gui_text.xml"</p>
ERROR: Failed to load shader 'fixed:gui_text'</p>

[Edit] I did not have "preferglsl=true". Now that I have it, it does not crash, but I have a strange picture: ​http://img15.hostingpics.net/pics/389942screenshot0010.png

Maybe a problem with my graphic card.

[Yves]

Overwriting config settings in local.cfg seems to be broken with the patch. It works with the same unmodified revision.

EDIT: That wasn't quite accurate. Setting Windowed to true in local.cfg works, but setting hotkey.fps.toggle = "AltGr+F" or hotkey.camera.reset = "P" doesn't work.

[Jorma Rebane]

Latest megapatch update. 1 known bugs: 1) zip archiver fails?

[historic_bruno]

An attempt at separating megapatch has begun here: ​http://git.wildfiregames.com/gitweb/?p=0ad.git;a=shortlog;h=refs/heads/projects/philip/megapatch-split

Any useful parts can be pulled out, reviewed, and if necessary new tickets created for them. Most of this ticket is no longer applicable.