Crash: infinite recursion in debug stack dump (Win 7/64)

[historic_bruno]

OS: Win 7 64-bit
Build: VS 2010 Pro - release build

While testing #1812 I encountered an assertion failure, but there was a hard crash before the error dialog was displayed:

Simulation2.cpp(329): Assertion failed: "0 && (L"Serialization test failure")"
First-chance exception at 0x522d7c77 in pyrogenesis.exe: 0xC00000FD: Stack overflow.
INVALID_UDT boost::unordered::detail::compressed<std::allocator<boost::unordered::detail::ptr_bucket>,std::allocator<boost::unordered::detail::ptr_node<boost::shared_ptr<CTexture> > > > 1 1
First-chance exception at 0x52322203 in pyrogenesis.exe: 0xC0000005: Access violation writing location 0x02e00de8.
Unhandled exception at 0x52322203 in pyrogenesis.exe: 0xC0000005: Access violation writing location 0x02e00de8.

Here's the call stack for all threads at the time of the crash: ​http://pastebin.com/Kc3aLKtP

I don't know what could cause this, but we currently have a bug where a thread is running constantly and frequently accessing timer_Time(), if that's relevant. This is not consistently reproducible but these steps work for me with r13045:

• Start the game in Atlas with serialization test mode: -editor -serializationtest
• Load "Acropolis 01" map
• Start simulation test
• Select a soldier and task him to build a house, triggering the serialization test failure, and sometimes the crash

[historic_bruno]

Still get this crash, less frequently, though the sound manager threading is supposedly fixed.

[historic_bruno]

As suggested by Philip, commenting out the stack dump part of debug_BuildErrorMessage() prevents the crash.

[historic_bruno]

Looked into this a bit more, it's definitely some Boost types giving the stack dump trouble. Here's an example:

m_TextureCache = 
	table_ = 

		...

	 buckets_ = 0x1F749910 -> 
		 next_ = 0x0948FCC8 -> 
			 next_ = 0x1F7FF210 -> 
		  next_ = 0x0948FC80 -> 
			  next_ = 0x1D54AA20 -> 
				  next_ = 0x1D4FDAE8 -> 
			   next_ = 0x094DD1E0 -> 
				   next_ = 0x1FF57A50 -> 
				next_ = 0x1D69C0D0 -> 
					next_ = 0x0948FBF0 ->  ...

It's not infinite recursion (I think) but a very very deep structure.

Decreasing maxIndirection to more like 128 prevents the stack overflow in this case, with VC++ 2010 release build. The problem is that also generates a bunch of warnings about "Symbol nesting too deep or infinite recursion", so maybe we can return the error code without warning (as a temporary fix).

[Jan Wassenberg]

This makes a lot of sense. It's trying to display a large data structure. It would be more elegant to provide a special case for that data type to avoid printing so much spam (perhaps just the number of entries). However, decreasing the max limit and avoiding warning when it is reached sounds very reasonable.

[historic_bruno]

Thanks for the quick response, Jan! :) I didn't notice we already had special cases of some types, so I will simply add one for boost::unordered as well. That's indeed a more elegant solution.

[ben]

In 13724:

Fixes stack overflow in stack dump logic, by special-casing boost::unordered types, fixes #1813.
Adds INIT_HAVE_DISPLAY_ERROR init flag to correctly override ah_display_error in Atlas (InitVfs was overriding it again), but don't use it because it's ugly and broken, refs #1729