Consider disabling frame-pointer omission in VC++ 2008 projects (breaks stack walker)

[historic_bruno]

RedFox and I were trying to debug problems with the stack walker in release builds. In VC++ 2008 which the autobuilder uses, StackWalk64 would fail on the first call, but in VC++ 2010 or 2012 it worked perfectly for me.

This is because in MSVC, ​frame-pointer omission is an implied option with some optimizations enabled, unless it's explicitly disabled. We do explicitly disable FPO on *nix via -fno-omit-frame-pointer, but not yet on Windows.

Here's the interesting part, apparently VC++ handling of FPO differs between versions. In VC++ 2008. if OmitFramePointers isn't specified in the project, it defaults to no but it may be implied. However in VC++ 2010, if OmitFramePointers isn't specified, it defaults to no and /Oy- appears on the command line, explicitly disabling it. Premake doesn't have an option to disable FPO, so we'll have to patch Premake or add the /Oy- option in premake4.lua. I think it's worth losing whatever value this optimization has, for a working stack dump in release builds.

[Jorma Rebane]

I think it's not simply a problem of FP omission. Even with default build settings /Oy- had no effect and stack dump still failed. By default Frame Pointers are generated (unless the function is inlined).

The error really becomes apparent when we do Whole Program Optimization, which causes link-time code generation and inlines functions across modules.

The stack walking code in Windows builds has a bug that makes the stack walk fail if the topmost function is inlined. In ticked #1995 I addressed this isssue by simply not filtering the stack walk and just dump all.

I think we need both /Oy- and fixed stack walk.

[historic_bruno]

Your bug isn't simply FPO, but you've added custom optimizations to your build that no one else uses :) I agree that should also be fixed properly, but disabling FPO is definitely good enough to fix VC++ 2008 projects with default options (and thus the autobuild).

[Jorma Rebane]

At any rate, /Oy- must be implied. We can use the "workaround" in #1995 to get around the edge cases where cross-linkage inlining breaks the stack walker.

Can you make the necessary changes in premake for /Oy- ?

[ben]

In 13779:

Disables frame-pointer omission in VC++ 2008 projects, fixes stack dump in release build, fixes #2078

[historic_bruno]

It may be worth adding a flag to Premake for this, it already has NoFramePointer that enables FPO, but no flag to explicitly disable it. Adding /Oy- for VC++ 2008 was good enough for A14 without risking breakage.

I suspect the problem with whole program optimization is something else, maybe inlining as you and Philip mentioned. I tried with that option and /Oy- had no effect, I also tried a newer version of dbghelp.dll with no success.