[PATCH] Adjust buildscripts and test SpiderMonkey 24 on Mac OS X

[Yves]

I don't own a Mac, so someone else needs to take over this task. You basically have to:

1. Apply the most recent patch in #1886
2. Copy the SpiderMonkey source tarbal to libraries/source/spidermonkey (download: ​https://ftp.mozilla.org/pub/mozilla.org/js/mozjs-24.2.0.tar.bz2)
3. Adjust libraries/osx/build-osx-libs.sh (the changes should be more or less the same as for Linux in libraries/source/spidermonkey/build.sh).
4. Build the library using the build script and build 0 A.D.
5. Check if the 0 A.D. build works or if there are any warnings or errors that are specific to the Mac OS X environment.

The following warnings are currently a know issue that probably also happens on the Mac (ignore them):

mozilla/NullPtr.h|29|warning: "_MSC_VER" is not defined [-Wundef]|.

It would be good to have this done as soon as possible. I would like to commit long enough before the next alpha release to get enough testing. I should be ready with the rest of the SpiderMonkey patch in about a week.

[Tom Brewe]

I tried to apply the patch and compile spidermonkey. After a few problems it seemed that it is necessary to use clang and clang++ instead of gcc and g++. As spidermonkey needs gcc 4.4 and Apple only delivers 4.2 in Mac OS 10.9.1. So I changed that temporarily. However, flags hat to be removed because clang doesnt support them. ( --enable-debug --disable-optimize --disable-tests --disable-shared-js )

After that, there still is an issue, which I am not sure how to circumvent other than compiling gcc 4.4:

~/Install/0ad/libraries/source/spidermonkey/mozjs24/js/src/build-debug ~/Install/0ad/libraries/source/spidermonkey/mozjs24/js/src ~/Install/0ad/libraries/source/spidermonkey ~/Install/0ad/libraries/osx
Apple LLVM version 4.2 (clang-425.0.28) (based on LLVM 3.2svn)
Target: x86_64-apple-darwin13.0.0
Thread model: posix
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -dynamic -arch x86_64 -macosx_version_min 10.9.0 -syslibroot / -o a.out ../configure -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/4.2/lib/darwin/libclang_rt.osx.a
ld: warning: ignoring file ../configure, file was built for unsupported file format ( 0x23 0x21 0x20 0x2f 0x62 0x69 0x6e 0x2f 0x73 0x68 0x a 0x a 0x a 0x a 0x a 0x a ) which is not the architecture being linked (x86_64): ../configure
Undefined symbols for architecture x86_64:
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ERROR: Spidermonkey build failed

[Tom Brewe]

modified osx build script so far

[historic_bruno]

Replying to nylki:

Can you please attach diff/patches when submitting modifications?

[Echelon9]

For those attempting to use the latest Mac OS X 10.9 as a build platform, I strongly suggest you first review and comment on the patch set that is queued up here: ​http://trac.wildfiregames.com/ticket/2304

That proposed patch seeks to address the pre-existing compilation issues on OS X 10.9, thus starting with a more stable base to facilitate the Spidermonkey testing.

[Tom Brewe]

Replying to Echelon9:

For those attempting to use the latest Mac OS X 10.9 as a build platform, I strongly suggest you first review and comment on the patch set that is queued up here: ​http://trac.wildfiregames.com/ticket/2304

That proposed patch seeks to address the pre-existing compilation issues on OS X 10.9, thus starting with a more stable base to facilitate the Spidermonkey testing.

nice! I'll try those patches, and apply the changes i have made to the build script, afterwards. Spidermonkey does compile here so far. But 0ad itself does not build. Maybe your linked patches could fix the issue.

[Tom Brewe]

After applying the patched in ​http://trac.wildfiregames.com/ticket/2304 and the ones linked here, I was able to build SM24. The 0ad make itself still seems to make some problems. After changing the include paths in the premake.lua, I was able to eliminate errors, but now I get some new ones.

./../../source/lobby/XmppClient.cpp:320:51: error: copying parameter of type 'JS::RootedObject' (aka 'Rooted<JSObject *>') invokes deleted constructor
        scriptInterface.EnumeratePropertyNamesWithPrefix(JS::RootedObject(scriptInterface.GetContext(), JSVAL_TO_OBJECT(dataval)), "", properties);
                                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../libraries/source/spidermonkey/include/mozjs-24/js/RootingAPI.h:671:5: note: function has been explicitly marked deleted here
    Rooted(const Rooted &) MOZ_DELETE;
    ^
54 warnings and 1 error generated.
make[1]: *** [obj/lobby_Release/XmppClient.o] Error 1
make: *** [lobby] Error 2

[Tom Brewe]

build_osx_libs.sh changes with succesful compiling mozjs24

[Yves]

I see where the problem is. It's not allowed to pass temporaries as JS::RootedObject. Declaring a JS::RootedObject variable and assigning it a value first and then passing it to the function should work. At the moment I'm working on some fixes for the Windows build. I think it would make sense to test it again with these fixes. Hopfully I'll have it done by this thursday evening.

[Yves]

It took a bit longer but now I'm ready. You can get the current version from github: ​https://github.com/Yves-G/0ad

[Yves]

I made a "blind" patch, meaning I have not tested it and I'm quite sure it won't work without corrections. It should at least give you some hints what's important.

I noticed that we build the library as static library on Mac OS X (--disable-shared-js instead of --enable-shared-js). I don't know the reason for this, so I assumed there is one and didn't change that. I don't know where NSPR comes from on the Mac, so I assumed it's already installed and pkg-config produces the correct output as on Linux, but that may be wrong. If we have to bundle NSPR, that would add some additional complexity. Some of the earlier patches contain code to build it manually, but that's not really automated and won't work on all systems (64 bit vs 32 bit for example).

I think it doesn't need any changes in premake4.lua or extern_libs4.lua if you copy the include directories from the debug and release builds to the correct location (as I hopefully did in the patch).

[historic_bruno]

There's no pkg-config on OS X by default, unless Homebrew or MacPorts are installed. NSPR won't be there either, Macs come with Safari installed and no Mozilla products. So if that's a new dependency we will likely have to build it (actually it would be nice if OS X devs could download the prebuilt libs as Windows devs can, and this was discussed re: the Git migration, but we still need a way to build them, and it's well beyond the scope of this ticket).

SM was built as a static lib like all the other libraries in build-osx-libs, the reason was that loading libraries at runtime gets a little nasty since they can be in a bundle structure or a dev checkout of SVN/Git. And it works pretty well :)

[Tom Brewe]

I tried Yves blind patch, I get some strange errors regarding dublicate symbols.

clang -arch x86_64 -o host_pathsub.o -c -Qunused-arguments  -Qunused-arguments -Wall -Wpointer-arith -Wdeclaration-after-statement -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-unused -std=gnu99 -fgnu89-inline -fno-common -ffunction-sections -fdata-sections -pthread -pipe  -DDEBUG -D_DEBUG -DTRACING -g -fno-omit-frame-pointer  -DXP_UNIX -DXP_MACOSX -DNO_X11  -DUNICODE -D_UNICODE  -I../../config -I. -I../dist/include  -I/usr/local/Cellar/nspr/4.10/include/nspr       -I/usr/local/Cellar/nspr/4.10/include/nspr  /Volumes/Daten/Install/0ad_SM24_git/libraries/source/spidermonkey/mozjs24/js/src/config/pathsub.c
/Volumes/Daten/Install/0ad_SM24_git/libraries/source/spidermonkey/mozjs24/js/src/build-debug/_virtualenv/bin/python /Volumes/Daten/Install/0ad_SM24_git/libraries/source/spidermonkey/mozjs24/js/src/build-debug/../config/nsinstall.py -m 644 "../js.msg" "dist/include"
clang -arch x86_64 -o nsinstall_real -Qunused-arguments  -Qunused-arguments -Wall -Wpointer-arith -Wdeclaration-after-statement -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-unused -std=gnu99 -fgnu89-inline -fno-common -ffunction-sections -fdata-sections -pthread -pipe  -DDEBUG -D_DEBUG -DTRACING -g -fno-omit-frame-pointer  -DXP_UNIX -DXP_MACOSX -DNO_X11  -DUNICODE -D_UNICODE -lobjc  -framework ExceptionHandling -Wl,-executable_path,../dist/bin host_nsinstall.o host_pathsub.o  
/Volumes/Daten/Install/0ad_SM24_git/libraries/source/spidermonkey/mozjs24/js/src/build-debug/_virtualenv/bin/python /Volumes/Daten/Install/0ad_SM24_git/libraries/source/spidermonkey/mozjs24/js/src/build-debug/../config/nsinstall.py -m 644 "../jsalloc.h" "dist/include"
duplicate symbol ___sputc in:
    host_nsinstall.o
    host_pathsub.o
ld: 1 duplicate symbol for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [nsinstall_real] Error 1
make[1]: *** [config/nsinstall] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [default] Error 2
ERROR: Spidermonkey build failed

I found two threads on mozillas bugzilla, with pretty much the identical error (also on os x 10.9): ​https://bugzilla.mozilla.org/show_bug.cgi?id=930584

​https://bugzilla.mozilla.org/show_bug.cgi?id=917526

In the second thread is a patch that might fix the issue: ​https://bug917526.bugzilla.mozilla.org/attachment.cgi?id=808090 I am not sure how to properly apply the patch, which directory etc.? I tried to hand edit libraries/source/spidermonkey/mozjs24/js/src/configure.in accordingly, but it had no obvious effect.

PS: (probably unrelated to this issues though ) I installed nspr and pkg-config with brew.

[Yves]

Thanks for testing! These bugs are all about using ASan (address sanatizer). There's the option "--enable-address-sanitizer" in configure but according to the description it's disabled by default. So maybe it's exactly the same problem as in these bugs but reading what was the problem there and what solved it could help. I have a look this weekend.

I think we can stick to the homebrew version of nspr and pkg-config for testing the rest of the build. But for the final version I'll have to integrate a bundled version into our build process.

[Tom Brewe]

Replying to Yves:

Thanks for testing! These bugs are all about using ASan (address sanatizer).

Just to be sure, but I don't think the second bug has to do with ASan, only the first?

[Yves]

I started writing this 5 minutes before I had to leave this morning... you are right, the second bug is not about ASan.

Just to be sure: If you edit that file and run the build-osx-libs.sh script, it will overwrite your file with the original file from mozjs-24.2.0.tar.bz2. Have you done anything to avoid that?

[Tom Brewe]

Replying to Yves:

I started writing this 5 minutes before I had to leave this morning... you are right, the second bug is not about ASan.

Just to be sure: If you edit that file and run the build-osx-libs.sh script, it will overwrite your file with the original file from mozjs-24.2.0.tar.bz2. Have you done anything to avoid that?

Oh right, I suppose I did not. However, in the next 3 weeks, I hardly have time to test much My weeks are completely occupied. :/

[historic_bruno]

The following patch allowed me to build and run the game from Yves' git repo on OS X Lion.

OS: Darwin LionBox.local 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23 16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64
Compiler (mozjs24): Apple LLVM version 4.2 (clang-425.0.28) (based on LLVM 3.2svn)
Compiler (everything else): i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
Xcode: 4.6.3

Changes from the "blind" patch:

• Adds NSPR build dependency
• Adds NSPR libs as Spidermonkey link deps (extern_libs4.lua)
  • I would rather this be done with js24-config tool, but it's kinda broken atm
• Adds perl commands to modify JS shell makefile, as it's hardcoded to link against libjs_static.a
• Fixes a few paths and variables, minor cleanup
• Removes "make install" from Spidermonkey build, 1) headers get installed with an extra "mozjs24" in them, 2) static libs get overwritten with a single `libmozjs_24.a
  • I think both of these could be fixed (nice, it would give us a working js24-config, and is more elegant than manually copying files)

Release build worked well. Debug build was either too slow or broken, it seemed to hang during loading, I don't know if that's a known issue.

[Yves]

Thanks very much, well done! I've applied your patch to the ​git repository. Now it would be good if as many people as possible test it. I'll mark the SpiderMonkey patch for review soon.

Replying to historic_bruno:

Release build worked well. Debug build was either too slow or broken, it seemed to hang during loading, I don't know if that's a known issue.

No, I don't know about such issues on Linux or Windows. What does loading refer to? Do you mean starting the game before it even reaches the main menu or starting a match or loading a savegame? Some gczeal settings make it very very slow, but enabling gczeal in the build should just make it available for the code and not automatically enable it.

[Tom Brewe]

nice historic_bruno! I tried to test it with 10.9.2, but nspr has given me some troubles. Running the configure alone (in the nspr directory) runs through without hassle! however when running build_osx_libraries.sh the nspr configure gets stuck at some point ("checking for dladdr..."). Yves and me think its probably somewhere around here in the nspr configure: (near line ~6570)

for ac_func in dladdr gettid lchown setpriority strerror syscall
do
echo $ac_n "checking for $ac_func""... $ac_c" 1>&6
echo "configure:5673: checking for $ac_func" >&5
if eval "test \"`echo '$''{'ac_cv_func_$ac_func'+set}'`\" = set"; then
  echo $ac_n "(cached) $ac_c" 1>&6
else
  cat > conftest.$ac_ext <<EOF

But its still strange that the configure executed alone causes no problem, but calling it from within our build_osx_libraries.sh does.

[historic_bruno]

If you can provide the config.log, both when it's run separately and as part of build-osx-libs.sh, that might reveal what the configure script is doing differently.

[historic_bruno]

Replying to Yves:

Replying to historic_bruno:

Release build worked well. Debug build was either too slow or broken, it seemed to hang during loading, I don't know if that's a known issue.

No, I don't know about such issues on Linux or Windows. What does loading refer to? Do you mean starting the game before it even reaches the main menu or starting a match or loading a savegame?

It never reached the main menu, freezing after "Sound: AlcInit success, using Built-in Output". I just tested with SVN, and it has the same problem, so it's not the SM upgrade and no longer relevant to this ticket :)

[historic_bruno]

Ok, I now have a clean install of Mavericks (10.9.2) to play with :)

Replying to nylki:

nice historic_bruno! I tried to test it with 10.9.2, but nspr has given me some troubles.

I didn't encounter this problem, so it could be some interference from Homebrew, or some other oddity of your build environment. We definitely need to see the config.log(s).

Replying to nylki:

I tried Yves blind patch, I get some strange errors regarding dublicate symbols.

Now, I did have this problem. The patch in ​Bugzilla #917526 (nice find!) applies to configure.in, so you would need to use autoconf to rebuild the configure script. However, I simply added the changes to the prebuilt script and made a new patch from it (to be applied by build-osx-libs.sh).

I had to make a few other changes:

• The configure script didn't like "-arch..." appearing in the CC/CXX variables, it failed with a misleading error message about zlib.
• I switched a few things to use provided config options.

The following patch should apply to the head of Yves' git repo, I built and ran the game using it (note: it doesn't fix the other Mavericks build issues). I apologize that it doesn't look very clean, there were line ending problems introduced in the last patch.

[historic_bruno]

Note the last patch has been committed in Yves' ​SpiderMonkey Github branch.

Results of building the branch with no modifications:

• [SUCCESS] 10.9.2
  • Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
• [SUCCESS] 10.8.5
  • Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
• [SUCCESS] 10.7.5
  • Apple LLVM version 4.2 (clang-425.0.28) (based on LLVM 3.2svn)
  • Built normally - Ran on 10.7.5
  • Built bundle with 10.7 SDK and 10.6 API - Ran on 10.7.5 and 10.6.8
• [FAILED] OS X 10.6.8
  • Upgraded Xcode from 3.2.6 to 4.2, to get latest available command line tools
  • Apple clang version 3.0 (tags/Apple/clang-211.10.1) (based on LLVM 3.0svn)
  • Upgraded Python from 2.6.1 to ​2.7.6 (Python 2.7 seems to be a particular dependency of SM 24)
  • Got the following error which was reported last year as ​Bug 862657 - Fix failures on Mac SpiderMonkey jobs, but not yet resolved

    In file included from /Users/ben/GitHub/0ad/libraries/source/spidermonkey/mozjs24/js/src/jsgc.cpp:9:
    In file included from ../jsgc.h:19:
    In file included from ../jsscript.h:15:
    In file included from ../jsinfer.h:16:
    In file included from ../gc/Barrier.h:12:
    ../gc/Heap.h:665:8: error: cannot initialize a parameter of type 'void *' with
          an rvalue of type 'volatile uintptr_t (*)[2016]'
    struct ChunkBitmap
           ^~~~~~~~~~~
    /Users/ben/GitHub/0ad/libraries/source/spidermonkey/mozjs24/js/src/jsgc.cpp:3078:9: note: 
          in instantiation of function template specialization
          'js::Swap<js::gc::ChunkBitmap>' requested here
            js::Swap(*entry, *bitmap);
            ^
    ./dist/include/js/Utility.h:730:7: note: implicit default copy assignment
          operator for 'js::gc::ChunkBitmap' first required here
        t = Move(u);
          ^
    1 error generated.
    make[1]: *** [jsgc.o] Error 1
    

Overall good results.

Failing on 10.6.8 is somewhat troubling because I was building against the 10.5 SDK to maintain compatibility, and that's not an option with newer OS X. It does seem possible to build on 10.7 and run on 10.6 (see above results) or even 10.5, but I tried that for A15 and there were issues reported (like #2429). So I don't know how reliable an option that is, maybe it's time to re-evaluate what versions of OS X we can support.

---

Also an annoyance: Spidermonkey's configure script ignores the CC/CXX variables when testing the compiler and seems to invoke gcc/c++ directly on OS X, which causes an error if those are really LLVM-GCC but you set clang-specific compiler flags like -stdlib=libstdc++. I don't know why they can't just accept standard variables like every other configure script...

[Yves]

In 14877:

Second (main) commit for the SpiderMonkey upgrade.

This commit contains all the required changes to our source files and build scripts (hopefully).
A next commit will remove the old stuff of SpiderMonkey 1.8.5.

Spcial thanks to:

• H4writer who helped a lot mainly with the performance issues we had/have, but also with other problems or questions.
• Leper for the review.
• Historic_bruno for implementing the build scripts on Mac OS X and testing on the Mac.
• The people from the #jsapi channel and from mozilla.dev.tech.js-engine who answered a lot of questions and helped solving problems.
• All the other people who helped

Refs #1886
Fixes #2442
Fixes #2416