Script to build OS X dependencies

[historic_bruno]

Developing the game on OS X is a pain, because not only does Xcode need to be installed to get a Unix build environment, but we currently need Mac Ports or equivalent to acquire all the ​dependencies, since OS X doesn't have a real package manager. This also has the side effect of making the game more difficult to distribute, as it's unknown what versions of the dependencies Mac Ports might have pulled in or how they were built. For Windows developers we already provide prebuilt libraries, the same can be achieved on OS X.

It's proposed to eliminate the Mac Ports requirement by having a script that downloads (optionally) and builds the game's dependencies from source. I'm working on a script now that does just that. This script is only for developers, not the end users. Currently it builds static libs (.a) which get linked into the engine. This avoids the need for install_name_tool fiddling later and other problems like dylib search paths.

We can even go so far as to distribute the prebuilt libraries on separate SVN directories, arranged by OS, similar to what ​Blender does. This would greatly speed the development process as most have no need or desire to build the dependencies.

[Juicyfruit]

I think we should link libxml2 zlib and maybe even libpng and jpeg dynamicly as reasonable versions are provided with leopard and above.

Please also include libopenal-soft if you can as that fixes the sound problems as far as i can check. (this would recuire a premake hack i think, not sure how to override the search location for openal)

SDL 1.2 I think a snapshot checkout is the best bet and link that static. I got some notes on how i build my libs I used for compiling if you want those.

[historic_bruno]

Replying to Juicyfruit:

I think we should link libxml2 zlib and maybe even libpng and jpeg dynamicly as reasonable versions are provided with leopard and above.

Currently my script assumes libxml2, libz, libcurl, and libpng reside on the system, in /usr/lib/, which I can confirm on SL and Lion.

One problem with libpng, it's only a symbolic link to the latest version available e.g. libpng.15.15.dylib on Lion. During the build, the symlink gets translated to the actual file name, creating a problem if you compile on Lion and try to run on SL, since the former has a newer version of libpng. I guess the others could have a similar problem, when e.g. OS X 10.8 comes along.

Please also include libopenal-soft if you can as that fixes the sound problems as far as i can check. (this would recuire a premake hack i think, not sure how to override the search location for openal)

This depends somewhat on what Kenny is doing (see #931). If he's not working on that anymore, then someone else can look into solving audio problems. I think premake tells the linker to use the OpenAL framework, and it searches in /Library/Frameworks followed by /System/Library/Frameworks. If we provide our own framework then we could use something like the -F and -Z linker options to override the default search paths, or we could build OpenAL-soft as something other than a framework. It's not something I've looked into.

SDL 1.2 I think a snapshot checkout is the best bet and link that static. I got some notes on how i build my libs I used for compiling if you want those.

I'm currently testing the 1.2.15 pre-release, linked statically as you say. When they release it officially, we can just update the build script to retrieve the correct source tarball. Unfortunately I can't test if the fullscreen fix works on Lion because my VM is challenged, but others say it works, so I'll believe them :)

[Juicyfruit]

Replying to historic_bruno:

Replying to Juicyfruit:

I think we should link libxml2 zlib and maybe even libpng and jpeg dynamicly as reasonable versions are provided with leopard and above.

Currently my script assumes libxml2, libz, libcurl, and libpng reside on the system, in /usr/lib/, which I can confirm on SL and Lion. One problem with libpng, it's only a symbolic link to the latest version available e.g. libpng.15.15.dylib on Lion. During the build, the symlink gets translated to the actual file name, creating a problem if you compile on Lion and try to run on SL, since the former has a newer version of libpng. I guess the others could have a similar problem, when e.g. OS X 10.8 comes along.

Can't we just add -isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 to the link and compile flags. I think that should link agains the sdk's libraries but keep the path in the normal (/usr/lib or /System/Frameworks fashion)

Please also include libopenal-soft if you can as that fixes the sound problems as far as i can check. (this would recuire a premake hack i think, not sure how to override the search location for openal)

This depends somewhat on what Kenny is doing (see #931). If he's not working on that anymore, then someone else can look into solving audio problems. I think premake tells the linker to use the OpenAL framework, and it searches in /Library/Frameworks followed by /System/Library/Frameworks. If we provide our own framework then we could use something like the -F and -Z linker options to override the default search paths, or we could build OpenAL-soft as something other than a framework. It's not something I've looked into.

The documentation of openal-soft advises strongly agains static linking. I am not sure what their rationale is

SDL 1.2 I think a snapshot checkout is the best bet and link that static. I got some notes on how i build my libs I used for compiling if you want those.

I'm currently testing the 1.2.15 pre-release, linked statically as you say. When they release it officially, we can just update the build script to retrieve the correct source tarball. Unfortunately I can't test if the fullscreen fix works on Lion because my VM is challenged, but others say it works, so I'll believe them :)

Wel maybe you could share your binaries and your script then Il test for lion. (or setup a quick github or gitorious so I can help you. I am also like 90% of the way of building a good relocatable bundle, so I would love to help.

[historic_bruno]

This is part of the planned build system overhaul outlined here.

I'm pretty happy with how the build script works, it's flexible enough to work on different versions of Xcode and command line tools and it can be used for app bundles (built against a specific SDK). The idea is this script gets dropped in libraries/osx, then run once by the developer to build all the libraries. After that they only need to build the game, unless the libraries get updated or they change paths. Currently it's a clean build every time you run it.

Explanations:

• SYSROOT and MIN_OSX_VERSION are supported for SDK builds.
• CXX and CC are set to fix broken configs.
• I didn't use the bundled build scripts from e.g. Spidermonkey because it would make those scripts very messy with OS X-specific handling, so I duplicated parts of them in this script as a lesser evil.
• It would be fairly straightforward to switch to dynamic libs, if there are useful benefits, but static libs avoid the need for install_name/rpath nastiness.

Areas for improvement:

• Detect partially successful builds, in case one library failed, not all need to be rebuilt - it's time consuming!
• How to handle updated libs? Again, a full rebuild is slow
• Somehow speed up the wxWidgets build? :(
• Improve maintainability / flexibility
• Any missing debug builds?

Other than that, the script just needs to be reviewed for bugs or ways to make it more robust. I'm sure there's lots of scripting cleverness I don't know about :)

[Kieran P]

The file seems to have some funny line endings. Trying to run it, I get:

zsh: ./build-osx-libs.sh: bad interpreter: /bin/bash^M: no such file or directory

[historic_bruno]

I uploaded it from my Windows machine, so it probably has Windows-style line endings. I'll try converting the line endings and re-attaching.

(BTW if you try running it, it will explode once it reaches Spidermonkey, because the path is different, but it should download and build the others)

[leper]

Some improvements:

You should replace #!/bin/sh with #!/bin/bash as you are using some bash specific syntax ([[ and ${var-_}) (that may or may not be available if running in a /bin/sh mode) and bash is installed on OS X by default.

You should replace tar with bsdtar as that one is automatically detecting the compression format.

Move the libname and version stuff into variables. You should add a url variable for every lib.

You could then move the

if [ ! -e filename ]; then
	echo "Downloading"
	curl -L -O $(url)$(filename)
fi

block into a function.

[historic_bruno]

Thanks for the tips, I will add them!

[historic_bruno]

Attached new version that addresses the above concerns. It turns out tar is bsdtar on OS X, I didn't realize it was different :)

[leper]

You shouldn't invoke sed twice if you make a backup (FCollada). Replace the second sed with mv Makefile.bak Makefile.

Some thoughts on rebuilding: We should touch a file for each lib to check when the lib was built. Then add a check if the built lib (or whatever the resulting file is) is newer than that file then skip it, otherwise we build that lib again (as something changed. We should add a command line parameter (--force or --rebuild) to force the rebuild of all libs.

The download_lib function streamlined the build process but there is still room for improvement. SDL, wxWidgets libjpeg, libpng, libogg, libvorbis, libxml2 and enet (though the last one needs to skip the download part) could call a function with their configure arguments as a parameter instead of duplicating the code for those. (Skip this if it creates too many problems with escaping stuff)
Boost, SpiderMonkey, nvtt and FCollada already differ so we can keep them separate.

[Kieran P]

Got the following when running the latest script:

Building Spidermonkey...

./build-osx-libs.2.sh: line 302: pushd: ../source/spidermonkey/: No such file or directory

Is the script designed to be run within the build folder of a copy of the 0 A.D. code?

I'd suggest a variable that points to the copy of the 0 A.D. source code.

[historic_bruno]

Replying to leper:

You shouldn't invoke sed twice if you make a backup (FCollada). Replace the second sed with mv Makefile.bak Makefile.

Good catch.

Some thoughts on rebuilding: We should touch a file for each lib to check when the lib was built. Then add a check if the built lib (or whatever the resulting file is) is newer than that file then skip it, otherwise we build that lib again (as something changed. We should add a command line parameter (--force or --rebuild) to force the rebuild of all libs.

Sounds like that would work, we do something similar with the Spidermonkey build script (except we have a separate "clean workspaces" script for forcing a rebuild). I'll try to learn exactly what magic is being used there.

Replying to k776:

Is the script designed to be run within the build folder of a copy of the 0 A.D. code?

See above comment :P Certain bundled libraries (e.g. Spidermonkey, ENet, NVTT, FCollada) will be moved to libraries/source to differentiate them from the proposed libraries/win32 and libraries/osx.

Also there's one requirement I forgot to mention for the build script: ​CMake is needed for NVTT.

[leper]

I did some testing and something like this should work for rebuilding:

#!/bin/bash

TESTFILE=libname-built
CREATEFILE=libname.so

if [[ ! -e $TESTFILE ]] || [[ $CREATEFILE -ot $TESTFILE ]];
then
  touch $TESTFILE
  echo "Build it".
  #simulate build
  touch $CREATEFILE
else
  echo "skipping already up-to-date"
fi

[historic_bruno]

Updated new version with some fixes (ld wants the min OS X version, some options were incorrectly used, Spidermonkey needs a special argument for cross-compiling to 10.5 32-bit). Haven't yet looked into leper's suggestion for smarter rebuilding.

[historic_bruno]

Made a GitHub account, if anyone wants to see how it all fits together: ​https://github.com/historicbruno/0ad/tree/osx-build

I'll try to keep that updated as I tweak things.

[historic_bruno]

Replying to leper:

I did some testing and something like this should work for rebuilding:

#!/bin/bash

TESTFILE=libname-built
CREATEFILE=libname.so

if [[ ! -e $TESTFILE ]] || [[ $CREATEFILE -ot $TESTFILE ]];
then
  touch $TESTFILE
  echo "Build it".
  #simulate build
  touch $CREATEFILE
else
  echo "skipping already up-to-date"
fi

Thanks, I used a modification of that concept to improve the build script. It can be seen on my Github repo ​here. Now it's much more efficient if only a few packages fail to build. That can be overridden with --force-rebuild. I'll see if I can add something similar to the bundle script.

Remaining TODOs:

• Log the build output instead of cluttering the terminal
• Don't replace the libs when building a bundle (against SDK), instead build them in a new directory or make a backup of the old ones. This will save time for developers
• Convert some of the environment variables to command line options
• Add an option for 32-bit 10.5 cross-compiling, it's a bit of a pain currently, I added some ​instructions
• I would really prefer offering the prebuilt OS X libs in SVN, as we do for win32, need to investigate the impact of this on the Premake scripts
• Directory structure is still weird, *nix users would be downloading win32 .libs for the contents of libraries/source. Not to mention the annoyance of all those .dlls and .exes in binaries/system.
• How to streamline the build process for different OSes? Do we want separate packages/SVN branches for dependencies?

[historic_bruno]

Time to add this :)

[ben]

In 13148:

NOTE: Requires update-workspaces!
Adds build script for OS X dependencies, fixes #1112. OS X builds no longer use MacPorts/Homebrew, check BuildInstructions.
Adds build script for OS X bundles. Adds OS X icon and DMG background image.
Moves bundled libraries to libraries/source.
Moves most headers and precompiled static libs for Windows to libraries/win32.
Updates Premake and packaging scripts to support this.

[leper]

In 14662:

Remove OS X related stuff from library build scripts. Building of those is handled in build-osx-libs.sh. Refs #1112.