#1112 closed task (fixed)
Script to build OS X dependencies
Reported by: | historic_bruno | Owned by: | historic_bruno |
---|---|---|---|
Priority: | Should Have | Milestone: | Alpha 13 |
Component: | Build & Packages | Keywords: | osx build |
Cc: | Patch: |
Description
Developing the game on OS X is a pain, because not only does Xcode need to be installed to get a Unix build environment, but we currently need Mac Ports or equivalent to acquire all the dependencies, since OS X doesn't have a real package manager. This also has the side effect of making the game more difficult to distribute, as it's unknown what versions of the dependencies Mac Ports might have pulled in or how they were built. For Windows developers we already provide prebuilt libraries, the same can be achieved on OS X.
It's proposed to eliminate the Mac Ports requirement by having a script that downloads (optionally) and builds the game's dependencies from source. I'm working on a script now that does just that. This script is only for developers, not the end users. Currently it builds static libs (.a) which get linked into the engine. This avoids the need for install_name_tool
fiddling later and other problems like dylib search paths.
We can even go so far as to distribute the prebuilt libraries on separate SVN directories, arranged by OS, similar to what Blender does. This would greatly speed the development process as most have no need or desire to build the dependencies.
Attachments (3)
Change History (30)
follow-up: 2 comment:1 by , 12 years ago
follow-up: 3 comment:2 by , 12 years ago
Replying to Juicyfruit:
I think we should link libxml2 zlib and maybe even libpng and jpeg dynamicly as reasonable versions are provided with leopard and above.
Currently my script assumes libxml2, libz, libcurl, and libpng reside on the system, in /usr/lib/
, which I can confirm on SL and Lion.
One problem with libpng, it's only a symbolic link to the latest version available e.g. libpng.15.15.dylib on Lion. During the build, the symlink gets translated to the actual file name, creating a problem if you compile on Lion and try to run on SL, since the former has a newer version of libpng. I guess the others could have a similar problem, when e.g. OS X 10.8 comes along.
Please also include libopenal-soft if you can as that fixes the sound problems as far as i can check. (this would recuire a premake hack i think, not sure how to override the search location for openal)
This depends somewhat on what Kenny is doing (see #931). If he's not working on that anymore, then someone else can look into solving audio problems. I think premake tells the linker to use the OpenAL framework, and it searches in /Library/Frameworks
followed by /System/Library/Frameworks
. If we provide our own framework then we could use something like the -F and -Z linker options to override the default search paths, or we could build OpenAL-soft as something other than a framework. It's not something I've looked into.
SDL 1.2 I think a snapshot checkout is the best bet and link that static. I got some notes on how i build my libs I used for compiling if you want those.
I'm currently testing the 1.2.15 pre-release, linked statically as you say. When they release it officially, we can just update the build script to retrieve the correct source tarball. Unfortunately I can't test if the fullscreen fix works on Lion because my VM is challenged, but others say it works, so I'll believe them :)
comment:3 by , 12 years ago
Replying to historic_bruno:
Replying to Juicyfruit:
I think we should link libxml2 zlib and maybe even libpng and jpeg dynamicly as reasonable versions are provided with leopard and above.
Currently my script assumes libxml2, libz, libcurl, and libpng reside on the system, in
/usr/lib/
, which I can confirm on SL and Lion. One problem with libpng, it's only a symbolic link to the latest version available e.g. libpng.15.15.dylib on Lion. During the build, the symlink gets translated to the actual file name, creating a problem if you compile on Lion and try to run on SL, since the former has a newer version of libpng. I guess the others could have a similar problem, when e.g. OS X 10.8 comes along.
Can't we just add -isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 to the link and compile flags. I think that should link agains the sdk's libraries but keep the path in the normal (/usr/lib or /System/Frameworks fashion)
Please also include libopenal-soft if you can as that fixes the sound problems as far as i can check. (this would recuire a premake hack i think, not sure how to override the search location for openal)
This depends somewhat on what Kenny is doing (see #931). If he's not working on that anymore, then someone else can look into solving audio problems. I think premake tells the linker to use the OpenAL framework, and it searches in
/Library/Frameworks
followed by/System/Library/Frameworks
. If we provide our own framework then we could use something like the -F and -Z linker options to override the default search paths, or we could build OpenAL-soft as something other than a framework. It's not something I've looked into.
The documentation of openal-soft advises strongly agains static linking. I am not sure what their rationale is
SDL 1.2 I think a snapshot checkout is the best bet and link that static. I got some notes on how i build my libs I used for compiling if you want those.
I'm currently testing the 1.2.15 pre-release, linked statically as you say. When they release it officially, we can just update the build script to retrieve the correct source tarball. Unfortunately I can't test if the fullscreen fix works on Lion because my VM is challenged, but others say it works, so I'll believe them :)
Wel maybe you could share your binaries and your script then Il test for lion. (or setup a quick github or gitorious so I can help you. I am also like 90% of the way of building a good relocatable bundle, so I would love to help.
comment:4 by , 12 years ago
Milestone: | Alpha 9 → Alpha 10 |
---|
comment:5 by , 12 years ago
Keywords: | osx,build → osx, build |
---|---|
Priority: | Should Have → If Time Permits |
comment:6 by , 12 years ago
Type: | enhancement → task |
---|
comment:7 by , 12 years ago
Keywords: | review added |
---|
This is part of the planned build system overhaul outlined here.
I'm pretty happy with how the build script works, it's flexible enough to work on different versions of Xcode and command line tools and it can be used for app bundles (built against a specific SDK). The idea is this script gets dropped in libraries/osx
, then run once by the developer to build all the libraries. After that they only need to build the game, unless the libraries get updated or they change paths. Currently it's a clean build every time you run it.
Explanations:
SYSROOT
andMIN_OSX_VERSION
are supported for SDK builds.CXX
andCC
are set to fix broken configs.- I didn't use the bundled build scripts from e.g. Spidermonkey because it would make those scripts very messy with OS X-specific handling, so I duplicated parts of them in this script as a lesser evil.
- It would be fairly straightforward to switch to dynamic libs, if there are useful benefits, but static libs avoid the need for install_name/rpath nastiness.
Areas for improvement:
- Detect partially successful builds, in case one library failed, not all need to be rebuilt - it's time consuming!
- How to handle updated libs? Again, a full rebuild is slow
- Somehow speed up the wxWidgets build? :(
- Improve maintainability / flexibility
- Any missing debug builds?
Other than that, the script just needs to be reviewed for bugs or ways to make it more robust. I'm sure there's lots of scripting cleverness I don't know about :)
follow-up: 9 comment:8 by , 12 years ago
The file seems to have some funny line endings. Trying to run it, I get:
zsh: ./build-osx-libs.sh: bad interpreter: /bin/bash^M: no such file or directory
comment:9 by , 12 years ago
I uploaded it from my Windows machine, so it probably has Windows-style line endings. I'll try converting the line endings and re-attaching.
(BTW if you try running it, it will explode once it reaches Spidermonkey, because the path is different, but it should download and build the others)
by , 12 years ago
Attachment: | build-osx-libs.sh added |
---|
comment:10 by , 12 years ago
Some improvements:
You should replace #!/bin/sh
with #!/bin/bash
as you are using some bash specific syntax ([[
and ${var-_}
) (that may or may not be available if running in a /bin/sh
mode) and bash is installed on OS X by default.
You should replace tar with bsdtar as that one is automatically detecting the compression format.
Move the libname and version stuff into variables. You should add a url variable for every lib.
You could then move the
if [ ! -e filename ]; then echo "Downloading" curl -L -O $(url)$(filename) fi
block into a function.
by , 12 years ago
Attachment: | build-osx-libs.2.sh added |
---|
comment:12 by , 12 years ago
Attached new version that addresses the above concerns. It turns out tar
is bsdtar
on OS X, I didn't realize it was different :)
comment:13 by , 12 years ago
You shouldn't invoke sed twice if you make a backup (FCollada). Replace the second sed with mv Makefile.bak Makefile
.
Some thoughts on rebuilding:
We should touch
a file for each lib to check when the lib was built. Then add a check if the built lib (or whatever the resulting file is) is newer than that file then skip it, otherwise we build that lib again (as something changed. We should add a command line parameter (--force or --rebuild) to force the rebuild of all libs.
The download_lib function streamlined the build process but there is still room for improvement.
SDL, wxWidgets libjpeg, libpng, libogg, libvorbis, libxml2 and enet (though the last one needs to skip the download part) could call a function with their configure arguments as a parameter instead of duplicating the code for those. (Skip this if it creates too many problems with escaping stuff)
Boost, SpiderMonkey, nvtt and FCollada already differ so we can keep them separate.
follow-up: 15 comment:14 by , 12 years ago
Got the following when running the latest script:
Building Spidermonkey... ./build-osx-libs.2.sh: line 302: pushd: ../source/spidermonkey/: No such file or directory
comment:15 by , 12 years ago
Replying to leper:
You shouldn't invoke sed twice if you make a backup (FCollada). Replace the second sed with
mv Makefile.bak Makefile
.
Good catch.
Some thoughts on rebuilding: We should
touch
a file for each lib to check when the lib was built. Then add a check if the built lib (or whatever the resulting file is) is newer than that file then skip it, otherwise we build that lib again (as something changed. We should add a command line parameter (--force or --rebuild) to force the rebuild of all libs.
Sounds like that would work, we do something similar with the Spidermonkey build script (except we have a separate "clean workspaces" script for forcing a rebuild). I'll try to learn exactly what magic is being used there.
Replying to k776:
Is the script designed to be run within the build folder of a copy of the 0 A.D. code?
See above comment :P Certain bundled libraries (e.g. Spidermonkey, ENet, NVTT, FCollada) will be moved to libraries/source
to differentiate them from the proposed libraries/win32
and libraries/osx
.
Also there's one requirement I forgot to mention for the build script: CMake is needed for NVTT.
follow-up: 22 comment:16 by , 12 years ago
I did some testing and something like this should work for rebuilding:
#!/bin/bash TESTFILE=libname-built CREATEFILE=libname.so if [[ ! -e $TESTFILE ]] || [[ $CREATEFILE -ot $TESTFILE ]]; then touch $TESTFILE echo "Build it". #simulate build touch $CREATEFILE else echo "skipping already up-to-date" fi
comment:17 by , 12 years ago
Milestone: | Alpha 10 → Alpha 11 |
---|
by , 12 years ago
Attachment: | build-osx-libs.3.sh added |
---|
comment:18 by , 12 years ago
Updated new version with some fixes (ld
wants the min OS X version, some options were incorrectly used, Spidermonkey needs a special argument for cross-compiling to 10.5 32-bit). Haven't yet looked into leper's suggestion for smarter rebuilding.
comment:19 by , 12 years ago
Made a GitHub account, if anyone wants to see how it all fits together: https://github.com/historicbruno/0ad/tree/osx-build
I'll try to keep that updated as I tweak things.
comment:20 by , 12 years ago
Priority: | If Time Permits → Should Have |
---|
comment:21 by , 12 years ago
Milestone: | Alpha 11 → Alpha 12 |
---|
comment:22 by , 12 years ago
Replying to leper:
I did some testing and something like this should work for rebuilding:
#!/bin/bash TESTFILE=libname-built CREATEFILE=libname.so if [[ ! -e $TESTFILE ]] || [[ $CREATEFILE -ot $TESTFILE ]]; then touch $TESTFILE echo "Build it". #simulate build touch $CREATEFILE else echo "skipping already up-to-date" fi
Thanks, I used a modification of that concept to improve the build script. It can be seen on my Github repo here. Now it's much more efficient if only a few packages fail to build. That can be overridden with --force-rebuild
. I'll see if I can add something similar to the bundle script.
Remaining TODOs:
- Log the build output instead of cluttering the terminal
- Don't replace the libs when building a bundle (against SDK), instead build them in a new directory or make a backup of the old ones. This will save time for developers
- Convert some of the environment variables to command line options
- Add an option for 32-bit 10.5 cross-compiling, it's a bit of a pain currently, I added some instructions
- I would really prefer offering the prebuilt OS X libs in SVN, as we do for win32, need to investigate the impact of this on the Premake scripts
- Directory structure is still weird, *nix users would be downloading win32 .libs for the contents of libraries/source. Not to mention the annoyance of all those .dlls and .exes in binaries/system.
- How to streamline the build process for different OSes? Do we want separate packages/SVN branches for dependencies?
comment:23 by , 12 years ago
Milestone: | Alpha 12 → Backlog |
---|
comment:26 by , 11 years ago
Keywords: | review removed |
---|
I think we should link libxml2 zlib and maybe even libpng and jpeg dynamicly as reasonable versions are provided with leopard and above.
Please also include libopenal-soft if you can as that fixes the sound problems as far as i can check. (this would recuire a premake hack i think, not sure how to override the search location for openal)
SDL 1.2 I think a snapshot checkout is the best bet and link that static. I got some notes on how i build my libs I used for compiling if you want those.