Update and cleanup OS X libraries

[fabio]

This patch is to keep track of the need library updates for the current milestone. Updates are done in ​https://trac.wildfiregames.com/browser/ps/trunk/libraries/osx/build-osx-libs.sh

• WxWidgets (3.0.5 -> 3.1.4)
• Libogg (1.3.3 -> 1.3.4) (Currently doesn't compile)
• Boost (1.74 -> 1.75)
• ...

[wraitii]

I created a branch that relies on Homebrew to download all non-bundled libs (except for gloox, as we do not use ssl and brew does).

​https://github.com/wraitii/0ad/tree/OSX_libs_brew

This seems to compile on my system (10.11.6), and has the advantages of:

• cutting down compile time substantially, particularly for wxwidgets/boost
• making sure we always use the latest versions

Now we voluntarily didn't use brew before. I'm not exactly sure why, but here are a few notes:

• Brew is largely the best and most common OS X package manager now
• It used to requires sudo at some point but no longer does
• It does not introduce conflicts with OSX's libraries, as it can provide "unlinked" versions (say, for libpng)
• it's got all the libraries we use.

I don't see why we should/could not rely on it now, at least.

NB: I removed libiconv because it seemed to be only used to compile the others, but I'm not actually sure about that.

[wraitii]

As a sidenote: using brew libraries and hot linking seems to work but fails to compile into a binary that's self-sufficient, it'll probably take more changes. As a side-sidenote, wxwidgets 3.0.2 no longer compiles on macOs 10.12 (Sierra), you have to use 3.1.0

edit:hm actually not either, we'll have to port an upstream patch.

[Itms]

Can we push that to A23 or is there an important library update that needs to be performed before packaging?

[fabio]

Most (all?) libraries in build-osx-libs.sh should be updated for a22, some due to security issues.

Wraitii use of Homebrew can wait for a future release.

[fabio]

Suggested for a22:

-ZLIB_VERSION="zlib-1.2.8"
-CURL_VERSION="curl-7.46.0"
-ICONV_VERSION="libiconv-1.14"
-XML2_VERSION="libxml2-2.9.3"
-SDL2_VERSION="SDL2-2.0.4"
-BOOST_VERSION="boost_1_60_0"
-WXWIDGETS_VERSION="wxWidgets-3.0.2"
+ZLIB_VERSION="zlib-1.2.11"
+CURL_VERSION="curl-7.54.0"
+ICONV_VERSION="libiconv-1.15"
+XML2_VERSION="libxml2-2.9.4"
+SDL2_VERSION="SDL2-2.0.5"
+BOOST_VERSION="boost_1_64_0"
+WXWIDGETS_VERSION="wxWidgets-3.0.3"
 # libpng was included as part of X11 but that's removed from Mountain Lion
 # (also the Snow Leopard version was ancient 1.2)
-PNG_VERSION="libpng-1.6.21"
+PNG_VERSION="libpng-1.6.29"
 OGG_VERSION="libogg-1.3.2"
 VORBIS_VERSION="libvorbis-1.3.5"
 # gloox is necessary for multiplayer lobby
-GLOOX_VERSION="gloox-1.0.14"
+GLOOX_VERSION="gloox-1.0.20"
 # NSPR is necessary for threadsafe Spidermonkey
-NSPR_VERSION="4.11"
+NSPR_VERSION="4.15"
 # OS X only includes part of ICU, and only the dylib
 # NOTE: remember to also update LIB_URL below when changing version
-ICU_VERSION="icu4c-56_1"
+ICU_VERSION="icu4c-59_1"
 ENET_VERSION="enet-1.3.13"
-MINIUPNPC_VERSION="miniupnpc-1.9.20151026"
+MINIUPNPC_VERSION="miniupnpc-1.9.20160209"

[Itms]

Alright, I hope wraitii can look into that.

I'm going to look into performing a few updates on Windows as well, they are long due.

[elexis]

Besides that zlib version not being available anymore (reported in #4639), nigel87 also uses OSX Sierra 10.12 which fails to build wxWidgets due to including quicktime which was dropped from that OS: ​http://trac.wxwidgets.org/ticket/17639 ​https://forums.wxwidgets.org/viewtopic.php?t=42856

He tried passing WXWIDGETS_VERSION="wxWidgets-3.0.3" and added --disable-qtkit for CONF_OPTS in L356, but that didn't help.

[elexis]

In 19825:

Update OSX zlib version to the most recent and now only one hosted.

Fixes #4639
Refs #4362
Differential Revision: ​https://code.wildfiregames.com/D644
Tested By: Tobbi

[elexis]

​Phab:D679

[elexis]

In 19848:

Update OSX libraries to the most recently released versions.

Remove wxWidgets patch from rP16155 that had been merged with the new release.

Differential Revision: ​https://code.wildfiregames.com/D679
Refs #4362
Based On Patch By: fabio
Discussed With: leper
Tested By: Tobbi, Itms

[elexis]

Four exploits in the latest release of libxml2 mentioned in ​https://code.wildfiregames.com/D679?id=2703#inline-12860 should be either patched or it should be confirmed that we are not affected. Rest of the library versions should be checked for exploitable vulnerabilities too.

[elexis]

​Phab:D699 for the libxml2 snapshot.

[elexis]

I set this to backlog because noone has the will to update libxml2 to the most recent dev snapshot before the alpha 22 release and because we will need a new ticket to keep track of updates for the next releases if this was closed as fixed.

[stan]

for macosx, im encountering error: 'connectx' is only available on macOS 10.11 or newer while installing libraries on curl-7.54.0, fix is to update to curl-7.56.0, see discussion here: ​https://github.com/VCVRack/Rack/pull/200 quick fix is to update this line to : CURL_VERSION="curl-7.56.0" in libraries/osx/build-osx-libs.sh

[Itms]

I am going to perform the curl change on macOS ASAP.

[Itms]

In 21501:

Update libcurl for OSX, in order to fix the build. Use the opportunity to update libxml2 which had vulnerabilities.
Refs #4362

[fabio]

Given #4790 was recently closed, here is an update of latest libraries and current status (to be A23) for ​OS X:

• zlib: 1.2.11 - OK;
• curl ​7.59.0 - 7.58.0;
• iconv: 1.15 - OK;
• libxml2: 2.9.8 - OK;
• libsdl2 ​2.0.6, ​2.0.7. ​2.0.8 - 2.0.5;
• boost ​1.66.0 - 1.64;
• wxWidgets ​3.0.4 - 3.0.3.1;
• libpng ​1.6.34 - 1.6.29;
• libogg ​1.3.3 - 1.3.2;
• libvorbis ​1.3.6 - 1.3.5;
• gloox 1.0.20 - OK;
• nspr ​4.19 - 4.15;
• icu4c ​60.2 - 59.1;
• enet 1.3.13 - OK;
• miniupnpc ​2.0.20180222 - 2.0.20170509.

Most are a bit behind and some are possibly security related. Patching is trivial, but it would likely need some testing.

[fabio]

• curl 7.58 -> 7.59 fixes: CVE-2018-1000120, CVE-2018-1000121, CVE-2018-1000122
• miniupnpc ​2.0.20180222 -> 2.0.20170509 fixes: "Fix buffer over run in minixml.c", "Fix uninitialized variable access in upnpreplyparse.c"
• libvorbis 1.3.5 -> 1.3.6 fixes: CVE-2018-5146, CVE-2017-14632, CVE-2017-14633

Will update some low risk libraries here.

[fabio]

In 21583:

Update some OS X libraries with security fixes and hopefully low regression risks. Refs #4362.

[fabio]

Given #4790 was recently closed, here is an update of latest libraries and current status (to be A23) for ​OS X:

• zlib: 1.2.11 - OK;
• curl 7.59.0 - OK;
• iconv: 1.15 - OK;
• libxml2: 2.9.8 - OK;
• libsdl2 ​2.0.6, ​2.0.7. ​2.0.8 - 2.0.5;
• boost ​1.66.0 - 1.64;
• wxWidgets ​3.0.4 - 3.0.3.1;
• libpng 1.6.34 - OK;
• libogg 1.3.3 - OK;
• libvorbis 1.3.6 - OK;
• gloox 1.0.20 - OK;
• nspr ​4.19 - 4.15;
• icu4c ​60.2 - 59.1;
• enet 1.3.13 - OK;
• miniupnpc 2.0.20180222 - OK.

Some are still a bit behind and possibly security related. Patching is trivial, but it would likely need some testing.

[Itms]

In 21683:

Update libcurl to 7.59.0 on Windows and enable SSL support on Windows and macOS.
Refs #3004, #4362.

[fabio]

@stanislas69 , @trompetin17

If you are testing OS X you may want to have a look at this. These two are still open:

• boost system may not be needed anymore; remove it if confirmed;
• verify how to use already built libpng for wxwidgets, rather than its builtin version (no need to provide two libpng, wx one is also older).

[Stan]

Hey Fabio, I think boost is still being used, I remember having issues with when I tried to switch to the VS2017 compiler.

Shouldn't we use the latest libpng possible ?

[fabio]

I meant just "boost system" should no longer be needed. On Linux it is no longer used. So I mean change:

--with-libraries=filesystem,system

to just:

--with-libraries=filesystem

And I would agree to update most/all libraries to their latest version, especially the ones with security implications.

It should be tested by someone having OS X.

[historic_bruno]

On a related note, it wouldn't be a bad idea to do checksum verification for all downloaded files within the script (especially since this is used for official releases), but that might belong in a separate ticket.

Replying to wraitii:

Now we voluntarily didn't use brew before. I'm not exactly sure why, but here are a few notes:

• Brew is largely the best and most common OS X package manager now
• It used to requires sudo at some point but no longer does
• It does not introduce conflicts with OSX's libraries, as it can provide "unlinked" versions (say, for libpng)
• it's got all the libraries we use.

I don't see why we should/could not rely on it now, at least.

The reason for the custom build script is mostly related to bundle distribution, where we want to target a specific SDK and minimal API version. If there's a package manager that can do that reliably, and not pick up incompatible libraries built against other SDKs or the local system libs, then we could certainly switch to it.

Or maybe have some sort of isolated package manager install just for the 0 A.D. build, but IMO the point is we can't just take any old packages built in other contexts and expect them to work in the bundles. And using static libs was 100% reliable, whereas picking up dylibs was kinda flaky back then (lots of major changes between OS X versions, plus Apple tended to not update their 3rd party libs).

What we don't want is a dev that already uses Homebrew for building random stuff, and then builds a 0 A.D. bundle using a Homebrew-based solution, and then it doesn't work on other macOS versions. That is what used to happen, both with Homebrew and MacPorts.

Another option would be to use e.g. Homebrew for non-release builds and use this script for bundle releases. But I thought having a single build path per OS was most sensible, even if the first full build does take a while (I forget how long it took, maybe 15-20 minutes back in 2015? on a quad-core 8GB RAM VM)

[wraitii]

FYI (This comment is 2 years old) - my current opinion is that x agree with you.

Still it's convenient for some libraries (wxwidgets notably)

[historic_bruno]

Replying to wraitii:

FYI (This comment is 2 years old) - my current opinion is that x agree with you.

Still it's convenient for some libraries (wxwidgets notably)

I figured, but couldn't remember if we ever discussed that or where, and wanted the original intent documented for posterity :)

I think looking back, my regret would be choosing bash instead of Python, not only because we might be able to reuse existing Python-based projects for this sort of thing (I've used at least one on a different project), but also the possibility of building many of the same libraries on Windows, which is currently not automated at all (even more of a pain and much more time-consuming).

I think eventually we could have a prebuilt package of macOS libraries with whatever the earliest SDK/API we support is. Similar to what we do with Windows now (in SVN), that way almost nobody would ever need to run this script at all. Just download the package and you're ready to build 0 A.D.

[elexis]

In r22455 by historic_bruno:

Fixes GnuTLS build on macOS.
Fixes macOS linker warning "PIE disabled absolute - addressing not allowed".

Updates nettle to 3.5.1, GnuTLS to 3.6.8, gloox to 1.0.22.
Disables TCP fast open feature of GnuTLS (requires 10.11, no SDK build support).
Fixes GnuTLS detection of GMP by adding it to LIBS flag.
Disables getaddrinfo on gloox 1.0.22. Lobby connections failed during server hostname resolution.
Adds --with-pic to GMP build to force consistent PIC usage.
Adds -N flag to patch commands to avoid reapplying them.
Removes unneeded build flags.
Documents --enable-fat configure flag: GMP and nettle detect CPU-specific features, fat binaries let us build and run them on different CPUs (see D1772).

Fixes #5453, #5489. Refs #5481.
Tested by: kali0ad, trompetin17
Reviewed by: trompetin17

Differential Revision: ​​​https://code.wildfiregames.com/D2057

[historic_bruno]

Note to self: replace http URLs with https.

[Itms]

r23302 updated libsodium to 1.0.18.

[Krinkle]

There's a post in the forums with a few ideas for lib upgrades.

https://wildfiregames.com/forum/index.php?/topic/28059-svn-public-alpha-24-version-on-macos-ui-problem/&tab=comments#comment-395406

I'm not sure there's a use for having this generic ticket though. Libs should be updated either because there is a problem of some sorts, of because we want to keep it closer with the versions used by other platforms (e.g. win/linux).

I suppose we could add a point to some sort of release checklist to see if there are patch releases we haven't applied and consider applying them, and if there are major upstream releases to file a ticket to look into whether that's useful etc.

I'd suggest closing this but open to comments from others :)

[Krinkle]

More possible lib updates by @Stan:

	ZLIB_VERSION="zlib-1.2.11"
	CURL_VERSION="curl-7.59.0" → 7.7.0
	ICONV_VERSION="libiconv-1.15" 1.16
	XML2_VERSION="libxml2-2.9.8"
	SDL2_VERSION="SDL2-2.0.5" →  2.0.12
	BOOST_VERSION="boost_1_64_0" → 1.73
	# NOTE: remember to also update LIB_URL below when changing version
	WXWIDGETS_VERSION="wxWidgets-3.0.3.1" →3.1.3
	
	# (also the Snow Leopard version was ancient 1.2)
	PNG_VERSION="libpng-1.6.34" → 37
	OGG_VERSION="libogg-1.3.3" → 1.3.4
	VORBIS_VERSION="libvorbis-1.3.6"
	# gloox requires GnuTLS, GnuTLS requires Nettle and GMP
	GMP_VERSION="gmp-6.1.2" → 6.2.0
	NETTLE_VERSION="nettle-3.5.1"
	# NOTE: remember to also update LIB_URL below when changing version
	GNUTLS_VERSION="gnutls-3.6.13" → 14
	GLOOX_VERSION="gloox-1.0.22" → 23
	# OS X only includes part of ICU, and only the dylib
	# NOTE: remember to also update LIB_URL below when changing version
	ICU_VERSION="icu4c-59_1" → 67 
	ENET_VERSION="enet-1.3.13" → .15
	MINIUPNPC_VERSION="miniupnpc-2.0.20180222" → 2.2
	SODIUM_VERSION="libsodium-1.0.18"
	# -----------------------------------------

[wraitii]

In 24257:

Upgrade Mac OS libraries

All are updated except:

• WxWidgets (not sure about compatibility)
• libogg (doesn't compile)
• zlib/libsodimum which have no update.

This drops supports for 10.11 and earlier, meaning we can now assume SSE4.1 is present.
Updates the build system to account for this.

Closes #5777 (by virtue of upgrading to SDL 2.12)

Refs #4362

Differential Revision: ​https://code.wildfiregames.com/D3111

[Stan]

Pushing to A25 as most updates have been performed already.

[Stan]

In 26788:

Update the LIB_URL for zlib.
Fixes: #6496
Refs: #4362
Patch by: @Langbart
Differential Revision: ​https://code.wildfiregames.com/D4603

[Stan]

Push back