Opened 3 years ago

Last modified 3 days ago

#4362 new task

Update and cleanup OS X libraries

Reported by: fabio Owned by: Itms
Priority: Should Have Milestone: Alpha 24
Component: Build & Packages Keywords: mac
Cc: stanislas69, trompetin17, Krinkle Patch: Phab:D1691

Description (last modified by Itms)

Update and cleanup build-osx-libs.sh:

  • update libraries to latest releases (should do for alpha22, some have security issues);
  • boost system may not be needed anymore; remove it if confirmed;
  • verify how to use already built libpng for wxwidgets, rather than its builtin version (no need to provide two libpng, wx one is also older).

Change History (34)

comment:1 Changed 3 years ago by wraitii

I created a branch that relies on Homebrew to download all non-bundled libs (except for gloox, as we do not use ssl and brew does).

https://github.com/wraitii/0ad/tree/OSX_libs_brew

This seems to compile on my system (10.11.6), and has the advantages of:

  • cutting down compile time substantially, particularly for wxwidgets/boost
  • making sure we always use the latest versions

Now we voluntarily didn't use brew before. I'm not exactly sure why, but here are a few notes:

  • Brew is largely the best and most common OS X package manager now
  • It used to requires sudo at some point but no longer does
  • It does not introduce conflicts with OSX's libraries, as it can provide "unlinked" versions (say, for libpng)
  • it's got all the libraries we use.

I don't see why we should/could not rely on it now, at least.

NB: I removed libiconv because it seemed to be only used to compile the others, but I'm not actually sure about that.

Last edited 3 years ago by wraitii (previous) (diff)

comment:2 Changed 3 years ago by wraitii

Keywords: rfc patch added

comment:3 Changed 3 years ago by wraitii

As a sidenote: using brew libraries and hot linking seems to work but fails to compile into a binary that's self-sufficient, it'll probably take more changes. As a side-sidenote, wxwidgets 3.0.2 no longer compiles on macOs 10.12 (Sierra), you have to use 3.1.0

edit:hm actually not either, we'll have to port an upstream patch.

Last edited 3 years ago by wraitii (previous) (diff)

comment:4 Changed 2 years ago by Itms

Description: modified (diff)

Can we push that to A23 or is there an important library update that needs to be performed before packaging?

comment:5 Changed 2 years ago by fabio

Most (all?) libraries in build-osx-libs.sh should be updated for a22, some due to security issues.

Wraitii use of Homebrew can wait for a future release.

comment:6 Changed 2 years ago by fabio

Suggested for a22:

-ZLIB_VERSION="zlib-1.2.8"
-CURL_VERSION="curl-7.46.0"
-ICONV_VERSION="libiconv-1.14"
-XML2_VERSION="libxml2-2.9.3"
-SDL2_VERSION="SDL2-2.0.4"
-BOOST_VERSION="boost_1_60_0"
-WXWIDGETS_VERSION="wxWidgets-3.0.2"
+ZLIB_VERSION="zlib-1.2.11"
+CURL_VERSION="curl-7.54.0"
+ICONV_VERSION="libiconv-1.15"
+XML2_VERSION="libxml2-2.9.4"
+SDL2_VERSION="SDL2-2.0.5"
+BOOST_VERSION="boost_1_64_0"
+WXWIDGETS_VERSION="wxWidgets-3.0.3"
 # libpng was included as part of X11 but that's removed from Mountain Lion
 # (also the Snow Leopard version was ancient 1.2)
-PNG_VERSION="libpng-1.6.21"
+PNG_VERSION="libpng-1.6.29"
 OGG_VERSION="libogg-1.3.2"
 VORBIS_VERSION="libvorbis-1.3.5"
 # gloox is necessary for multiplayer lobby
-GLOOX_VERSION="gloox-1.0.14"
+GLOOX_VERSION="gloox-1.0.20"
 # NSPR is necessary for threadsafe Spidermonkey
-NSPR_VERSION="4.11"
+NSPR_VERSION="4.15"
 # OS X only includes part of ICU, and only the dylib
 # NOTE: remember to also update LIB_URL below when changing version
-ICU_VERSION="icu4c-56_1"
+ICU_VERSION="icu4c-59_1"
 ENET_VERSION="enet-1.3.13"
-MINIUPNPC_VERSION="miniupnpc-1.9.20151026"
+MINIUPNPC_VERSION="miniupnpc-1.9.20160209"

comment:7 Changed 2 years ago by Itms

Alright, I hope wraitii can look into that.

I'm going to look into performing a few updates on Windows as well, they are long due.

comment:8 Changed 2 years ago by elexis

Besides that zlib version not being available anymore (reported in #4639), nigel87 also uses OSX Sierra 10.12 which fails to build wxWidgets due to including quicktime which was dropped from that OS: http://trac.wxwidgets.org/ticket/17639 https://forums.wxwidgets.org/viewtopic.php?t=42856

He tried passing WXWIDGETS_VERSION="wxWidgets-3.0.3" and added --disable-qtkit for CONF_OPTS in L356, but that didn't help.

comment:9 Changed 2 years ago by elexis

In 19825:

Update OSX zlib version to the most recent and now only one hosted.

Fixes #4639
Refs #4362
Differential Revision: https://code.wildfiregames.com/D644
Tested By: Tobbi

comment:10 Changed 2 years ago by elexis

Keywords: rfc removed

comment:11 Changed 2 years ago by elexis

In 19848:

Update OSX libraries to the most recently released versions.

Remove wxWidgets patch from rP16155 that had been merged with the new release.

Differential Revision: https://code.wildfiregames.com/D679
Refs #4362
Based On Patch By: fabio
Discussed With: leper
Tested By: Tobbi, Itms

comment:12 Changed 2 years ago by elexis

Four exploits in the latest release of libxml2 mentioned in https://code.wildfiregames.com/D679?id=2703#inline-12860 should be either patched or it should be confirmed that we are not affected. Rest of the library versions should be checked for exploitable vulnerabilities too.

comment:13 Changed 2 years ago by elexis

Phab:D699 for the libxml2 snapshot.

comment:14 Changed 2 years ago by elexis

Milestone: Alpha 22Backlog

I set this to backlog because noone has the will to update libxml2 to the most recent dev snapshot before the alpha 22 release and because we will need a new ticket to keep track of updates for the next releases if this was closed as fixed.

comment:15 Changed 19 months ago by stan

for macosx, im encountering error: 'connectx' is only available on macOS 10.11 or newer while installing libraries on curl-7.54.0, fix is to update to curl-7.56.0, see discussion here: https://github.com/VCVRack/Rack/pull/200 quick fix is to update this line to : CURL_VERSION="curl-7.56.0" in libraries/osx/build-osx-libs.sh

Last edited 19 months ago by stan (previous) (diff)

comment:16 Changed 17 months ago by Itms

Keywords: patch removed
Milestone: BacklogAlpha 23
Owner: set to Itms

I am going to perform the curl change on macOS ASAP.

comment:17 Changed 17 months ago by Itms

In 21501:

Update libcurl for OSX, in order to fix the build. Use the opportunity to update libxml2 which had vulnerabilities.
Refs #4362

comment:18 Changed 17 months ago by Itms

Milestone: Alpha 23Alpha 24

comment:19 Changed 16 months ago by fabio

Given #4790 was recently closed, here is an update of latest libraries and current status (to be A23) for OS X:

Most are a bit behind and some are possibly security related. Patching is trivial, but it would likely need some testing.

Last edited 16 months ago by fabio (previous) (diff)

comment:20 Changed 16 months ago by fabio

  • curl 7.58 -> 7.59 fixes: CVE-2018-1000120, CVE-2018-1000121, CVE-2018-1000122
  • miniupnpc ​2.0.20180222 -> 2.0.20170509 fixes: "Fix buffer over run in minixml.c", "Fix uninitialized variable access in upnpreplyparse.c"
  • libvorbis 1.3.5 -> 1.3.6 fixes: CVE-2018-5146, CVE-2017-14632, CVE-2017-14633

Will update some low risk libraries here.

comment:21 Changed 16 months ago by fabio

In 21583:

Update some OS X libraries with security fixes and hopefully low regression risks. Refs #4362.

comment:22 Changed 16 months ago by fabio

Given #4790 was recently closed, here is an update of latest libraries and current status (to be A23) for OS X:

  • zlib: 1.2.11 - OK;
  • curl 7.59.0 - OK;
  • iconv: 1.15 - OK;
  • libxml2: 2.9.8 - OK;
  • libsdl2 2.0.6, 2.0.7. 2.0.8 - 2.0.5;
  • boost 1.66.0 - 1.64;
  • wxWidgets 3.0.4 - 3.0.3.1;
  • libpng 1.6.34 - OK;
  • libogg 1.3.3 - OK;
  • libvorbis 1.3.6 - OK;
  • gloox 1.0.20 - OK;
  • nspr 4.19 - 4.15;
  • icu4c 60.2 - 59.1;
  • enet 1.3.13 - OK;
  • miniupnpc 2.0.20180222 - OK.

Some are still a bit behind and possibly security related. Patching is trivial, but it would likely need some testing.

comment:23 Changed 16 months ago by Itms

In 21683:

Update libcurl to 7.59.0 on Windows and enable SSL support on Windows and macOS.
Refs #3004, #4362.

comment:24 Changed 7 months ago by fabio

Cc: stanislas69 trompetin17 added

@stanislas69 , @trompetin17

If you are testing OS X you may want to have a look at this. These two are still open:

  • boost system may not be needed anymore; remove it if confirmed;
  • verify how to use already built libpng for wxwidgets, rather than its builtin version (no need to provide two libpng, wx one is also older).

comment:25 Changed 7 months ago by stanislas69

Hey Fabio, I think boost is still being used, I remember having issues with when I tried to switch to the VS2017 compiler.

Shouldn't we use the latest libpng possible ?

comment:26 Changed 7 months ago by fabio

I meant just "boost system" should no longer be needed. On Linux it is no longer used. So I mean change:

--with-libraries=filesystem,system

to just:

--with-libraries=filesystem

And I would agree to update most/all libraries to their latest version, especially the ones with security implications.

It should be tested by someone having OS X.

comment:27 Changed 7 months ago by stanislas69

Patch: Phab:D1691

comment:28 Changed 5 weeks ago by Krinkle

Cc: Krinkle added

comment:29 in reply to:  1 Changed 2 weeks ago by historic_bruno

On a related note, it wouldn't be a bad idea to do checksum verification for all downloaded files within the script (especially since this is used for official releases), but that might belong in a separate ticket.

Replying to wraitii:

Now we voluntarily didn't use brew before. I'm not exactly sure why, but here are a few notes:

  • Brew is largely the best and most common OS X package manager now
  • It used to requires sudo at some point but no longer does
  • It does not introduce conflicts with OSX's libraries, as it can provide "unlinked" versions (say, for libpng)
  • it's got all the libraries we use.

I don't see why we should/could not rely on it now, at least.

The reason for the custom build script is mostly related to bundle distribution, where we want to target a specific SDK and minimal API version. If there's a package manager that can do that reliably, and not pick up incompatible libraries built against other SDKs or the local system libs, then we could certainly switch to it.

Or maybe have some sort of isolated package manager install just for the 0 A.D. build, but IMO the point is we can't just take any old packages built in other contexts and expect them to work in the bundles. And using static libs was 100% reliable, whereas picking up dylibs was kinda flaky back then (lots of major changes between OS X versions, plus Apple tended to not update their 3rd party libs).

What we don't want is a dev that already uses Homebrew for building random stuff, and then builds a 0 A.D. bundle using a Homebrew-based solution, and then it doesn't work on other macOS versions. That is what used to happen, both with Homebrew and MacPorts?.

Another option would be to use e.g. Homebrew for non-release builds and use this script for bundle releases. But I thought having a single build path per OS was most sensible, even if the first full build does take a while (I forget how long it took, maybe 15-20 minutes back in 2015? on a quad-core 8GB RAM VM)

Last edited 2 weeks ago by historic_bruno (previous) (diff)

comment:30 Changed 13 days ago by wraitii

FYI (This comment is 2 years old) - my current opinion is that x agree with you.

Still it's convenient for some libraries (wxwidgets notably)

Last edited 13 days ago by wraitii (previous) (diff)

comment:31 in reply to:  30 Changed 13 days ago by historic_bruno

Replying to wraitii:

FYI (This comment is 2 years old) - my current opinion is that x agree with you.

Still it's convenient for some libraries (wxwidgets notably)

I figured, but couldn't remember if we ever discussed that or where, and wanted the original intent documented for posterity :)

I think looking back, my regret would be choosing bash instead of Python, not only because we might be able to reuse existing Python-based projects for this sort of thing (I've used at least one on a different project), but also the possibility of building many of the same libraries on Windows, which is currently not automated at all (even more of a pain and much more time-consuming).

I think eventually we could have a prebuilt package of macOS libraries with whatever the earliest SDK/API we support is. Similar to what we do with Windows now (in SVN), that way almost nobody would ever need to run this script at all. Just download the package and you're ready to build 0 A.D.

comment:32 Changed 10 days ago by elexis

In r22455 by historic_bruno:

Fixes GnuTLS build on macOS.
Fixes macOS linker warning "PIE disabled absolute - addressing not allowed".

Updates nettle to 3.5.1, GnuTLS to 3.6.8, gloox to 1.0.22.
Disables TCP fast open feature of GnuTLS (requires 10.11, no SDK build support).
Fixes GnuTLS detection of GMP by adding it to LIBS flag.
Disables getaddrinfo on gloox 1.0.22. Lobby connections failed during server hostname resolution.
Adds --with-pic to GMP build to force consistent PIC usage.
Adds -N flag to patch commands to avoid reapplying them.
Removes unneeded build flags.
Documents --enable-fat configure flag: GMP and nettle detect CPU-specific features, fat binaries let us build and run them on different CPUs (see D1772).

Fixes #5453, #5489. Refs #5481.
Tested by: kali0ad, trompetin17
Reviewed by: trompetin17

Differential Revision: ​​https://code.wildfiregames.com/D2057

comment:33 Changed 10 days ago by Krinkle

Keywords: mac added

comment:34 Changed 3 days ago by historic_bruno

Note to self: replace http URLs with https.

Note: See TracTickets for help on using tickets.