Opened 4 years ago

Closed 4 years ago

#5729 closed defect (fixed)

gnutls-3.6.8 fails to build on macOS 10.15 Catalina

Reported by: martin_2m Owned by: Krinkle
Priority: Release Blocker Milestone: Alpha 24
Component: Build & Packages Keywords: macos
Cc: Patch: https://code.wildfiregames.com/D2716, https://code.wildfiregames.com/D2717

Description (last modified by Krinkle)

ld: weak import of symbol '___darwin_check_fd_set_overflow' not supported because of option: -no_weak_imports for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[4]: *** [gnutls-cli-debug] Error 1
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [all-recursive] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
ERROR: GnuTLS build failed

I would like to recommend the following changes to build the libraries. I found the suggestions for these changes here https://gitlab.com/gnutls/gnutls/-/issues/966

I have set the versions for 'gnutls' to 3.6.13. The download URL and the version of 'icu' I set to 66.1.

Here is the patch file for build-osx-libs.sh

--- original_0ad/0ad/libraries/osx/build-osx-libs.sh    2020-04-23 14:23:04.000000000 +0200
+++ 0ad/libraries/osx/build-osx-libs.sh 2020-04-23 13:57:39.000000000 +0200
@@ -37,11 +37,11 @@
 GMP_VERSION="gmp-6.1.2"
 NETTLE_VERSION="nettle-3.5.1"
 # NOTE: remember to also update LIB_URL below when changing version
-GNUTLS_VERSION="gnutls-3.6.8"
+GNUTLS_VERSION="gnutls-3.6.13"
 GLOOX_VERSION="gloox-1.0.22"
 # OS X only includes part of ICU, and only the dylib
 # NOTE: remember to also update LIB_URL below when changing version
-ICU_VERSION="icu4c-59_1"
+ICU_VERSION="icu4c-66_1"
 ENET_VERSION="enet-1.3.13"
 MINIUPNPC_VERSION="miniupnpc-2.0.20180222"
 SODIUM_VERSION="libsodium-1.0.18"
@@ -656,7 +656,6 @@
 LIB_ARCHIVE="$LIB_VERSION.tar.xz"
 LIB_DIRECTORY="$LIB_VERSION"
 LIB_URL="https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/"
-
 mkdir -p gnutls
 pushd gnutls > /dev/null

@@ -695,7 +694,7 @@
           --without-p11-kit \
           --disable-tests \
           --disable-nls \
-    && make ${JOBS} && make install) || die "GnuTLS build failed"
+    && make ${JOBS} LDFLAGS= install) || die "GnuTLS build failed"
   popd
   touch .already-built
 else
@@ -754,7 +753,7 @@
 LIB_VERSION="${ICU_VERSION}"
 LIB_ARCHIVE="$LIB_VERSION-src.tgz"
 LIB_DIRECTORY="icu"
-LIB_URL="http://download.icu-project.org/files/icu4c/59.1/"
+LIB_URL="https://github.com/unicode-org/icu/releases/download/release-66-1/"

 mkdir -p $LIB_DIRECTORY
 pushd icu > /dev/null

Attachments (2)

build-osx-libs.sh.patch (1.4 KB ) - added by martin_2m 4 years ago.
but--.png (149.5 KB ) - added by Krinkle 4 years ago.
Render failure (after GnuTLS and ICU fixes)

Download all attachments as: .zip

Change History (20)

by martin_2m, 4 years ago

Attachment: build-osx-libs.sh.patch added

comment:1 by martin_2m, 4 years ago

Component: Core engineBuild & Packages
Milestone: BacklogAlpha 24
Priority: Should HaveMust Have

comment:2 by Stan, 4 years ago

Component: Build & PackagesCore engine
Milestone: Alpha 24Backlog
Priority: Must HaveShould Have

Hey there,

Patches should be uploaded (with context) to https://code.wildfiregames.com for review.

If you need more info see wiki:SubmittingPatches and wiki:Phabricator

You can also add yourself to the credits when you do so.

Thanks for the patch!

comment:3 by Stan, 4 years ago

Component: Core engineBuild & Packages
Milestone: BacklogWork In Progress

comment:4 by martin_2m, 4 years ago

Milestone: Work In ProgressAlpha 24
Priority: Should HaveMust Have
Summary: build-osx-libs update[PATCH] build-osx-libs update

comment:5 by martin_2m, 4 years ago

Summary: [PATCH] build-osx-libs update[PATCH] build-osx-libs.sh update

comment:6 by Krinkle, 4 years ago

Keywords: macos added

comment:7 by Krinkle, 4 years ago

Description: modified (diff)

comment:8 by Krinkle, 4 years ago

Summary: [PATCH] build-osx-libs.sh updategnutls-3.6.8 fails to build on macOS 10.15 Catalina

comment:9 by Krinkle, 4 years ago

Description: modified (diff)

comment:10 by Krinkle, 4 years ago

TLDR: I can reproduce the compilation failure on macOS 10.15 as well, however the proposed patch does not fix it for me.


The initial failure for me was:

ERROR: GnuTLS build failed
 … autogen: command not found
 … autogen: command not found

I fixed this by installing autogen from Homebrew. The build instructions on the wiki at BuildInstructions#macOS document that only CMake is needed (e.g. brew install cmake). And that worked for me until now on macOS 10.14 Mojave. So this probably means that Movaje and/or a previous version of CMake included autogen. Assuming that this is indeed needed and not a cascading problem, we'll need to update the wiki to reflect this dependency.

After installing autogen, I see the same reported failure:

ld: weak import of symbol '___darwin_check_fd_set_overflow' not supported because of option: -no_weak_imports for architecture x86_64
clang: error: linker command failed with exit code 1
…
ERROR: GnuTLS build failed

However, after applying the proposed patch it still fails for me, albeit in a different way:

…
ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
In procedure dynamic-link: file: "/Users/krinkle/Development/0ad/libraries/osx/gnutls/gnutls-3.6.13/guile/src/guile-gnutls-v-2", message: "file not found"
make[2]: *** [modules/gnutls.go] Error 1
make[2]: *** Waiting for unfinished jobs....
  1209:24  3 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
   285:10  2 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
In unknown file:
           1 (load-extension "/Users/krinkle/Development/0ad/…?" ?)
In ice-9/boot-9.scm:
   752:25  0 (dispatch-exception _ _ _)

ice-9/boot-9.scm:752:25: In procedure dispatch-exception:
In procedure dynamic-link: file: "/Users/krinkle/Development/0ad/libraries/osx/gnutls/gnutls-3.6.13/guile/src/guile-gnutls-v-2", message: "file not found"
make[2]: *** [modules/gnutls/extra.go] Error 1
make[1]: *** [install-recursive] Error 1
make: *** [install-recursive] Error 1
ERROR: GnuTLS build failed

comment:11 by Krinkle, 4 years ago

The full patch I needed to make GnuTLS compile on macOS 10.15 Catalina also required me to disable the guil component.

I don't know GnuTLS or Guil very well, so I'm not sure whether this is actually something 0AD needs. However, I feel somewhat good that Homebrew's system-wide install for GnuTLS also disables Guil, so if they are able to still satisfy dependencies from arbitrary software needing GnuTLS without Guil, then that gives us a good chance we don't either.

https://github.com/Homebrew/homebrew-core/blob/415bf2782eeb2e9c44aac01d72c72f8c0c1ebc3f/Formula/gnutls.rb#L33-L44

--- a/libraries/osx/build-osx-libs.sh
+++ b/libraries/osx/build-osx-libs.sh
@@ -37,11 +37,11 @@ VORBIS_VERSION="libvorbis-1.3.6"
 GMP_VERSION="gmp-6.1.2"
 NETTLE_VERSION="nettle-3.5.1"
 # NOTE: remember to also update LIB_URL below when changing version
-GNUTLS_VERSION="gnutls-3.6.8"
+GNUTLS_VERSION="gnutls-3.6.13"
 GLOOX_VERSION="gloox-1.0.22"
 # OS X only includes part of ICU, and only the dylib
 # NOTE: remember to also update LIB_URL below when changing version
-ICU_VERSION="icu4c-59_1"
+ICU_VERSION="icu4c-59_2"
 ENET_VERSION="enet-1.3.13"
 MINIUPNPC_VERSION="miniupnpc-2.0.20180222"
 SODIUM_VERSION="libsodium-1.0.18"
@@ -694,8 +694,9 @@ then
           --with-included-libtasn1 \
           --without-p11-kit \
           --disable-tests \
+          --disable-guile \
           --disable-nls \
-    && make ${JOBS} && make install) || die "GnuTLS build failed"
+    && make ${JOBS} LDFLAGS= install) || die "GnuTLS build failed"
   popd

Unfortunately, after this the compilation fails on the next library, which is ICU. Their server went down last month and so the urls have changed.

The patch in the ticket description from Martin updates these urls, but also upgrades from ICU 59 to ICU 66 which I'm not sure we should do at the same time without much more testing, and presumably also doing it for Linux and Windows at the same time.

Using ICU 59 from the new location worked fine for me, and was able to compile all libraries, workspaces, and make pyrogenis without errors.

--- a/libraries/osx/build-osx-libs.sh
+++ b/libraries/osx/build-osx-libs.sh
-ICU_VERSION="icu4c-59_1"
+ICU_VERSION="icu4c-59_2"
 ENET_VERSION="enet-1.3.13"
 MINIUPNPC_VERSION="miniupnpc-2.0.20180222"
 SODIUM_VERSION="libsodium-1.0.18"
@@ -754,7 +755,7 @@ echo -e "Building ICU..."
 LIB_VERSION="${ICU_VERSION}"
 LIB_ARCHIVE="$LIB_VERSION-src.tgz"
 LIB_DIRECTORY="icu"
-LIB_URL="http://download.icu-project.org/files/icu4c/59.1/"
+LIB_URL="https://github.com/unicode-org/icu/releases/download/release-59-2/"

(ICU 59.1 is not avaliable on the new location, but 59.2 is.. I guess that's a compromise we'll have to accept.)

comment:13 by Krinkle, 4 years ago

Patch: D2716, D2717

comment:14 by Krinkle, 4 years ago

Patch: D2716, D2717D2716

comment:15 by Krinkle, 4 years ago

Patch: D2716https://code.wildfiregames.com/D2716, https://code.wildfiregames.com/D2717

comment:16 by Krinkle, 4 years ago

Owner: set to Krinkle
Priority: Must HaveRelease Blocker
Status: newassigned

by Krinkle, 4 years ago

Attachment: but--.png added

Render failure (after GnuTLS and ICU fixes)

comment:17 by Krinkle, 4 years ago

After the GnuTLS and ICU fixes, the libs, workspaces and game code all compile fine. However, the end result unplayable due to a serious rendering defect. The game is rendered at a tiny size, with the screen rendered red (see attached PNG).

Related: https://wildfiregames.com/forum/index.php?/topic/28059-svn-public-alpha-24-version-on-macos-ui-problem/

It's suggested that updating SDL2 might fix this problem. I'll look at that next.

comment:18 by wraitii, 4 years ago

Resolution: fixed
Status: assignedclosed

In 23772:

build-osx-libs: Update GnuTLS to 3.6.13 (unbreak macOS 10.15)

This includes a workaround that should be removed when it is fixed upstream (LDFLAGS=).
It disables guile support, since we do not need it and it fails to compile too.

Patch by: Krinkle

Reviewed By: wraitii

Fixes #5729

Differential Revision: https://code.wildfiregames.com/D2716

Note: See TracTickets for help on using tickets.