Disable compression of public.zip

[Philip Taylor]

Using zip compression when generating public.zip for release packages has a significant detrimental effect on download size (maybe around 25%), because it conflicts with the (much better) bz2/lzma package compression. People would probably have to play the game dozens of times for the gain in startup performance to outweigh the extra download time, so the tradeoff doesn't seem good at our current stage. So we should disable compression now, for the clear improvement in download size, and then later we can spend more time thinking about and measuring and optimising startup performance again.

[fabio]

Where the improvements to the startup performance from the public.zip come from? I suspect it is from the many smaller files that get a higher compression ratio.

What about:

• removing all the files that actually go uncompressed (ogg, png) from the zip file so that bzip2/lzma has a greater chance of compressing them later on;

•  create the public.zip using 7z (in zip mode, also available on linux with p7zip/p7zip-full) using best options (7z a -mx=9 -mfb=258 -mpass=15 public.zip files...) which has a greater (10%) compression ratio;

With these the difference between using or not public.zip should be a lot smaller. Then also:

• create the xz dist files adding the -e (extreme) option;

• create the bz2 dist files using 7z a -mx=9 -mpass=10 file.bz2 files... ;

• create the gz dist files again using 7z (rather than gz) with 7z a -mx=9 -mfb=258 -mpass=15 file.zip files... .

[Philip Taylor]

See ​Jan's old work on IO performance. The main benefit seems to be from simply packing everything into a single file, avoiding the OS/filesystem overhead of opening thousands of individual files, rather than from the compression.

We already store some files without compression - IsFileTypeIncompressible in archive_zip.cpp says

    L".zip", L".rar",
    L".jpg", L".jpeg", L".png", 
    L".ogg", L".mp3"

Looks like we actually compress with Zlib with Z_BEST_SPEED, which is silly, due to misconfiguration. Fixed in r8740. 7-Zip's implementation would probably be better but it doesn't seem trivial (given that the zip generation is currently embedded in the engine, and if we used external tools instead then we'd have to be careful about keeping some files uncompressed and about controlling the order of files).

I'll try out those options for generating the release distribution files.

[fabio]

OK, at a quick search I thought compression was done with source/tools/archbuilder/archbuild.pl

But why adding these features (public.zip compression and png -> dds conversion) to the same files that get used by players rather than using a script or a dedicated executable?

Anyway having the ogg files out of public.zip shouldn't increase too much the number of opened files while it could improve a bit the compression since these are a big part of public.zip size (I didn't do any test however).

[Philip Taylor]

(Deleted old tool in r8761.)

We want to provide good easy-to-use cross-platform modding support, so users should be able to edit content and generate mod packages (as zip files) without installing any additional tools. Those are the same features we need for developing and distributing the official game content. Integrating those features into the main game executable means there are fewer technical problems than when building and calling an external tool, and it allows a lot of code sharing (reducing complexity and file size etc).

The ogg files are uncompressed inside public.zip so I don't think there would be any major difference when removing them from the zip. It's basically the same as storing them all in a .tar file (except with the .zip file format instead) - the xz/bz2/gz compression doesn't care about the distinction between individual files, it'll see basically the same stream of bytes in either case.

[Philip Taylor]

I did some tests for this, using the current data files but without the cached texture conversion (since I was too impatient and it shouldn't affect much):

• zip compressed: 189MB (the Z_BEST_SPEED made it 7MB worse than that)
• zip uncompressed: 320MB
• zip compressed + xz -9: 184MB
• zip uncompressed + xz -9: 162MB
• zip uncompressed + xz -e9: 161MB

So it'll save about 28MB download size, and use 130MB more disk space. Probably worthwhile for now.

[philip]

(In [8785]) Disable compression of public.zip (fixes #671).

[Philip Taylor]

Replying to fabio:

• create the gz dist files again using 7z (rather than gz) with 7z a -mx=9 -mfb=258 -mpass=15 file.zip files... .

Using 7z -mx=9 saves about 3.5MB (vs gzip -9), which is good. Adding -mfb=258 -mpass=15 takes about half an hour longer and saves about 50KB, so that part doesn't seem worthwhile.