Introduction

The VFS (Virtual File System) is an abstraction layer that simplifies file access for game data. Its main purpose is to have platform independent file access methods, decrease the cost of file access, and modding support. It also provides hotloading support to allow fast iteration of files during development.

File Access Cost

Games typically encompass thousands of files. Such heavy loads expose 2 problems with current file systems:

  • wasted disk space. An average of half a cluster (>= 1 sector, typically 512 bytes) is lost per file due to internal fragmentation.
  • lengthy file open times. Permissions checks and overhead added by antivirus scanners combine to make these slow. Additionally, files are typically not arranged in order of access, which induces costly disk seeks.

The solution is to put all files in archives: internal fragmentation is eliminated since they are packed end-to-end; open is much faster; seeks can be avoided by arranging in order of access. For more information, see 'Archive Details' below.

Note that a good file system could also provide the above. However this depends on machine specific settings, while our used approach has no such disadvantage.

Hotloading

During development, artists and programmers typically follow a edit/see how it looks in-game/repeat methodology. Unfortunately, changes to a file are not immediately noticed by the game. The usual workaround is to restart the map (or worse, entire game) to make sure they are reloaded. Since decreases in edit cycle time improve productivity, we want changes to files to be picked up immediately. To that end, we support hotloading - as soon as the OS reports changes, all Handle objects that ensued from that file are reloaded.

The VFS's part in this is registering "watches" that report changes to any mounted real directory. Since the file notification backend (or some OS specific APIs) cannot watch an entire directory tree, we need to do so for every single directory. The VFS traverses and stores data for them anyway, we do so here.

Modding

Motivation

When users tweak game parameters or even create an entirely new game principle with the same underlying engine, it is called modding. As evidenced by the Counterstrike mod for Half-Life, this can greatly prolong the life of a game. Additionally, since we started out as a mod group, great value is placed on giving users all the tools to make modding easy.

Means

The actual method of overriding game data is quite simple: a mod directory is mounted into the file system with a higher priority than original data. These files therefore temporarily (as long as the mod is active) replace the originals. This allows multiple mods to be active at the same time and also makes switching between them easy. The same mechanism could also be used for patches to game data.

File Replacement

A newly mounted mod has its files added to the VFS if the files have a higher priority, or they have the same priority but a newer timestamp (e.g. update packs, development), or the used file loader (archives are preferred over loose files) is more efficient.

File Removal

Mods can remove files from lower priority mods. To remove file a the mod has to provide a file named a.DELETED (contents do not matter, file should probably be empty to save space). Removing directories follows a similar approach where a directory b in a lower priority mod is removed if a file named b.DELETED is in the mod. (Not yet included; see #2641) All files with a lower priority than the .DELETED file in this directory (and all subdirectories) are then removed, and empty subdirectories are also removed. Note that the directory cannot be removed unconditionally since it might contain higher priority (or same priority) files which should be kept.

Rationale

Older games did not provide any support for modding other than directly editing game data. Obviously this is risky and insufficient. Requiring mods to provide a entire new copy of all game logic/scripts would obviate support from the file system, but is too much work for the modder (since all files would first have to be copied somewhere). Allowing overriding individual files is much safer (since game data is never touched) and easier (more fine-grained control for modders).

Alternatives to the patch archive approach would be to completely replace the game data archive (infeasible due to size) or apply a binary patch (complicated and brittle WRT versioning). We are therefore happy to use the already existing mod mechanism.

Note however that multiple patches do impact performance (despite constant-time VFS path -> file location lookup) simply due to locality; files are no longer arranged in order of access. Fortunately there is an easy way to avoid this: simply run the archive builder script; all patched files will be merged into the archive.

For more information, see 'Mount Details' below.

Mount Details

"Mounting" is understood to mean populating a given VFS directory (the "mount point") with the contents of e.g. a real directory or archive (the "mounted object"). It is important to note that the VFS is a full-fledged tree storing information about each file, e.g. its last-modified time or actual location. The advantage is that file open time does not increase with the number of mounts, which is important because multiple patches and mods may be active. This is in contrast to e.g. PhysicsFS, which just maintains a list of mountings and scans it when opening each file.

Each file object in the VFS tree stores its current location; there is no way to access files of the same name but lower priority residing in other mounted objects. For this reason, the entire VFS must be rebuilt (i.e. repopulating all mount points) when a mounting is removed. Fortunately this is rare and does not happen in-game (apart from the mod selector), so we optimize for the common case.

Archive Details

Rationale

An open format (.zip) was chosen instead of a proprietary solution for the following reasons:

  • interoperability: anyone can view or add files without the need for special tools, which is important for modding.
  • less work: freely available decompression code (ZLib) eases implementation. Disadvantages are efficiency (only adequate; an in-house format would offer more potential for optimization) and lacking protection of data files.

Interoperability is a double-edged sword, since anyone can change critical files or use game assets. However, obfuscating archive contents doesn't solve anything, because the application needs to access them and a cracker need only reverse-engineer that. Regardless, the application can call its archives e.g. ".pk3" (as does Quake III) for minimal protection.

Archive Builder

The game supports building archives via the -archivebuild command line parameters. This adds all files in the specified mod (and their cached variants) to a .zip archive. Optionally the archive can be compressed thereby slightly improving file access times (less data to read, decompression overhead is not an issue since it is done in parallel with IOs).

Access Order Arrangement

To speed up access times the archived files could be sorted in access order, but this optimization is currently not needed (and it would only work well for some (relatively) common cases). However, the VFS can log all file open calls into a text file (one per line). This could then be processed by an archive builder script, which needs to collect all files by VFS lookup rules, then add them to the archive in the order specified in that file (all remaining files that weren't triggered in the logging test run should be added thereafter).

Note that the script need only be a simple frontend for e.g. infozip, and that a plain user-created archive will work as well (advantage of using Zip); this is just a possible optimization.

Last modified 15 months ago Last modified on Jan 13, 2016 8:14:21 PM