Changes between Initial Version and Version 1 of Virtual_File_System

Timestamp:

Feb 23, 2008, 4:18:59 AM (16 years ago)

Author:

trac

Comment:

--

Virtual_File_System

@@ +1 @@
+[KEEP IN SYNC WITH VFS.H!]
+
+== Introduction ==
+
+The VFS (Virtual File System) is a layer between the application and file.cpp's API. Its main purpose is to decrease the cost of file access; also provided for are "hotloading" and "modding" via overriding files
+(explained below).
+The interface is almost identical to that of file.cpp, except that it works with Handles for safety (see [wiki:handle_manager h_mgr.h]).
+
+
+== File Access Cost ==
+
+Games typically encompass thousands of files. Such heavy loads expose 2 problems with current file systems:
+ * wasted disk space. An average of half a cluster (>= 1 sector, typically 512 bytes) is lost per file due to internal fragmentation.
+ * lengthy file open times. Permissions checks and overhead added by antivirus scanners combine to make these slow. Additionally, files are typically not arranged in order of access, which induces costly disk seeks.
+
+The solution is to put all files in archives: internal fragmentation is eliminated since they are packed end-to-end; open is much faster; seeks are avoided by arranging in order of access. For more information, see 'Archive Details' below.
+
+Note that a good file system (Reiser3 comes close) could also deliver the above. However, this code is available now on all platforms; there is no disadvantage to using it and the other features remain.
+
+
+== Hotloading ==
+
+During development, artists and programmers typically follow a edit/see how it looks in-game/repeat methodology. Unfortunately, changes to a file are not immediately noticed by the game; the usual workaround is to
+restart the map (or worse, entire game) to make sure they are reloaded. Since decreases in edit cycle time improve productivity, we want changes to files to be picked up immediately. To that end, we support hotloading -
+as soon as the OS reports changes, all Handle objects that ensued from that file are reloaded.
+
+The VFS's part in this is registering "watches" that report changes to any mounted real directory. Since the file notification backend (currently SGI FAM and a Win32 port) cannot watch an entire directory tree,
+we need to do so for every single directory. The VFS traverses and stores data for them anyway, we do so here.
+
+
+== Modding ==
+
+=== Motivation ===
+
+When users tweak game parameters or even create an entirely new game principle with the same underlying engine, it is called modding. As evidenced by the Counterstrike mod for Half-Life, this can greatly prolong the life of a game. Additionally, since we started out as a mod group, great value is placed on giving users all the tools to make modding easy.
+
+=== Means ===
+
+The actual method of overriding game data is quite simple: a mod directory is mounted into the file system with a higher priority than original data. These files therefore temporarily (as long as the mod is active) replace the
+originals. This allows multiple (non-overlapping!) mods to be active at the same time and also makes switching between them easy. The same mechanism is also used for patches to game data.
+
+=== Rationale ===
+
+Older games did not provide any support for modding other than directly editing game data. Obviously this is risky and insufficient. Requiring mods to provide a entire new copy of all game logic/scripts would obviate support from the file system, but is too much work for the modder (since all files would first have to be copied somewhere). Allowing overriding individual files is much safer (since game data is never touched) and easier (more fine-grained control for modders).
+
+Alternatives to the patch archive approach would be to completely replace the game data archive (infeasible due to size) or apply a binary patch (complicated and brittle WRT versioning). We are therefore happy to use the already existing mod mechanism.
+
+Note however that multiple patches do impact performance (despite constant-time VFS path -> file location lookup) simply due to locality; files are no longer arranged in order of access. Fortunately there is an easy way to avoid this: simply run the archive builder script; all patched files will be merged into the archive.
+
+
+For more information, see 'Mount Details' below.
+
+
+== Mount Details ==
+
+"Mounting" is understood to mean populating a given VFS directory (the "mount point") with the contents of e.g. a real directory or archive (the "mounted object" - for a list of supported types, see enum !MountType).
+
+It is important to note that the VFS is a full-fledged tree storing information about each file, e.g. its last-modified time or actual location. The advantage is that file open time does not increase with the number of
+mounts, which is important because multiple patches and mods may be active. This is in contrast to e.g. PhysicsFS, which just maintains a list of mountings and scans it when opening each file.
+
+Each file object in the VFS tree stores its current location; there is no way to access files of the same name but lower priority residing in other mounted objects. For this reason, the entire VFS must be rebuilt (i.e. repopulating all mount points) when a mounting is removed. Fortunately this is rare and does not happen in-game; we optimize for the common case.
+
+
+== Archive Details ==
+
+=== Rationale ===
+
+An open format (.zip) was chosen instead of a proprietary solution for the following reasons:
+ * interoperability: anyone can view or add files without the need for special tools, which is important for modding.
+ * less work: freely available decompression code (ZLib) eases implementation. Disadvantages are efficiency (only adequate; an in-house format would offer more potential for optimization) and lacking protection of data files.
+Interoperability is a double-edged sword, since anyone can change critical files or use game assets. However, obfuscating archive contents doesn't solve anything, because the application needs to access them and a cracker
+need only reverse-engineer that. Regardless, the application can call its archives e.g. ".pk3" (as does Quake III) for minimal protection.
+
+=== Archive Builder ===
+
+Arranging archive contents in order of access was mentioned above. To that end, the VFS can log all file open calls into a text file (one per line). This is then processed by an archive builder script, which needs to
+collect all files by VFS lookup rules, then add them to the archive in the order specified in that file (all remaining files that weren't triggered in the logging test run should be added thereafter).
+
+Note that the script need only be a simple frontend for e.g. infozip, and that a plain user-created archive will work as well (advantage of using Zip); this is just an optimization.
+
+=== Misc. Notes ===
+
+To ease development, files may additionally be stored in normal directories. The VFS transparently provides access to the correct (newest) version.
+
+One additional advantage of archives over loose files is that I/O throughput is increased - since files are compressed, there is less to read from disk. Decompression is free because it is done in parallel with IOs.