#4072 closed defect (fixed)
Remove Vfs Cache
Reported by: | wraitii | Owned by: | elexis |
---|---|---|---|
Priority: | Should Have | Milestone: | Alpha 23 |
Component: | Core engine | Keywords: | |
Cc: | Patch: | Phab:D587 |
Description (last modified by )
Reloading mods call InitVFS, which tries to allocate 200MB of contiguous memory each time. On windows 32-bit, this quickly ends up crashing because the memory can't be found. Reponsible function is in GameSetup.cpp See IRC on 26-06-2016. Maybe we should just remove the cache?
Change History (10)
comment:1 by , 8 years ago
Milestone: | Alpha 21 → Backlog |
---|
comment:2 by , 7 years ago
Keywords: | simple added |
---|---|
Priority: | Should Have → Nice to Have |
Summary: | Reloading mods several times crashes on Windows because the VFS Cache can't find memory → Remove Vfs Cache |
12:11 < Philip> Oh, the VFS cache seems silly to me 12:11 < Philip> The OS already caches files, we don't need our own cache on top of that 12:13 < Philip> (janwas still thought it was a good thing to have but I don't think I ever understood why)
Indeed not only the harddisk but also the OS does cache things. Removing the cache shouldn't be hard, see file_cache.cpp
and m_fileCache
in vfs.cpp
.
comment:3 by , 7 years ago
Component: | UI & Simulation → Core engine |
---|
comment:4 by , 7 years ago
Priority: | Nice to Have → Must Have |
---|
20:14 < Philip> The OS has a buffer cache, where it tries to use all your free RAM to cache parts of the hard disk 20:14 < Philip> (The hard disk's own cache is irrelevant) 20:15 < Philip> Having our own file cache in the game means there will be less free RAM, so the OS won't be able to cache as much stuff itsef 20:15 < Philip> and the OS is smarter at caching than we are 20:15 < Philip> (That assumes we don't open files with '''O_DIRECT''', which I don't think we do - maybe we used to in the past) 20:15 < Philip> Also the game just doesn't re-open files frequently 20:16 < Philip> since there are loads of higher-level caches of templates, textures, etc 20:16 < Philip> s/re-open files/re-read pages from public.zip/ 20:17 < Philip> The zip file has a sort of directory listing of all its contents, so we read that into RAM, and then when the game wants to read a particular file it looks it up in that directory and then just reads the relevant section from within the zip file 20:19 < Philip> The game's 'file' caching is at a higher level - it caches the files after they've been read out of the zip (and decompressed, if necessary) 20:20 < Philip> The OS's caching is at a lower level, it just caches ~4KB pages that can be in the middle of a file or can be filesystem metadata etc
For windows: https://msdn.microsoft.com/en-US/library/windows/desktop/aa364218(v=vs.85).aspx
TODO: find some specs for linux and OSX to verify that our target platforms are covered
comment:5 by , 7 years ago
Linux
Write
The writing data operations are first written in Page Cache and in another moment are transferred to drive unity. We could remember the command sync that transfers all page cache to devices and them we can unplug the pen-drive, for example. Newer kernels have one thread by hard device to be faster do transfer operations. Since 2.6 if my memory is good
Read
In the same way of write, the read operation sends the data to page cache and then you read them. When another read are made, the page cache it is consumed. The page cache data is cleared when additional memory is required by some application.
Well, I found a link that explains better than me: https://www.thomas-krenn.com/en/wiki/Linux_Page_Cache_Basics
comment:6 by , 7 years ago
Description: | modified (diff) |
---|---|
Milestone: | Backlog → Work In Progress |
Patch: | → Phab:D587 |
comment:7 by , 6 years ago
Keywords: | simple removed |
---|
comment:8 by , 6 years ago
Milestone: | Work In Progress → Alpha 23 |
---|---|
Priority: | Must Have → Should Have |
comment:10 by , 5 years ago
Here janwas paper "Optimizing File Accesses via Ordering and Caching":
http://wassenberg.dreamhosters.com/articles/study_thesis.pdf
https://code.wildfiregames.com/F859317
The error isn't triggered as often anymore on 32bit windows as of #4190. Backlogging due to lack of progress.