Opened 8 years ago

Closed 6 years ago

Last modified 5 years ago

#4072 closed defect (fixed)

Remove Vfs Cache

Reported by: wraitii Owned by: elexis
Priority: Should Have Milestone: Alpha 23
Component: Core engine Keywords:
Cc: Patch: Phab:D587

Description (last modified by Sandarac)

Reloading mods call InitVFS, which tries to allocate 200MB of contiguous memory each time. On windows 32-bit, this quickly ends up crashing because the memory can't be found. Reponsible function is in GameSetup.cpp See IRC on 26-06-2016. Maybe we should just remove the cache?

Change History (10)

comment:1 by elexis, 8 years ago

Milestone: Alpha 21Backlog

The error isn't triggered as often anymore on 32bit windows as of #4190. Backlogging due to lack of progress.

comment:2 by elexis, 7 years ago

Keywords: simple added
Priority: Should HaveNice to Have
Summary: Reloading mods several times crashes on Windows because the VFS Cache can't find memoryRemove Vfs Cache
12:11 < Philip> Oh, the VFS cache seems silly to me
12:11 < Philip> The OS already caches files, we don't need our own cache on top of that
12:13 < Philip> (janwas still thought it was a good thing to have but I don't think I ever understood why)

Indeed not only the harddisk but also the OS does cache things. Removing the cache shouldn't be hard, see file_cache.cpp and m_fileCache in vfs.cpp.

comment:3 by elexis, 7 years ago

Component: UI & SimulationCore engine

comment:4 by elexis, 7 years ago

Priority: Nice to HaveMust Have
20:14 < Philip> The OS has a buffer cache, where it tries to use all your free RAM to cache parts of the hard disk
20:14 < Philip> (The hard disk's own cache is irrelevant)
20:15 < Philip> Having our own file cache in the game means there will be less free RAM, so the OS won't be able to cache as much stuff itsef
20:15 < Philip> and the OS is smarter at caching than we are
20:15 < Philip> (That assumes we don't open files with '''O_DIRECT''', which I don't think we do - maybe we used to in the past)
20:15 < Philip> Also the game just doesn't re-open files frequently
20:16 < Philip> since there are loads of higher-level caches of templates, textures, etc
20:16 < Philip> s/re-open files/re-read pages from public.zip/
20:17 < Philip> The zip file has a sort of directory listing of all its contents, so we read that into RAM, and then when the game wants to read a particular file it looks it up in that directory and then just reads the relevant section from within the zip file
20:19 < Philip> The game's 'file' caching is at a higher level - it caches the files after they've been read out of the zip (and decompressed, if necessary)
20:20 < Philip> The OS's caching is at a lower level, it just caches ~4KB pages that can be in the middle of a file or can be filesystem metadata etc

For windows: https://msdn.microsoft.com/en-US/library/windows/desktop/aa364218(v=vs.85).aspx

TODO: find some specs for linux and OSX to verify that our target platforms are covered

Last edited 7 years ago by elexis (previous) (diff)

comment:5 by Bezerra, 7 years ago

Linux

Write

The writing data operations are first written in Page Cache and in another moment are transferred to drive unity. We could remember the command sync that transfers all page cache to devices and them we can unplug the pen-drive, for example. Newer kernels have one thread by hard device to be faster do transfer operations. Since 2.6 if my memory is good

Read

In the same way of write, the read operation sends the data to page cache and then you read them. When another read are made, the page cache it is consumed. The page cache data is cleared when additional memory is required by some application.

Well, I found a link that explains better than me: https://www.thomas-krenn.com/en/wiki/Linux_Page_Cache_Basics

comment:6 by Sandarac, 7 years ago

Description: modified (diff)
Milestone: BacklogWork In Progress
Patch: Phab:D587

comment:7 by elexis, 6 years ago

Keywords: simple removed

comment:8 by elexis, 6 years ago

Milestone: Work In ProgressAlpha 23
Priority: Must HaveShould Have

comment:9 by elexis, 6 years ago

Owner: set to elexis
Resolution: fixed
Status: newclosed

In 20639:

Remove VFS cache, because it is less effective and less efficient than the OS cache (and partially redundant with higher level application caches).

Patch By: Sandarac
Discussed with: Philip, echotangoecho, Bezerra
Fixes #4072.
Differential Revision: https://code.wildfiregames.com/D587

comment:10 by elexis, 5 years ago

Here janwas paper "Optimizing File Accesses via Ordering and Caching":
http://wassenberg.dreamhosters.com/articles/study_thesis.pdf
https://code.wildfiregames.com/F859317

Note: See TracTickets for help on using tickets.