[PATCH] Caching / texture loading issue with release data

[historic_bruno]

Mythos and I both notice this playing A11: when starting a new game, some textures start gray as if they need to be converted or cached from PNG to DDS. This should never happen because all textures should be pre-cached in public.zip. I can also verify no new textures are being cached by wiping out my local cache folder beforehand, playing the game, then checking that the cache folder is still empty.

It seems the only textures that have this problem are those that get applied to terrain (e.g. selection overlays, decals, terrains with materials).

For what it's worth, I'm testing with all default graphics config, so effects like specular, AO, and normal mapping should have no effect.

[myconid]

All textures have CTexture::Prefetch() called on them, so the engine knows it has to preload them. My guess is that that function tries to be clever and do the loading in a separate thread, thus the apparent delay you are experiencing.

Without looking at the code, the solution will probably be along the lines of modifying the texture manager to load all cached textures in the main thread, and only do the conversions in a separate thread (as opposed to doing both the loading and converting in a separate thread).

[myconid]

Right, so one solution might be to replace the CTexture::Prefetch() calls in graphics/ObjectEntry.cpp with CTexture::TryLoad(). It works, though I'm not sure if I trust that function to do what its name suggests... 

I guess a similar thing could be done for terrain textures, though the thing in Atlas where it loads up the previews could cause some issues.

[historic_bruno]

Replying to myconid:

All textures have CTexture::Prefetch() called on them, so the engine knows it has to preload them. My guess is that that function tries to be clever and do the loading in a separate thread, thus the apparent delay you are experiencing.

Hmm but that's nothing new, is it? This is very noticeable with all decal textures since Alpha 11, might it have to do with changes to terrain blending or something? I can't recall seeing this prior to Alpha 11.

[myconid]

Replying to historic_bruno:

Hmm but that's nothing new, is it? This is very noticeable with all decal textures since Alpha 11, might it have to do with changes to terrain blending or something? I can't recall seeing this prior to Alpha 11.

This isn't a blending thing. It just takes longer for the textures to load since there's more of them, so the delay is more noticeable.

Attached a patch that should fix it, though it probably needs testing for unintended consequences.

[historic_bruno]

Philip, k776, and I discussed this in ​IRC today. Philip's suggestions:

22:44 < Philip`> historicbruno: For loading the resources needed for the initial render, we should probably just tell it to render the scene once before closing the loading screen, and then it'll have loaded all the textures and meshes etc that it'll need
22:46 < Philip`> historicbruno: Textures being loaded in a separate thread shouldn't make the latency any worse, and it makes it possible to do nicer things like keep the loading screen's progress bar updating while it's all loading
22:46 <@historicbruno> Philip`, although there are a few it won't have, like selection overlays
22:47 < Philip`> historicbruno: Yeah, and anything that's not visible from the default camera position, and anything for units/buildings that haven't been constructed yet, etc
22:48 < Philip`> historicbruno: which shouldn't be too bad to load incrementally at runtime (like they are now), since you don't get a lot suddenly becoming visible at once, and if it is a problem then we could do some fancier preloading (e.g. tell it to start loading textures as soon as a unit/building is in the production queue)
22:48 < Philip`> historicbruno: (or make it load a low-resolution proxy texture first, and swap in the high-res version once it's loaded from disk, etc)

[ben]

In 13181:

Render initial game view during loading screen, to begin loading assets and eliminate ugly terrain blending artifacts. Refs #1663

[Kieran P]

Marking as fixed. Public releases have a public.zip package which contains the converted files, and the game, when loading a map, now prerenders the screen during the loading screen, which should help eliminate the short period where textures are grey. Any further changes required can be done in another ticket.

[historic_bruno]

Replying to k776:

Marking as fixed. Public releases have a public.zip package which contains the converted files, and the game, when loading a map, now prerenders the screen during the loading screen, which should help eliminate the short period where textures are grey. Any further changes required can be done in another ticket.

It's not fixed for decals, selection overlays, etc. Even on a fast system with an SSD, I get noticeable delays loading those textures (whether converting + caching, or only loading from cache, I don't know yet)

[historic_bruno]

Replying to historic_bruno:

It's not fixed for decals, selection overlays, etc. Even on a fast system with an SSD, I get noticeable delays loading those textures (whether converting + caching, or only loading from cache, I don't know yet)

I should point out the delays are still noticeable with the attached patch, even if somewhat shorter. It's probably too late to do anything about this for A13 and also the "bug" was introduced in A11.

[Josh]

Is this still a valid issue? I haven't noticed any particularly slow texture loading and the last activity on this ticket was over a year ago.

[Stan]

I don't think so, It only happens on SVN since textures are not cached.

[leper]

Replying to Josh:

Is this still a valid issue?

Yes.

[historic_bruno]

Replying to Josh:

Is this still a valid issue? I haven't noticed any particularly slow texture loading and the last activity on this ticket was over a year ago.

You have to watch carefully at the start of games, also on some systems it may not be as obvious. I think until someone deliberately fixes the problem, it will be broken (not likely to be fixed by accident or as a side-effect of something else).