Generating Shader Cache

[dred]

Hello, 

We are having quite a bit of hitching when moving though our scene, a big part of that appears to be generating shaders on the fly.  currently we are using unigine 2.4 and aren't really in a place to update to 2.6.  I was looking into pre-generating the shader cache, but have run into some problems.  when attempting to use the method outlined in this thread  where i run shader_cache_d3d11.usc  the shader cache that gets generated is 1kb large and next to it i see 5306 text files with errors, ill attach a sample.

 

The errors files don't make much sense to me.  I was hoping to try and see if i could figure out whats going wrong but i couldn't find any source for usc_x64.exe

 

I have also tried to use the config file setting render_manager_create_shaders, this does generate a shader cache which work alright, but about half the time i run the application from the same machine or others I will still get some hitches here and there and the shader cache grows.   I don't want to enforce this flag on the users machines as it does add quite a bit of time on startup.  

 

Any help you can offer would be much appreciated.

Dusty

FFF50093_C8C0.txt

FFF50093_C8C0_errors.txt

[silent]

Hi Dred,

Shader cache behavior was significantly improved in 2.6. For 2.4, I'm afraid you will have to write some custom logic (similar that we have in Superposition) to really warm-up the all states of the shaders in scene. Also, in 2.4 there is a difference between shaders that compiled on Intel / nVidia / AMD GPUs due to special define for each GPU.

We do not recommend to use the generation script (due to complexity of mesh_base material) and compilation process will never ends.

Sorry for the inconvenience.

[dred]

Thank you for the information Silent, 

Would it be possible to take one shader cache from an intel machine to a nVidia machine then to an AMD machine?  will it compile all of the shaders into that one working cache, or will it overwrite the old shader data each time?

I did look at the 2.6 source and it does look like a much faster approach.  from a quick peak it looked very extensive, I'm guessing it would be difficult to merge in just the shader caching changes from 2.6 into 2.4?

Dusty

[silent]

Hi Dred,

13 hours ago, dred said:

I did look at the 2.6 source and it does look like a much faster approach.  from a quick peak it looked very extensive, I'm guessing it would be difficult to merge in just the shader caching changes from 2.6 into 2.4?

Dusty

I would assume that backporting shader cache changes from 2.6 to 2.4 will be not generally possible due to huge materials changes. So the only valid approach for now would be the manual warming-up process.

Quote

Would it be possible to take one shader cache from an intel machine to a nVidia machine then to an AMD machine?  will it compile all of the shaders into that one working cache, or will it overwrite the old shader data each time?

Cache will be added each time (not overwritten).

Thanks!

 

[craig.biggs_]

On a related note, we have been generating shader cache files as part of our build process for Unigine 1 based simulators. It has been taking us on average ~2 hours to generate the cache files which, whilst time consuming is generally acceptable.

Having attempted the similar process for a Unigine 2.6 based simulator, we are finding that the cache file is taking 2 days to generate! Is this to be expected? Are there any tips to reduce the amount of time it takes to generate the shader cache?

 

[silent]

Hi Craig,

I'm afraid this is expected behavior. Shader cache is now includes not only static pass but alpha fade / skinned and gpu-independed. We found also that opengl cache is generate way much faster.

We will see what else we can do in the future releases (not in 2.7 I'm afraid).

Btw, how do you generate cache right now? We've recently updated our documentation page about it: https://developer.unigine.com/en/docs/2.6/code/shaders_cache

Thanks for the feedback.

[craig.biggs_]

Thanks for your reply.  I will get back to you in January on how we are currently generating the shader cache. We are trialling a few different methods for our product.

One point I did want to clarify - you mention the process is GPU independent? I am led to believe that this wasn't the case in Unigine 1.

Does this mean with Unigine 2.6 we could generate the shader cache on say a NVidia GTX-680 and then use it effectively on a GTX-1080 (for example)?

[silent]

Quote

Does this mean with Unigine 2.6 we could generate the shader cache on say a NVidia GTX-680 and then use it effectively on a GTX-1080 (for example)?

Yes, for now you can generate shader cache on AMD and it will work for nVidia cards as well.

[dred]

So from reading the documentation it says that if you use different video settings to compile for each preset.  We have a settings menu that allows users to change a bunch of video settings,  does this mean we need to run the shader cache on each combination of video options?  at this point that's over 10000 combinations of settings and worlds we would need to run it on.  I noticed that along with shaders_create there is also shaders_create_all, will that create caches for all defines? or is that just all materials(including ones that aren't loaded) and wouldn't help in this case.

[morbid]

Hello dred,

in your case shaders_create_all wouldn't be useful. This command generates cache for all materials but doesn't cover all of possible states\parameters combinations. It's simply would take cosmic amount of time.

Currently we have no fully automated solution for cache generation. 

You can organize your render settings into presets and generate cache for each preset with shaders_create.