Jump to content

Material and texture unloading on camera rotation, and never coming back


photo
Go to solution Solved by silent,

Recommended Posts

Posted (edited)

Hello everyone,

I didn't really know where to put this problem, I'm not sure it could be considered a bug, it might be us making mistakes.

We are migrating to 2.19.1 our application, reaching the final steps. But while exploring our worlds, I noticed a lot of material/texture unloading in our environment while moving. While we do have a relatively big world, we have a solid 80 FPS and I never saw that behavior on Unigine 2.18.1. The problem is not consistent, it might be difficult to reproduce sometimes, and at other times it breaks for good.

Let's look at an example I managed to capture. Here are my vehicles, parked:

image.thumb.png.a0bae27900acc6a905c616d8967f713e.png

I turn my head 180, and rotate back.
image.thumb.png.f3908de7f2d49a566205ee5fa7f5b5cb.png

Generally, the materials/texture come back after a few seconds. However sometimes, like in this particular occasion, they never come back. It happens on all kinds of objects, animated or static. These events are only occuring when moving the camera (rotating/translating) and not correlated to any other event (nothing is going on anyway).

I had the same problem using DLAA or not using any upscaling. We only have one camera rendering, no weird configuration or any setting modifying what the camera sees. We are not modifying materials or textures. Not generating anything.

All assets on the project have been upgraded to Unigine 2.19.1 (by actually opening Unigine with all the assets accessible. We let the editor upgrade the assets). The log doesn't show any complaint regarding material or texture loading. Here is the beginning of the log file (the rest is filled with our own logs, no error).

 

Quote

09:52:43 ---- Engine ----
09:52:43 Version: 2.19.0.3 Release Double ver-2.19.1-c8c5226 Jan 29 2025 (2.19.1 SDK)
09:52:43 Binary: Windows x86_64 Release WinSDK(10.0.10011.16384) Compiler(MSVC:1940 Toolchain:Unknown Toolset:Unknown VisualStudio:Unknown)
09:52:43 Features: Direct3D11.1 Direct3D12.0 Vulkan OpenAL VR OpenVR OpenXR Varjo SplashScreen CustomSplashScreen CustomProxy QudroSync Geodetic DoubleCoords HalfTexCoords Release
09:52:43 
09:52:43 ---- OS ----
09:52:43 Windows 11 (build 26100)
09:52:43 
09:52:43 ---- CPU ----
09:52:43 Intel(R) Core(TM) Ultra 7 165H
09:52:43 Extensions:  MMX SSE SSE2 SSE3 SSSE3 SSE41 SSE42 AVX AVX2 HTT
09:52:43 Frequency: 3071MHz
09:52:43 Cores: 16 Threads : 22
09:52:43 RAM: 31.5GB
09:52:43 
09:52:43 ---- GPU ----
09:52:43 GPU 0 Active : NVIDIA RTX 2000 Ada Generation Laptop GPU Memory: 7960MB Type:Discrete    Vendor:NVidia   CommonHeaps:true Priority:32 LUID:00000000000174a2 DeviceID:000028b8 Driver:573.44
09:52:43 GPU 1        : Intel(R) Graphics                         Memory:18361MB Type:Integrated  Vendor:Intel    CommonHeaps:true Priority:31 LUID:00000000000167dc DeviceID:00007dd5 Driver:101.6078
09:52:43 GPU 2        : Microsoft Basic Render Driver             Memory:18361MB Type:Other       Vendor:Unknown  CommonHeaps:true Priority:30 LUID:0000000000017287 DeviceID:00017287 Driver:Unknown
09:52:43 GAPI: Direct3D 12.0
09:52:43 
09:52:43 
09:52:43 ---- Application ----
09:52:43 
09:52:43 ---- System Proxy ----
09:52:43 SystemProxy initialization (Time: 114.6ms, Memory: 0B)
09:52:43 
09:52:43 ---- VR ----
09:52:43 VR initialization (Time: 0.1ms, Memory: 0B)
09:52:43 
09:52:43 
09:52:43 ---- Render ----
09:52:48 Renderer API: Direct3D 12.0
09:52:48 Maximum texture size: 16384
09:52:48 Maximum texture units: 16
09:52:48 Quadro Sync feature supported
09:52:48 DLSS is supported
09:52:48 DLSS Streamline version: 2.4.0
09:52:48 DLSS NGX version: 3.7.0
09:52:48 FSR is supported
09:52:48 FSR Version: 2.2.2
09:52:48 FSR Max Contexts: 8
09:52:48 FSR RAM Scratch Size: 10 MB
09:52:48 Render initialization (Time: 4.9s, Memory: 0B)
09:52:48 
09:52:48 
09:52:48 ---- VR Subsystems ----
09:52:48 VR Subsystems initialization (Time: 0.1ms, Memory: 0B)
09:52:48 
09:52:48 
09:52:48 ---- Filesystem begin ----
09:52:48 App path:  D:/Projects/Oktal/SimNext_UnigineMigration/deploy/Bin/
09:52:48 Data path: D:/Projects/Oktal/SimNext_UnigineMigration/deploy/data/vision/
09:52:48 Save path: D:/Projects/Oktal/SimNext_UnigineMigration/deploy/Bin/
09:52:48 
09:52:48 Runtimes loaded: 333 (Time: 54.2ms, Memory: 0B)
09:52:48 Filesystem initialization (Time: 479.9ms, Memory: 0B)
09:52:48 ---- Filesystem end ----
09:52:48 
09:52:48 
09:52:48 ---- Sound ----
09:52:48 Sound: null
09:52:48 Sound initialization (Time: 0.4ms, Memory: 0B)
09:52:48 
09:52:48 ---- Physics ----
09:52:48 
09:52:48 ---- PathFind ----
09:52:48 
09:52:48 Loading controls "D:/Projects/Oktal/SimNext_UnigineMigration/deploy/data/vision/configs/default.controls"...
09:52:48 Loading config "D:/Projects/Oktal/SimNext_UnigineMigration/deploy/data/vision/oksygen/configs/sim.config"...
09:52:49 D3D12Particles::createBuffers (Time: 35.6ms, Memory: 0B)
09:52:49 
09:52:49 ---- Materials begin----
09:52:49 
09:52:49 ---- MeshManager begin ----
09:52:49 
09:52:49 ---- Properties begin ----
09:52:49 Properties loaded: 40/40 (Time: 37.2ms, Memory: 0B)
09:52:49 ---- Properties end ----
09:52:49 
09:52:49 Base materials loaded: 166/166
09:52:50 Materials loaded: 2670/2670
09:52:50 Load Materials (Time: 1.3s, Memory: 0B)
09:52:51 Preload materials loaded: 9917 (Time: 1.4s, Memory: 0B)
09:52:52 Initialization meshes (Time: 2.9s, Memory: 0B)
09:52:52 Total resources: 16343
09:52:52 ---- MeshManager end ----
09:52:52 
09:52:57 Shaders compiled: 4874 (Time: 6.1s, Memory: 0B)
09:52:57 Total materials loaded: 12753 (Time: 8.8s, Memory: 0B)
09:52:57 ---- Materials end ----

What changed from 2.18.1 to 2.19.1 that could cause this kind of bug ?

I don't really have the time right now to make a simple project to showcase the bug, I have not yet checked if it happens with smaller scenes. But If any other information could help you identify what's going on, I would happy to give it.

 

Edit: The problem gets worse with time and the amount of area I explored (amount of assets loaded ?). I can "reset" the problem by changing upscale mode during runtime, but the problem will again worsen with time. Also, I'm using multihreaded rendering.

Edited by K.Wagrez
Posted

Hi Kevin,

Over recent releases, we’ve updated how the engine manages VRAM resources and asset streaming, with more improvements planned. These changes are necessary to ensure better long-term performance and stability on modern graphics APIs (DirectX 12 and Vulkan). While the goal is improved behavior overall, they can occasionally surface issues like the one you’re encountering.

Better results you already should observe with the recent 2.20 update. And we aim to improve this behavior even further in 2.21 coming later this year.

A bit more information on VRAM usage and how you can affect it you can find in this forum post: 

It worth checking the dedicated VRAM usage when this issue is happens. More likely you are hitting the dedicated VRAM limits and streaming is not able to recover from this. However, with normal streaming work and correctly tuned content there should not be no issues like that.

To help you move forward, our first goal is to pinpoint why VRAM is being exhausted on your side. In our experience, this typically comes from one (or a mix) of the following:

  1. Content & scene configuration (e.g., oversized textures, streaming settings, LODs).
  2. Upscaling or runtime issues (e.g., allocator fragmentation, resource leaks, and yes, upscalers are also use additional VRAM).
  3. Environment factors (driver/runtime versions, GPU-specific behavior, background tools).

If possible, please share the exact scene and reproduction steps with us. We’re happy to sign an NDA (if required). With access to your scene, our QA and dev teams can reproduce the issue, profile VRAM usage, and identify the root cause and provide a set of recommendations to reduce or fully eliminate this issue.

If that's not possible, you can try to use adjustments from the post mentioned above and perform additional profiling of your content / memory usage in this scenario. Maybe you will find options to reduce the VRAM usage without big scene and content changes.

Thank you!

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Posted

thanks for your answer, and for the link silent :)

However it doesn't look I'm using that much VRAM (sorry, I cannot change my laptop's language):
 

image.thumb.png.68ed5e58418537a84dea66eaf42747a9.png

This is what the current render is at the time:

image.thumb.png.f61448ae2e1f7f1acbbb3a2962a8174c.png

As you can imagine, the grey blocks are supposed to have textures on them. And they had, at some point during runtime. The behavior is kind of inconsistent. I managed to launch twice the application with barely any problem. But in this session, everything looks broken. I will keep investigating based on the link you provided, to profile what's going on behind the scenes.

Posted (edited)

Okay the problem may run deeper, I checked my world's content in the editor, and, like in runtime, my materials seem to break, as I'm not having any preview in the asset folder. It still is inconsistent and some materials still function.

 

Also the bug is much less prevalent after restarting the computer. I will keep you notified if the problem reappears. (why it appeared on the first place is unclear, but I might found out with time)

Edited by K.Wagrez
Posted

That's interesting.

It's recommended to use .texture format for runtimes textures. Additionally, some materials may break when migrating from very old engine versions. We’ve seen cases where material files reference the wrong textures, which in turn causes incorrect mipmap behavior. We’ve documented a straightforward fix in the following forum thread that addresses the texture references in the material files:

You might be experiencing a similar issue. Please try running the provided .usc script, it automatically fixes affected materials.

Unfortunately, without access to the specific content that triggers this behavior, it’s difficult to determine the root cause. If you can share the problematic scene or a minimal repro (we can sign an NDA if needed), we’ll investigate and pinpoint the issue.

Thanks!

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Posted

The problem doesn't appear at first, it's not linked to the current state of a world or scene. It's something that gradually appears. I will try to find a process to generate the problem at will. 

Posted
2 minutes ago, silent said:

Does this issue appear only on NVIDIA GeForce RTX 20xx mobile GPUs? If so, it could point to a hardware or driver-specific problem.

It's on my "to-check" list.

Posted (edited)

Still haven't identified when it starts appearing, however it might be correlated with another side effect, and may be caused by one of our behavior.

When launching a world, we use intersections to place a lot of our dynamic objects on meshes and terrains. And strangely, any intersection that should work with a surface fails. All of our objects which are above terrain are projected on the terrain, but any object that is under the terrain and trying to intersect with a surface fails to do so.

I checked with bounding boxes to make sure it was at the right place and yes, our objects should intersect. This behavior (failure of intersection) appears correlated with our problem in delayed assets loading. I tried to force load in RAM or VRAM the meshes it should intersect with, but the methods return false (which means it failed to do so ?).

I'm going to try using the microprofiler to see anything weird going on behind the scenes. I don't have access to a lot of GPUs here, so I'm not sure if I can check wether or not it's GPU related.

I'll keep posting each time I have news of interest.

Edit: Is that the thread responsible for streaming assets ?
image.thumb.png.22a3b40d55a6d64fbf69e6713302045f.png

Edit 2: Hmm, I believe this is weird. Negative number for CPU ram, GPU ram ? What is going on ?
image.png.039aa889f799299845331a530550d8da.png

By the way, I have Unigine 2.9.1.2.
Edit 3: I used the "dangerous commands" from the first post you mentioned, and it fixes everything. I think for now that is going to be my fix.

Edited by K.Wagrez
Posted

Thanks for the additional screenshots. Looks like engine detects that it's lacking RAM / VRAM memory.

First of all, could you please do a screenshot of RAM tab from the Windows Task Manager? We need to check RAM committed values:

image.png 

We hope that it at least will give us an additional insight why engine reports negative RAM numbers on that device.

We also need to check the Intel GPU metrics from the task manager as well. Integrated GPUs usually reserves physical RAM from the system. Once RAM is reserved no other application can allocate this RAM for their needs. Please try adjusting the amount of memory reserved for the Intel integrated GPU in the BIOS/UEFI, or disable the iGPU if it isn’t needed. With either change, the engine should stop reporting negative values in the profiler.

Also negative VRAM / RAM numbers may indicated that for some reason system page file size is not growing. That making the OS unable to move background applications to page file and let another application to allocate RAM correctly. That leads to the situations where engine is being not able to request more memory for the process. It may be due to some internal Windows logic, but often this happens because there are simply no free space left on drive C:.

To resolve this potential issue we also recommend to manually set page file size to something like 16GB and make sure that you always have free space on disk C: so it would grow as expected over time. Making this adjustments to the system may result in more stable work and proper RAM / VRAM detection.

Please try to adjust page file size and disable iGPU to see if that will make streaming work as expected. If so, you can disable dangerous commands and use the engine as-is.

Thank you!

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Posted (edited)

Well Okay, the RAM is pretty bad. For comparison, here is our memory without Unigine (but with all of our other required processes running):

image.thumb.png.1aeb7e27c1725e5e1324d4ff1b814d25.png

 

And with Unigine, loading the world I believe to be the heaviest (the one from our examples and with the negative reading):
image.thumb.png.c750f12e640e83c3859ca3ddb3acac61.png

This is taken in a situation where everything works well again. I'm not sure why it changed compared to the last execution, except maybe that I launched previously a level with the render_streaming commands the way you mentionned. And here, CPU ram seem to be positive.

image.thumb.png.bffed6267ccfc4cd2a3377cf34c3701e.png

I will try a few more times to launch the world, to see if there is some leak somewhere.

This machine is a development rig, I believe our simulator are more powerful, but that is yet to be verified. I will try to make measurments on a bigger machine. The problem is here that we use nearly the minimum of what we ask, we have only one render, I didn't even start the simulation...

Edit: Okay, I managed to reproduce the error, after loading a different world. Nothing seem much more troublesome than the previous RAM graphics

image.thumb.png.67d909bcb27ca66e1dd80ce32832231e.png

And this time memory readings from the profiler are negative (you can see shiny objects behind who lost their texture):

image.thumb.png.c36f8d04edac84a1ac9129c0fb15ec98.png

I will try to verify if it's the action to load a different world that generated this problem. It's not linked to one particular world.

Edited by K.Wagrez
  • Solution
Posted

Yeah, apparently a lot of RAM has been already taken by 3rd party processes. As you can see from the RAM section Committed memory is already exceeding the current physical memory available (42 GB / 37GB). Committed RAM is the only valid indicator of how much free memory you have. Even if you have 88% free displayed in Performance section, that basically is not true.

If you are running development engine build you can disable microprofile tool on startup to save several gigs of RAM. To do so just pass an additional start-up parameter:

  • -microprofile_enabled 0

Maybe doing just this one step will result in major improvements. Also you can additionally control how much physical RAM application will try to use:

  • render_streaming_usage_limit_ram (how many to allocate in %)
  • render_streaming_free_space_ram (how much to keep free in MB)

For the first command Instead of 100% you can use like 90 or 95. And for the second command you can keep like 100-200MB. But you must monitor Commited RAM carefully after that so you will not reach this limits.

Also after loading world you can perform memory_clear console command to revoke some memory back. That also can clear some RAM / VRAM.

Beyond this, our options are limited. Upgrading the physical RAM may help, and for troubleshooting, working with smaller worlds (with less content) could make testing more manageable.

Increasing page file to even bigger values may also improve stability a lot (like 30-50GB and even bigger values).

Thanks!

  • Thanks 1

How to submit a good bug report
---
FTP server for test scenes and user uploads:

×
×
  • Create New...