[SOLVED] Excessive Image memory usage despite calling destroy()

[ataylor]

I have an application that does not fully instantiate the Unigine engine but merely creates and saves meshes / textures in Unigine format for later use with a separate full Unigine engine application.

 

I am creating and destroying instances of the Image class with the following example code:

#include <UnigineImage.h>
...
Unigine::ImagePtr myImage=Unigine::Image::create("D:\\test.tga");
myImage->createMipmaps();
myImage.destroy();

D:\test.tga is a 16MB uncompressed targa file. Calling createMipmaps() after loading this image increases the application's memory usage from roughly 10MB up to 110MB, according to WIndows Task Manager. This additional memory is never released until the application is closed. Repeating this process again results in an additional ~100MB each time. Eventually, the system runs out of memory and the app dies with the following message:

 

SystemAllocator::allocate(): can't allocate 33554464 bytes

 

The documentation for Memory Management led me to believe that .destroy() was the correct way to delete something after calling a static create() function, but clearly I missed something crucial as these Image instances never seem to be deallocated. What am I doing wrong?

 

Windows 7, Unigine SDK 2.3beta (similar problem with other SDK versions), 32-bit, single precision.

[silent]

Hi Adam,

 

Thank you for the detailed description. I've tried this code (AppWorldLogic.cpp in newly generated project) and I don't have memory leaks:

#include "AppWorldLogic.h"

#include <UnigineImage.h>
#include <windows.h>

/*
 */
AppWorldLogic::AppWorldLogic() {
	
}

AppWorldLogic::~AppWorldLogic() {
	
}

/*
 */
int AppWorldLogic::init() {
	static bool inited = false;

	if(!inited) {
		for(int i = 0; i < 1000; ++i) {
			Unigine::ImagePtr myImage = Unigine::Image::create("D:\\test.tga");
			myImage->createMipmaps();
			Sleep(10);
		}

		inited = true;
	}

	return 1;
}

int AppWorldLogic::shutdown() {
	return 1;
}

int AppWorldLogic::destroy() {
	return 1;
}

/*
 */
int AppWorldLogic::update() {

	return 1;
}

int AppWorldLogic::render() {
	return 1;
}

int AppWorldLogic::flush() {
	return 1;
}

/*
 */
int AppWorldLogic::save(const Unigine::StreamPtr &stream) {
	UNIGINE_UNUSED(stream);
	return 1;
}

int AppWorldLogic::restore(const Unigine::StreamPtr &stream) {
	UNIGINE_UNUSED(stream);
	return 1;
}

Also, please, make sure that you are running you code after Engine init. For example (unigine_project.cpp) - that code will cause memory leak:

#include <UnigineEngine.h>
#include <UnigineImage.h>
#include <windows.h>

#ifdef _WIN32
int wmain(int argc,wchar_t *argv[]) {
#else
int main(int argc,char *argv[]) {
#endif

	for(int i = 0; i < 100; ++i) {
		Unigine::ImagePtr myImage = Unigine::Image::create("D:\\test.tga");
		myImage->createMipmaps();
		myImage.destroy();

		Sleep(100);
	}

	Unigine::EnginePtr engine(UNIGINE_VERSION,argc,argv);
	
	return 0;
}

If you will move Unigine::EnginePtr engine(UNIGINE_VERSION,argc,argv); before the for loop - everything will be just fine.

 

This happens mostly because engine memory allocator is not working until the engine init.

 

Thanks!

[justin.ead]

We are intentionally avoiding setting up an instance of the engine as doing so would be overkill for our needs. This is only used as part of a separate tool we have. I take it the only way around this is to delete the raw pointers themselves or to work directly with Image.h instead of UnigineImage.h as is done with the Resource Editor?

[ataylor]

As Justin said above (and as I mentioned in my initial post), this application does not instantiate the Unigine engine -- it merely uses its Image API. The only Unigine header included in my reproduction example is UnigineImage.h. If possible, we would like to keep it this way.

 

So, is there a way to forcefully execute a garbage collection / memory management function after calling Image::destroy() without creating an Engine instance? 

[silent]

Hi Adam,

 

You also can check ImageDDS tool sources - it uses directly Image.h (not using the API). If you are using engine's API methods, you'll have to init engine, I'm afraid.

 

We will also continue to  investigate what we can do with that after 2.3 release. Maybe dev team will find more elegant solution.

 

Thanks!

[silent]

Hi  Adam,

It seems that we've fixed this in our internal builds. Fix will be available together with 2.3 SDK stable release.

 

You can try to rebuild engine with modified utils/Memory.cpp. Just replace two deallocate methods around 1560 lines:

void Memory::deallocate(void *ptr) {
	if(ptr == NULL) return;
	#if defined(_LINUX) && defined(__ARM_EABI__)
		if(sizeof(size_t) == 4 && !IS_ALIGNED16(ptr)) {
			ptr = static_cast<size_t*>(ptr) - 2;
		}
	#endif
	if(allocator != NULL) allocator->deallocate(ptr);
	else SystemAllocator::deallocate(ptr);
}

void Memory::deallocate(void *ptr,size_t size) {
	if(ptr == NULL) return;
	#if defined(_LINUX) && defined(__ARM_EABI__)
		if(sizeof(size_t) == 4 && !IS_ALIGNED16(ptr)) {
			ptr = static_cast<size_t*>(ptr) - 2;
			size += sizeof(size_t) * 2;
		}
	#endif
	if(allocator != NULL) allocator->deallocate(ptr,size);
	else SystemAllocator::deallocate(ptr);
}

Your feedback is highly appreciated :)

 

Thanks!

[ataylor]

Thank you. The new deallocate functions seem to be an improvement, but not fix the problem 100%.

 

I have recompiled the Unigine DLL with the updated Memory::deallocate functions provided above and am using it with my test application. Creating an Image from a 16MB .tga and calling createMipmaps now permanently consumes about 20MB instead of the previous 100MB. This allows me to run our image and mesh conversion process roughly four times as many times as previously before the application dies due to a failed memory allocation.

[silent]

Hi Adam,

 

Could you please provide a full code for reproduction? Probably, issue not in System deallocator.

 

Thanks!

[ataylor]

Sure. Attached is new win32 project created with Visual Studio 2013 with only a few lines of code added.

 

1) Place a large .TGA file in d:\test.tga (or edit path if needed). The file I am using is 16,385KB at 2000x2000 px.

2) Ctrl + shift + esc to open Task Manager.

3) Run, note CleanupTest_GUI memory usage is low, about 1MB.

4) Help->About to invoke Unigine::Image creation and mipmap generation. Monitor RAM usage in Task Manager. It will increase.

5) Repeat step 4 many times. Memory usage increases each time.

 

The DLLs I am running this with on my end are compiled from sim_windows_2.2.1-2_src with the deallocator fix included.

CleanupTest.zip

[silent]

Hi Adam,

 

Thank you for the test scene! Will try to find out what's happening inside.

[silent]

Hi Adam,

 

You should not have memory leaks in 2.3-beta with that deallocate patch.

 

Due to small bug in Image::create() API method in 2.2.1 you need to manually call myImage->grab(); after myImage->createMipmaps();

 

Also, calling destroy() method is not necessary in this case, because Ptr will be destroyed automatically.

 

Thanks!

[ataylor]

Okay, great. Thank you for investigating.

 

I haven't yet tested this locally -- Can I expect this same approach to work with other classes as well? The Image class is one of a few different Unigine classes (Image, Mesh, Xml, possibly some others) that we are using in our asset conversion process. Though I imagine even if there is a memory leak issue, the mesh and xml will be far less problematic...

[ataylor]

It seems that this is working at an acceptable level and can be marked as solved.