rendering to multiple textures

[esger.abbink_]

I've run into a problem with rendering into textures. I am trying to render rearward facing views (mirrors of a vehicle) into textures and settting those texture to the materials of the mirror mesh objects.

 

Materials:

    <material name="newHollandT6160_mirror_left" parent="mesh_base">
        <texture name="albedo">textures/mirror_placeholder_left.tga</texture>
        <parameter name="metalness">0.5</parameter>
        <parameter name="roughness">0.5</parameter>
        <parameter name="specular">0.2</parameter> 
        <parameter name="normal_scale">0.33</parameter>
    </material>
    <material name="newHollandT6160_mirror_right" parent="mesh_base">
        <texture name="albedo">textures/mirror_placeholder_right.tga</texture>
        <parameter name="metalness">0.5</parameter>
        <parameter name="roughness">0.5</parameter>
        <parameter name="specular">0.2</parameter> 
        <parameter name="normal_scale">0.33</parameter>
    </material>
    <material name="newHollandT6160_mirror_mid" parent="mesh_base">
        <texture name="albedo">textures/mirror_placeholder_mid.tga</texture>
        <parameter name="metalness">0.5</parameter>
        <parameter name="roughness">0.5</parameter>
        <parameter name="specular">0.2</parameter> 
        <parameter name="normal_scale">0.33</parameter>
    </material>

Nodes:

<?xml version="1.0" encoding="utf-8"?>
<nodes version="2.3">
    <materials>
        <library>materials/newhollandT6160.mat</library>
    </materials>
    <node type="ObjectMeshStatic" id="191218668" name="newHollandT6160_mirror_left">
        <mesh_name>nodes/newHollandT6160/meshes/newHollandT6160_mirror_left.mesh</mesh_name>
        <surface name="newHollandT6160_mirror_left" material="newHollandT6160_mirror_left" property
        <transform>0.999999880791 0 0 0.0 0 0.999999880791 0 0.0 0 0 0.999999940395 0.0 0 0 0 1.0</
    </node>
</nodes>

<?xml version="1.0" encoding="utf-8"?>
<nodes version="2.3">
    <materials>
        <library>materials/newhollandT6160.mat</library>
    </materials>
    <node type="ObjectMeshStatic" id="1233247209" name="newHollandT6160_mirror_right">
        <mesh_name>nodes/newHollandT6160/meshes/newHollandT6160_mirror_right.mesh</mesh_name>
        <surface name="newHollandT6160_mirror_right" material="newHollandT6160_mirror_right" proper
        <transform>0.999999940395 0 0 0.0 0 0.999999940395 0 0.0 0 0 0.999999940395 0.0 0 0 0 1.0</
    </node>
</nodes>

<?xml version="1.0" encoding="utf-8"?>
<nodes version="2.3">
    <materials>
        <library>materials/newhollandT6160.mat</library>
    </materials>
    <node type="ObjectMeshStatic" id="914307996" name="newHollandT6160_mirror_mid">
        <mesh_name>nodes/newHollandT6160/meshes/newHollandT6160_mirror_mid.mesh</mesh_name>
        <surface name="newHollandT6160_mirror_mid" material="newHollandT6160_mirror_mid" property="
        <transform>0.999999940395 0 0 0.0 0 0.999999940395 0 0.0 0 0 0.999999940395 0.0 0 0 0 1.0</
    </node>
</nodes>

Each mirror is setup individually with its own TextureRender, Texture and Node as follows (error checking removed for brevity):

Unigine::NodeReferencePtr mirror_node      = Unigine::NodeReference::create (mirror_setup.node_filename.c_str ()) ;
Unigine::ObjectMeshStaticPtr mirror_object = Unigine::ObjectMeshStatic::cast (mirror_node->getReference ()) ;
int32_t surface_id                         = mirror_object->findSurface (mirror_setup.surface_name.c_str ()) ;
Unigine::MaterialPtr material              = mirror_object->getMaterial (surface_id) ;
int32_t texture_id                         = material->findTexture ("albedo") ;

// loaded node and found surface
size_t width = mirror_setup.texture_width ;
size_t height = width / mirror_setup.aspectratio ;

Unigine::TextureRenderPtr texture_renderer = Unigine::TextureRender::create () ;
texture_renderer->create2D (width, height) ;

Unigine::TexturePtr texture = Unigine::Texture::create () ;
texture->create2D (width, height, Unigine::Texture::FORMAT_RGBA8, Unigine::Texture::WRAP_CLAMP | Unigine::Texture::FILTER_LINEAR) ;

texture_renderer->setColorTexture (0, texture) ;

material->setImageTexture (texture_id, texture) ;

The mirror textures are then rendering as follows in the vehicle frameStep ():

double h = 2.0 ;

for (auto& mirror : m_mirrors)
{
    double hfov = 60 ;
    double vfov = hfov / mirror->setup.aspectratio ;
	
    Unigine::Math::mat4 projection = Unigine::Math::perspective (vfov, 2.0, 0.05, 10000) ;
	
    m_camera->setProjection (projection) ;
	
    Unigine::Math::dmat4 modelview = Unigine::Math::translate (Unigine::Math::dvec3 (0, h, 0)) * Unigine::Math::rotateZ (-90.0) * Unigine::Math::rotateY (90.0) ; //Unigine::Math::translate (Unigine::Math::dvec3 (-10, 0, 0)) * Unigine::Math::rotateY (90.0) ;
	
    m_camera->setModelview (modelview) ;
	
    mirror->texture_renderer->enable () ;
	
    m_viewport->render (m_camera) ;
	
    mirror->texture_renderer->flush () ;
    mirror->texture_renderer->disable () ;
	
    // viewport states
    renderstate->clearStates () ;
    renderstate->clearBuffer (Unigine::RenderState::BUFFER_ALL, Unigine::Math::vec4 (0.0f)) ;
    renderstate->setPolygonCull (Unigine::RenderState::CULL_NONE) ;
    renderstate->flushStates () ;
	
    h += 1.0 ;
}

The CameraPtr and the ViewportPtr are contained in the vehicle class and are re-used for each mirror. The execution order is: App::update (), App::render (), vehicle::frameStep (), App::update (), App::render (), vehicle::frameStep (), etc

 

With just one mirror enabled this works fine, as can be seen in the composited image below:

 

 

However, when I enable more than one mirror something goes wrong. The mirror images shimmer, and look sort of 'hazed':

 

 

What am I missing / doing wrong?

[silent]

Hi Esger,

 

For each camera you need to have own viewport. You can't reuse the same viewport for multiple cameras.

 

You can also verify if that's the main issue by disabling post materials (render_skip_post_materials 1). If image will return to normal it's probably the root cause.

 

Otherwise we will need some small test scene with the reproduction.

 

Thanks!

[esger.abbink_]

Hi Silent,

 

actually I have just the one camera and the one viewport (for the mirrors), created in the vehicle class constructor:

m_camera = Unigine::Camera::create () ;
m_viewport = Unigine::Viewport::create () ;

m_camera->setViewportMask (~0) ;
m_camera->setReflectionMask (~0) ;

How is the relationship between camera and viewport established? Currently they only meet during rendering by doing;

m_viewport->render (m_camera) ;

Which is the only line where the two meet. 

 

I'll try the post materials test. 

[esger.abbink_]

setting render_skip_post_materials to 1 indeed fixed the problem.

 

I have now added individual Camera and Viewport for each mirror and that solves the problem as well (with render_skip_post_materials back to 0)

 

I am curious though, what exactly the problem is here? Both the naming of the function and the description don't imply a making a lasting connection between the two.

void Viewport::render (const Ptr<Camera>& camera)

Renders an image from the specified camera in the viewport.

• const Ptr<Camera> & camera - A camera an image from which should be rendered in the viewport.

 

 

Now that the basic thing is working correctly, what would be the best suggestions to minimize impact from rendering the additional mirrors on performance?

[silent]

Hi Esger,

 

Each viewport internally keeping the previous frames camera's projection and view matrices. They are required for correct velocity buffer calculations. Post processing effect such as motion blur is using the velocity buffer (as well as Temporal Anti-Aliasing (TAA)) internally. When you use single viewport with multiple cameras you are mixing different projection / view matrices between frames and basically made the velocity buffer unusable.

 

You still can use single viewport and different cameras if you are not using post-processing effects at all. Otherwise you need to keep the single Viewport <-> Camera connection.

 

We will update documentation to clarify that in the next SDK update.

 

Now that the basic thing is working correctly, what would be the best suggestions to minimize impact from rendering the additional mirrors on performance?

You can try to decrease viewport's resolution for example orr not to use post processing effects in such mirrors if that is not critical for you (check the setSkipFlags methods as well).

 

Thanks!

[esger.abbink_]

Hi Silent,

 

thanks, that clarifies indeed and makes sense as well.

 

As for performance, we won't be rendering them at 4K no ;) Will check the setSkipFlags.

[lin.xianming]

Hi Silent,

 

I've learned a lot from this topic. I'm still looking forward to further insides.

 

1> While I use the (TextureRender，Camera，Viewport）method to render a scene into texture, the loop "engine->main(&system_logic, &world_logic, NULL)" is running, isn't it? 

As far as I know, in this case Esger should render two textures which belong to two mirrors in advance, and then render the vehicle. So Actually Unigine renders the same scene three times(twice for mirrors by Viewport::render, once for vehicle by App::render). is it correct?

 

2> In my case, we have program and preview outputs supposed to be rendered simultaneously，and the loaded world is the same，but they differ with the camera positions and (for instance) some objects' positions. Since I have to update those objects' positions between two renderring, It seems a bit different to the above case. Dose Unigine allow me to uptade some objects' positions between two Viewport::render or between one Viewport::render and one App::render?

 

3> What's the most efficient way to carry my point? a)calling Viewport::render twice, 2)Viewport::render + App::render, 3) App::render twice.

 

Thank you.

[silent]

lin.xianming

 

1> While I use the (TextureRender，Camera，Viewport）method to render a scene into texture, the loop "engine->main(&system_logic, &world_logic, NULL)" is running, isn't it? 

 

As far as I know, in this case Esger should render two textures which belong to two mirrors in advance, and then render the vehicle. So Actually Unigine renders the same scene three times(twice for mirrors by Viewport::render, once for vehicle by App::render). is it correct?

 

Yes, this is correct.

 

2> In my case, we have program and preview outputs supposed to be rendered simultaneously，and the loaded world is the same，but they differ with the camera positions and (for instance) some objects' positions. Since I have to update those objects' positions between two renderring, It seems a bit different to the above case. Dose Unigine allow me to uptade some objects' positions between two Viewport::render or between one Viewport::render and one App::render?

It's better to render all the additional stuff in render() inside AppWorldLogic.

 

3> What's the most efficient way to carry my point? a)calling Viewport::render twice, 2)Viewport::render + App::render, 3) App::render twice.

It's hard to say without seeing the actual code and content. Probably they are all will be mostly the same. You will have to choose one based on your observations.

 

Thanks!

[lin.xianming]

Hi Silent,

 

 

 

It's better to render all the additional stuff in render() inside AppWorldLogic.

Do you mean I should call Viewport::render twice inside one AppWorldLogic::render? Dose Unigine allows me to Update some objects' positions between two callings of Viewport::render inside one AppWorldLogic::render? What does all the additional stuff mean, the whole scene or just the objects with different positions? 

 

Thanks

[lin.xianming]

Hi Silent,

 

I'd like to provide more details to clarify the usage of mine, which pretty much look like as below:

class AppUser : public App {
	// ...
}
int main(int argc,char **argv) {
	AppUser appUser;
	Engine::init(UNIGINE_VERSION,0,&appUser,argc,argv);
	
	// run the function that manages the main loop
        if(!appUser.isDone())
        {
            //Update player's position as the *program* camera, update some objects' position
            Game::get()->setPlayer(&player);
            appUser::update(); appUser::render(); appUser::swap();

            //Update player's position as the *preview* camera, update some objects' position
            Game::get()->setPlayer(&player);
            appUser.update(); appUser.render(); appUser.swap();
        }
	Engine::shutdown();
}

It works. I can find the outcome from App::m_renderTarget.

I try to replace App::render; with Viewport::render.

int main(int argc,char **argv) {
	AppUser appUser;
	Engine::init(UNIGINE_VERSION,0,&appUser,argc,argv);
	
	// run the function that manages the main loop
        if(!appUser.isDone())
        {
		//Update camera_pgm's position as the program camera, update some objects' position
		texture_renderer_pgm->enable();
                texture_renderer_pgm->setColorTexture (0, texture);
		viewport_pgm->render(camera_pgm);
		texture_renderer_pgm->flush();
		texture_renderer_pgm->disable();
                //If check the outcome of texture, it dosen't work.

		//Update camera_pst's position as the preview camera, update some objects' position
		texture_renderer_pst->enable();
                texture_renderer_pst->setColorTexture (0, texture);
		viewport_pst->render(camera_pst);
		texture_renderer_pst->flush();
		texture_renderer_pst->disable();
                //If check the outcome of texture, it dosen't work.
        }
	Engine::shutdown();
}

Then It can't work properly.

 

Strangely，when I modified the codes like this way，the outcome appeared.

		//Update camera_pgm's position as the program camera, update some objects' position
		appUser.App::updata();
		{
			texture_renderer_pgm->enable();
                	texture_renderer_pgm->setColorTexture (0, texture);
			viewport_pgm->render(camera_pgm);
			texture_renderer_pgm->flush();
			texture_renderer_pgm->disable();
		}
		appUser.App::render();
		appUser.App::swap();
                //If check the outcome of texture, It works.

Actually，the texture with outcome has nothing to do with App class at all.

I don't know where the problem is.

Thanks

[alexander]

Hi, lin.xianming!

 

Viewport::render() uses some info from the last App::render() call.

So, the best game loop is the following:

while(!appUser.isDone())
{
	// change your nodes
        // ...

	appUser::update();
	appUser::render();

	// render to textures with Viewport
        // ...

	appUser::swap();
}

Best regards,

Alexander

[lin.xianming]

Hi Alexander,

 

I get your idea, Thanks. Your explanation makes sense.

 

I guess we are getting close to the heart of the matter: 

I have try two ways to render the program and preview outcomes.

 

1>

            //Update player's position as the *program* camera, update some objects' position
            //...
            Game::get()->setPlayer(&player);
            appUser::update(); appUser::render(); appUser::swap();
            //Get program outcome from the renderTarget
            //...

            //Update player's position as the *preview* camera, update some objects' position
            //...
            Game::get()->setPlayer(&player);
            appUser.update(); appUser.render(); appUser.swap();
            //Get preview outcome from the renderTarget

2>

	//Update camera_pgm's position as the program camera, update some objects' position
	//...
        appUser::update();
	appUser::render();
	// render program to textures with Viewport
        // ...
	appUser::swap();
        //Get program outcome from texture_pgm
        //...

	//Update camera_pvw's position as the program camera, update some objects' position
	//...
        appUser::update();
	appUser::render();
	// render preview to textures with Viewport
        // ...
	appUser::swap();
       //Get preview outcome from texture_pvw

But both have the same problem: When the cameras' positions are different significantly，The ghost shadows appear.

 

No matter the render_skip_post_materials is 0 or 1，the ghost shadows are still there.

But when I set render_antialiasing from 2 to 0 or 1, then the ghost shadows disappear.

Please help me to check it out, Thanks.

[silent]

lin.xianming,

 

A small test scene based on a newly created project via SDK browser will help to understand what is going on. Could you please prepare some?

Thanks!

[lin.xianming]

Hi engine,

 

Here is the Players.cpp file. You can replace the original one (Api\Nodes\Players\Players.cpp) with it.

 

After the execution, you can find two .png files in disk C: by the names of render1_1st.png and render2_1st.png. They are supposed to be exactly the same thing. But they are not.

Players.zip

Please drag a data folder from any new project, and drop it into Players' folder. in order to let it run properly.

 

 

Thanks.

[silent]

Hi Lin,

The first frame rendered with TAA enabled will always produce this artifacts. You need to render at least two frames (or disable TAA) to avoid such effects. That happens because velocity buffer have no information about previous frame, therefore it looks incorrect. After the first frame velocity buffer is starting to work as expected.

 

Thanks!

[lin.xianming]

Hi Silent,

 

I understand your idea. But the key point is I have to render a scene from two or more different cameras(or Players), and the TAA is the only acceptable AA for quality consideration.

What you mentioned is the velocity buffer must contain the infomation about previous frame coming from the same camera. I agree. But why not just give developers an option to allow them establish the additional velocity buffer explicitly?

My suggestion is put the velocity buffer into Camera(or Player). In my case, I can create multi-Players(I've done that way), otherwise just share one. And It will make the Camera(or Player) class more explicit, To my knowledge, now there aren't many differences whether I create one Player or two.

So far I have no alternative way to deal with the issue, because the quality is always the priority. 

 

Thanks!

[silent]

lin.xianming,

 

There is already a velocity buffer assigned for each new pair of Viewport and Camera. If you need to render static screenshot - you will have to render 2 frames instead of one to avoid artifacts - there is no solution to fix this behavior at all. If you need to constantly render through this additional viewport - there will be no issues, because number of rendered frames will be always > 1.

 

We will also try to do some more research in this direction, but chances that we can somehow fix it are very tiny.

 

Sorry for the inconvenience caused.

[lin.xianming]

Hi Silent,

 

Sorry, I didn't catch what you said. Maybe I should clarify my needs a little bit more.

 

If you need to constantly render through this additional viewport - there will be no issues.

 

Yes, We need to render the additional viewport, not switch from one to the other then stick in the latter one, but alternately render one and the other frame by frame, back and forth, both TAA, hopefully without any interference with each other. And the frame rate is an other crucial priority,

All in all, we have to present both of them in high quality simultaneously at realtime. 

In this case, Is any solution available? My sample just illustrates this issue. 

If the answer is Yes. Please give me your piece of code, and let me figure it out.

The first frame rendered with TAA enabled will always produce this artifacts. You need to render at least two frames (or disable TAA) to avoid such effects. 

Because we’re designing some live production tools for TV Industry, our customers wouldn't tolerate any minor fault at all.

 

BTW, My curiosity is if you have already utilized muti-velocity buffers mechanism, why the restriction is still there？

 

Thank you for your consideration.

[lin.xianming]

Hi Silent and Alexander,

 

After I described my needs, would you please give me any solutions or advices？

 

I'm looking forword to getting further information. Thank you!