Dynamic Cubemaps / Reflection Probes / Render To Texture in 2020

twe · November 19, 2020, 5:16pm

Hello everyone,
I am currently trying to get dynamic cubemaps to work and have already scoured most of this forum, to no avail. What I need is the following:

create an entity in the middle of a room
a script renders the cubemap ONLY on application start
the cubemap gets assigned to the skybox

Having more than one CubemapRenderer in the scene would be preferable for my use-case.
Sadly, simply rendering the cubemap offline (in a 3d program) is NOT an option for my project.
Severe road blocks I have encountered so far:

because of the newer layer system, most old-school camera target approaches do not work
using the layer system means creating a dedicated render layer and assigning every single object to it for rendering (even if that were an option, much of the lighting in my scene is lost, because the ‘Skybox’ layer cannot be copied or reassigned)

Still, I have been able to cobble together some code that, IN THEORY, should work (if I interpreted it corretly) - but, as I am not a programmer, most of this stuff is beyond me. The code certainly does something, seeing as

the skybox actually gets replaced (with a plain white one, but still…)
if slightly modified, it works with just one camera and one texture which is assigned to a material

var RenderCubemap = pc.createScript('renderCubemap');

RenderCubemap.prototype.initialize = function() {
    // define camera angles (posx, negx, posy, negy, posz, negz)
    var angles = [
                [ 0, 90, 180 ],
                [ 0, -90, 180 ],
                [ 90, 0, 0 ],
                [ -90, 0, 0 ],
                [ 0, 180, 180 ],
                [ 0, 0, 180 ]
                ];    
    
    //var face = [pc.CUBEFACE_POSX, pc.CUBEFACE_NEGX, pc.CUBEFACE_POSY, pc.CUBEFACE_NEGY, pc.CUBEFACE_POSZ, pc.CUBEFACE_NEGZ];
     
    // create new Cubemap texture
    this.colorBuffer = new pc.Texture(this.app.graphicsDevice, {
        cubemap: true,
        rgbm: true,
        fixCubemapSeams: true,
        format: pc.PIXELFORMAT_R8_G8_B8_A8,
        width: 128,
        height: 128
    });    
    
    this.colorBuffer.minFilter = pc.FILTER_LINEAR_MIPMAP_LINEAR;
    this.colorBuffer.magFilter = pc.FILTER_LINEAR;      
    
    // for each of the 6 sides of the Cubemap...
    for (var i = 0; i < 6; i++) {
       
        var renderTarget = new pc.RenderTarget(this.app.graphicsDevice, this.colorBuffer, {
            depth: true,
            face: i
        });
        
        // create and position a camera        
        var e = new pc.Entity();
        e.addComponent('camera', {
            clearColor: new pc.Color(1, 1, 1),
            aspectRatio: 1,
            fov: 90
        });
                
        this.entity.addChild(e);
        e.setLocalEulerAngles(angles[i][0], angles[i][1], angles[i][2]);        
        e.setPosition(this.entity.getPosition());        

        this.setRt(undefined, renderTarget);    // set the renderTarget of the currently active camera to renderTarget
        this.app.renderer.renderComposition(this.app.scene.layers);     // render all layers      
        this.setRt(renderTarget, undefined);    // reset the renderTarget of the current camera
        
        e.destroy();    // destroy camera    
    }    
    
    // compute Cubemap (see: github.com/playcanvas/engine/pull/202)
    var options = {
        device: this.app.graphicsDevice,
        sourceCubemap: this.colorBuffer,
        method: 1,
        samples: 4096,
        cpuSync: true,
        filteredFixed: [],
        singleFilteredFixed: true,
    };
    pc.prefilterCubemap(options);
    this.app.scene.setSkybox(options.filteredFixed);
};


RenderCubemap.prototype.setRt = function(prevRt, newRt) {
    var comp = this.app.scene.layers;
    var layers = comp.layerList;
    for(var i=0; i<layers.length; i++) {
        if (layers[i].renderTarget === prevRt && layers[i].id !== pc.LAYERID_UI) {
            layers[i].renderTarget = newRt;
        }
    }
};

Please - if anybody has an idea on how to approach this, I’d be sooo thankful! I’ve been at this for hours on end now and simply cannot work it out

I based my code on several projects and posts, including (but not limited to):
https://forum.playcanvas.com/t/how-to-make-a-dynamic-cubemap-reflexion-project/2437/7
https://playcanvas.com/project/532659/overview/render-low-res-world-layer
https://playcanvas.com/project/528262/overview/render-to-texture-test
https://playcanvas.com/project/605131/overview/capturing-screenshot-from-camera
https://playcanvas.com/project/578359/overview/tutorial-render-to-texture-once
https://playcanvas.com/project/560797/overview/tutorial-render-to-texture

yaustar · November 19, 2020, 5:22pm

There’s some work in the PR to create dynamic cubemaps: https://github.com/playcanvas/engine/pull/2523

mvaligursky · November 19, 2020, 6:32pm

I worked on the PR @yaustar listed to get this working as an engine example, and even though that works, I don’t like the need to create 6 layers, 6 cameras, and especially having to add all objects to those layers. This is even worse for multiple cameras.

And so I’m exploring the option to modify engine’s layer system to allow render target to be specified on the camera instead of the layer. The new cubemap / texture rendering based on this, that I have working locally, is a lot simpler now, but this is not part of PR as that needs few more things to be finished, and it would need a lot of testing as it might break compatibility for some projects. But the hope is that in the not too far future this will get easy to do.

twe · February 10, 2021, 9:18am

@mvaligursky @yaustar

As of Engine v1.39.0, according to changelog, there is now a Render-to-Cubemap option, as shown in this example: https://github.com/playcanvas/engine/pull/2757 which seems to do exactly what I tried to achieve last year.
However, upon trying to view the code, the site simply throws 404 page not found. Could you please look into that and provide the correct link?

yaustar · February 10, 2021, 9:25am

The examples source code are in the Engine repo (in case this happens again) https://github.com/playcanvas/engine/tree/master/examples

Here is the source code to the render to cubemap: https://github.com/playcanvas/engine/blob/master/examples/graphics/render-to-cubemap.html

I have a quick look to see why the view source button isn’t working

mvaligursky · February 10, 2021, 9:33am

and here’s the script it uses internally - this is what you need if you want to get it to work easily in the Editor

github.com

playcanvas/engine/blob/master/scripts/utils/cubemap-renderer.js

// How to use in the Editor:
// - create entity with Camera component - position of the entity defines where the cubemap is rendered from
//   and properties of the Camera are used to render cubemap (adjust near / far distance, clearing, layers and other properties)
//   Note: the layers should contain all layers visible by cubemap camera.
// - to use generated cube map, you can access it like this using script:
//   material.cubeMap = entity.script.cubemapRenderer.cubeMap;

var CubemapRenderer = pc.createScript('cubemapRenderer');

CubemapRenderer.attributes.add('resolution', {
    title: 'Resolution',
    description: 'Resolution of one side of a cubemap. Use power of 2 resolution if you wish to use Mipmaps.',
    type: 'number',
    default: 64
});

CubemapRenderer.attributes.add('mipmaps', {
    title: 'Mipmaps',
    description: 'If set to true, mipmaps will be allocated and autogenerated.',
    type: 'boolean',

This file has been truncated. show original

twe · February 10, 2021, 9:40am

@mvaligursky @yaustar
Thanks a lot!

PS: one more question of interest: is it possible to use box projection as usual with this method?

mvaligursky · February 10, 2021, 9:49am

What do you mean by a box projection? What are you trying to do?

twe · February 10, 2021, 10:05am

Basically I’m trying to render a cubemap in a (most likely not quadratric) room, let’s say 2,5 x 1,5 x 2,5 m. Then I want to use the Box Projection Setting on all materials in the room (shiny floor etc.) with the correct measurements / Half Extends of the room so that the reflection of the environment matches the rooms dimensions exactly.

mvaligursky · February 10, 2021, 10:40am

This should work, and it’s on my list to look at adding this as an engine example in the near future.

twe · February 10, 2021, 11:09am

Ok, that’s nice to hear. I have several more questions if you are willing to bear with me:

Is it possible to only render the cubeMap ONCE (e.g. on initialize) and keep it in memory while the 6 cameras are destroyed / disabled?
Can I or how can I use the cubeMap like I would use a prefiltered CubeMap for rough surfaces like discussed in this thread: [SOLVED] Assigning Cubemaps to Materials at Runtime

mvaligursky · February 10, 2021, 11:15am

yes, that’s the best way to go about it now. Disable camera to stop rendering, the cubemap will still be usable of course.
See how the viewer does prefiltering. You might need to do something similar to handle rough surfaces. At some point in the near future I’d like to wrap all this in some easier to use API, but that has not been done yet. I hope to have something as simple as Camera.RenderCubemap(cubemap, layers, prefilter); or similar.
https://github.com/playcanvas/playcanvas-viewer/blob/master/src/viewer.ts#L345

twe · February 10, 2021, 12:10pm

Where would I do this then? When I disable the camera(s) after this part in the code:

        // add the camera as a child entity
        this.entity.addChild(e);

        // set up its rotation
        e.setRotation(cameraRotations[i]);

the material which uses the cubeMap simply turns black.

So basically what’s done here is rendering the same Cubemap several times in several sizes and then calling pc.reprojectTexture to … kinda combine them , correct?

mvaligursky · February 10, 2021, 1:12pm

I added this to render-to-cubemap example to execute after inside app.on(“update” …, to stop cubemaps from updating. But make sure this is done on the second frame, as first one needs to render it. The cubemap at that time continues to be rendered on the sphere.
This disables 6 child cameras that render individual faces.

                shinyBall.children.forEach(function(child) {
                    if (child.camera) {
                        child.camera.enabled = false;
                    }
                })

from the viewer code, this block is relevant: " // generate prefiltered lighting data"
it uses single loaded cubemap, and renders it with varying levels of blur into a set of cubemaps, which are then used as a skydome.

twe · February 10, 2021, 1:38pm

I came up with the following solution: use a setTimeout with a short delay and then disable the cameras. However, in the short period of time where the cameras are enabled, every camera throws an error from launch.js seemingly every frame:

Trying to bind current color buffer as a texture

If I don’t disable them at all, this error gets thrown continously. Any idea what I may have done wrong? I mostly just copied the code from above with the minor adjustment that I made it so it could be applied to a model and have the camera as a seperate entity.

Demo is here: https://playcanvas.com/editor/scene/1037157

mvaligursky · February 10, 2021, 3:31pm

I would say that an object you put cubemap on is getting rendered into cubemap … and you cannot use cubemap both for read and write at the same time. In the example I did, the sphere with cubemap is on excluded layer, so it does not render into cubemap.

mvaligursky · February 10, 2021, 3:34pm

also, set timeout is not ideal … it can render few times, or not at all, depending on system’s performance.
Just have some boolean variable, flip it in the update, and use it inside next update to disable cameras.

twe · February 10, 2021, 4:58pm

After a lot of trial and error with the correct way of prefiltering the cubemap, it works fine now. Thanks, @mvaligursky

Here’s the Demo if anyone is interested:
https://playcanvas.com/editor/scene/1037157

mvaligursky · February 10, 2021, 5:17pm

nice! glad all that worked.

twe · February 11, 2021, 6:35pm

After a full day of debugging, I came to the conclusion that it doesn’t really work, after all.
In my simple test scene I tried to render a CubeMap and project it to a hollow cube (room-wall-floor-thingy) but the projection doesn’t look right at all. The most glaring issue is the blue cone which is facing to the right instead to the left where it should be facing.
What am I doing wrong this time?