Casting shadows with custom shader

Our project is using a custom shader along with hardware instancing to render some custom models. Our models render as expected, but they do not cast shadows properly. I have created a small repro project with simplified shaders to demonstrate the issue. This repro project contains the cube that is provided by the Model Viewer Start Kit template along side a wall of instanced tiles rendered with our shaders.

https://playcanvas.com/editor/scene/974723

When we add Directional or Spot lights to the scene, shadows appear correctly when using PCF shadow maps, but when we use Variance Shadow Maps, the shadows are incorrect.

When we add a Point light to the scene, we do not cast correct shadows regardless of the shadow map type. Instead, our shadows appear to cut-out the shadow from other models in the scene.

Below is an image demonstrating these three lighting scenarios. The top image shows shadows from a Directional light with PCF shadow maps (this is what we expect the shadows to look like). The middle image shows shadows from a Directional light with VSM shadows (blur=1). The bottom image shows shadows from a Point light with PCF shadows (note how the shadow from the cube is missing where the shadow from our tiles should be).

Any help getting our shadows working properly would be greatly appreciated. Thanks!

1 Like

Hi @nathanael-omnivor and welcome,

Very good explanation and repro project. I think the issue here is that VSM shadows require a certain shader to be included in the material to work.

I’d say that you can easily resolve this if you use the pc.StandardMaterial class and override the relevant shader chunks to add your custom GLSL code. Instead of using an empty material (pc.Material).

Check the hardware instancing example on the engine repo on how to enable hardware instancing on a standard material:

And here is a simple example on how to override a shader chunk:

https://developer.playcanvas.com/en/tutorials/tutorial-plasma-shader-chunk/

2 Likes

Hi @Leonidas, thanks for the advice!

I tried modifying our setup to use shader chunks and the pc.StandardMaterial and it seems promising, but there is one aspect of our shader code that I cannot figure out how to get working with the shader chunk system. Our shaders require some extra attributes (besides the default attributes defined by the pc.StandardShader), but I cannot figure out how to map our attributes to a semantic.

I’ve read through some other forum posts discussing attributes in shader chunks (Custom vertex attributes in chunk), but haven’t found a solution yet. Do you know if this is possible?

The good news is that I was able to modify our shaders (in a way that won’t work for our end goal) to remove our dependency on custom attributes, and I did confirm that all the shadows work properly.

Good point @nathanael-omnivor, did some research but I wasn’t able to find a straightforward/official way of editing the attributes.

They are being referenced by the shader program internally here:

Calling @mvaligursky, he may be able to comment on this.

Welcome, @nathanael-omnivor

Perhaps, this thread could be of use as well.

I believe I have something working now!

I added a “customAttributes” flag to the pc.StandardMaterial options, like so:

var material = new pc.StandardMaterial();
material.onUpdateShader = function (options) {
    options.useInstancing = true;
    options.customAttributes = true;
    return options;
};

And then I monkey patched the createShaderDefinition function to add extra attributes:

var origFunc = pc.programlib.standard.createShaderDefinition;
pc.programlib.standard.createShaderDefinition = function createShaderDefinition(device, options) {
    var result = origFunc.apply(this, [device, options]);
    if (options.customAttributes) {
        result.attributes['MY_CUSTOM_ATTRIBUTE_1'] = pc.SEMANTIC_TEXCOORD6;
        result.attributes['MY_CUSTOM_ATTRIBUTE_2'] = pc.SEMANTIC_ATTR2;
        ...
        result.attributes['MY_CUSTOM_ATTRIBUTE_5'] = pc.SEMANTIC_ATTR5;
    }

    return result;
}

This may be similar to what was suggested in this post: Custom vertex attributes in chunk

I’d love to hear back if there is a better way to do this, but I think we’re unblocked for now. Thanks for the help and suggestions, everyone.

3 Likes

@Leonidas I was able to get shadows working much better after switching to using shader chunks and the pc.StandardMaterial, as you suggested. I’m now hitting a new issue with shadows, to which I cannot figure out a solution.

If I add two instances of our custom model to the scene, they both cast shadows but the shadow for the second instance appears in the location of the first instance. It is as if the world transform for the first instance is being used to render the shadows for both instances. If only one instance is enabled, its shadow renders in the correct location. Below is an image illustrating this.

The top two images show the correct shadows when only one of two instances is enabled. The bottom image shows the incorrect shadows when both instances are enabled.

I have updated the repro project (https://playcanvas.com/editor/scene/974723) to include two instances of our custom model for debugging.

Any help would be much appreciated!

Thanks.

1 Like

Ah, that’s interesting, seems like a bug or something missing in the shadow shader.

@mvaligursky may have a point on how to fix this.

I had a quick look at the capture using https://spector.babylonjs.com/ - the shader used to render into both shadow map and color framebuffer for these objects has the same getPosition() part using your instancing attributes and so positions should match. Must be something else I’m missing there. I’d suggest you to install SpectorJS browser plugin, capture your preview and study the differences - in both shaders and uniforms and attached buffers to see what could cause it.

2 Likes

@mvaligursky I installed SpectorJS, and its been pretty helpful!

I’m still debugging this, but by examining the uniforms in draw, it seems that during renderShadows, the uniforms for the first instance are incorrectly being set to the second instance. During renderForward the correct uniforms are set.

I haven’t figure out where the uniforms are being incorrectly overwritten yet.

One work around for us could be to give each of our uniforms a unique name at runtime, so there is no overlap in uniform naming between instances of our models.

how do you set those uniforms for the two instances?
one way is to set them on material, for this you’d need to clone / create two materials.
the other way is to set them up on meshInstance directly - this way you can have a single material.

give me a name of the uniform that is getting overwritten, I’ll add debug logging on setting these on device and see what values we’re setting in engine.

by the way - curious about what you are trying to do, perhaps there is a simpler way. You could just generate mesh with what you need and render it without instancing? Or you could have extra properties in the texture and sample those in the vertex shader (not all HW supports this).

Sorry, I think my use of the term “instance” is getting overloaded. When I said we have two instances of our custom model, I meant that we are creating, from scratch, two occurrences of our custom model.

Each custom model (see repro_loader.js) creates a new pc.Mesh, a new pc.StandardMaterial, then sets our custom chunks on it, and creates a new pc.MeshInstance, set to render 256 instances of the pc.Mesh. We set the uniforms on the MeshInstance’s material.

In the repro project, the uniform that is being overwritten in the shadow pass is called model_transform. But in our production project, I can see that all of the uniforms are being overwritten, including a sampler2D that is used in both the vertex and fragment shaders.

I think my repro project has stripped away so much of our logic, that it is hard to see the motivation for doing things this way. In our actual project, the shaders and final output are much more complex.

Just an update - I’m investigating this at the moment, no conclusion yet.

ok I found a workaround. Change relevant line to this (added last parameter)

self.renderEntity_.model.model.meshInstances[0].material.setParameter('model_transform', Array.from(worldTransform), 0xFFFFFFFF);

this makes parameter to be applied in all passes, including shadow pass.

I don’t like this behaviour much at all, a parameter like this should not be required, and I’ll look at doing that in the near future.

2 Likes

here’s a change https://github.com/playcanvas/engine/pull/2391 that removes the need to pass 0xFFFFFFFF as last parameter. It it gets approved, it should be out probably sometimes next week, and you can stop passing the value in (it will be simply ignored if you do, no harm).

I tried out your fix and it works perfectly. Thanks so much for your help!

1 Like