Adding new attribute for instancing

[nikita_osyak]

So I want to add extra instancing attributes to my geometry instancing setup.

I know that I already have transformation matrix arranged for me. I tried to follow the logic of this thread Hardware instancing and color, and did the following:

• overriden instancingVS chunk with

attribute vec4 instance_line1;
attribute vec4 instance_line2;
attribute vec4 instance_line3;
attribute vec4 instance_line4;

attribute vec4 block_uv; //xy : uv uffset, zw : uv scale

• overriden uv0VS chunk with:

vec2 getUv0() {
    vec2 resultVertex = vertex_texCoord0 * block_uv.zw;
    return resultVertex + block_uv.xy;
}

• added extra vertex format semantic:

const formatDesc = [
            //
            // transformation matrix property block
            { semantic: pc.SEMANTIC_TEXCOORD2, components: 4, type: pc.TYPE_FLOAT32 },
            { semantic: pc.SEMANTIC_TEXCOORD3, components: 4, type: pc.TYPE_FLOAT32 },
            { semantic: pc.SEMANTIC_TEXCOORD4, components: 4, type: pc.TYPE_FLOAT32 },
            { semantic: pc.SEMANTIC_TEXCOORD5, components: 4, type: pc.TYPE_FLOAT32 },

            //
            // uv offset\scale
            { semantic: pc.SEMANTIC_TEXCOORD6, components: 4, type: pc.TYPE_FLOAT32 }
        ];

and then layed out all of the data in float array as expected. However, I’m seem to be missing attribute definition in shader itself, since I get:

Vertex shader attribute "block_uv" is not mapped to a semantic in shader definition.

so. aside from having a fully custom shader for this, how can I access and add attributes to existing standard shader, if that’s even possible?

[Leonidas]

Calling @mvaligursky, he would be the best to answer for the current recommended way.

[nikita_osyak]

I found that there is _shader field in meshInstance, but seetting attribute there doesn’t do anything

[mvaligursky]

This needs to be done. I’m not sure there’s any good workaround without it, unless you start patching the engine code or similar: Custom attributes on material's shader definitions. · Issue #3485 · playcanvas/engine · GitHub

[nikita_osyak]

so. I was able to set up fully custom shader to support instancing, so I can use any semantics I want. However, I noticed super weird situation.

1. instanced attributes for matrices must be in semantic slots 12-15, otherwise they dont’ work
2. if I add additional vertexFormat data (with whatever semantics), shaders go apeshit and get stretches into infinity

works fine with default vertex buffer:

now I’m adding extra vertex format thingy, not even doing anything with that data:

{
    semantic: pc.SEMANTIC_ATTR11,
    components: 4,
    type: pc.TYPE_FLOAT32
},

ofc I have bigger block data, and add it to float32array, but I’m not actually using this instanced data in shader:

Now I have stretched to infinity things, though I’m sure with my vertex shader code, it’s very simple, and I changed nothing but added to vertex buffer.

would be super to get some insight on how this might be done @mvaligursky

[nikita_osyak]

When I was in process of making shader, I noticed that setting semantic like ‘ATTR16’ causes shader compiler to report that it occupies same slot as vertex_position, which is super weird. I totally don’t understand how semantics work in playcanvas specifically and would like to hear how do I use additional instancing attribute here, since it’s like first thing you want to have with instancing.

[mvaligursky]

There are these at least two hardcoded parts in the engine, that the issue I mentioned needs to address. You would need to workaround those somehow:

playcanvas/engine/blob/a51a3e2d66d3d6a983c95243f8982e2880bfcc87/src/scene/shader-lib/programs/lit-shader.js#L245-L251

playcanvas/engine/blob/main/src/scene/shader-lib/chunks/lit/vert/instancing.js

As mentioned, adding custom attributes for instancing is not currently supported, and needs to be addressed before this can be done without changes / workarounds to the engine. Custom attributes on material's shader definitions. · Issue #3485 · playcanvas/engine · GitHub

[mvaligursky]

@nikita_osyak

When I was in process of making shader, I noticed that setting semantic like ‘ATTR16’

We only have semantics up to ATTR15 … 16 does not seem to exist?

In general, the engine exposes more semantic names (around 31-32) then there is actual slots on the devices to use them, and so usually two of those get mapped to the same locations. Which means you cannot use together semantics that map to the same locations, for example both COLOR and ATTR4 both map to location 4, and so cannot be used at the same time. See the mapping here:

playcanvas/engine/blob/a51a3e2d66d3d6a983c95243f8982e2880bfcc87/src/platform/graphics/constants.js#L1196-L1226

[nikita_osyak]

thanks, seeing this mappings help me understand how and why I can have assign to same slot.

So. You saying that rn it’s impossible to add custom instancing attribute? Since the stuff that you mentioned as hard coded, I did addressed in my local solution. Are there more parts to it? I just need at least hard possible or impossible, since I’m very stuck rn and need to understand if I can invest more time in this solution or not.

[nikita_osyak]

Because from conversation and seeing other forum threads and github ticket it feels like I’ve addressed every angle that needed to be addressed

[mvaligursky]

It’s definitely possible by making changes to the engine, as its open source and easy to build. Apart from that you could also monkey patch the engine and change few functions to what you need them to be. But I’m not sure there is an easy to use way to do what you need - that’s why the issue was created to address this.

There was some example on the forums by people adding COLOR attribute to instancing. Instancing really only needs 3x4 matrix, but whole 4x4 matrix is passed in, and they changed the shader chunk to only use 3x4, and use the data in the remaining vec4 for other purspose in some other shader chunk. That is doable. Adding data on top of 4x4 I’m not sure, not without even more hacks.

[nikita_osyak]

I guess my question was about ‘where should i even start looking’ then, I see that it requires hacks, but I need to understand what exact amount of hacks, since look

• I have vertex format with added field
• I have custom shader that contains additional attribute for instanced data
• I have my semantics in order and matching between material and vertex format
• Vertex buffer seems fine also

I just don’t understand where to go from here and what to look at, what else is happening with data to become instanced then?

[mvaligursky]

To be honest, it would require a quite a bit of time spent looking at the engine code to exactly figure out what needs to change for this to work - I don’t have the exact answer ready, apart from pointing to bits of code that are relevant. You can grab the engine repo, build the engine, build the instancing example we have, and start looking around and make changes. I’m not sure there is a better answer.

[nikita_osyak]

I see.

For now I gave up and will use existing instancing slots that I have, 16 floats is actually not that bad, for my particular case at least. I can fit everything.

I will not mark this thread as solved and later will return to this case to understand how to really do it. Thanks for answers.

[LeXXik]

It is now possible to set attributes on standard material (in the next release, 1.66 probably). This allows you to provide additional data per each instance. The method is simple - when creating a vertex buffer, instead of the default vertex format, you create your own one and then set the semantics on the material.

Here is an example of passing two pc.Vec3 vectors:

// 16 for TRS matrix and +3 for each additional vector
const data = new Float32Array(instanceCount * (16 + 6));

let index = 0;
for (let i = 0; i < instanceCount; i++) {
    matrix.setTRS(pos, pc.Quat.IDENTITY, pc.Vec3.ONE);

    // add some custom vectorA
    data[index++] = vectorA.x;
    data[index++] = vectorA.y;
    data[index++] = vectorA.z;

    // add vectorB
    data[index++] = vectorB.x;
    data[index++] = vectorB.y;
    data[index++] = vectorB.z;

    for (let m = 0; m < 16; m++) {
        buffer[bufferIndex++] = matrix.data[m];
    }
}

// create vertex format, 10-11 - our vectorA and vectorB, 12-15 - instancing data
const format = new pc.VertexFormat(device, [
    { semantic: pc.SEMANTIC_ATTR10, components: 3, type: pc.TYPE_FLOAT32 },
    { semantic: pc.SEMANTIC_ATTR11, components: 3, type: pc.TYPE_FLOAT32 },
    { semantic: pc.SEMANTIC_ATTR12, components: 4, type: pc.TYPE_FLOAT32 },
    { semantic: pc.SEMANTIC_ATTR13, components: 4, type: pc.TYPE_FLOAT32 },
    { semantic: pc.SEMANTIC_ATTR14, components: 4, type: pc.TYPE_FLOAT32 },
    { semantic: pc.SEMANTIC_ATTR15, components: 4, type: pc.TYPE_FLOAT32 },
]);            

// create vertex buffer
const vertexBuffer = new pc.VertexBuffer(device, format, instanceCount, pc.BUFFER_STATIC, data);

// add attributes on material to map semantics
standardMaterial.setAttribute('vectorA', pc.SEMANTIC_ATTR10);
standardMaterial.setAttribute('vectorB', pc.SEMANTIC_ATTR11);

// create mesh instance, using that material and provide vertex buffer for instancing
const instance = new pc.MeshInstance(mesh, material);
instance.setInstancing(vertexBuffer);

Then in your vertex chunk, e.g. transformVS

attribute vec3 vectorA, vectorB;
vec4 getPosition()
{
    vec4 something = vec4(vectorA, 1.);
}