Hi again folks,
The pc.Picker seems to work well with default Materials, but for some reason it does not work for me when mesh instances use a new ´pc.Material´ with a custom shader.
When trying to access the selection, I get an error.
const selected = this.picker.getSelection(event.offsetX * this.pickAreaScale, event.offsetY * this.pickAreaScale);
const node = selected[0].node;
Uncaught TypeError: Cannot read properties of undefined (reading 'node')
Any idea why?
Would you be able to post a public example project?
It’s a pure engine project, so not very easily, sadly.
this.picker = new pc.Picker(this.app, canvasWidth * this.pickAreaScale, canvasHeight * this.pickAreaScale);
this.picker.resize(canvasWidth * this.pickAreaScale, canvasHeight * this.pickAreaScale);
this.picker.prepare(this.cameraComponent, this.app.scene);
const selected = this.picker.getSelection(event.offsetX * this.pickAreaScale, event.offsetY * this.pickAreaScale);
const node = selected[i].node;
The above code is quite indicative of all code regarding the Picker itself.
const vShader = `
attribute vec3 aPosition;
attribute vec2 aUv0;
uniform mat4 matrix_model;
uniform mat4 matrix_viewProjection;
varying vec2 vUv0;
void main(void)
{
vUv0 = aUv0;
gl_Position = matrix_viewProjection * matrix_model * vec4(aPosition, 1.0);
}
`;
const fShader = `
precision ${this.app.graphicsDevice.precision} float;
varying vec2 vUv0;
uniform sampler2D uDiffuseMap;
void main(void)
{
vec4 tex = texture2D(uDiffuseMap, vUv0);
gl_FragColor.rgb = vec3(1.0, 0.0, 0.0);
gl_FragColor.a = 1.0;
}
`;
const shaderDefinition = {
attributes: {
aPosition: pc.gfx.SEMANTIC_POSITION,
aUv0: pc.gfx.SEMANTIC_TEXCOORD0
},
vshader: vShader,
fshader: fShader
};
const material = new pc.Material();
material.shader = new pc.Shader(this.app.graphicsDevice, shaderDefinition);
material.setParameter("uDiffuseMap", this.diffuseMapAsset.resource);
return material;
And here’s the Shader, which is assigned directly to a pc.MeshInstance.material.
I’ve created a repro of the issue here with the following code:
function example(canvas: HTMLCanvasElement, deviceType: string): void {
const assets = {
'bloom': new pc.Asset('bloom', 'script', { url: '/static/scripts/posteffects/posteffect-bloom.js' }),
helipad: new pc.Asset('helipad-env-atlas', 'texture', { url: '/static/assets/cubemaps/helipad-env-atlas.png' }, { type: pc.TEXTURETYPE_RGBP }),
};
const gfxOptions = {
deviceTypes: [deviceType],
glslangUrl: '/static/lib/glslang/glslang.js',
twgslUrl: '/static/lib/twgsl/twgsl.js'
};
pc.createGraphicsDevice(canvas, gfxOptions).then((device: pc.GraphicsDevice) => {
const createOptions = new pc.AppOptions();
createOptions.graphicsDevice = device;
createOptions.mouse = new pc.Mouse(document.body);
createOptions.touch = new pc.TouchDevice(document.body);
createOptions.componentSystems = [
// @ts-ignore
pc.RenderComponentSystem,
// @ts-ignore
pc.CameraComponentSystem,
// @ts-ignore
pc.ScriptComponentSystem
];
createOptions.resourceHandlers = [
// @ts-ignore
pc.ScriptHandler,
// @ts-ignore
pc.TextureHandler
];
const app = new pc.AppBase(canvas);
app.init(createOptions);
// Set the canvas to fill the window and automatically change resolution to be the same as the canvas size
app.setCanvasFillMode(pc.FILLMODE_FILL_WINDOW);
app.setCanvasResolution(pc.RESOLUTION_AUTO);
const vShader = `
attribute vec3 aPosition;
attribute vec2 aUv0;
uniform mat4 matrix_model;
uniform mat4 matrix_viewProjection;
varying vec2 vUv0;
void main(void)
{
vUv0 = aUv0;
gl_Position = matrix_viewProjection * matrix_model * vec4(aPosition, 1.0);
}
`;
const fShader = `
precision ${app.graphicsDevice.precision} float;
varying vec2 vUv0;
uniform sampler2D uDiffuseMap;
uniform vec3 uColor;
void main(void)
{
gl_FragColor.rgb = uColor;
gl_FragColor.a = 1.0;
}
`;
const shaderDefinition = {
attributes: {
aPosition: pc.gfx.SEMANTIC_POSITION,
aUv0: pc.gfx.SEMANTIC_TEXCOORD0
},
vshader: vShader,
fshader: fShader
};
const assetListLoader = new pc.AssetListLoader(Object.values(assets), app.assets);
assetListLoader.load(() => {
app.start();
// setup skydome
app.scene.skyboxMip = 2;
app.scene.envAtlas = assets.helipad.resource;
app.scene.skyboxIntensity = 0.1;
// use a quarter resolution for picker render target (faster but less precise - can miss small objects)
const pickerScale = 0.25;
let mouseX = 0, mouseY = 0;
// generate a box area with specified size of random primitives
const size = 30;
const halfSize = size * 0.5;
for (let i = 0; i < 300; i++) {
const shape = Math.random() < 0.5 ? "cylinder" : "sphere";
const position = new pc.Vec3(Math.random() * size - halfSize, Math.random() * size - halfSize, Math.random() * size - halfSize);
const scale = 1 + Math.random();
const entity = createPrimitive(shape, position, new pc.Vec3(scale, scale, scale));
app.root.addChild(entity);
}
// handle mouse move event and store current mouse position to use as a position to pick from the scene
new pc.Mouse(document.body).on(pc.EVENT_MOUSEMOVE, function (event: any) {
mouseX = event.x;
mouseY = event.y;
}, this);
// Create an instance of the picker class
// Lets use quarter of the resolution to improve performance - this will miss very small objects, but it's ok in our case
const picker = new pc.Picker(app, canvas.clientWidth * pickerScale, canvas.clientHeight * pickerScale);
// helper function to create a primitive with shape type, position, scale
function createPrimitive(primitiveType: string, position: pc.Vec3, scale: pc.Vec3) {
// create material of random color
const material = new pc.Material();
material.shader = new pc.Shader(app.graphicsDevice, shaderDefinition);
material.setParameter('uColor', [0.0, 1.0, 0.0]);
material.update();
// create primitive
const primitive = new pc.Entity();
primitive.addComponent('render', {
type: primitiveType,
material: material
});
// set position and scale
primitive.setLocalPosition(position);
primitive.setLocalScale(scale);
return primitive;
}
// Create main camera
const camera = new pc.Entity();
camera.addComponent("camera", {
clearColor: new pc.Color(0.1, 0.1, 0.1)
});
// add bloom postprocessing (this is ignored by the picker)
camera.addComponent("script");
camera.script.create("bloom", {
attributes: {
bloomIntensity: 1,
bloomThreshold: 0.7,
blurAmount: 4
}
});
app.root.addChild(camera);
// function to draw a 2D rectangle in the screen space coordinates
function drawRectangle(x: number, y: number, w: number, h: number) {
const pink = new pc.Color(1, 0.02, 0.58);
// transform 4 2D screen points into world space
const pt0 = camera.camera.screenToWorld(x, y, 1);
const pt1 = camera.camera.screenToWorld(x + w, y, 1);
const pt2 = camera.camera.screenToWorld(x + w, y + h, 1);
const pt3 = camera.camera.screenToWorld(x, y + h, 1);
// and connect them using white lines
const points = [pt0, pt1, pt1, pt2, pt2, pt3, pt3, pt0];
const colors = [pink, pink, pink, pink, pink, pink, pink, pink];
app.drawLines(points, colors);
}
// sets material emissive color to specified color
function highlightMaterial(material: pc.Material, color: number[]) {
material.setParameter('uColor', color);
material.update();
}
// array of highlighted materials
const highlights: pc.Material[] = [];
// update each frame
let time = 0;
app.on("update", function (dt: number) {
time += dt * 0.1;
// orbit the camera around
if (!camera) {
return;
}
camera.setLocalPosition(40 * Math.sin(time), 0, 40 * Math.cos(time));
camera.lookAt(pc.Vec3.ZERO);
// turn all previously highlighted meshes to black at the start of the frame
for (let h = 0; h < highlights.length; h++) {
highlightMaterial(highlights[h], [0.0, 1.0, 0.0]);
}
highlights.length = 0;
// Make sure the picker is the right size, and prepare it, which renders meshes into its render target
if (picker) {
picker.resize(canvas.clientWidth * pickerScale, canvas.clientHeight * pickerScale);
picker.prepare(camera.camera, app.scene);
}
// areas we want to sample - two larger rectangles, one small square, and one pixel at a mouse position
// assign them different highlight colors as well
const areas = [
{
pos: new pc.Vec2(canvas.clientWidth * 0.3, canvas.clientHeight * 0.3),
size: new pc.Vec2(100, 200),
color: pc.Color.YELLOW
},
{
pos: new pc.Vec2(canvas.clientWidth * 0.6, canvas.clientHeight * 0.7),
size: new pc.Vec2(200, 20),
color: pc.Color.CYAN
},
{
pos: new pc.Vec2(canvas.clientWidth * 0.8, canvas.clientHeight * 0.3),
size: new pc.Vec2(5, 5),
color: pc.Color.MAGENTA
},
{
// area based on mouse position
pos: new pc.Vec2(mouseX, mouseY),
size: new pc.Vec2(1, 1),
color: pc.Color.RED
}
];
// process all areas
for (let a = 0; a < areas.length; a++) {
const areaPos = areas[a].pos;
const areaSize = areas[a].size;
const color = areas[a].color;
// display 2D rectangle around it
drawRectangle(areaPos.x, areaPos.y, areaSize.x, areaSize.y);
// get list of meshInstances inside the area from the picker
// this scans the pixels inside the render target and maps the id value stored there into meshInstances
const selection = picker.getSelection(areaPos.x * pickerScale, areaPos.y * pickerScale, areaSize.x * pickerScale, areaSize.y * pickerScale);
// process all meshInstances it found - highlight them to appropriate color for the area
for (let s = 0; s < selection.length; s++) {
if (selection[s]) {
//console.log(selection[s]);
const material = selection[s].material as pc.Material;
highlightMaterial(material, [1.0, 0.0, 0.0]);
highlights.push(material);
material.update();
}
}
}
});
});
});
}
With this example project: https://playcanvas.github.io/#/graphics/area-picker
Ah, yes it seems to break the example. The Picker ceases to select anything.
Created ticket for this: Area picker doesn't work with custom shader on pc.Material · Issue #5265 · playcanvas/engine · GitHub
I added a comment on the ticket.
Thanks for taking a gander, guys