Need help with pc.Picker


I’m trying to build an AR project / proof of concept where user can tap on the screen and have different items appear (and stay). Example of what I’m talking about: place-ground.

Since im very new, and this is a bit overwhelming, I’ve decided to divide everything into smaller milestones, and here they are:

  1. Create a simple project with couple of primitive entities. :white_check_mark:
  2. Implement something like an “event listener” that would console log entities that I click on.
    2.2 Do something with the entity once it is clicked, e.g. change its color.
  3. Have the event listener spawn a new entity, rather than console logging an existing entity.
  4. Have all of the above work on a phone (when user taps on the screen).
  5. Have all of the above work in an AR environment.

So, I’m currently at #2 and I’ve been sitting for few hours unable to progress.

Having looked at the forums and the docs/tutorials sections, I see that there are at least two ways of selecting an entity:

  • Using collision picking (raycasting).
  • Using frame buffer picking.

I’ve come to the conclusion that it would be beneficial for me to know both ways, so I started with the latter.

I am following this tutorial: entity picking, and although some of the in the snippets seem deprecated vs what the docs say, I’ve managed to get it to work.

Here is my current project: /overview/augmented-reality-basics

At this stage, could anybody kindly help me understand in plain English how the whole frame buffer picking works?

Following the documentation:

  1. What is a “pick buffer”?
  2. Why is 1024 width and 1024 height hardcoded in the tutorial (pick buffer’s width/height)?
  3. How does the getSelection method work exactly?
    3.1 I know it has 2 required and 2 optional arguments: x, y = the left and top edges of the “rectangle” and width, height of the rectangle. But what it the rectangle and how is it drawn? Is it something like a “selection box” in RTS games where you can click and drag your mouse to select multiple units?

This is the most difficult part for me right now; since I cannot image / don’t know how the virtual “rectangle” is drawn, I don’t know what I’m really doing in the code.

Here is my code so far:
(console logs are just for debugging purpose).

// jshint esversion: 6

const pickViaFrameBuffer = pc.createScript('frameBuffer');

pickViaFrameBuffer.prototype.initialize = function () {
    const mouse =;
    this.picker = new pc.Picker(, 1024, 1024);
    mouse.on("mousedown", this.handleClick, this);

pickViaFrameBuffer.prototype.handleClick = function(e) {
    const canvas = document.querySelector('#application-canvas');
    const canvasWidth = canvas.clientWidth;
    const canvasHeight = canvas.clientHeight;

    const camera =;
    const scene =;

    const picker = this.picker;

    const selection = picker.getSelection(e.x, e.y, canvasWidth, canvasHeight);
    console.log(`mouseX: ${e.x}, mouseY: ${e.y}` );
    console.log(`picker width: ${picker.width}, picker heigth: ${picker.height}`);

    selection.forEach(meshInstance => console.log(meshInstance ? : "undefined element"));


This is how it works right now:

I want to have it so I can click on an entity and log its name/details/whaterver.

The way it is done in the tutorial, it seems like the first MeshInstance is picked from the array of MeshInstances.

if (selected.length > 0) {
        // Get the graph node used by the selected mesh instance
        var entity = selected[0].node;

I think I’m fine with this solution, but I would appreciate if somebody could kindly explain to me how this method works.

Also, I’m curious as to why is there always one undefined element in the array. It picks up the ground, sphere, box, capsule and the cone, but also an undefined element and I don’t know what that is.

Apologies for the lengthy post and I hope my questions although very basic are still appropriate for these forums.

1 Like

Hi @Kris,

Good job in putting everything together. Some answers to your questions:

A buffer is a render surface where the engine can render models on. Pick buffer is a special render surface on which the engine will render all models in view by replacing their materials with very uniquely colored simple diffuse materials.

When you use the getSelection() the picker class will render to that buffer and based on the color of the pixel or pixels (for multi-select) selected it will try and find the selected models.

That is the resolution of the picker. For maximum precision you could just pass the width/height of your active window, so each picker pixel matches the actual rendered view of your app/game.

Yes, this class can select pc.MeshInstances, from there as the example does, you will have to move up the hierarchy until you find the parent entity.

Each pc.MeshInstance contains a single pc.GraphNode, it’s like a bare bones entity with limited functionality (e.g. contains only translation data, can’t hold components etc.). But they are still part of the entity hierarchy so traversing the hierarchy upwards eventually will lead you to the parent entity.


You did well in trying to figure it out. Hopefully @Leonidas answer helps you. I just wanted to add a few comments to complement his answer.

When you click with your mouse on something, you are basically clicking on a 2D image, which does not carry any useful info on where that click happened in the game world. In order for the engine to figure out which entity that pixel you clicked corresponds to, it can use a webgl picking method. It is a practice, where the engine would keep all the ids of all the mesh instances in a special image, with each id having own color. So, when you click anywhere, it will know that you clicked color “blue” (for example), and find that “blue” color is an id of your cylinder mesh instance.

So, the higher the resolution of your picker image, the more memory it will require to manage it, and the higher the precision of your clicks will be. And yes, I think an RTS selection analogy is a good one. It will take the complete image and read only a limited region from it, described by your rectangle. Any colors (ids) of mesh instances inside the rectangle will be returned to you in an array.


@Leonidas, @LeXXik

Thank you so much for taking the time to respond to me; I really appreciate that.

I now have better understanding and will try to make the #2 on my list work. I might have some additional questions later on; hope that is ok.


Quick question1:

I have changed the picker resolution from 1024 x 1024 to window.innerWidth x window.InnerHeigth and now it works like I wanted it to work, so it correctly chooses the right object.

However, when I resize the window, the picker resolution no longer matches my (now resized) window resolution.

Would it be a good idea to put some code in the update method to check the window size every frame? Or is there a better way of doing that?

Quick question2:

Does it make sense in my example to set the width and height of the selection rectangle to 1 pixel If I don’t care for multiple selection right now?

const selection = picker.getSelection(e.x, e.y, 1, 1);

1 Like

So, what we usually do is add an event listener to the resizecanvas event kindly provided by the pc.GraphicsDevice class:'resizecanvas', function(){
   // resize the pc.Picker instance
}, this);

Yes, exactly, that is the proposed way of doing single selections using the pc.Picker.


Please note that this method stalls the graphics renderer as it needs to render an extra frame every time you pick at the screen.


Yeah, what’s basically heavy isn’t so much the pick buffer rendered, but the readPixels() that is executed internally to pick the selected pixel color. The GPU has to wait for the CPU to get the result of the rendered buffer in memory.

1 Like

Thank you so much guys. With your help I have managed to get these items work:

  1. Create a simple project with couple of primitive entities. :white_check_mark:
  2. Implement something like an “event listener” that would console log entities that I click on. :white_check_mark:
    2.2 Do something with the entity once it is clicked, e.g. change its color. :white_check_mark:

Here is the current progress:

I’ll try to repeat these steps now, but now using the raycasting method as this is probably what I’ll be using the final iteration with AR.


I’ve managed to implement raycasting however I’ve faced the following issue:

Once I click the object and move it, the ray seems to “think” that the object is still in its original position. So it “doesn’t know” that the object has moved. I’m not really sure how to fix this.

Here is the code:

// jshint esversion: 6, unused: true, varstmt: true

const Raycast = pc.createScript('raycast');

Raycast.prototype.initialize = function () {'mousedown', this.handleClick, this);

Raycast.prototype.handleClick = function (e) {
    const camera =;
    const rayStart = camera.screenToWorld(e.x, e.y, camera.nearClip);
    const rayEnd = camera.screenToWorld(e.x, e.y, camera.farClip);

    const target =, rayEnd);

    if (target) {
        const { entity } = target;
        const { x, y, z } = entity.getPosition();
        entity.setPosition(x * -0.9, y, z);



Here is the project:

And here is a short demonstration of what I mean:

1 Like

So, you can think of it as having 2 worlds - Playcanvas world and Ammo.js world. When you use Ammo.js, it creates a physics world internally, where it keeps track of all the rigid bodies, their collisions, positions, etc. When you do a raycast, the hit test is done by the physics engine in its physics world. If there is a hit, it reports it to Playcanvas world, which then informs your script. When you move an entity with .setPosition() method, you are only moving it inside Playcanvas world, but not in the physics world. So the rigidbody in Ammo.js opinion is still at the old position. You can use .teleport() method of the Rigidbody Component to move it inside the physics world (and in Playcanvas as a result):

entity.rigidbody.teleport(position, [rotation]);
1 Like

Aha! That makes a lot of sense. Thank you so much for the insight.
/edit: works like a charm :slight_smile:

/edit2: I have managed to accomplish another milestone: spawning entities where the mouse points (using raycast). Yey!

Time for AR, which is probably going to be a lot more difficult.

1 Like

Hiya, I just wanted to share my progress so far:

This is what I currently have:
This is my end goal:

Now, im wondering, going further, would this place be still appropriate to ask questions about AR? For example, how to make it so the cubes I spawn, appear like they are actually placed in physical world, rather than just levitating :smiley:

Or is this something I would need to ask 8th Wall folks? Right now, I’m trying to figure things out by looking at the code in the “end goal” project.

For now, I have a transparent box entity (floor) and spawning a cube where the ray hits the floor. Seems like the other project is doing the same, but to much greater effect.

Not sure what you are looking for here? Apart from the mesh, they both look the same to me.

Edit: Adding shadows will help ground them to the real world.

1 Like

You know what, I think you’re right. I think I gave that project too much credit. I thought it had something like a “flat surfaces detection”, but looking into it further I think it it is literally placing objects on a transparent plane, rather than actually detecting flat surfaces from your camera feed.

What normal happens with AR libraries like WebXR, 8th Wall, etc, is that it detects planes of float horizontal surfaces. The app can then place planes in the world that can be raycasted against.

The planes aren’t necessarily transparent, they can just be ‘data’. More commonly, they are planes that are transparent but can receive shadows so the shadows of objects can be seen.

Eg (different AR example)


As for surface placement when using 8th Wall as your AR engine, it really is as simple as placing your 3d model such that its base sits at Y=0 in your scene.

Check out

Now that you have raycasting working, get the X/Z coordinate of the user tap against the (transparent) “ground” entity in your scene, and place the model at (X, 0, Z), assuming the pivot point of the model is at the bottom. If not, adjust Y-position accordingly so that the bottom of the object sits at Y=0.

In your cube example, if that’s a unit cube you’d spawn your models at (X, 0.5, Z)

Hope this helps!


@yaustar, @atomarch - Thank you for your reply!

I have played with the project a bit and got better results now. Having said that, I still am facing some issues.

  • If I set my “ground” entity completely transparent, it does not receive shadows. I have to set its intensity to at least 0.1. This results in the “ground” entity to being visible (barely, but still) when you tilt your phone camera to a certain angle.

  • Entities created on tap, do not “anchor” to the ground. When I move camera, they will sway and move with the camera quite a bit.

  • Entities only cast shadows in the area where I initially point the camera. So it seems like the light I placed in the scene, sits statically in one place and does not move with the camera.

I don’t know if I understand this correctly, so let me try to paraphrase and please let me know if I got it:

  • AR Libraries, 8th Wall in my case, detect float horizontal surfaces from the camera feed(?).
  • If a flat horizontal surface is detected (based on camera feed(?)), then the app will then place the plane(“ground”) in the real world that can be raycasted against
  • We are talking about the same plane/ground that I created beforehand, either in the editor or programmatically
  • If the flat horizontal surface is not detected, then the plane is not placed in the world and therefore the use cannot place the object because the location is “inappropriate”

Also, would you guys happen to know, if I use PlayCanvas as my framework of choice and load 8th Wall engine via external scripts, do I get to use all of the 8th Wall’s features or is this limited somehow?

You will need the custom material that is in the project I linked to earlier. See this file and line:

You shouldn’t be limited, no. It’s using the same SDK. You may need to hook/integrate some things yourself into PlayCanvas though.


It will give you the data, you will have to do the PlayCanvas side of things such as placing a plane down.

You can still place objects in the AR world, you just can’t test against any flat surfaces because it it hasn’t found them yet.

Is the light a child of the camera or moved in code? If not, it won’t move with the camera.

This is something you have to debug and start looking at console logs. Is the correct flat surface detected? Are you definitely placing the cube on the surface and not in the air?

1 Like

Thank you so much for your reply and for your patience with me :stuck_out_tongue:

I see, I will try to play with that later and see if I can make it work.

Thank you for your insight. I still need a bit of help to understand this, though.

When you say that I will get the “data” what do you mean? Is the data a JavaScript object that I can intercept and, for example, console log it?

I understand that it is this “data” that represents a “correct flat surface” that has been detected by the 8th Wall AR Engine (“instant flat surface detection”), right? So how would I intercept this data? Right now, my understanding is that, I setup my scene in a way where I place a huge 500 x 500 ground entity. When I open the camera, the ground entity covers all screen, and when I tap, I shoot a ray that hits the entity.

I’m struggling to understand where does the “data” about the flat surface detected/not detected come from.

It’s not the child of the camera nor it is moved in the code. Thanks for the hint, I will try to work on this.

Without looking at the 8th Wall documentation, I assume it will give you an object with a position vector and a normal to the plane.

You would have to check the 8th Wall documentation/forums/Slack, I haven’t had a chance to use 8th Wall yet.

1 Like

Check out and

Surface placement in 8th Wall works differently than you may be used to if you’ve ever built an
ARKit/ARCore based app - there is no “wave the phone around and wait for a surface to be detected” mechanism with 8th Wall’s AR engine.

8th Wall provides you with an “instant” surface. As I mentioned in my previous comment, all you need to do is position the bottom of your 3D model at Y=0 at it will sit on the surface in front of you. There is no “data” you need to query in order to do this and you don’t have to wait for anything to be “found”.

If your 3D model doesn’t look like it’s anchored to the surface, it’s likely that the bottom of the model isn’t sitting at Y=0 in your scene. An entity position of Y=0 may not be sufficient (e.g. if the pivot point of the 3d model is in the center, you’ll need to adjust so that the bottom is at Y=0). To convince yourself, temporarily make the plane non-transparent and compare the position of your 3D models to it.

The transparent “ground” plane is really only used to A) raycast against and B) receive shadows.