Mouse to Screen Space

Help & Support
1

Hi all,

So I’m in the peculiar situation where the game I’m making is entirely UI - as in, the camera only exists to render a 2D screen, but doesn’t do anything in and of itself.

The issue I’m facing is that I want to drag one of my image elements. I’ve got my mouse events handled properly, but I’m having trouble converting the mouse position to world space, because the camera screenToWorld() method specifically functions off the camera, and if the camera doesn’t really matter, then the positions it is using to translate to world space don’t actually point to the real world space of my UI objects.

At best, I seem to be able to use an orthographic camera, but given the fact that the 2D screen sets its dimensions to pixels and the camera sets its Ortho Height to world points, you can’t match the two.

Ideally, I’m imagining the solution to be a screenToWorld() method that actually uses the screen as a reference point, rather than the camera, but I’m not sure how I would make that happen.

Hoping someone has some ideas, the end result is ideally a simple element drag.

2

Hi @TingleyWJ,

If you simple want to drag an element then you can subscribe to the element mouse/touch events and translate its position accordingly when there is cursor/touch input.

This is pseudo code:

    // mouse events
    entity.element.on('mousedown', this.onPress, this);
    entity.element.on('mousemove', this.onMove, this);
    entity.element.on('mouseup', this.onRelease, this);
    entity.element.on('mouseleave', this.onRelease, this);

MyScript.prototype.onMove = function(e){
    
    var myEntity = e.element.entity;
    
    if( myEntity.tags.has('selected') ){        

        myEntity.translateLocal(e.dx, -e.dy, 0);
    }
};

For this to work the mouse cursor has to be on top of the element when moving to register mousemove events. This will require some sync to get it perfect but it’s a good start. More correctly you could add a generic app mousemove listener and translate the mouse coords to entity/element coords.

3

Experimenting with this now. It is a start, but it’s rocky. I think I could fix the jittery-ness with some clever uses of interpolation though. My main concern is the fact that myEntity.translateLocal(e.dx, -e.dy, 0); seems to be contigent on mouse DPI. I’m not aware if I can access that attribute via JS, but if I can’t then I don’t know how to account for varying mouse speeds.

Could you elaborate on what you mean by this?

translate the mouse coords to entity/element coords

4

I don’t think you can access that, but you can definitely smooth this out using interpolcation/lerping.

Mouse input comes as X/Y coordinates from window top/left, whereas entity/element are placed in either screen local space.

5

Found a solution, so I figured I’d post it for anyone who comes across this in the future.

  1. Bind this.app.mouse.on('mousemove', this.SaveMousePos, this); in one of your scripts. The callback is simple, saving the mouse position to a Vec2: this.MousePos = new pc.Vec2(e.x, e.y);
  2. On the script of the element you want to drag, bind this.entity.element.on('mousedown', this.MouseDown, this); and this.entity.element.on('mouseup', this.MouseUp, this);

In MouseDown() (set this.MouseIsDown = false and this.MouseTimer = 0 in Initialize)

this.MouseIsDown = true;
this.MouseTimer = 0;

In MouseUp()

this.MouseIsDown = false;
  1. On the element you want to drag, in the script’s update function, you’ll want:
if (this.MouseIsDown) {
        this.MouseTimer += dt;

        //You can change 0.1 to however long you want the drag to register. I have this here because I also want to check against mouse clicks.
        if (this.MouseTimer >= 0.1) {
            //Note: My Screen's Scale Mode is set to Blend at a value of 1, and these calculations reflect it as such. They'll need editing if you use Blend at a value of 0, but should work fine with Blend at 0.5
            let origResolution = this.entity.element.screen.screen._resolution.clone();
            let resolution = origResolution.clone();
            resolution.x *= this.entity.element.screen.screen.scale;

            let mousePos = new pc.Vec2([YourMousePosScript].MousePos.x, [YourMousePosScript].MousePos.y);
            mousePos.x -= ((origResolution.x - resolution.x) / 2) * ((origResolution.x / 2 - mousePos.x) / (resolution.x / 2));

            //Note: My resolution is set to 1920x1080, and because Blend is at 1, it can be hardcoded.
            //Note: The entity's parent's center was not the application's (0, 0), and thus I had to calculate the Y value with some modifications. In this case, it was originally 250 pixels up from 1080, resulting in 4.32 as the edited value.
            this.entity.setLocalPosition(mousePos.x - (origResolution.x / 2), -(mousePos.y - (resolution.y - (resolution.y / 4.32))) * (1080 / resolution.y), 0);
        }
    }

The last bit is definitely a bit of math, but with minor adjustments for a given project, it should be usable for others as well.

End Result: Draggable elements.

1 Like
6

You can further optimize it by not creating new vectors on each frame or on every mouse move event. Instead, create the vector once on initialization, and only update it when the update loop is running.

// in your mouse script
initialize: function() {
    this.mousePos = new pc.Vec3();
},
updateMousePos: function(e) {
    this.mousePos.x = e.x;
    this.mousePos.y = e.y;
}

// in your other script
initialize: function() {
    this._tempPos = new Vec3();
},
update: function() {
    let mousePos = this._tempPos.copy(mouseScript.mousePos);
    // do something with the new mouse position
}

Creating new vectors via new pc.Vec2() is generally an expensive operation, so you should try to minimize its usage and re-use already created objects.

2 Likes