Interacting with UI elements during AR Session

Help & Support
1

Hello.

I am currently working on a small project. Basically, I want to have a cube in AR, which I can move, rotate, and resize, similar to this:

I got the moving part working and right now I want to have a button that will rotate my object on press.

This code is to test if the button interaction works in the first place.

// jshint esversion: 6, unused: true, varstmt: true
const RotateControls = pc.createScript('rotateControls');

RotateControls.attributes.add('rotateButton', { type: 'entity' });

RotateControls.prototype.initialize = function() {
    const rotateButton = this.rotateButton.element;
    console.log(rotateButton);
    rotateButton.on('touchstart', () => alert('hello') , this);
};

The code works only if the AR session has not started yet. But as soon as I start the AR session, the button does not do anything. I think this may have something to do with the hit test for AR which probably takes precedent over the button event but I don’t know yet how to fix this.

Any ideas?

2

I would be surprised if the XR session is consuming a touch event. Is this in WebXR or 8th Wall?

1 Like
3

This is WebXR.

4

Hmm could be a bug, will need someone to deep dive into it.

5

In case someone does not have a phone that is compatible for viewing this project, this is how it looks like on an emulator:

6

I still have not figured this out, however I have been doing some research and it seems that I need something called “WebXR DOM Overlay”: https://immersive-web.github.io/dom-overlays/.

Also, in this video: https://youtu.be/ypSkIYpJjE8?t=1320 Ada, who is Immersive Web chair talks about that feature.

So, it seems like I need to:

  1. Use html + css assets to serve as my UI rather than native PlayCanvas UI.

  2. Need to somehow specify that I want to use DOM Overlay as an optional feature (although this seems to me like something that is too low level and not accessible through PlayCanvas(?)):

https://immersive-web.github.io/dom-overlays/#initialization

let uiElement = document.getElementById('ui');
navigator.xr.requestSession('immersive-ar', {
    optionalFeatures: ['dom-overlay'],
    domOverlay: { root: uiElement } }).then((session) => {
    // session.domOverlayState.type is now set if supported,
    // or is null if the feature is not supported.
  }
}
  1. Attach event listener to to my HTML UI element and listen to a special event called beforexrselect:

https://immersive-web.github.io/dom-overlays/#onbeforexrselect

document.getElementById('button-container').addEventListener(
  'beforexrselect', ev => ev.preventDefault());

So I think that XR session consuming a touch event is not a bug but an expected behavior? I might be wrong here, but this is what it seems like.

I don’t know how to implement all of the above yet, especially how to enable [dom-overlay] special feature.

Right know I’ve copied some code from an HTML button tutorial but:

  • My html elements are not connected to my CSS, don’t know why.
  • They disappear when I start AR session (probably because the [dom-overlay] is not enabled).

Here is the editor with all the parts: https://playcanvas.com/editor/scene/1009448

I just wanted to share my finding, hoping for help/suggestions. Will try to figure this on my own, but if you guys have any ideas, I would very much appreciate that.

7

I wonder if you can forward the event on to the canvas once you get it from the WebXR DOM Overlay

8

Hi, I’ve had some experience with this since it came out. Currently I’m doing a bit of a monkey-patch to PlayCanvas’ internal function to pass in the DOM overlay optionalFeature. Here’s how it looks:

// Small adjustments to PlayCanvas' internal function for dom-overlay support
    pc.app.xr.start = function (camera, type, spaceType, options) {
        var self = this;
        var callback = options;

        if (typeof(options) === 'object')
            callback = options.callback;

        if (! this._available[type]) {
            if (callback) callback(new Error('XR is not available'));
            return;
        }

        if (this._session) {
            if (callback) callback(new Error('XR session is already started'));
            return;
        }

        this._camera = camera;
        this._camera.camera.xr = this;
        this._type = type;
        this._spaceType = spaceType;

        this._setClipPlanes(camera.nearClip, camera.farClip);

        var optionalFeatures = [];

        if (type === pc.XRTYPE_AR) {
            optionalFeatures.push('light-estimation');
            optionalFeatures.push('hit-test');
            optionalFeatures.push('dom-overlay');
        } else if (type === pc.XRTYPE_VR) {
            optionalFeatures.push('hand-tracking');
        }

        if (options && options.optionalFeatures) {
            optionalFeatures = optionalFeatures.concat(options.optionalFeatures);
        }

        navigator.xr.requestSession(type, {
            requiredFeatures: [spaceType],
            optionalFeatures: optionalFeatures,
            // Pass in the document body as an overlay into the WebXR session. 
            // Probably not the best practice to include the whole body but it includes all HTML / CSS you write
            // as well as the canvas element so that this.app.touch events work as normal
            domOverlay: {root: document.body}
        }).then(function (session) {
            self._onSessionStart(session, spaceType, callback);
        }).catch(function (ex) {
            self._camera.camera.xr = null;
            self._camera = null;
            self._type = null;
            self._spaceType = null;

            if (callback) callback(ex);
            self.fire('error', ex);
        });
    };

You can pass whatever you want into the domOverlay object, but for this.app.touch events to work you need to include the application-canvas. Make sure to set the canvas’ opacity to 0 when it starts or you’ll see two canvases.

If the PC devs know a better way of doing this please let me know :slight_smile:

1 Like
9

I’ve just made a PR to integrate DOM Overlay API into engine.

PR: https://github.com/playcanvas/engine/pull/2614
And demo: https://playcanv.as/p/S01LYTIU/

Will be available in Editor once merged and enging released.

3 Likes
10

That’s great news!

11

Is there a way to use this even with PlayCanvas Button entities?

Right now my PlayCanvas buttons work in regular mode but don’t work in XR mode. In XR mode, they are visible but don’t work.

12

They should work. There’s a tutorial here https://developer.playcanvas.com/en/tutorials/webxr-ray-input/

13

I’m using React.js for the UI, and I’m using the “WebXR AR Starter Kit” template for PlayCanvas.
I’m embedding PlayCanvas using an iframe with the “allow” required (xr-spatial-tracking; vr; camera; fullscreen; gyroscope; accelerometer;) and I have the communication between React and Playcanvas working fine.

The page works just fine, as expected. But when I go into AR mode, all the UI elements made in React dissapear. And when I exit AR mode the UI comes back.

So I tried the solution of this thread before starting the AR mode, but I get a black screen instead of the phone camera view. The page goes full-screen, and I can see the UI made in react along with the UI made in PlayCanvas (the AR button), but the background is just black (it should show me the smarphone camera view)

This is the code I added in “AugmentedRealityManager.js” in the “_startAugmentedRealitySession” function

const reactElement = window.parent.document.querySelector(‘.ui-container’);
if(this.app.xr.domOverlay && reactElement){
this.app.xr.domOverlay.root = reactElement;
}
this.arCameraEntity.camera.startXr(pc.XRTYPE_AR, this.spaceType);

So any tips what I may be doing wrong?

14

Just a few minutes after I posted this I found the problem.
So I’m posting the solution in case it can be of help for other people.

The iframe can’t be inside the dom element used in domOverlay.root

my structure was something like this:

< div className=“ui-container”…>
__< div className=“some-other-classes” >
____< iframe src=“…playcanvasindex.html”/>
__< /div>
< /div>

So I changed it to something like this:

< div className=“new-class”…>
__< iframe src=“…playcanvas.html”/>
__< div className=“ui-container”…>
____< div className=“some-other-classes” />
__< /div>
< /div>

This way the iframe is no longer inside “ui-container”, and now it works, I can see the React UI on top of the 3D scene in AR mode.