WebXR - AR camera not working

I was working on a project on playcanvas which was using webXR for AR bit, everything was working fine till yesterday but it suddenly stopped providing camera feed and access after the recent updates of playserviceAR and chrome browser services. Looking for a quick support on it. Old projects working fine but any new exported project or build doesn’t work

Can you add one of the PlayCanvas team to the project please as read permissions? ‘yaustar’ for me.

We did do an engine release yesterday that may have affected things :frowning:

This PR might fix your issue: https://github.com/playcanvas/engine/pull/2205

Will know more once I get access to your project :slight_smile:

WebXR api was working fine till yesterday but it suddenly stopped the work on mobile Chrome browser. I am using this code. Please resolve my problem.

this.buttonVr.button.active = this.app.xr.supported && this.app.xr.isAvailable(pc.XRTYPE_VR);
this.buttonAr.button.active = this.app.xr.supported && this.app.xr.isAvailable(pc.XRTYPE_AR);

this.clearColor = this.cameraEntity.camera.clearColor.clone();
this.clearColorAR = new pc.Color(0, 0, 0, 0);

// click button
this.buttonVr.element.on('click', function() {
    // check support
    if (this.app.xr.isAvailable(pc.XRTYPE_VR)) {
        // start session
        this.cameraEntity.camera.startXr(pc.XRTYPE_VR, pc.XRSPACE_LOCALFLOOR);
    }
}, this);

this.buttonAr.element.on('click', function() {
    // check support
    if (this.app.xr.isAvailable(pc.XRTYPE_AR)) {
     
        this.cameraEntity.camera.startXr(pc.XRTYPE_AR, pc.XRSPACE_VIEWER);

    }
}, this);

this.app.keyboard.on('keydown', function(evt) {
    if (evt.key === pc.KEY_ESCAPE) {
        this.app.xr.end();
         this.Splashmenu.enabled = true;
    }
}, this);

this.app.xr.on('start', function() {
    this.buttonVr.enabled = false;
    this.buttonAr.enabled = false;
    
    if (this.app.xr.type === pc.XRTYPE_AR) {
        this.cameraEntity.camera.clearColor = this.clearColorAR;
        this.app.scene.root.findByTag('nonAr').forEach(function(e) {
            e.model.enabled = false;
        });
    }
}, this);

this.app.xr.on('end', function() {
    this.buttonVr.enabled = true;
    this.buttonAr.enabled = true;
    
    this.cameraEntity.camera.clearColor = this.clearColor;
    this.app.scene.root.findByTag('nonAr').forEach(function(e) {
        e.model.enabled = true;
    });
}, this);

this.app.xr.on('available', function() {
    this.buttonVr.button.active = this.app.xr.supported && this.app.xr.isAvailable(pc.XRTYPE_VR);
    this.buttonAr.button.active = this.app.xr.supported && this.app.xr.isAvailable(pc.XRTYPE_AR);
}, this);

Thanks @yaustar for the swift response. I have added you to the project. please look into it and revert me on same at the earliest

It doesn’t seem to be a bug with the engine as using the older version of the engine didn’t work either.

Sounds like something has changed and perhaps the engine needs to be updated to match. WebXR is still in the experimental stage so things can break at any point.

@moka May know more as he add support for it in the engine.

@yaustar thanks for this. Can you or @moka suggest me a way out on this as my project is almost done. Need to resolve this on very high priority.

If this is a commercial product that depends on WebXR, that is a risky position to take IMHO as any update on the browser API for WebXR could break published builds. There’s not much we can do to prevent it on the PlayCanvas side TBH

@yaustar I really appreciate the efforts, Let me know what maximum is possible from play canvas side and if you can point me to the right direction, this would be really helpful as for now I am completely lost on the situation.

I was wrong, our last engine release did break WebXR :frowning:

Looking into a fix now. Please add the param of the old engine for now while developing for the launch URL where XXXXXX is the screen id.

eg https://launch.playcanvas.com/XXXXX?use_local_engine=https://code.playcanvas.com/playcanvas-1.28.6.js

Thanks for this, let me try this and please update me as soon as you fix this

There’s a fix in PR at the moment: https://github.com/playcanvas/engine/pull/2210

I also suggest integrating WebXR polyfill to support devices that don’t have native WebXR enabled: https://github.com/immersive-web/webxr-polyfill

It’s working. Thank you @yaustar.

@yaustar and @moka We are workin on this and there is a issue that we are facing, I am facing an issue with screen record and screen capture in AR mode. Can you help me on same Please.

Unfortunately, I’m pretty rammed at the moment so I won’t be able to provide direct support for that. What have you tried so far? Can you provide a public project that you are attempting this with please?

Thanks for your response and we completely understand that you are rammed but just to let you know - this is a client project with 20 or 30 projects in the pipeline and we want to move this client to playcanvas but all of their projects require screen capture so if you could help us then this is a large client for both you and us and we would greatly appreciate it.
As per your instruction we have provided a public link project below and here are the details of what we have already tried:
1.

this._window = window.open('', '');
 this._window.document.title = "Screenshot";
 this._window.document.body.style.margin = "0";
 var canvas = this.app.graphicsDevice.canvas;
 var img = new Image();
 img.src = canvas.toDataURL();
 this._window.document.body.appendChild(img);
var texture = new pc.gfx.Texture(this.app.graphicsDevice);
var image = this.app.graphicsDevice.canvas.toDataURL('image/png');
 var img = new Image();
    img.onload = function () {
        texture.minFilter =  pc.XRTYPE_AR.FILTER_LINEAR;
        texture.magFilter =  pc.XRTYPE_AR.FILTER_LINEAR;
        texture.addressU = pc.gfx.ADDRESS_CLAMP_TO_EDGE;
        texture.addressV = pc.gfx.ADDRESS_CLAMP_TO_EDGE;
        texture.setSource(img);
    };
   img.src = image;
this._window.document.body.appendChild(img);

So here’s the tricky part. This code works when I’m still not in AR mode, it takes a screenshot and renders into the texture successfully.
However, if I enter XR mode (with the device’s camera) and do the same thing, I only get a black texture.
Is it possible that “Preserve Drawing Buffer” is disabled while in AR mode somehow?could you told me how to get texture form that framebuffer?

Here is a public build as requested https://playcanvas.com/project/689747/overview/ardevelopment

A possibility is that the canvas is not used at all in VR/AR hence getting a black image.

I would check the engine source code to see how it is setup when it enters XR, there might be some clues from Moka.

I would also try using the camera’s render to texture method (eg: https://playcanvas.com/editor/scene/719944)

Thank you. But its not working.

Currently looking into this and it might not be possible yet with WebXR: https://github.com/immersive-web/proposals/issues/36

Reading more into it: https://github.com/immersive-web/webxr-samples/issues/36
@ArnaudHambenne is right, the current Chrome immersive-ar implementation does not allow JS-side access to camera pixels”

Edit: An alternative here is to use something like 8th Wall who does have an API to capture a screen grab.