AR Camera Portrait Orientation Mismatch

In portrait mode, the perspective of AR marker content doesn’t quite line up with the real world camera. Augmented objects skew out of alignment one way or another when the phone is moved. Switching the phone to landscape and viewing it that way produces correct results. So it seems the game content and video are being scaled differently in portrait mode, but correctly in landscape.

I tried adjusting a few things in the scene rendering and camera settings. These changes either didn’t work or caused other problems. Is there some other setting or something in the script I should be looking for?

Edit: Found the orientation code. Maybe the FOV isn’t actually being set properly here?

ArCamera.prototype.onResize = function () {
    if (!this.arController) return;
    
    var device = this.app.graphicsDevice;
    var cw = device.width;
    var ch = device.height;
    var vw = this.video.videoWidth;
    var vh = this.video.videoHeight;

    // Resize the video texture
    if (this.entity.model && this.entity.model.model) {
        var material = this.entity.model.model.meshInstances[0].material;
        material.setParameter('uVideoSize', new Float32Array([vw, vh]));
        material.setParameter('uCanvasSize', new Float32Array([cw, ch]));
    }

    // Resize the 3D camera frustum (via the fov)
    var camMatrix = this.arController.getCameraMatrix();
    var fovy = 2 * Math.atan(1 / camMatrix[5]) * 180 / Math.PI;

    this.arController.orientation = (vw < vh) ? 'portrait' : 'landscape';
    if (vw < vh) {
        this.entity.camera.fov = Math.abs(fovy) * (vh / vw);
    } else {
        if (cw / ch > vw / vh) {
            // Video Y FOV is limited so we must limit 3D camera FOV to match
            this.entity.camera.fov = Math.abs(fovy) * (vw / vh) / (cw / ch);
        } else {
            // Video Y FOV is limited so we must limit 3D camera FOV to match
            this.entity.camera.fov = Math.abs(fovy);
        }
    }
};

Edit: Screenshots attached to demonstrate the problem. It’s more obvious when you move around to different vantage points where the object is always slightly too close to the camera, misaligned from surrounding lines of perspective, and seeming like it’s dipping into the ground away from the camera. Meanwhile in landscape mode, it’s consistently lines up with the marker’s transformation from any angle.

Which AR project are you basing this off? WebXR or the AR toolkit with the Hiro Marker?

Using ARKit based on this:

https://playcanvas.com/project/481413/overview/ar-starter-kit

Just found the 2020 version. Same result with that.

I had a go and it looks like the position is correct for me

If you keep it in the middle I find it’s pretty consistent. When it’s in the upper or lower area of the screen it diverges.

Ah I see. It does look it drifts when the marker is close to the bottom and the top of the screen. :thinking:

@will Can you take a look at this please?

I don’t think it’s the FOV. I set up some buttons to adjust it manually and the default is as correct as it gets.

I tried gradually adjusting the position and rotation of the AR Camera in the editor. At a certain point the portrait orientation starts matching better, but the landscape view becomes misaligned instead. If I can make sure it’s consistent enough in portrait mode, I’ll just swap the camera transformations on orientation change as a workaround for now.

These are the transformations where it started lining up:

Position Y: -0.036
Position Z: -0.195
Rotation X: 0.29

Edit: Actually, I have just been watching the edges of the regular camera view, even in the camera app (on a Galaxy S9 and iPad) and noticed that objects do distort slightly toward the edges. So maybe the problem is that PlayCanvas’ rendering isn’t distorted. I imagine there’s no easy way to implement that distortion to match the camera so all we can do is get it close enough.

I wonder if it’s related that our PlayCanvas camera only does one axis of FOV rather than both?

I don’t know much about FOV but not handling one of the axes sounds like something that would affect this.

It also depends on the device. On iPad Air, around 50 FOV seems to be comfortable with default camera transformations. Still out a bit but mostly behind the object where you don’t notice it. 55 looks better for the S9. They both default to around 57.4.