Proof of concept: 3DGS scene + head-tracked perspective (pseudo-holographic display)

Showcase
1

Hey guys!

So here is the idea - if we track user’s face with device front-facing camera, we can create a head-tracked perspective aka “pseudo-holographic” display. You’ve probably seen some examples on youtube, so here I reproduced this “digital window” effect in a live web demo:

https://www.cognitive.plus/apps/3dgs-parallax/

Side notes:

:large_orange_diamond: The scene was created with Apple’s ML Sharp model and is ~1.2M splats. It was converted to .sog format and takes ~10.5MB (~6x reduction in size)

:large_orange_diamond: You can fly around using mouse / keyboard / touch

:large_orange_diamond: 2 modes are supported:

  • Head Tracking (front camera) - implemented with MediaPipe
  • World Tracking (back camera) - implemented with 8th Wall

Additional URL params:

:large_blue_diamond: ?q / ?quality = high | low | 0.125 ... 3
Set it to high if you want high-rez 3DGS rendering, or to low if you’re on potato device :slightly_smiling_face:

:large_blue_diamond: ?blur / ?faceblur = true | 0 ... 3
Enable it if you’re shy and want to record a video, but with your face blurred out. Alternatively, you can always close face cam with a dedicated close button

:large_blue_diamond: ?fs / ?fullscreen = true
Enable full screen (requires tap / click)

:large_blue_diamond: ?mode = all
Enable both World and Head Tracking on your device. By default Head Tracking is enabled only on laptops and tablets, and World Tracking - on tablets and mobiles

4 Likes
2

Great work!
Congrats

3

Really cool and great scene btw.

I can’t quite tell, and forgive me if I’m wrong, but it looks like you’re mapping head rotation and position to the camera transform itself. This feels quite natural, but to get that “holographic” effect scene-in-your-display effect, you actually want to transform the projection matrix itself so the projection axis is asymmetric. The camera transform itself remains unchanged. This is what makes it look wrong for everyone else but uncannily real at the focal point.

Anyway you may already be doing this, but just thought I’d offer my 2 cents :blush:

4

Thanks Mark! Yep, may be you’re right, it’s not textbook implementation with skewed projection matrix, etc :slight_smile: I’m applying camera transform based on detected face position / orientation, with some remapping. I’m also limiting view rotation based on distance - the further you are, the more clamped your rotation is. That felt more natural to me during testing.

Alternative approach with actual skewed matrix sounds interesting, I could try it in some spare time as well :ok_hand:

5

This is great. I wanted to do a horror house for halloween where if you walked past the window a camera would track you so it looked like you looking into a haunted room.

6

Interesting, but why stop there? The entire navigation around the house could be done with this approach + click to teleport (like in 360 tours). Would be so cool to work on commercial project like that! :jack_o_lantern:

7

I can definitely see this being use more at theme parks.