We are very excited to release our showcase demo for Microsoft Mixed Reality Capture Studios (MRCS) volumetric video technology.
PlayCanvas now supports MRCS volumetric video with a playback library for captured footage at their studios. Watch it on desktop, mobile with AR or even in a WebXR-enabled VR headset, all from a single URL!
The library can be easily added to any PlayCanvas project and used to create fantastic immersive mixed reality experiences.
Read how we made it here: PlayCanvas now supports Microsoft volumetric video playback | PlayCanvas Blog
Retweet us!
8 Likes
Hey,
I run into some problems while trying to set things up on my end. I’ve got a video file (.hcap and .mp4) as well as the devkit library. However I don’t really understand how to serve those file for the player to work. Getting 404s for the output files of both the dash player and the holo video object library. Any documents you can point to for how to set up / serve those files?
Paul
Have you followed the set up steps here?
1 Like
Yes, followed those steps. Getting the following output from the console running the build project locally and also pointing the holo-player to the .hcap locally:
I’m using a video from a customer, are there any additional files that might be missing? Only have a .hcap and .mp4.
It seems your local host is not delivering the files / at the correct url? For example the preloader.js
preloader.js isn’t required, just make an empty file to fix that asset error.
You need way more than .hcap and .mp4, simply use Network tab in any browser and search for “azure”:
soccer_20k1504_web/soccer_20k15040.bin
soccer_20k1504_web/hls/soccer_20k15042.ts
soccer_20k1504_web/hls/soccer_20k15043.ts
soccer_20k1504_web/hls/soccer_20k15041.ts
soccer_20k1504_web/hls/soccer_20k15040.ts
soccer_20k1504_web/hls/soccer_20k15044.ts
soccer_20k1504_web/hls/soccer_20k1504.m3u8
soccer_20k1504_web/soccer_20k1504.hcap
soccer_20k1504_web/dash/soccer_20k1504_dash_track0_init.mp4
soccer_20k1504_web/dash/soccer_20k1504_dash_track1_3.m4s
soccer_20k1504_web/dash/soccer_20k1504_dash_track1_4.m4s
soccer_20k1504_web/dash/soccer_20k1504_dash_track0_3.m4s
soccer_20k1504_web/dash/soccer_20k1504_dash_track0_4.m4s
soccer_20k1504_web/dash/soccer_20k1504_dash_track1_5.m4s
soccer_20k1504_web/dash/soccer_20k1504_dash_track1_init.mp4
soccer_20k1504_web/dash/soccer_20k1504_dash_track1_2.m4s
soccer_20k1504_web/dash/soccer_20k1504_dash_track0_5.m4s
soccer_20k1504_web/dash/soccer_20k1504_dash_track0_2.m4s
soccer_20k1504_web/dash/soccer_20k1504_dash_track0_1.m4s
soccer_20k1504_web/dash/soccer_20k1504_dash_track1_1.m4s
soccer_20k1504_web/dash/soccer_20k1504.mpd
soccer_20k1504_web/soccer_20k15041.bin
soccer_20k1504_web/soccer_20k15045.bin
soccer_20k1504_web/soccer_20k15042.bin
soccer_20k1504_web/soccer_20k15043.bin
soccer_20k1504_web/soccer_20k15044.bin
And that should be all you need (51MB).
1 Like
I needed a test project to display RuntimeTypeValidator.js, so I picked this one, as it’s quite interesting.
I found and fixed several issues, the whole bug fixing story is here:
floating-point-exception/NaN bugs, resource allocation bugs and…
I hope you enjoy it, have fun watching! 
1 Like
There is another bug in the ui-video-texture.js script.
We have a line like: video.addEventListener('canplaythrough', ...);
But the event listener is never deleted. When the event is triggered, the script deletion crashes.
- Open the project: Volumetric Video Microsoft MRCS - PLAYCANVAS
- Open DevTools, pick the right frame context
- Run this code:
pc.app.root.destroy();
- Also possible:
pc.app.scenes.changeScene("Main Scene") (twice)
Error:
Fix:
this.bound_canplaythrough = this.canplaythrough.bind(this);
video.addEventListener('canplaythrough', this.bound_canplaythrough);
UiVideoTexture.prototype.canplaythrough = function(e) {
//debugger;
console.log("canplaythrough");
//this.entity.element.texture = this.videoTexture;
this.entity.element.color = pc.Color.WHITE;
if (this._playing) {
video.play();
}
}
Another issue was this, this line doesn’t need to be in canplaythrough… at least not in Chrome (maybe in other browsers):
// was in canplaythrough, but triggers first with texture dim of 0x0,
// which divides 0 by 0 for aspect ratio = Infinity
this.entity.element.texture = this.videoTexture;
3 Likes
Great to see MS Volumetric support in Playcanvas! 
I’m getting the following error when running the project, with holo-video-object-umd.js added from the repo.
And the same, when running locally (error is repeated for every update):
No idea why the preloader is not included in the download or what the /null reference is about.
I can see that all the files seem to load fine though (both online and local):
Is the holo-video-object-umd.js from the repo above not the correct one? Or have I missed something else?
Also tried loading the holo-video-object-umd.js before others, which didn’t help.
This is the HoloVideoPlayer script that we wrote, which calls into Microsoft’s SDK, and it crashes inside updateBuffers call. It’d be worth placing a breakpoint on that line, and inspecting if setBuffers perhaps passes some nulls / undefines or similar.
Thanks, seems like you’re right, both this.norView and this.texture.impl._glTexture resolve to null. I’ll try to figure out what’s going on… might be a bit over my head.
I traced it back to this line:
let hvo = this.hvo = new MRCS.HoloVideoObject(this.app.graphicsDevice.gl, {
disableTransformFeedback: true
});
Which doesn’t seem to set the hvo correctly, is that right?:
Enabling hvo.logLevel = 3 gave me this in console:
Looking at that script, norView is null when the file does not have normals … and as the script seems to be aware of this possibility, I think passing null for norView is ok.
_glTexture can be null for few initial frames, the comment mentions that too:
// some number of ticks will pass before both static preview frame is loaded and 'texture.impl._glTexture' is initialized, at that point we'll finally be able to display something
So I think think these two parametes are not causing problems.
But I am not sure what is … as this is inside Microsoft SDK … you might need to ask them.
Not sure, it seems odd everything is undefined, but it could be expected, maybe following calls initialize those members. Not sure, this is inside Microsoft SDK.
Just double check that this.app.graphicsDevice.gl that you pass in seems valid.
Ok thanks, will do. It’s weird that the example runs fine and all I’ve done is forked it and added the holo-video-object-umd.js from the repo…
Definitely strange, perhaps their SDK has changed since the time the example was created, not sure.
Would someone be willing to check their version of the holo-video-object-umd.js that it matches mine?
Name: holo-video-object-umd.js
Size: 114862 bytes (112 KiB)
SHA256: 8d0b5522bc1a9c7ea0c442f4931bbba768d8a676d3dc8745a6f6d941f1d7bedf
MD5: 6b19e88e29544580deae86b652fd7ede
Can’t you just go back in commit history and try an older umd file? Personally I just took the MRCS from the PlayCanvas project and cleaned it up a bit, for 100% compatibility (and I can’t be bothered to register etc.).
But yes, someone with access to the Microsoft repo should be able to link the used commit id.
