Hi,
I used the PlayCanvas video texture script to play some videos. Since I have to switch between videos during runtime, I modified it a bit. I simultaneously start video playback with some 3D animations and a separate audio file. The audio file and the 3D animation are in sync, but it sometimes happens that the video is out of sync or strangely stops while playing. Since the video is also a main part of the experience, it would be important that it works in sync with the other elements. I don’t know if I can make any adjustments to fix the problem.
basically the setup is from the excample. In the console i also receive “webgl-texture.js:656 WebGL: INVALID_VALUE: texImage2D: no video” but the video source is set…
furthermore a question: because im on mobile, the fps can variate from device to device - could this also cause the problem, that the video isnt synchron? Can this somehow be fixed or is this just a limitation, because its hardware related?
the code im using for the video texture:
var VideoTexture2 = pc.createScript('videoTexture2');
// This video texture is compatible with stencil buffer - so i this for video rendering
VideoTexture2.attributes.add('controllerEntity', { type: 'entity' , description: 'Controller Entity'});
VideoTexture2.attributes.add('video', {
title: 'Video',
description: 'MP4 video asset to play back on this video texture.',
type: 'asset'
});
VideoTexture2.attributes.add('materialAsset', {type: 'asset', assetType: 'material'});
var video;
// initialize code called once per entity
VideoTexture2.prototype.initialize = function() {
var app = this.app;
// Create HTML Video Element to play the video
video = document.createElement('video');
//video.loop = true;
// muted attribute is required for videos to autoplay
video.muted = true;
// critical for iOS or the video won't initially play, and will go fullscreen when playing
video.playsInline = true;
// needed because the video is being hosted on a different server url
video.crossOrigin = "anonymous";
// autoplay the video
video.autoplay = true;
// iOS video texture playback requires that you add the video to the DOMParser
// with at least 1x1 as the video's dimensions
var style = video.style;
style.width = '1px';
style.height = '1px';
style.position = 'absolute';
style.opacity = '0';
style.zIndex = '-1000';
style.pointerEvents = 'none';
document.body.appendChild(video);
// Create a texture to hold the video frame data
this.videoTexture = new pc.Texture(app.graphicsDevice, {
format: pc.PIXELFORMAT_R8_G8_B8,
minFilter: pc.FILTER_LINEAR_MIPMAP_LINEAR,
magFilter: pc.FILTER_LINEAR,
addressU: pc.ADDRESS_CLAMP_TO_EDGE,
addressV: pc.ADDRESS_CLAMP_TO_EDGE,
mipmaps: true
});
this.videoTexture.setSource(video);
this.materialAsset.resource.emissiveMap = this.videoTexture;
video.addEventListener('canplaythrough', function (e) {
video.play();
}.bind(this));
/*
// Check if the video has ended
video.addEventListener('ended', function() {
this.onVideoEnded();
console.log("videoEnded");
}.bind(this));
*/
// set video source
video.src = this.video ? this.video.getFileUrl() : this.videoUrl;
document.body.appendChild(video);
video.load();
};
VideoTexture2.prototype.onVideoEnded = function() {
this.controllerEntity.script.gameController.changeVideo(0); // set idle animation if video ended
};
// Function to set the video source and loop attribute
VideoTexture2.prototype.setVideoSource = function(source, loop, _domVideoIdReference) {
// make sure we have at last a source for fallback
if (source) {
// try to get existing video element (loaded already in index.html)
if (document.getElementById(_domVideoIdReference) != null) {
video = document.getElementById(_domVideoIdReference);
this.videoTexture.setSource(video);
}else // play fallback video file from pc assets
video.src = source;
}
if (typeof loop === 'boolean') {
video.loop = loop;
}
if(video)
{
video.currentTime = 0;
video.play();
}
};
// Function to toggle video playback
VideoTexture2.prototype.togglePlayback = function() {
if (video.paused) {
video.play();
} else {
video.pause();
}
};
// update code called every frame
VideoTexture2.prototype.update = function(dt) {
// Transfer the latest video frame to the video texture
if (video.readyState === video.HAVE_ENOUGH_DATA) {
// Transfer the latest video frame to the video texture
this.videoTexture.upload();
}
};
The “setVideoSource” Function receives the video asset (fallback), if its looped and the id of the video source i preloaded in a custom loader to avoid preloading / buffering in the experience. The reason for this is: in the experience the videos are used at the very beginning and have to be “ready”. Furthermore I assume that the “canplaythrough” eventlistener in the initialization handles that the video is also ready.
I would be happy about every suggestion - or ideas for optimizing (but like i said the init is basically the playcanvas template). Right now i could not avoid, that the video is sometimes async or weirdly interrupts after a few seconds (the videos are 20 secs long and are around 8mb) (maybe its buffering? but then the canplaythrough doesnt work correct i guess).
Thank you very much!