I’ve noticed some very strange FPS effects when I try to use a video texture. Basically, the FPS starts to go all over the place (FPS calculated with 1/dt). I’ve seen it range from 40 all the way up into the 120s.
It seems to only be happening in Google Chrome.
I’ve created a simple scene for debugging:
https://playcanvas.com/project/477566/overview/testing-video-textures
Here’s the published url for that project:
Here’s a screenshot of what I get:
Any ideas why this could be the case?
I just recorded a trace in the Timeline tab of Dev Tools:
The thing to notice is:
23.45ms for WebGL to extract the pixel data of the current video frame and copy it into the texture. As far as I’m aware, the only way to reduce this cost is to reduce the resolution of the video. That video is 1280x720. Consider making it quarter resolution.
Also, I notice that if you switch the format of the texture that you stream the video to to pc.PIXELFORMAT_R8_G8_B8, the cost of texImage2D falls dramatically. This will use twice the VRAM as pc.PIXELFORMAT_R5_G6_B5, but presumably, pixel conversions don’t have to be performed on the CPU (because the video pixel data is already 888).
However, notice that it’s still running at 30FPS (on my machine). The reason seems to be because the GPU is still having to do a lot of work. Investigating…
Ah. I am driving a 4K monitor from my laptop and any WebGL content is locked to 30FPS. So yeah, looks like the things to try are:
pc.PIXELFORMAT_R8_G8_B8
Thanks so much for looking into that @will. I’ll give those suggestions a try! I really appreciate it!
I’m still totally confused as to why the FPS was going up (and delta time was presumably going down) for me when using the video texture. A consistent drop in framerate I can understand–but any idea why I could start to see numbers like 100+ fps? If I disable the video texture I get a consistent 60fps, give or take a frame.
Also important to note–when I was getting those super high reported fps numbers it appeared to the eye that the web page fps was lower than usual; probably 30fps or so. It makes me wonder if something strange was going on with dt in general.
(Also, I tried on two different computers and saw FPS ranging from super high to super low on both computers.)
Okay; that profiler is awesome. I’d somehow missed it before!
Anyways, my results were as follows: there seems to be no direct correlation between the 1/dt fps calculation and the profiler fps. I also included chrome’s native FPS counter in this screenshot. (I also opened two tabs of the playcanvas project–shown in the screenshot–to help push down the profiler fps and accentuate the fact that the two fps numbers are so drastically different.)
So, here are the numbers:
Profiler: 35 fps
Chrome native: 34.4 fps
1/dt method: 80.69 fps
Here’s the full fps calculation code I’m using (credit goes to whoever wrote it for the WebVR lab project!)
// Calculate frames per second
if (dt) {
var fps = 1/dt;
this.array.push(fps);
}
var sum = 0;
for(var i = 0; i < this.array.length; i++) {
sum += this.array[i];
}
if (this.array.length) {
sum /= this.array.length;
}
if (this.array.length > 100) {
this.array.shift();
}
var fps = sum.toFixed(2);Just tested, and when using PIXELFORMAT_R8_G8_B8 I seem to get predictable/consistent framerates in Chrome. So that’s great!
But still–it seems like there was something wacky going on with delta time when using PIXELFORMAT_R5_G6_B5 (maybe revealed by the heavy CPU load–and will maybe return in other CPU heavy situations).
Your code was only uploading the video every other frame. Meaning your dt was probably alternating between 0.016 and 0.032 or even 0.048. The spiky nature of your dt was maybe causing problems. But ultimately, I think it’s in the method used for calculation, because the PlayCanvas Profiler frame rate is fine and accurate. Seems to be calculated here:
Ohh okay gotcha–so the engine literally counts frames in one second intervals. Definitely seems the most accurate.
Thanks for all the help and info! Purely for sake of curiosity I’m going to do some dt logging and see how it’s coming up with the numbers I’m getting. It seems strange that it could average to such a high number.
Soo, I did some investigating and found out that I’m getting some super low dts on some frames. On on particular FPS calculation that showed an fps of 100.48 fps, these were the 100 dts (and individual fps calculations) that resulted in that high fps:
// Yielded an average fps of 100.48
[
{ "dt":"0.009005", "1/dt":"111.05" },
{ "dt":"0.037000", "1/dt":"27.03" },
{ "dt":"0.006000", "1/dt":"166.67" },
{ "dt":"0.052000", "1/dt":"19.23" },
{ "dt":"0.013000", "1/dt":"76.92" },
{ "dt":"0.031000", "1/dt":"32.26" },
{ "dt":"0.010995", "1/dt":"90.95" },
{ "dt":"0.030000", "1/dt":"33.33" },
{ "dt":"0.012000", "1/dt":"83.33" },
{ "dt":"0.033005", "1/dt":"30.30" },
{ "dt":"0.017995", "1/dt":"55.57" },
{ "dt":"0.027000", "1/dt":"37.04" },
{ "dt":"0.005000", "1/dt":"200.00" },
{ "dt":"0.036005", "1/dt":"27.77" },
{ "dt":"0.012995", "1/dt":"76.95" },
{ "dt":"0.034005", "1/dt":"29.41" },
{ "dt":"0.016000", "1/dt":"62.50" },
{ "dt":"0.030995", "1/dt":"32.26" },
{ "dt":"0.004005", "1/dt":"249.69" },
{ "dt":"0.029000", "1/dt":"34.48" },
{ "dt":"0.003995", "1/dt":"250.31" },
{ "dt":"0.030005", "1/dt":"33.33" },
{ "dt":"0.003000", "1/dt":"333.33" },
{ "dt":"0.029000", "1/dt":"34.48" },
{ "dt":"0.003995", "1/dt":"250.31" },
{ "dt":"0.028005", "1/dt":"35.71" },
{ "dt":"0.005995", "1/dt":"166.81" },
{ "dt":"0.030000", "1/dt":"33.33" },
{ "dt":"0.004005", "1/dt":"249.69" },
{ "dt":"0.016000", "1/dt":"62.50" },
{ "dt":"0.015000", "1/dt":"66.67" },
{ "dt":"0.030000", "1/dt":"33.33" },
{ "dt":"0.003995", "1/dt":"250.31" },
{ "dt":"0.028005", "1/dt":"35.71" },
{ "dt":"0.005995", "1/dt":"166.81" },
{ "dt":"0.028005", "1/dt":"35.71" },
{ "dt":"0.005000", "1/dt":"200.00" },
{ "dt":"0.029000", "1/dt":"34.48" },
{ "dt":"0.003995", "1/dt":"250.31" },
{ "dt":"0.029000", "1/dt":"34.48" },
{ "dt":"0.004005", "1/dt":"249.69" },
{ "dt":"0.017995", "1/dt":"55.57" },
{ "dt":"0.016005", "1/dt":"62.48" },
{ "dt":"0.035000", "1/dt":"28.57" },
{ "dt":"0.014000", "1/dt":"71.43" },
{ "dt":"0.040000", "1/dt":"25.00" },
{ "dt":"0.010995", "1/dt":"90.95" },
{ "dt":"0.042005", "1/dt":"23.81" },
{ "dt":"0.006995", "1/dt":"142.96" },
{ "dt":"0.053005", "1/dt":"18.87" },
{ "dt":"0.014000", "1/dt":"71.43" },
{ "dt":"0.057995", "1/dt":"17.24" },
{ "dt":"0.019000", "1/dt":"52.63" },
{ "dt":"0.042000", "1/dt":"23.81" },
{ "dt":"0.014005", "1/dt":"71.40" },
{ "dt":"0.061995", "1/dt":"16.13" },
{ "dt":"0.005000", "1/dt":"200.00" },
{ "dt":"0.042000", "1/dt":"23.81" },
{ "dt":"0.008005", "1/dt":"124.92" },
{ "dt":"0.029000", "1/dt":"34.48" },
{ "dt":"0.003995", "1/dt":"250.31" },
{ "dt":"0.032000", "1/dt":"31.25" },
{ "dt":"0.004005", "1/dt":"249.69" },
{ "dt":"0.046995", "1/dt":"21.28" },
{ "dt":"0.003000", "1/dt":"333.33" },
{ "dt":"0.038005", "1/dt":"26.31" },
{ "dt":"0.010000", "1/dt":"100.00" },
{ "dt":"0.027995", "1/dt":"35.72" },
{ "dt":"0.005000", "1/dt":"200.00" },
{ "dt":"0.045005", "1/dt":"22.22" },
{ "dt":"0.005000", "1/dt":"200.00" },
{ "dt":"0.027995", "1/dt":"35.72" },
{ "dt":"0.005000", "1/dt":"200.00" },
{ "dt":"0.028005", "1/dt":"35.71" },
{ "dt":"0.005995", "1/dt":"166.81" },
{ "dt":"0.033005", "1/dt":"30.30" },
{ "dt":"0.016995", "1/dt":"58.84" },
{ "dt":"0.030005", "1/dt":"33.33" },
{ "dt":"0.003995", "1/dt":"250.31" },
{ "dt":"0.024000", "1/dt":"41.67" },
{ "dt":"0.009005", "1/dt":"111.05" },
{ "dt":"0.025995", "1/dt":"38.47" },
{ "dt":"0.007000", "1/dt":"142.86" },
{ "dt":"0.032000", "1/dt":"31.25" },
{ "dt":"0.003005", "1/dt":"332.78" },
{ "dt":"0.015000", "1/dt":"66.67" },
{ "dt":"0.017000", "1/dt":"58.82" },
{ "dt":"0.025995", "1/dt":"38.47" },
{ "dt":"0.007000", "1/dt":"142.86" },
{ "dt":"0.030000", "1/dt":"33.33" },
{ "dt":"0.004005", "1/dt":"249.69" },
{ "dt":"0.029000", "1/dt":"34.48" },
{ "dt":"0.003995", "1/dt":"250.31" },
{ "dt":"0.028005", "1/dt":"35.71" },
{ "dt":"0.005000", "1/dt":"200.00" },
{ "dt":"0.031000", "1/dt":"32.26" },
{ "dt":"0.006000", "1/dt":"166.67" },
{ "dt":"0.016000", "1/dt":"62.50" },
{ "dt":"0.013000", "1/dt":"76.92" },
{ "dt":"0.032000", "1/dt":"31.25" },
{ "dt":"0.004000", "1/dt":"250.00" }
]
So, there are some frames that have oddly high/low values, such as this one for example:
{ "dt":"0.003000", "1/dt":"333.33" },
I’m mainly just posting this out of curiousity; not looking for a solution per se. 
This isn’t inhibiting my work. Is there any plausible explanation as to why a delta time could get as low as 0.003000?