I’ve been stumped by this issue for a couple of weeks now and I’m hoping someone has some ideas. In short, ~10% of players will encounter an issue where numerous batch groups on the world layer aren’t being rendered.
The game is an open world driving game it has been soft launched on poki.com for a few weeks. You can play it here. It should look like this:
We can see that the UI layers are rendering correctly, however many batches on the World layer are not visible. The green directional arrows are on the same World layer but are assigned to their own batch group and they are visible, strangely. I have 37 dynamic batch groups with maxAABBs of 1,000. I wrote an editor script that automatically iterates through all render components and assigns a unique batch to every render component with a unique material set. My current suspicion is that the invisible batch groups are hitting some kind of vertice limit and simply not rendering.
I’m fairly sure that there is no console error thrown when this issue occurs. Looking at the aggregated console errors across all players, none of them occur frequently enough to be associated with this render issue.
I can confirm the model assets are definitely loading correctly. We know this because the minimap in the corner is rendering correctly and it uses the same map asset that should be rendered to the World layer. Additionally, the vehicle, pedestrians and buildings are all loaded via separate render assets, it’s highly unlikely they would all be failing to load simultaneously this consistently.
PlayCanvas is supposed produce multiple batches so that the batching rules are maintained. I understand that the max number of mesh instances per dynamic batch is hardware dependant and PlayCanvas should simply create more batch groups as needed. This issue appears to be mostly happening on Windows 10 devices, Chrome and Edge browsers. I haven’t yet seen any mobile users report this issue.
I have not been able to recreate this issue locally. If anyone from the PlayCanvas community has any ideas around what might be causing this, or if you have any ideas of things I can try to recreate this bug locally, please let me know.
Update: Over the last two weeks, I’ve deployed a number of changes which I hoped would fix it, but unfortunately this bug is still occurring and I still can’t recreate it locally. 10% of all the game’s feedback submitted through Poki is complaining about this bug.
Here is what I’ve tried:
Added a delayed batcher.generate() call after the game finishes loading. This had no effect.
Setting all the Batch Groups’ Max AABB values to 75 on one attempt and 10,000 on another attempt. Neither value affected the frequency of this bug.
Refactored the Layer settings to the PlayCanvas defaults. The only effect this had was hiding the minimap. Previously the minimap render and camera were on a different layer and were still visible whenever this bug occurred (see original screenshots). After moving them to the World layer, they are no longer rendered. It’s now just the blank HUD element. (Bottom left of this screenshot).
Set every other green arrow to a new test batch group. Now only the arrows in the original Arrows batch group are showing, every render entity added to the test batch group is invisible. Not sure if this is significant, but the original Arrows batch group appears first in the list of batch groups and has one of the lowest numbered groupIds. All of the invisible groups have higher numbered batchIds.
Added a series of test render entities underneath the map. I gave them different materials and different batch groups, some with no batch group at all. My idea was that when this bug occurs, some of them may still be visible and I could figure out which material/batch settings were on the visible ones (similar to the green arrows). Unfortunately, whenever this bug occurs, all the test render entities are invisible too.
I’ve noticed this bug disproportionately affects players in these countries in order:
Venezuela
Mexico
Vietnam
Brazil
Argentina
Almost no one from the US or Canada has reported this bug. The obvious common denominator here is poor internet quality, which I can back up with this list. It’s possible this bug is caused when an asset fails to load, or loads very slowly. Here is what I tried:
Blocking the request URL of certain assets in Chrome Dev tools. When I do this, the game just gets stuck at the loading screen.
Building the PlayCanvas engine locally and placing random setTimeout() delays on asset requests. This slows down the loading speed, but the game eventually starts normally.
Used a tool called Clumsy to simulate poor network quality on my local machine. I opened the game across dozens of browser tabs using a variety of different low quality network settings. While there were many connection timeouts, the game loaded and started correctly most of the time. After hundreds of tests on multiple machines, this bug didn’t occur once.
I’m going through the PlayCanvas engine source code and adding setTimeout()s, logging and commenting out certain functions to see what happens. My goal is to recreate this bug locally and work backwards from there.
I’m going to keep this thread updated as I learn more. But if any PlayCanvas devs have some suggestions of more things I can try, please let me know.
Its like reading a detective story Fascinating and eagerly waiting for the next chapter!
Sorry to hear about your troubles, though. If I get some ideas to try, I will definitely let you know.
When you say that you call batch generate after app has loaded, what do you mean specifically? How do you identify that the app has loaded? The thought here is that if you call it while some textures are still on the way, then it won’t update the related batch groups.
Thanks for reading my mystery novel! It’s certainly becoming quite the investigation.
I set up my own callback which was fired after the player’s car has spawned and the game is waiting for their input. By then all the assets should have loaded. In fact, the callback would never be reached if certain assets like the vehicle or the map failed to load. I deliberately placed it as late as I possibly could hoping it would force the batches to update. Unfortunately it had no effect.
Right, so loading it later doesn’t guarantee it, unfrotunately. What I would probably try is using an asset list loader to trigger the batch reset. If you are not familiar with AssetListLoader, you can check this example for instance:
It is a helper that would guarantee some code execution would take place only after all the listed assets are loaded.
On the other hand, I think if the batch group gets reset before the texture is loaded, it should be a default standard material, not being empty like that. Not sure.
This project was created using the editor, not an engine-only so it’ll take me some time to add the AssetListLoader. I’ve also been digging through a locally-built copy of the engine and adding debugging code. I’m looking for how the project is currently loading the list of assets because the AssetListLoader isn’t being constructed at all right now.
I did some quick and dirty asset request failure testing last week by simply blocking specific texture urls using Chrome Dev Tools. The game still loads and runs, just with the default standard material, as you said. I could not recreate this bug by simulating an asset load failure.
Update: Disabling realtime shadows has fixed it! I wish I had a more technical explanation as to why this worked, but I was never able to recreate this bug locally.
So in the unlikely event anyone has a project with this same issue, turning off realtime shadows should be the first thing to try!
Its strange that you don’t get any error logs in that case, though. @mvaligursky any idea how a real time shadow is related to missing meshes on screen on low-end devices. Is it only a debug version that can log it?
On the Poki platform, error logs aren’t currently captured when players submit bug reports. The Poki team is working on adding this soon, however. I only have the aggregated console errors across all players and I wasn’t able to match any errors to this bug. It’s also possible console errors are being thrown but they’re not being captured for some reason.
I’ll update this thread if more information becomes available, but for now I’m just glad we have a solution.
10% of all the game’s feedback submitted through Poki is complaining about this bug.
Fascinating and eagerly waiting for the next chapter!