When will browsers kill the page of playcanvas application?

[scarlex]

Some of our playcanvas applications run on mobile devices, they are often killed by iOS safari.
Here is one of them: Art Gallery - PLAYCANVAS

Reproductive steps:

1. Open the page with your iOS safari

2. Click a frame
   

   

3. Click right side button to upload image
   

   

4. Capture image by camera or select image from gallery
   

   

5. When the paint of the frame change, go to step 3.
   

   

6. Repeat about ten times, the browser will kill the page.

We have trouble with this issue, and have no idea how to solve it.
I don’t know when will the browser kill the page, and how to optimize the application to prevent it.
Can anyone give me tips ?

---------- UPDATE --------------
I test on android tablet (on Firefox) just now, it works fine, the browser do not kill the page.

[whydoidoit]

Sounds like you are probably running the system out of memory. iOS will kill things that use more than 50% of the memory on the device. Are you storing a whole load of pictures there? Putting them in the gallery? If they are the raw pictures then they will be consuming huge amounts of memory. One trick is to redraw the captured image onto a much smaller canvas and then use that. Perhaps you could upload the whole picture to a web site and then grab it again on demand if you need it to be full screen?

[scarlex]

@whydoidoit Before the image convert to texture, I will do the following steps:

1. calculate its EXIF data, rotate the canvas, make sure it is “Portrait mode”.
2. cut the image’s size to 256 x 256.

And here is the code of implementation:

RightSideBar.prototype.onFileChange = function(e) {
    var file = e.target.files[0];
    
    if (!file) return;

    var self = this;

    EXIF.getData(file, function() {
        var orientation = this.exifdata.Orientation;

        var reader = new FileReader();

        reader.onload = function(e) {
            var image = new Image();
            image.onload = function(e) {
                var canvas = document.createElement('canvas');
                var context = canvas.getContext('2d');
                var imageType = image.width >= image.height ? 'landscape' : 'portrait';
                // var clipSize = Math.max(image.width, image.height);
                var clipSize = Math.min(image.width, image.height);
                var canvasWidth = Math.min(clipSize, 256);
                var canvasHeight = Math.min(clipSize, 256);

                // var dx = imageType === 'landscape' ? 0 : (canvasWidth - image.width) / 2;
                // var dy = imageType === 'landscape' ? (canvasHeight - image.height) / 2 : 0;
                var dx = imageType === 'landscape' ? (image.width - clipSize) / 2 : 0;
                var dy = imageType === 'landscape' ? 0 : (image.height - clipSize) / 2;

                canvas.width = canvasWidth;
                canvas.height = canvasHeight;

                switch (orientation) {
                    case 2:
                        // horizontal flip
                        context.translate(canvas.width, 0);
                        context.scale(-1, 1);
                        break;
                    case 3:
                        // 180° rotate left
                        context.translate(canvas.width, canvas.height);
                        context.rotate(Math.PI);
                        break;
                    case 4:
                        // vertical flip
                        context.translate(0, canvas.height);
                        context.scale(1, -1);
                        break;
                    case 5:
                        // vertical flip + 90 rotate right
                        context.rotate(0.5 * Math.PI);
                        context.scale(1, -1);
                        break;
                    case 6:
                        // 90° rotate right
                        context.rotate(0.5 * Math.PI);
                        context.translate(0, -canvas.height);
                        break;
                    case 7:
                        // horizontal flip + 90 rotate right
                        context.rotate(0.5 * Math.PI);
                        context.translate(canvas.width, -canvas.height);
                        context.scale(-1, 1);
                        break;
                    case 8:
                        // 90° rotate left
                        context.rotate(-0.5 * Math.PI);
                        context.translate(-canvas.width, 0);
                        break;
                }

                //context.drawImage(image, imageStart[0], imageStart[1], image.width, image.height);
                context.drawImage(image, dx, dy, clipSize, clipSize, 0, 0, canvas.width, canvas.height);
                
                var texture = new pc.Texture(self.app.graphicsDevice);
                texture.setSource(canvas);
                self.updatePaint(texture);
            };
            image.src = reader.result;
        };

        reader.readAsDataURL(file);

    });
};

[scarlex]

One more question, will the browser kill the page when vram usage out of amounts?
If it will, what’s the amounts of vram usage ?

[whydoidoit]

So here’s the nightmare about iOS and VRAM…

You use both main memory and VRAM for images. It’s double (often triple) counted (but only one of the tools ever shows this, you can think everything is fine and then “BAM” you get killed for out of memory).

Also for the sake of clarity doesn’t matter about the image file size, in memory for display it is 12 x number of pixels in the image no matter what (unless you use graphics card compression - like “Compress” in the PlayCanvas settings).

Firstly in your example code, I’d start by only having one canvas, not making one every time (although garbage collection should fix that, it’s better practise anyway). Make a global canvas and keep reusing it - you’ll thank me for that when you get smoother movement because of reduced garbage collection.

Next I’ve got to say, this kind of thing is something that “seems” easy - but this kind of graphics manipulation is where the thing falls apart because images use loads of memory. I’d be saying only have a limited number of textures that you use, make a “rolling buffer” of textures let’s say 8.

If you follow this approach you can actually save the images as JPGs when they aren’t being displayed (using a tiny amount of memory compared - even if you base64 encode them for ease). So you put all of your images into strings (perhaps if your gallery gets big enough you make a buffer of these too and stick them on the web somewhere).

With all of your images in strings you work out as the camera moves which ones you need to display, you then set the textures of the paintings on display to the JPG and keep on rolling. Clearly there’s some smart code to be written there to not bother updating things that are already right (as they will be most of the time) and to choose the right “spare” texture to change to be the new image when another item comes into view.

Really I’ve been there and had EXACTLY this kind of problem before (although that was on Unity, it has the same issues)

[whydoidoit]

BTW it will eventually die on Android as well. To make this fully scalable you need to write smart image/memory management. Just iOS will die earlier.

[scarlex]

First, thanks for your tips!

I am not familiar with the optimization, and here are my questions:

Firstly in your example code, I’d start by only having one canvas, not making one every time (although garbage collection should fix that, it’s better practise anyway). Make a global canvas and keep reusing it - you’ll thank me for that when you get smoother movement because of reduced garbage collection.

Texture.setSource can take canvas element as its parameter, if I use a global canvas element, every new texture will reference it, when the canvas’s image change, shouldn’t it affect other textures? I am not sure, correct me if I am wrong.

Next I’ve got to say, this kind of thing is something that “seems” easy - but this kind of graphics manipulation is where the thing falls apart because images use loads of memory. I’d be saying only have a limited number of textures that you use, make a “rolling buffer” of textures let’s say 8.

I don’t know your meaning of “limited number of textures”, what’s that? My gallery has many textures, the amounts should more than 8. And I don’t know what is “rolling buffer” too…

If you follow this approach you can actually save the images as JPGs when they aren’t being displayed (using a tiny amount of memory compared - even if you base64 encode them for ease). So you put all of your images into strings (perhaps if your gallery gets big enough you make a buffer of these too and stick them on the web somewhere).

The image is uploaded (just a local file, not upload to the server) by the user. It seems that the iOS will save image as PNG format. How can I implement your saying “put all of your images into strings” ?

With all of your images in strings you work out as the camera moves which ones you need to display, you then set the textures of the paintings on display to the JPG and keep on rolling. Clearly there’s some smart code to be written there to not bother updating things that are already right (as they will be most of the time) and to choose the right “spare” texture to change to be the new image when another item comes into view.

Do you mean I need to calculate which texture should be displayed ? I think this job should be done in the engine…

Really I’ve been there and had EXACTLY this kind of problem before (although that was on Unity, it has the same issues)

Yes, my parter also occur this issue on Unity too…

[whydoidoit]

So firstly I’m just a PlayCanvas user not related to the business

From looking at the engine code it would appear that the texture is uploaded from the canvas and then not automatically modified from it unless you call upload() again. So changing the canvas contents will not cause an issue for other textures.

The thing here is that you are making an engine to create a gallery - so yeah it does need to be handled in the engine, sadly it’s your engine that has to do that! PlayCanvas is just providing you with a set of things on which you can draw textures etc. If you need to be able to handle many textures then it’s up to you to deal with the fact that this will run the phone out of memory.

My suggestion is that you can have many images in the gallery and that you will store these images in an array of strings as their JPG representation. You will have a limited number of images that can be seen at any time, you will convert these images from the JPG to a Texture for PlayCanvas. As you move around the scene you will have new images becoming visible and others becoming invisible - you will write code to manage the textures.

So you create lets say 8 textures. This will mean that the maximum number of pictures that can be seen at the same time is 8.

In an update() function you will work out which gallery pics can be seen (they are in the camera frustum and oriented towards the camera).

So you can get the frustum from the current camera and then you could iterate through the paintings and check a bounding sphere to see if it’s near viewable.

See this for a bit of help with frustums https://developer.playcanvas.com/en/api/pc.Frustum.html

There are lots of ways you could then manage the textures and images to ensure you allocate visible images to textures and allocate those textures to the right planes. I could go further with this but it would take some time to do that.

Basically I’m saying that there is quite a lot of work to do to create a non-naive implementation of a gallery. The way you are doing it is fine for a demo prototype but to allow more pictures you are going to have to bite the bullet and handle the fact your images are big when they are in a texture. So you’ve got to keep them out of the textures when they aren’t visible and store them as JPGs.

This is really the meat of developments like this!!

By the way the way to store them as strings is to draw them to the canvas and then use the canvas function to encode them as a base64 version.

https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/toDataURL

Make sure you pass ‘image/jpeg’ as the type as PNGs are a lot bigger.,

[scarlex]

Thanks for your guide. I will try to implement it by myself.

[cameraMan]

A little addition to @whydoidoit’s complete reference:
You can also:
Null uploaded Images after usage. This will make them available to browser’s Garbage Collection and will free up memory
and resize uploaded images in the server, before make them available to browser.

[scarlex]

Null uploaded Images after usage.

What do you mean? Any code snippets?

[cameraMan]

Here you are:

var uploaded_image = new Image();
// load it, manipulate it and then:
uploaded_image = null;

[scarlex]

Oh, thank you!

[scarlex]

@whydoidoit
I see this post just now. It says ammo.js perform badly on mobile. My gallery also use ammo.js to detect collision (just for testing which frame the use click by raycast), I am not sure app crash is related to ammo.js too …

I also notice your library of Collision Detection ! It seems great ! But I can’t find examples about selecting entity by raycast…

Could you please give me some examples? Thanks !

[yaustar]

@scarlex

I am not sure app crash is related to ammo.js too

It’s unlikely as ammo.js performance issues are more about processing on the CPU rather than memory.

If you are just doing raycasting with AABB, OBBs and/or spheres, you can use the shapes that are included in the engine. https://developer.playcanvas.com/en/tutorials/entity-picking-without-physics/

More example projects can be found here: https://developer.playcanvas.com/en/tutorials/?tags=raycast

[scarlex]

It’s unlikely as ammo.js performance issues are more about processing on the CPU rather than memory.

Oh, I see.

If you are just doing raycasting with AABB, OBBs and/or spheres, you can use the shapes that are included in the engine. https://developer.playcanvas.com/en/tutorials/entity-picking-without-physics/

I know this way to select entities !

As the result base on the loop of entities array, if the raycast intersects multiple entities, how can I get the nearest one to the camera (if multiple cameras support)?

[yaustar]

@scarlex

As the result base on the loop of entities array, if the raycast intersects multiple entities, how can I get the nearest one to the camera (if multiple cameras support)?

I did this in WebVR Labs where I just kept track of which entity had the closest intersection point to the origin of the ray. PlayCanvas | HTML5 Game Engine (look at the raycast function)