So problem here is that your big box behind has blending on it, so it enables depth sorting for it. As avatars have sorting as well - they will both compete in order of drawing.
Order is defined by z-value from camera frustum of AABB of an entity.
So if you rotate camera, centre of box will be closer to camera near plane than z value of an avatar. That will lead to different rendering order.
Ways to go around - if you don’t really need a blending on box, disable it. Then it will be solid.
If you do need blending, then consider splitting it into smaller boxes, so that X and Y would be on same axes as avatars, that way it wont flicker on extreme angles.