Cloning the project solved this.
One time, the app crashed while switching to VR mode. From that point on, the problem started occurring.
I think the Pixel phone just marked my specific app as "cardboard only" from that point on.
As I said, simply cloning the project fixed the issue.