Camera Rotation Issue with User Physical Movement in VR Project

In my VR project, I’ve set up camera movement and rotation based on user input by adjusting the Camera Offset’s position and rotation. It works fine until the user physically moves in the real world and then tries to rotate. Since the Camera Offset stays at (0,0,0), the rotation anchor point doesn’t match the user’s actual position.

How can I fix this? I understand that the camera position and rotation are determined by the user’s device, so I can’t directly set the camera rotation through code. Any suggestions?