Check your profiler and specifically the shader compilation time. I’m pretty sure what’s happening is the moment you move your camera to that part of your scene, the rendered for the first time materials compile their shaders.
That’s a common issue with shader compilation, there are a number of topics around on how to work around it. A common approach is outlined here: