Here’s the AI analysis, and it seems spot on to me:
Root Cause: Depth Prepass Lacks Stencil Buffer
When CameraFrame is used with SSAO (or TAA/DOF), a depth prepass is automatically enabled. This prepass renders depth to a separate texture that SSAO reads from, but its render target has no stencil buffer, so stencil masking is completely ignored.
Here’s the chain of events:
1. SSAO forces prepass on:
frame-pass-camera-frame.jsLines 170-178
sanitizeOptions(options) {
options = Object.assign({}, _defaultOptions, options);
// automatically enable prepass when required internally
if (options.taaEnabled || options.ssaoType !== SSAOTYPE_NONE || options.dofEnabled) {
options.prepassEnabled = true;
}
return options;
}
2. Prepass render target has depth but NO stencil:
render-pass-prepass.jsLines 74-83
const renderTarget = new RenderTarget({
name: ‘PrepassRT’,
colorBuffer: this.linearDepthTexture,
// use depth buffer, but this can be discarded after the prepass as the depth is stored in the linearDepthTexture
depth: true,
// always single sampled
samples: 1
});
No stencil: true is passed, so RenderTarget defaults it to false.
3. Prepass renders ALL mesh instances with depthWrite:
render-pass-prepass.jsLines 120-131
for (let j = 0; j < meshInstances.length; j++) {
const meshInstance = meshInstances[j];
// only collect meshes that update the depth
if (meshInstance.material?.depthWrite) {
tempMeshInstances.push(meshInstance);
}
}
renderer.renderForwardLayer(camera, renderTarget, null, undefined, SHADER_PREPASS, this.viewBindGroups, {
meshInstances: tempMeshInstances
});
The “inside of the hole” mesh gets rendered here because it has depthWrite enabled. Since there’s no stencil buffer on this render target, the stencil test can never reject fragments. The depth is written for all fragments unconditionally (with respect to stencil).
4. SSAO reads from this prepass depth, not the main depth buffer:
render-pass-prepass.jsLines 91-94
after() {
// Assign the linear depth texture to the uniform
this.device.scope.resolve(DEPTH_UNIFORM_NAME).setValue(this.linearDepthTexture);
}
Summary
-
The main forward pass correctly applies stencil masking – colors and depth to the main render target are properly masked. The user’s stencil setup is working as intended there.
-
But the prepass runs before the main pass, into a separate framebuffer without stencil, so the “inside of the hole” writes its depth into the linear depth texture unconditionally.
-
SSAO (and any other effect using uSceneDepthMap) reads from this prepass linear depth texture, so it sees the unmasked depth of the hole interior.
Workaround for the User
The simplest workaround: set depthWrite: false on the “inside of the hole” material. Since it’s rendered only where stencil passes (inside the lid mask), it likely doesn’t need to contribute to the depth buffer. The prepass only includes meshes where material.depthWrite is true (line 124), so this would exclude it entirely.
Potential Engine Fix
To properly fix this, the prepass would need stencil support – either by adding a stencil attachment to PrepassRT and replicating the stencil-writing passes, or by providing a way for users to exclude specific mesh instances from the prepass. Both are non-trivial changes.
