Hi, i need a working example of a video being chromakeyed to transparency (from green screen video). While i found a handful of examples, none of them work. Not on mac, not on windows. Planes just stay black. May be they are to old.
Anybody having a working example?
i solved it myself. for anybody interested here is what you need to do:
create a scene with one entity you want the image to apear on. Typically this is a plane.
create a shader file. Name it “veChromakey”
and paste this code inside the file:
attribute vec3 aPosition;
attribute vec2 aUv0;
uniform mat4 matrix_model;
uniform mat4 matrix_viewProjection;
varying vec2 vUv0;
void main(void)
{
vUv0 = aUv0;
gl_Position = matrix_viewProjection * matrix_model * vec4(aPosition, 1.0);
}
Thats the vertex shader. Nothing special.
varying vec2 vUv0;
uniform sampler2D uDiffuseMap;
uniform float uTime;
void main(void)
{
vec3 tColor = texture2D(uDiffuseMap, vUv0).rgb;
float a = (length(tColor - vec3(0.1,0.9,0.2)) - 0.5) * 7.0;
gl_FragColor = vec4(tColor, a);
}
the color value to kay out is hardened here, because i could not manage to do this with attributes/parameters.
Lastly create a script file and paste this:
var ChromaKey = pc.createScript('chromaKey');
ChromaKey.attributes.add('vs', {
type: 'asset',
assetType: 'shader',
title: 'Vertex Shader'
});
ChromaKey.attributes.add('fs', {
type: 'asset',
assetType: 'shader',
title: 'Fragment Shader'
});
ChromaKey.attributes.add('diffuseMap', {
type: 'asset',
assetType: 'texture',
title: 'Diffuse Map'
});
// initialize code called once per entity
ChromaKey.prototype.initialize = function() {
var app = this.app;
var model = this.entity.model.model;
var gd = app.graphicsDevice;
var diffuseTexture = this.diffuseMap.resource;
var vertexShader = this.vs.resource;
var fragmentShader = "precision " + gd.precision + " float;\n";
fragmentShader = fragmentShader + this.fs.resource;
// A shader definition used to create a new shader.
var shaderDefinition = {
attributes: {
aPosition: pc.SEMANTIC_POSITION,
aUv0: pc.SEMANTIC_TEXCOORD0
},
vshader: vertexShader,
fshader: fragmentShader
};
// Create the shader from the definition
this.shader = new pc.Shader(gd, shaderDefinition);
// Create a new material and set the shader
this.material = new pc.Material();
this.material.setShader(this.shader);
// Set the initial time parameter
this.material.setParameter('uTime', 0);
// Set the diffuse texture
this.material.setParameter('uDiffuseMap', diffuseTexture);
this.material.blendType = pc.BLEND_NORMAL;
// Replace the material on the model with our new material
model.meshInstances[0].material = this.material;
};
// update code called every frame
ChromaKey.prototype.update = function(dt) {
};
Now attach this script to your object you wish to have a chromakey texture on. Then populate the slots with the vertex and fragment shader. lastly you should have a dummy green screen texture. Means an image. Green background - which will be keyed out - and then some other colours in there you wish to see.
And the main trick to get this to work is this line:
this.material.blendType = pc.BLEND_NORMAL;
Hope it helps.
This has way better quality in keying than any example i could find for play canvas. By the way, it is just a port of a shader from A-Frame.
and it also works with video but then you need to rewrite the script with the video part
Thanks for posting the solution! Much appreciated!
Hi sorry for the newbie question
i keep getting this error, could it be because i done something wrong with the script
Hi @Eugene_Ooi,
What error? Can you post the project that is giving you this error?
Hi Sorry,
mayb i should be more specific, is there a way to key an iframe video ?
Im using the code from marjanp but “Cannot read properties of undefined (reading ‘model’)” this is the error
I don’t think so, you need access to the raw stream (e.g. MP4) and follow the code above to update your texture per frame.
That way the chromakey shader can work.
so there is no way for me to key an iframe video away? erm ok, meaning i can only key video textures like an MP4 right ? erm is there a way for me to input a webcam & key it ?
Yes, I think you can. I’ve made this example in the past that projects the webcam video feed on a texture.
You can use it as a starting point.
https://playcanvas.com/project/698371/overview/webcam-video-texture
Another question ya, sorry im extremely new to this, so once i have attached the script from your project then i attached the script from marjanp ?
Hi,
I have two questions about this chromakey shader.
When using this code I kept getting the same error:
> Cannot read properties of undefined (reading 'model')
After a while I got it working. I saw that my plane did not have a model component with a material attached. It does have a render component though. (I saw somewhere in another thread that model is now replaced by render.)
So then I changed these lines:
var model = this.entity.model;
// Replace the material on the model with our new material
model.model.meshInstances[0].material = this.material;
to:
var render = this.entity.render;
// Replace the material on the model with our new material
render.meshInstances[0].material = this.material;
This seems to be working as I can see my transparent video.
However, it won’t play anymore. It seems to be stuck on the first frame.
How do I make it play?
My second question is; why will this video clip other images and how do I prevent this?
Here’s my current project:
https://playcanvas.com/project/871003/
Thanks in advance!
Edit: The keyed “video” is not the video. It seems to be the dummy diffuse map picture.
Hi @Ramon,
For the clipping issue I think a quick fix is to discard all pixels that are close to a transparent threshold like this (frChromakey):
varying vec2 vUv0;
uniform sampler2D uDiffuseMap;
uniform float uTime;
void main(void)
{
vec3 tColor = texture2D(uDiffuseMap, vUv0).rgb;
float a = (length(tColor - vec3(0.1,0.9,0.2)) - 0.5) * 7.0;
if(a <= 0.00000001) discard;
gl_FragColor = vec4(tColor, a);
}
For your video not playing issue, I am looking at your project right now.
I see you are using a combination of a chromakey script together with the original video texture script from the example? They will have to be updated to work together.
You can take a look at my project here, I’ve updated the chromakey shader and it correctly autoplays the video:
https://playcanvas.com/project/523575/overview/video-chromakeying
Hi Leonidas,
First of all, thank you for looking at my project.
Your first response fixed the clipping issue.
In your second response I see that you’re not using the frChromakey and veChromakey shaders from this thread so I disabled the Chromakey script. I copied your VideoTexture script and applied it to the Root entity. Also, I copied your Chromakey shader. Then I set up VideoTexture and added the Material, Shader and videoclip.
Now the video is playing. However, it is still not keying out the green. I am not sure what I’m doing wrong here.
Could you look at my project again? Many thanks!
Hey, you are almost there. The only issue is your material is ignoring the shader chunk that is overridden (opacity), it’s not included because of the default settings.
Just enable opacity by setting a blend mode (e.g. Alpha) and it will work:
Hi guys, the effect is great, I need some help, i am trying use my camera for made it in realtime but i can’t get it. The texture from camera is rendered but the glsl shader likes not work. Some ideas? Thanks in advance
My code:
var cameraChroma = pc.createScript('cameraChroma');
cameraChroma.attributes.add('screenMaterial', {
title: 'Screen Material',
description: 'The screen material of the TV that displays the video texture.',
type: 'asset',
assetType: 'material'
});
cameraChroma.attributes.add('chromaKeyShader', {
title: 'Chroma Key Shader',
description: 'The shader for chroma keying.',
type: 'asset',
assetType: 'shader'
});
cameraChroma.prototype.initialize = function () {
var app = this.app;
var self = this;
var canvas = app.graphicsDevice.canvas;
var video = document.createElement('video');
video.width = canvas.width;
video.height = canvas.height;
video.autoplay = true;
video.playsInline = true;
this.videoTexture = new pc.Texture(app.graphicsDevice, {
format: pc.PIXELFORMAT_R8_G8_B8,
minFilter: pc.FILTER_LINEAR_MIPMAP_LINEAR,
magFilter: pc.FILTER_LINEAR,
addressU: pc.ADDRESS_CLAMP_TO_EDGE,
addressV: pc.ADDRESS_CLAMP_TO_EDGE,
mipmaps: true
});
this.videoTexture.setSource(video);
// Get the material that we want to play the video on and assign the new video
// texture to it.
var material = this.screenMaterial.resource;
material.emissiveMap = this.videoTexture;
material.opacityMap = this.videoTexture;
material.chunks.opacityTexPS = this.chromaKeyShader.resource;
material.update();
// Set up the camera stream and attach it to the video element.
navigator.mediaDevices.getUserMedia({ video: true, audio: false }).then(function (stream) {
video.srcObject = stream;
}).catch(function (err) {
console.error('Camera Error:', err);
});
};
cameraChroma.prototype.update = function () {
// Upload the video texture every frame.
this.videoTexture.upload();
};
You will have to be more specific with your issue. A reproducible project would help a lot too. Maybe the tolerance for ‘green’ is too small to work with a webcam?