3D Face Tracking in PlayCanvas using TensorFlowJS

A small demo I made while trying to learn tensorflow.js.

9 Likes

This is one of the best PlayCanvas+TFJS projects I’ve seen @Gamer_Wael! Well done, looks amazing!

1 Like

I’m having a bit of trouble with this.

I’m trying to create an entity at the center of the face to make it easier to add face masks and objects.

I’ve got it to follow the user’s head movements, yaw and roll. But it doesn’t follow pitch.
This is how I’m doing it:

//Left cheek
pos1 = cubes[205].getPosition();
//Right Cheek
pos2 = cubes[425].getPosition();

dist = pos1.distance(pos2);
dist = dist / 2;
center.setPosition(pos1);
center.lookAt(pos2);
center.translateLocal(0, 0, -dist);
center.rotateLocal(0, 90, 0)

I then tried modifying it but now it only follows yaw and pitch but not roll:

//Left Cheek
pos1 = cubes[205].getPosition();
//Right Cheek
pos2 = cubes[425].getPosition();
dist = pos1.distance(pos2);
dist = dist / 2;
center.setPosition(pos1);
center.lookAt(pos2);
center.translateLocal(0, 0, -dist);
//center.rotateLocal(0, 90, 0);

//Forehead
pos3 = cubes[200].getPosition();
//Chin
pos4 = cubes[9].getPosition();
var ver = new pc.Vec3().sub2(pos3, pos4);
var hor = new pc.Vec3().sub2(pos2, pos1);
var cross = new pc.Vec3().cross(ver, hor);
center.lookAt(cross);

From a quick look, you need a reference point so you can create a local up vector. LookAt takes world up by default which is why you aren’t getting pitch.

If you pass in an up vector to lookAt, it will use that to change the entity local up.

2 Likes

Thanks a lot :smiley:

I changed the last line to

center.lookAt(cross, ver);

(ver is a vector from chin to head)

And now it works!

I had thought I would need to rewrite my entire code and try a different approach. I didn’t think it would be this simple!

1 Like