Controlling a humanoid rig using the Microsoft Kinect 2

Hey everybody,

we’re recently researching in the Microsoft Kinect, more specifically [this model][1].
Now we started out, trying to use Gesture Tracking to control one of our applications and while that seems to work quite well, we got a little distracted on the way :grimacing:

Since the Kinect provides you with a rig much like this one:

We thought it should be easy to control a rig using the read data… Well turns out its not just plug and play.

So we started out with a model of a police officer

we constructed in Adobe Fuse, it had a standard rig, however a much higher number of bones. So as we couldn’t make it work entirely, we made a second model of a guy in a trench-coat

(we lovingly call him inspector gadget here :smile: )

His rig was a little more simplified as his hands were not completely rigged out.
However, much like the first model, after a while and after some testing we were able to make him move:

Now to the tricky part, the bone rotations:
Until now all we did was moving the bones, not rotationg them, this of course leads to such funny results as the above, and well sometimes even worse…
Searching further as we tried and tried to add rotation to the bones and experiencing results like this :grin:

Now we found out at a very early state that Microsoft uses another coordinate system, so we invert the Vec3 we get for the position, however that doesn’t work for the Quaternion for the bones, as they did something very weird with their rig according to some sources we found:

Apparently they (to us) randomly, rotated some bones 90deg and then others 180 etc etc, so that its not really possible to just apply the information received from the Kinect stream on a standard rig…

Would anybody know about some way to convert the data to be applicable or has anybody else worked with the Kinect before?

We’re thinking of getting and storing the initial position and rotation of each bone to then calculate the offset for the new position so we can apply that offset to the initial position of the standard rig.

But before going that way we were curious to hear if anybody got any experience with this before.