Writing in the air, tracked with a webcam

IBM shot a series of educational videos where the presenter appears to write on glass in front of them, facing the camera. I wanted to know how that was done, and the version I could build was tracking a coloured marker through a webcam and drawing where it went.

1. 1
   Tracks a coloured object frame by frame through OpenCV colour-space masking, and treats its centroid as a pen position.
2. 2
   Accumulates positions onto a transparent canvas layer composited over the live feed, so the strokes persist while the video keeps moving.
3. 3
   The hard part is not the drawing — it is deciding when the pen is down. Lighting changes shift the mask, and a lost frame breaks a stroke in half.

1. 1
   Gesture-based pen up and pen down, rather than tracking continuously.
2. 2
   Stroke smoothing — raw centroid positions are noisier than a human hand.