Fingertoy is a Linux software bundle for computer vision based interactive
surfaces, with strong focus on DI/FTIR based multitouch devices. Currently it consists of set of cross-platform GStreamer plugins forming an processing pipeline, D-Bus service, client library, configuration applications and bindings to existing toolkits.

Processing pipeline

Processing pipeline is designed to be fully customizable by the end-user to fit most installations needs. There are five main parts - source, image filtering, detection, tracking and sinks.

Source part is responsible for acquiring and assembling of the raw camera data.

Filtering part performs source data segmentation and transforms source data to a binary image on which feature detection can be performed. Filtering is the most computationaly expensive part of the pipeline and for that reason it can be done by GPU (using OpenCL).

Detection part takes the filtered binary image and passes it to the chain of detection elements defined by user. Each chain link can than be enabled or disabled when needed by an application. Currently there are elements for detecting fingertips, fiducial markers, rectangular objects, shapes and generic blobs. All detection elements extends a common base class which does all GStreamer related processing and provides support for optional preview buffer. Creating a new detection element is easy as overriding single base class method.

Tracking is responsible for establishing relations between consecutive frames and detected features.

Sinks as the last part of pipeline are responsible for sending detected features to client application.

Configuration and system integration

Pipeline configuration utilizes GConf and has several frontends. Two python based frontends PyGtk+ and Django, and the Clutter based Calibrator. D-Bus service is responsible for integration within the user’s desktop session. It provides pipeline control api and features like automatic startup on login. Lastly there is also init.d script for startup and shutdown of projector via RS232.

Testing & Development

Besides standard development tools provided by GStreamer, like python bindings and , the Calibrator application provides environment for developing and debugging individual pipeline elements - displaying overlays with source, filtered result and preview outputs, pipeline graph and logging console.

Toolkit bindings

There are currently only bindings for PyMT and Clutter, but GTK+ and basic XInput support are on the way. The goal is to deliver full Gnome desktop integration in future.

PyMT Pong with shape tracking (Xvid, cca 10MB)

GStreamer plugin packages

clp - OpenCL Processing

  • clcolorspace
  • cldownload
  • clflip
  • climageop
  • clsmooth
  • clthreshold
  • clupload

fingertoy - Base package

  • autoblobfilter
  • blobdump
  • finders
    • blobfinder
    • shapefinder
    • fingertipfinder
    • fiducialfinder
  • opencv
    • cvblobfilter
    • cvcontourfinder
  • opencl
    • cvblobfilter
  • tuio
    • blobtuio
    • blobtuio2