We have been working for many years on a real-time system that allows a computer to play pinball.  It came to our attention that the system could be used for not only
controlling a pinball machine, but also to control other devices.  With that in mind, we set out to design a modular infrastructure for the system to interact with the real
world.  With this modularity, we are able to have a programmer create a wide range of learning tools by incorporating various components of the system.

       A typical student does not ordinarily have the opportunity to engage in the challenges of real-time programming.  Throughout a student?s journey into the world of
programming, they may hear the term Real-time System or learn about a particular system, but rarely do they get the chance to actually work or design a real-time system.  It is typically a burden on a school or laboratory to house such a system.  Both cost and size of such systems deter many institutions from providing activities of this kind.

       To allow students the means of working with a real-time system first hand, we have developed our own inexpensive real-time test bed.  The Real-time Object Oriented
Tool-kit (ROOT) we developed allows a novice programmer to manipulate real world devices attached to a PC.  The goal of our project is to develop a computer package (hardware and software) that can be put together with commonly available computer parts requiring minimal technical skills.  Our project is designed to facilitate students in developing programs to control physical devices that interact with the real world.  Real-time systems generally involve very expensive equipment.  Our goal was to develop a system that can easily be brought into the classroom or laboratory.

       The system is written primarily in Java, except for the camera driver written in the C language.  The camera is a QuickCam by Connectix (now Logitech) which connects to the parallel port of the computer.   Connected to one of the serial ports on the computer is a digital I/O board, ADR2200 by Ontrak Control Systems Inc., which controls the external devices.   The I/O board has a series of relays that control the switches on the devices.  No driver is required to use the I/O board.  It is a simple serial device that accepts ASCII commands.

       The computer interacts with the world by seeing the environment from the camera and controlling devices connected to the digital I/O board.  As mentioned at the beginning of this paper, the original idea of the system was to have the computer play pinball.  It became clear that the basic mechanics of the system could be applied to other devices besides that of a pinball machine.  In particular the image analysis section of the program can be applied to tracking other things besides
a ball on a pinball playing field.

Since Java is object-oriented, it allows for various changes to go on while still preserving the original design.  Our system is designed modularly and various parts are replaceable.  Experiments in various tasks are outlined below:

        Laser Pointer Mouse: Using a common pen laser pointer, a user is able to control the mouse pointer.  The camera is fitted with a red filter over its lens.  The user points the laser pointer on a white wall.  The lens blocks out all the other light, except for the laser pointer.  Since the camera does not see color, the laser point appears as a white light.  Movement does not come into play with the analysis.  The computer while analyzing the image looks for pixels that are pure white in a
single frame.  The mouse pointer on the computer is moved to reflect the position of the laser pointer in the frame. Clicking the laser pointer on and off was the original idea for clicking the mouse.  However the computer would miss seeing the light periodically and would create extra mouse clicks.  To prevent this, we hold a computer mouse and used its mouse buttons instead.

        Hand Interaction: We modified the analysis part to track a person?s right hand.  The analysis finds the furthest change in the right hand part of the image.  With a static background, this would be the person?s right hand.  The display inverts the image, so that a person sees their mirror image.  With the system able to track a person?s right hand, someone could interact with the display.  A simple interaction was chosen: touch the squares.  There are two squares on the display
that when a user reached up and touches them, the squares change color.

        Robot: We have connected an OWI Incorporated Movit Soccer Robot to the I/O board.  The system sees the robot moving and can track its progressive movement.  As a simple test, we set the system to detect if the robot is making any progress moving.  We do not detect the possibility of a collision, only that a collision has occurred.  The robot has a fence that prevents it from going of bounds.  When the robot hits the fence, it is stopped from making any progress in its current direction.  When the system detects this, its movement is reversed. The robot store carries them. Link to product.

        Morse Code Conversion: We hooked up a light to the I/O board so the computer could send Morse code light pulses.  Using the camera we can detect when the light is being transmitted.  The computer then reassembled the dots and dashes as characters and displays the message sent.  The Morse Code program uses the same analysis code that the laser pointer uses; it detects white light without movement.  With the room lights dimmed, the Morse code light becomes pure
white and is the only object found in the frame.


Go into the project which has the Makefile and run:
make all

After we short while, it will compile all the parts of the system. If some part has not been made, you may need to compile that directory by hand using javac.


Go into the root directory containing the pinball packages. Then do:

$ java pinball.player.PLAYER_PACKAGE.PLAYER

Replace PLAYER_PACKAGE with the package that contains the player class.

Replace PLAYER with the player class.

Project Overview

Users Guide

Programmers Guide


Copyright 1997-2001 Dayton Clark & Lawrence Goetz