Project SIKULI: Evolutionary approach to computing

How do you know what to do when you have GUI?
Visual cues are key to navigating and doing things which has made computer experience so easy for general public.
Sikuli Script is an innovative approach developed at MIT to program and control any GUI applications.

Well, let me quote some more:

Sikuli is a visual technology to search and automate graphical user interfaces (GUI) using images (screenshots). The first release of Sikuli contains Sikuli Script, a visual scripting API for Jython, and Sikuli IDE, an integrated development environment for writing visual scripts with screenshots easily. Sikuli Script automates anything you see on the screen without internal API's support. You can programmatically control a web page, a desktop application running on Windows/Linux/Mac OS X, or even an iphone application running in an emulator.

Head on to Project SIKULI:

http://sikuli.csail.mit.edu/documentation.shtml#doc/pythondoc-python.edu.mit.csail.uid.Sikuli.html