What Did You Wear to the 2004 JavaOne Conference? A Conversation with Technology Evangelist, Simon Ritter

Print-friendly Version

At James Gosling's keynote at the 2004 JavaOne Conference, Sun Microsystems' Technology Evangelist Simon Ritter demonstrated "wearable Java" technology, much to the delight of the audience. (Click on General Session, Part One, of James Gosling presentation.)

James Gosling and Simon Ritter at 2004 JavaOne Conference

Ritter arrived on stage with a laptop strapped around his waist, in addition to a video display unit, much like the viewfinder on a video camera. A keyboard and mouse were strapped to one wrist, and a webcam, with a microphone for audio input, was attached to his head. Moving around the stage, the webcam displayed whatever was in Ritter's field of vision in a video display unit. James Gosling held up a sign in Chinese, which Ritter highlighted on his video screen. He then called up a voice recognition program, which translated the sign in a digitized voice that said "ticket office". He focused on Gosling while requesting information: "The person is James Gosling," reported the device.

Ritter ended his demo by illustrating how wearable Java technology could aid a frustrated supermarket shopper trying to find a Snickers bar. The video displayed a map of the supermarket, a shelf of chocolates, and a subsection of Snickers bars.

Ritter holds a Bachelor's of Science degree in Physics from Brunel University in the U.K. Originally working in UNIX development for AT&T UNIX System Labs and then Novell, Simon moved to Sun in 1996. Since then, he has worked both in Java technology development and consultancy. We last spoke with him in 2002 about his experiences working with Java technology and Lego robots.

Recently, we caught up with him to get the latest news on wearable Java technology.

What were you wearing at the 2004 JavaOne Conference?

I was demonstrating the uses of Java technology in producing what is known as "augmented reality". The basic idea is to overlay additional information onto the user's field of view. I used a system produced by Xybernaut Ltd, which is a small laptop computer repackaged to make it possible to wear* the different parts of the system strapped to your body, which frees your hands to perform other tasks. The main component of the system is a very modest 400MHz CPU with 192MB RAM and a 20GB hard drive. I ran the Java Desktop System from Sun. Linux does not have many drivers for webcams, so it would be great if someone could produce a driver for one of the very small webcams on the market today. This would be more comfortable than the rather large model that I used in the demo.

"The performance of the Java software is better than earlier versions in natively compiled C++." - Simon Ritter, Technology Evangelist, Sun Microsystems

The images from the webcam were captured using the Java Media Framework. These were then rendered onto the display using a combination of the Swing and Java 2D APIs. To obtain more information, I used the mouse to focus attention around areas of interest in the display, like the Chinese sign that James Gosling held up. To communicate the additional required information, I used the Java Speech API in conjunction with the Sphinx-4 project, which is an open source, pure Java speech recognition system.

It's worth noting that, according to Sphinx, the performance of the Java software is better than earlier versions in natively compiled C++. This is a good example of something that is very intensive and yet runs faster in Java software than it does in natively compiled code. When a word is recognized by the system, an operation, like translating a sign, or facial recognition, can be performed. The results can be displayed both on the screen for the user to see, or spoken to the user, again through the Java Speech API and the FreeTTS engine. FreeTTS is another open source pure Java project that provides voice synthesis.

Uses of Wearable Computers

Where can wearable computers make a contribution?

This technology is useful in areas in which people need access to information while remaining able to move around with at least one hand free -- such as utility engineers, people who have to sort out power lines, etc. They need to be both mobile and have access to information, such as schematic diagrams, and various bits of data.

For me, it's a fun toy, but some people at MIT Media Research labs treat it very seriously and wear these devices all the time. They're experiencing the world through the computer rather than directly, and doing a lot more extreme things than I've done. People at MIT have a project involving an augmented reality system to help people play billiards (like pool) better by figuring out the best angle, etc., for the shot and then overlaying the information onto the user's field of view. I've done a proof of concept to illustrate Java technology's suitability for this kind of thing.

Augmented Reality Computing

Why is Java technology a good fit for augmented reality computing?

Java technology works for augmented computing because of its numerous APIs. Augmented computing involves many peripherals so it's not simply a laptop type of environment, with a keyboard, a mouse, and a display. If this were the case, then you could use virtually any kind of application. But this technology requires such peripherals and applications as a webcam, voice recognition, and voice synthesis -- all outside the customary interaction that laptops provide.

"This technology is useful in areas in which people need access to information while remaining able to move around with at least one hand free." - Simon Ritter, Technology Evangelist, Sun Microsystems

With Java Media Framework, it did not take long to figure out how to get the video images from the webcam into my Java application. But C or C++ would have required writing a lot of complicated code. Because of the Java Media Framework, a developer can write 50 to 60 lines of code to get images into the program process, and use the Java 2D APIs, and Java advanced imaging APIs to manipulate the images. It frees the developer to focus on the higher-level abstractions as opposed to having to write all of the low-level code as well.

Similarly, with voice recognition and voice synthesis, Java technology eased my job. I could take advantage of open source projects, like the FreeTTS for speech synthesis, and the Sphinx project for speech recognition, all within the Java language. You can just drop them onto the laptop, or the wearable running Linux, and straightaway you've got a voice recognition system. It's very simple from the API point of view. With a few lines of code, I've got the computer talking to me, and with a few more lines, it recognizes certain commands. Java technology is very powerful.

FreeTTS and the Sphinx project

Tell us more about FreeTTS and the Sphinx project.

FreeTTS stands for Free Text To Speech. FreeTTS allows you to convert text to speech and to create speech synthesis. You can use it in various ways. You get better results with a fixed set of words, such as a system for telling time which only requires numbers, morning, afternoon, evening, P.M., A.M., etc. You can use pre-recorded words to make it easy to understand. Trying to get voice synthesis to create or read any kind of text is more complex. And so, the results with FreeTTS are very good. The Sphinx project is completely open source, written in 100% Java software. This is a good example of how powerful Java technology can be, because speech recognition is difficult to achieve. Only in the last ten years have desktop type machines had enough power to allow speech recognition. Nowadays, systems exist that are quite good at recognizing speech, even in fairly poor conditions where there is background noise, and various voices. Typically, you must train a system to recognize your particular voice. But I didn't have to do that with the Sphinx system. I only had to ask it to recognize certain distinct key words, such as "translate" or "recognize". The system was able to pick those words out. I was impressed.

Java Speech API

What should we know about the Java Speech API?

"Java technology works for augmented computing because of its numerous APIs." - Simon Ritter, Technology Evangelist, Sun Microsystems

The Java Speech API sits on top of the underlying speech recognition system or speech synthesis system, giving you an easy-to-use cross-platform way to handle different speech systems. For example, the code I wrote to work with Sphinx and with FreeTTS is completely portable if I want to switch to something like the IBM ViaVoice product. So, you could move to the Windows platform, take exactly the same code and work with the IBM ViaVoice system, and you wouldn't have to rewrite anything. It's that easy. So, the Java Speech API is a great thing for cross-platform functionality.

The Claims of Artificial Intelligence

20 years ago, artificial intelligence experts were claiming that someday soon a student would be able to go to a lecture carrying a small device that would understand and remember the lecture and help them learn what was covered. It was supposed to share life experience and provide special insight and wisdom. Is this a realistic possibility?

Microsoft is doing research in this particular area, trying to create something wearable which is designed to record events throughout the day. It records various sounds, and takes occasional pictures, with the idea that you can document and record your entire life. I'm not sure how interesting this will be. I believe that a system that illuminates what's going on, and remembers things and provides more information is off in the future. Artificial intelligence still has a lot of challenges. We're doing better, but computers are still nowhere near as good as the human brain at doing many things. We're still a long way from the film, 2001, with Hal, the computer.

Wearable Java Code and Technical Details

Anyone who wants code and more technical details about wearable Java is welcome to write Simon Ritter at: simon.ritter@sun.com

See Also

• Simon Ritter Demo at 2004 JavaOne Conference (Click on General Session, Part One, of James Gosling presentation.)
• Xybernaut
• MIT Wearable Computing
• Boeing
• Java Media Framework
• Java Speech API
• Sphinx-4
• FreeTTS