The Future of Virtual Reality: Part Two of a Conversation with Jaron Lanier

Print-friendly Version

Jaron Lanier

Jaron Lanier is well known among developers as the co-inventor of "virtual reality," a term he coined in the 1980s. Renowned as a composer, musician, and artist, he has taught at many university computer science departments around the country, including Yale, Dartmouth, Columbia and Penn. He recently served as the lead scientist for the National Tele-Immersion Initiative, which is devoted, among other things, to using computers to enable people in different cities to experience the illusion that they are in the same room at the same time.

In this, the second part of a two-part interview, we asked him to talk about virtual reality, tele-immersion, and Java technology.

You are known for claiming that virtual reality (which you are credited with naming) would eventually enhance our lives. How's it going?

About ten years ago, I predicted that virtual reality would be accessible to consumers by about 2010. I still think that's true. Virtual reality is a somewhat broad idea, and the definition isn't fixed. It means different things to different people. As used in industrial technology, there's no question that virtual reality has already been a success. You can't buy a car today that wasn't designed using it. And you can't put gas in that car that wasn't made out of oil that was discovered using virtual reality through an oil field simulation. Most new drugs are made in a process assisted by virtual reality. There are many other examples.

"What I envision is not so much a pre-programmed virtual world that you might play as a game, but rather a virtual world that you can change from the inside that people use as a form of expression in which they're creating things together." - Virtual Reality Pioneer Jaron Lanier

What most people are curious about, though, isn't so much these industrial uses; rather, they want to experience some new level of cultural expression that arises out of virtual worlds. And what we currently have, in this regard, is the video game world. The difference between video games and my sense of what virtual reality would be like relates only partly to the intensity of the experience. Certainly we'd like to feel like we're inside the virtual world, and that world would be more vivid and fluid and detailed -- more computationally intensive -- than what we see today on PlayStation games. But the main difference isn't any of that stuff.

The main element lacking in video games (compared to what I hope we'll see in virtual reality) is an expressive power. And so, what I envision is not so much a pre-programmed virtual world that you might play as a game, but rather a virtual world that you can change from the inside; a world that people use as a form of expression, in which they're creating things together. Just as people make up their own Web pages, they would make up little realities and visit each other's realities, or co-create them. And I think that level of activity would give rise to really, really wonderful new sorts of human relationships and experiences. I still believe in that.

Jaron Lanier and Sun Microsystems In February of 1998, Sun acquired the patent portfolio of VPL Research Inc., Jaron Lanier's former company that pioneered the development of virtual reality. At the time Lanier remarked, "I'm delighted that Sun has acquired VPL's assets. Sun's commitment to open systems and the Java paradigm will provide a superb context for the formulation of competitive strategies by both VR users and developers. The next generation of applications will have to deal with a level of complexity that other leading platforms cannot address. Virtual reality-based applications will be needed in order to manage giant databases and networks, advanced medical imaging, and fast turn-around mechanical design. And all of these mega-applications will have to support real time collaboration over the net. Sun is in an ideal position to enable this new level of productivity."

And you know, the problem with digital stuff is always the same: the hardware gets cheaper and becomes more available, but it takes a really long time to figure out how to write good software. We still don't have good personal computer software. So, how long will it take for us to get good virtual reality software? It's not going to happen overnight. It's a big project.

Tele-Immersion - The Illusion of Being in the Same Room

You're the chief scientist for the National Tele-Immersion Initiative, which is connected to something called Internet 2. Tell us about this.

It's a fun project. The Internet 2 project is a large-scale collaboration that has provided a laboratory platform for people to do academic research on things like quality-of-service protocols and new applications that couldn't be done on the regular Internet. Another way to think about it is that it provides universities with their own network at a good price. And as a side effect, it has subsidized the rise of Napster, which is kind of amusing.

Tele-immersion is the lead application for Internet 2, the driver application to push Internet 2. So, we would try to experiment to get tele-immersion to work. I was the chief scientist at Advanced Networks, which was also the engineering office of Internet 2.

I was charged with coming up with some applications that simply couldn't be done on a lower level of Internet access. So, we had our own proprietary net, which was called Abilene. And we could write our own protocols on it and route it as we wanted. We could do all kinds of things to it. And that created an ideal opportunity to try out the next level of human experience over the net, which has been awkwardly called tele-immersion. We need a better name for it. I don't like the name, but at any rate, tele-immersion is a form of virtual reality that, instead of putting you in an imaginary world, connects two people -- who are significantly far apart from each other in the real world -- and gives them the illusion that they're in the same room.

Tele-immersion is ultra-realistic, full-scale 3D video-conferencing in which people feel like they are actually together. It's really hard to do on a lot of levels. The Internet implementation side of it is really hard, because it requires an extreme level of performance. And even the slightest latency issues become extremely important. It's also really difficult to do on a sensing and display level; it's not easy to sense the world in this sort of three-dimensional way, and it's hard to recreate it realistically.

So, there was a consortium of a dozen universities that worked for a few years to make a first demo. And the first demos weren't perfect, but they looked pretty good.

And it's gotten better and better. It's not ready for prime time by any means, because it's still an extremely expensive laboratory technology. But I believe that, eventually, you will be able to have the illusion of being present with another person, to the degree that you can walk around them and shake their hand. Though this will take a while, soon you will see a substantial improvement in the sense of contact you get from video-conferencing and telephones.

And I think it will be worth it. I think it will be a reasonable alternative to air travel for a lot of things -- for meetings, and for helping keep families together when people move around. I think it will be a useful addition to the means we have to connect with other people, which is really what this is all for. I'm really happy with how it turned out.

"Tele-immersion is ultra-realistic, full-scale 3D video-conferencing in which people feel like they are actually together." - Virtual Reality Pioneer Jaron Lanier

Consulting with Hollywood

You were a consultant for some of the futuristic aspects of the movie "Minority Report," which depicted a frightening world in which advertising signs could recognize you as you walked past them, and remember what you had previously bought and then make very personalized sales pitches to you.

It was kind of funny what happened. Because Hollywood people mostly respond to what they can see, rather than abstract visions, all the information technology that actually made it into the movie was very strongly based on demos that had already been created -- some of them quite old. Some of the things, such as the interface gloves, go back two decades or more. There's a literal-mindedness to Hollywood that's hard to overcome. I presented all sorts of ideas for what information technology might look like in fifty years, but the least noble of these were the only ones that stuck.

Nowhere in "Minority Report" do we see people using technology to interact creatively with each other. We don't see children inventing their own technological culture, which is a reality today. Philip K. Dick, who wrote the short story on which the film was based, didn't live long enough to see this reality. If he had, I believe he would have written a different kind of story.

The characters in "Minority Report" are either consumers (who are used by the advertisements, the animated cereal box and so forth) or elite controllers (the pre-crime officers who get to use a zippy interface). Three-dimensional displays are used for recorded images, but not for live contact.

As you can see, I'm a little bit of a Hollywood antagonist. I'll work with them, but I'm not too impressed.

Musical Instruments as Ideal Interfaces

You once said, "If there's any object in human experience that's a precedent for what a computer should be, it's a musical instrument: a device where you can explore a huge range of possibilities through an interface that connects your mind and your body, allowing you to be emotionally authentic and expressive."

I still believe that. I think musical instruments are the only good user interfaces that have ever been designed. I have an enormous collection of them and I'm always trying to learn to play new ones. I think that the musical instrument is the precedent for how computers should be designed. So, having said that, somebody might ask me, "Well, how could you specifically improve a computer based on your experience with instruments?" And here I have to say I'm still working on it.

Can you give us some clues?

Essentially, you want to discover your own high-level variable when you're controlling a complex system. Here's what I mean by that. If you're working with an instrument, you gradually train your own body and mind to the instrument. Your own body and mind are pattern recognition engines and you're always training yourself to the world and adapting to the world in wonderful, complex, subtle, and deep ways. So, you start doing that with the instrument. But each musician finds a different set of high-level parameters that they're continuing to refine on, sometimes consciously and sometimes not consciously. And the areas of action for them that are continuing to change and grow will vary from musician to musician. So, they become adaptive in some ways and less adaptive in other ways. And that's how people become specialized in their connection to the world.

"I think Java technology is fantastic and Java 3D deserves a lot more attention at this point." - Virtual Reality Pioneer Jaron Lanier

That's how people become good at things -- and some people are perhaps inherently good at some things. We have to be able to recreate that effect with computers. And right now, we can't do it, except to a limited degree; of course, there are some people who are better at playing fast video games and that sort of thing. But there are just so many things wrong, it's hard to know where to begin. The first thing is that, in humans, the mind and the body aren't really separate. There's a continuity. And there's a continuity between step-by-step conscious thinking and physical action and emotion. And you can think of physical action as a form of intelligence in which your brain has created patterns that take into account implicit patterns in your body's physiology. And that's a very sophisticated form of computation, which shouldn't be dismissed at all.

Similarly, emotion can be thought of as an extremely sophisticated form of computation in which you're looking at the very highest level patterns of life experience. And computers continuously draw us down to the sort of tweedly tedious level at which these things that we really excel at can't be relevant. It's extremely frustrating. Through their physicality, musical instruments allow us to incorporate those aspects of ourselves that involve high levels of pattern recognition. And we have to get to that point with computers some day. I think it's going to be hard. I don't expect this to happen overnight at all. But that's certainly what I'm going for. I don't know if I'll see it in my lifetime.

Java Technology

You once said that, while it is commonly thought that small start-ups come up with new ideas, Sun is noteworthy as a big company that created the Java programming language. How is Sun doing at applying Java technology and other ideas to virtual reality technology?

I think Java technology is fantastic and Java 3D deserves a lot more attention at this point. Having more layers of software systems distributed is a good thing, and having more open systems is a good thing. And there's another element, which is really important to me -- working on biological metaphors. If you look at the viruses that are floating around and how expensive they are, the kind of monoculture we have on PCs is a terrible security risk. What you really want is some balance between sanitization and variety so that you have robustness. I think Java systems achieve that balance much more effectively than anchored OS systems. And I think that's going to be a surprise driving software at some point in the future.

See Also

Jaron Lanier's Home Page
(http://www.jaronlanier.com/)

Brief Biography of Jaron Lanier
(http://people.advanced.org/~jaron/general.html)

One Half of a Manifesto
(http://www.edge.org/3rd_culture/lanier/lanier_index.html)

Slashdot Discussion of Jaron Lanier
(http://developers.slashdot.org/article.pl?sid=03/01/25/1511236&mode=thread&tid=156)

Coding from Scratch
(http://java.sun.com/features/2003/01/lanier_qa1.html)

Java 3D API
(http://java.sun.com/javase/technologies/desktop/java3d/)