 | |
The world of computing is in a state of rapid transition, but what we are (or should be) transitioning to is very much open to debate. Sun Microsystems' Ron Goldman believes we are heading towards a world of interoperable computing in which stand-alone applications are diminishing in importance. He has been working with colleague Richard Gabriel to develop ways to make large systems more robust, stable, and better able to take care of themselves -- what they call "conscientious software." Prior to coming to Sun in 1998, Goldman developed a program to generate and manipulate visual representations of complex data for use by social scientists as part of a collaboration between NYNEX Science & Technology and the Institute for Research on Learning. Goldman has a Ph.D. in computer science from Stanford University, where he was a member of the robotics group. He currently works as a senior staff engineer at Sun Microsystems Laboratories where he researches alternative software development methodologies and new software architectures, with a particular focus on what makes software more robust. Other research interests include biologically inspired computing, open source, programming language design, and user interfaces. In Part Two of this interview, we will explore his insights into open source, which he and Gabriel have articulated in their book, Innovation Happens Elsewhere: Open Source as Business Strategy.
Another important trend is that software continues to become more and more complex. We now have greater expectations about how our software should behave, and meeting those expectations requires more code. The problem though is that our programming languages, tools, and methods are more suited to creating small software artifacts. For large scale software they just don't help us. As a result we end up with brittle software that is not as robust as it needs to be. We also end up with a large number of software projects that fail.
Software today is produced according to a manufacturing model: A finished product is constructed at the factory and shipped to its destination where it's expected to act like any other machine --reliable, but oblivious to its surroundings and its own welfare. Once deployed, it is on its own, along with the people who use it. This results in brittle, buggy software, in which the end-user is left to hope that the next release will fix their problems.
Until recently that was the best we could do -- just getting the code to implement the functional requirements was a challenge and the resulting software often used up all the available cycles on the hardware. Now there will always be some applications that need the full processing power of the CPU (and are hungry for even more), but for most programs current hardware speed is quite sufficient. We think it's time that our software start using some of those CPU cycles to actively monitor its own activity and environment, to continually perform self-testing, to catch errors and automatically recover from them, to automatically configure itself during installation, to participate in its own development and customization, to pay attention to how humans use it and become easier to use over time, and to protect itself from damage when patches and updates are installed. Such dynamic software systems will be self-contained, including within themselves their entire source code, code for testing, and anything else needed for their evolution. We have coined the term "conscientious software" to describe code that takes responsibility for itself and its future.
That's the good news. The bad news is that we think it requires a different way of thinking to create conscientious software -- a lot of the basic underlying assumptions of our current programming models need to change. It won't be easy to convince people that they need to accept that software will always have errors or that performance is of secondary importance. Admitting Inevitable Failure: Bugs and the Misguided Focus on Performance
Rather than putting all of the programming effort into trying to prevent errors from occurring, we feel it is better to devote runtime resources to detecting that an error has occurred and recovering from it. Let me give a historical example: garbage collection, or automatic memory management. It's helpful to look at where garbage collection came from. When John McCarthy was designing the LISP language, one of the programs he wrote was an elegant algorithm to do symbolic differentiation. He recognized that the code would be using up memory and if it wasn't released that memory would eventually run out. And he deliberately decided that he didn't want to mess up his elegant algorithm for differentiation with a lot of record keeping and bookkeeping for memory, which had nothing to do with the problem he really cared about. So he did something that we're considering doing in a number of other places. He accepted the idea that all programs have bugs and created a system that can repair and clean up unused memory and, in a sense, that can recycle it and make it available. Making memory management a separate process independent of the application gives you a much cleaner, simpler-to-write program, and it is also easier to focus on good memory management. Repairing and acknowledging the problem makes the system more robust. If we're serious about wanting robustness, reliability, and security in our programs, we need to devote resources to them, resources above and beyond what the application uses.
Although automatic memory management has been around for more than 40 years, a lot of people still don't want it in their systems because it seems inelegant. It just strikes people as wrong -- "It's my memory. I should be releasing it when I know it's no longer being used. "Theoretically that may be true, but in practice programmers continue to forget to free up memory when they are done with it or, even worse, try to free it up while it's still in use. The results? Buggy code that is apt to crash unexpectedly. Why force programmers to do things for which they are not suited? Why not instead have the computer take over as many of those tedious chores as we can? Our software should be an active participant in maintaining its integrity. Our programming languages can deal with low-level concerns like array bounds checking and preventing buffer-overflow attacks. For higher-level matters such as maintaining system-wide constraints, such as providing a specified quality of service, we need new mechanisms.
We need to expand our definition of performance to go beyond just the time it takes to do a task. We need to include usability, robustness, security, and all the other factors that define the behavior of our programs. The Inspiration of Biology
One idea that seems promising is to separate the software that does the work from the software that keeps the system alive. In computing, perhaps 5% of our code deals with exception handling and error correction, which seems like a lot, while 95% tries to get the basic job done. Biology appears to reverse this, with 5% doing the basic metabolism and 95% functioning to make sure that the 5% can do its job. Think about keeping your heart beating -- is that overhead? Or is that a core activity that's part and parcel of who you are? Think of your body doing the work of keeping your mind and brain functioning. That's not overhead. Likewise, maintaining your computer system's health to make sure that all of its components are functioning is not overhead. That's just what is required to have a robust system. Allopoietic and Autopoietic Computing
Our current software applications, say banking systems, are allopoietic systems. They take various inputs and perform some desired functionality. Our programming languages, like C or the Java programming language, have been designed to create allopoietic programs. If we try to make the system more robust by adding exception handlers and error detection code, the program becomes hard to understand and very difficult to maintain. It goes beyond what people can really write.
A second approach that people have tried is to start with a self-organizing, autopoietic system that generates and controls its own development. But there's a problem: How do you write a banking application in such a system? In both approaches you're locked into a certain type of expression. We want to take the allopoietic components, and embed them in a larger autopoietic space with components that are working to maintain the integrity of the system. For instance, a web server may devote a lot of resources to serving up web pages, but resources are also devoted to such things as logging and indexing the web site, and doing internal editing. If you have been slashdotted [/..] and suddenly get heavy traffic, you may want to dynamically shift, and devote more resources to serving up web pages, and put off indexing until later. So how is that done? Is it something that's built deeply into every component, or is there a controlling layer outside the sensing layer that enables the web server to say, "Boy, I'm working a lot." And then the controller picks up on that, and says, "Oh, there's a hot spot right now, let's inform these other jobs that don't need to be done right now, let's throttle back, and devote more resources to the hot spot. Let's clone some extra copies of these web page servers." The system adjusts itself. And it may be that the language we use to describe that process is not the same as our normal programming languages -- for example, we may not be able to write a sort algorithm in it. We want something that has feedback loops in it, since biological systems seem to be robust in part because of their extensive use of nested feedback loops. Of course we can program feedback loops using low-level features of our current programming languages, but they're not a high-level construct that we have right from the get go. We need a language that lets us easily construct complex feedback networks.
Software that Sees
Here's a simple example that shows how we see visibility improving robustness: Each component in a system can describe its internal state by constantly muttering about what it is doing as it goes about its business -- muttering is like logging, only not just about reporting errors, and is probably not persistent. Another thread can periodically check that each component's mutterings indicate it is healthy and making progress; if not, the thread can reset or restart the failing component. Our inspiration about the importance of visibility comes in large part from the use of stigmergy as a powerful self-organizing mechanism in biological systems. Stigmergy is where individual parts of a system communicate with each other indirectly by modifying their local environment. For example, foraging ants deposit a trail of pheromones to indicate that they have located a source of food. Other ants sense the trail and follow it to the food source. In computing, an example of this would be JavaSpaces technology where a process writes data into a space that other processes can then read, do computation based on what they've read, and then possibly write new data into the space. This idea of visibility is also related to what Jaron Lanier is aiming at in what he calls phenotropic computing. Instead of a rigid, minimalistic API between two modules, he wants surfaces that can be pattern-recognized or sampled. We are considering shared blackboards with simple textual pattern-matching, extensions of something like Common Lisp's keyword/optional argument lists and calling conventions, or even passing XML documents. The key idea being that, instead of one agent reaching inside another and commanding it to do some function (e.g., a remote procedure call), it instead deposits a request that the second agent can then interpret and deal with as best it can. Software Like Children
Our software is like a child. At first, parents must provide everything and be ready to intervene to prevent the next disaster. But, after a while, the child grows up and is able to take care of itself, to cope, and hopefully to contribute something new. The child becomes responsible. With luck, persistence, or good upbringing, the child may become conscientious. We hope the same for our software. See Also
|
| ||||||||||||||||||||||||||||||||
| |
Oracle is reviewing the Sun product roadmap and will provide guidance to customers in accordance with Oracle's standard product communication policies. Any resulting features and timing of release of such features as determined by Oracle's review of roadmaps, are at the sole discretion of Oracle. All product roadmap information, whether communicated by Sun Microsystems or by Oracle, does not represent a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. It is intended for information purposes only, and may not be incorporated into any contract.
| ||||||||||||
Print-friendly Version
Give us your assessment of the state of software today.
I think that we are in the middle of a very interesting transition from stand-alone applications to applications that rely on networked services. Whether it's a mash up of Google Maps with your address book, or a web page displaying information pulled from several databases, instead of one software application we now have multiple services, created by different organizations, that need to interact. In this world of interdependent applications and services, nobody's in charge anymore -- no one person can say it's time to throw the switch to do a global recompile. A new version of software often unintentionally breaks other programs that rely on it. There's a paradox: to be useful, software must depend on other software; but to be reliable, it must be independent -- able to maintain its integrity in a changing environment.
