Intermediate Images

Articles Index

In a previous life, I worked in the 3D graphics arena writing applications, APIs, and drivers for realtime 3D graphics visualization. Fun stuff.

At that time (this is 5+ years ago), there was an idea kicking around the 3D graphics world of "Image Based Rendering". This approach would play tricks with images in order to simulate real-time 3D viewing. So, for example, instead of rendering the model associated with a building on every frame as the viewer walked around that world, you might render that model once from a single viewpoint, cache that result in an image, and warp that image thereafter. So as the viewer moved around, these images would be warped to "appear" as though they were being re-rendered on every frame, but it was actually just neat image rendering tricks to make it look "good enough".

This was indeed a neat trick; you could get great performance because you avoided complex model re-rendering and could do simple image operations instead. Of course, you would need to re-render models occasionally when the errors got bad enough (you can't walk around to the other side of a building model and not expect to re-render that building a few times in the process; for one thing, the original rendering of that model didn't include any details on the side or back of that building, so looking at it from the other side would result in some pretty messed-up artifacts).

Some example uses of this approach to rendering include Apple's Quicktime VR (there are several 3D views of a scene and you can smoothly animate around that scene by interpolating between the various pre-rendered images), various interesting academic papers at graphics conferences, and even proposals for hardware that would use this approach (notably, Microsoft's Talisman proposal).

Sadly, graphics hardware moved on, 3D rendering got orders of magnitude faster, and the need for using image-based tricks instead of re-rendering 3D geometry went away (check out the comment by Id Software's Brian Hook on the demise of Talisman). After all, with current graphics hardware able to render tens of millions of multitextured triangles per second, why should anyone have to play tricks and deal with associated rendering artifacts?

I love neat performance tricks, so part of me is sad to see this go away (although the rest of me is overjoyed because it means the whole system has gotten so much faster in the meantime. I don't think we would have such humanitarian and educational projects as Doom 3 if we were still trying to figure out how to warp images to get around polygon-count limitations...).

Fortunately, Image Based Rendering still has a place in the world, at least in my current world of 2D graphics. While not as complex or fascinating as the 3D Image Based Rendering techniques, the technique I'll discuss here does have similar performance advantages in the 2D realm.

I call this approach "Intermediate Images".

The main idea here is that it is a lot faster to copy an image than to perform a complex rendering operation. In general, a simple image copy (Graphics.drawImage(img, x, y, null)) operation is at least as fast as a blazing memcpy(), and may even end up being accelerated in hardware and video memory in the best case. On the other hand, more involved operations such as transforms, complex GeneralPath drawing, or even drawing text involve lots of operations per-pixel and end up being a bottleneck for what could otherwise be simple and fast code.

Okay, so that's the "why" of the equation; simple image copies are simply faster. What about the "how"?

The basic approach with intermediate images is to create an image of the type and size that you need, perform your expensive operations to that intermediate image, and thereafter copy from that image instead of performing your rendering operations directly.

For example, let's say you have an image of some size and you want it to be scaled to size scaleW x scaleH into some JComponent. You could do this in your paintComponent() method like so:

    public void paintComponent(Graphics g) {
        g.drawImage(img, 0, 0, scaleW, scaleH, null);
    }

This will cause Java2D to scale the image every time through this function (assuming your original image is not actually at size scaleW x scaleH).

You could, instead, use the Intermediate Image approach, where you create an image of the appropriate size, scale the original image to that new temporary image, and then do a simple copy from that temporary image:

    Image intermediateImage = null;
    public void paintComponent(Graphics g) {
        if (intermediateImage == null ||
            intermediateImage.getWidth() != scaleW ||
            intermediateImage.getHeight() != scaleH)
        {
            intermediateImage = createImage(scaleW, scaleH);
            Graphics gImg = intermediateImage.getGraphics();
            gImg.drawImage(img, 0, 0, scaleW, scaleH, null);
        }
        g.drawImage(intermediateImage, 0, 0, null);
    }

Note that, in this approach, intermediateImage is cached between calls to paintComponent() and will only be recreated/re-rendered when either the image is null (the first time through that method) or the scaling size (scaleW x scaleH) has changed. Otherwise, all "scaling" operations from that original image have been reduced to a simple copy operation.

This approach is not limited to scaling transforms; you can use the same approach for arbitrary transformations. Note, however, that some transforms of an image may produce non-rectangular results, thus you will need an image with a transparent background so that the intermediate image background does not show up during the copy operation. See some of the code below for this transparent- background image approach.

We've shown that you can do this for pre-transforming images, saving on the cost of scaling, rotating, or whatever you want to do with an image. But Intermediate Images are not limited to image-based operations; you can use the same technique for arbitrary rendering operations, such as complex shapes or even text.

The idea here is to create an image of the appropriate size with a transparent background, render your graphics operations into it once (or whenever they change), and thereafter just call drawImage() from that image instead of doing the actual rendering. Note that you will have to position your intermediate image appropriately to match where the rendering operations would have been drawn.

Suppose you want to have a "Happy!" icon represented by some insipid smiley face, such as this:

You could render that smiley with the following graphics operations:

    public void paintComponent(Graphics graphics) {
       Graphics2D g = (Graphics2D)graphics.create();
       // Yellow face
       g.setColor(Color.yellow);
       g.fillOval(0, 0, 100, 100);
       // Black eyes
       g.setColor(Color.black);
       g.fillOval(30, 30, 8, 8);
       g.fillOval(62, 30, 8, 8);
       // Black outline
       g.setStroke(new BasicStroke(3.0f));
       g.drawOval(0, 0, 100, 100);
       // Black smile
       g.drawArc(20, 20, 60, 60, 190, 160);
       g.dispose();
   }

Rendering this once isn't a problem. But suppose you had some application where you needed to render this same graphic several times every time you painted the component (maybe it's a graphical chat room application with lots of happy people in it). At some point, it just doesn't make sense to keep re-doing all of the same rendering over and over; you may as well cache the rendering results in an intermediate image and do simple image copies instead.

Note that in this case, the graphics you are rendering are non-rectangular. So when you cache the graphics as an image, you need to make the background of the image transparent so that copying that image will only result in copying the colors from the graphics, not the colors from the background of the image. You can make this work by creating a BITMASK transparent image, filling it with a transparent color and then rendering your graphics.

Here is code that will accomplish the same thing as above, only using an transparent- background intermediate image instead of drawing the graphics directly to the component:

    Image intermediateImage;
    
    public void renderSmiley(Graphics2D graphics) {
       // Create a copy of the Graphics object so that we can change
       // its attributes with reckless abandon
       Graphics2D g = (Graphics2D)graphics.create();
       // Yellow face
       g.setColor(Color.yellow);
       g.fillOval(0, 0, 100, 100);
       // Black eyes
       g.setColor(Color.black);
       g.fillOval(30, 30, 8, 8);
       g.fillOval(62, 30, 8, 8);
       // Black outline
       g.setStroke(new BasicStroke(3.0f));
       g.drawOval(0, 0, 100, 100);
       // Black smile
       g.drawArc(20, 20, 60, 60, 190, 160);
       g.dispose();
   }
    
    public void paintComponent(Graphics g) {
        if (intermediateImage == null) {
            GraphicsConfiguration gc = getGraphicsConfiguration();
            intermediateImage = gc.createCompatibleImage(100, 100, Transparency.BITMASK);
            Graphics2D gImg = (Graphics2D)intermediateImage.getGraphics();
            gImg.setComposite(AlphaComposite.Src);
            gImg.setColor(new Color(0, 0, 0, 0));
            gImg.fillRect(0, 0, 100, 100);
            renderSmiley(gImg);
            gImg.dispose();
        }
        g.drawImage(intermediateImage, 0, 0, null);
    }

(Note that we are only rendering the graphics in one size at a single position; this is a simplifying assumption just for making the sample code more readable).

Along the same lines as the arbitrary shape above, you could also use intermediate images to draw static text. This is obviously not suitable for very text-heavy applications such as word-processing; there is simply too much text and the text is too dynamic to make this a workable solution. But many applications use text for such things as static labels; they might benefit from intermediate images. For example, a game that wanted the best performance possible might want to draw its "Score:" label with an image, to avoid the slower path of text rasterization.

The main work to do in caching text as images is in getting the image size correct (based on the size of the text in a given font) and positioning that text image correctly. Note that an image is positioned through its upper-left coordinate, whereas a string is drawn starting from the lower left (actually, this lower-left point is the point of the base line; descending characters in the font (such as lower case "y" and "g") may go below this line). Note too that the approach below for sizing and positioning the string image correctly is too simplistic for the general case; those interested in more correct (and more involved) text manipulation should check out the javadocs for java.awt.Font and java.awt.font.TextLayout for starters.

Here is some sample code that draws some extremely interesting text in the upper-left corner of the component:

    Image intermediateImage;
    String theText = "Extremely interesting text";
    
    public void renderText(Graphics g, int x, int y) {
        g.setColor(Color.black);
        g.drawString(theText, x, y);
    }
    
    public void paintComponent(Graphics g) {
        if (intermediateImage == null) {
            // First, measure the size of the text
            FontRenderContext frc = new FontRenderContext(null, false, false);
            TextLayout layout = new TextLayout(theText, g.getFont(), frc);
            Rectangle2D rect = layout.getBounds();
            int imageW = (int)(rect.getWidth() - rect.getX() + .5);
            int imageH = (int)(rect.getHeight() - rect.getY() + .5);
            // We must also account for text "descent" in determining where to draw string in image
            int descent = (int)(layout.getDescent() + .5f);
            // Now, create the intermediate image
            GraphicsConfiguration gc = getGraphicsConfiguration();
            intermediateImage = gc.createCompatibleImage(imageW, imageH, 
                                                         Transparency.BITMASK);
            // And render the transparent background and the text into the image
            Graphics2D gImg = (Graphics2D)intermediateImage.getGraphics();
            gImg.setComposite(AlphaComposite.Src);
            gImg.setColor(new Color(0, 0, 0, 0));
            gImg.fillRect(0, 0, imageW, imageH);
            renderText(gImg, 0, imageH - descent);
            gImg.dispose();
        }
        // Drawing the text image at (0, 0) is equivalent to drawing the string 
        // at (0, imageH - descent))
        g.drawImage(intermediateImage, 0, 0, null);
    }

You can take the text example one step further and use images to render anti-aliased text as well. With a slight tweak to the above code, we can create the right kind of image, render anti-aliased text to it, and get image copies that will respect the smooth properties of the text rendering that you asked for:

    Image intermediateImage;
    String theText = "Extremely interesting text";
    static boolean AA = false;
    public void renderText(Graphics g, int x, int y) {
        g.setColor(Color.black);
        g.drawString(theText, x, y);
    }
    
    public void paintComponent(Graphics g) {
        if (intermediateImage == null) {
            // First, measure the size of the text
            FontRenderContext frc = new FontRenderContext(null, AA, false);
            TextLayout layout = new TextLayout(theText, g.getFont(), frc);
            Rectangle2D rect = layout.getBounds();
            int imageW = (int)(rect.getWidth() - rect.getX() + .5);
            int imageH = (int)(rect.getHeight() - rect.getY() + .5);
            // We must also account for text "descent" in determining where to draw string in image
            int descent = (int)(layout.getDescent() + .5f);
            // Now, create the intermediate image
            GraphicsConfiguration gc = getGraphicsConfiguration();
            if (!AA) {
                // non-Anti-Aliased text; only need transparent-background image
                intermediateImage = gc.createCompatibleImage(imageW, imageH, 
	                                                     Transparency.BITMASK);
            } else {
                // anti-aliased text needs translucent image
                intermediateImage = gc.createCompatibleImage(imageW, imageH, 
	                                                     Transparency.TRANSLUCENT);
            }
            // And render the transparent background and the text into the image
            Graphics2D gImg = (Graphics2D)intermediateImage.getGraphics();
            gImg.setComposite(AlphaComposite.Src);
            gImg.setColor(new Color(0, 0, 0, 0));
            gImg.fillRect(0, 0, imageW, imageH);
            if (AA) {
                 // Set up Anti-Aliasing for text rendering
                 gImg.setRenderingHint(RenderingHints.KEY_TEXT_ANTIALIASING,
	                               RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
            }
            renderText(gImg, 0, imageH - descent);
            gImg.dispose();
        }
        // Drawing the text image at (0, 0) is equivalent to drawing the string 
        // at (0, imageH - descent))
        g.drawImage(intermediateImage, 0, 0, null);
    }

The changes made to the prior text code are in bold above. Basically, we need to create a translucent image instead of the prior transparent-background image. Anti-aliased rendering works by rendering the text with alpha values in the colors at the edges of the text; this makes those edge colors blend with the background colors. If we create a Transparency.TRANSLUCENT image, then this image will preserve the alpha values in the anti-aliased text and the text-image will look the same as the text when drawn directly.

The examples above are merely meant to illustrate the concept; I'll leave it as extra credit for you to figure out how to apply the technique in your particular situation. For example, you could take the anti-aliased text example above a step or two further and cache rendering of any anti-aliased or translucent operations.

There are some important things to note about this technique:

• Cache Cow: In creating intermediate images, you are necessarily taking up more space on the memory heap. For example, if you are pre-scaling an image to 500 x 500 and that image has a color depth of 32 bits, then you just allocated about a MB of memory on the heap for that one image. While intermediate images can be quite beneficial if used correctly, you may not want to go about creating them all over the place, especially if your intermediate images will be large, because your application will suddenly be using a much bigger memory footprint than it would otherwise. So be aware of the tradeoff between size and speed for this approach.
  
  
• Translucent image performance: Currently, Java2D does not benefit from hardware acceleration for translucent image copies by default on any platform. So if your main goal is to get hardware acceleration for some rendering operations (pre-scaled images, text, whatever), be aware that if you convert your graphics operations to translucent image copies you may not get what you came for. Note that we do offer hardware acceleration for this type of operation through our OpenGL rendering pipeline (available on jdk 5.0 on all Sun platforms, but disabled by default and only enabled by the -Dsun.java2d.opengl=true flag) and through a command-line switch on our Direct3D rendering pipeline on Windows (-Dsun.java2d.translaccel=true). Also, we are hoping to enable hardware acceleration for this feature in an upcoming release; it's one of the most important features we have yet to accelerate. But for now, just be aware of the issue.

That's all there is to it. The technique is not very difficult to understand or to program (as the simple examples above hopefully demonstrate). The only tricks here are getting the image type correct (to account for transparent background and translucent situations) and getting the cached image positioned correctly (to match the positioning of the original graphics).

My main point here is that you can apply this technique across all of your graphics operations; anything that you draw repeatedly in the same way (static text, images scaled to the same size, complex shapes, whatever) is a candidate for Intermediate Image rendering. Okay, so it's not quite as geeky-cool as the 3D Image Based Rendering algorithms, but I figure anything that gives better graphics performance is pretty worthwhile anyway.

• Image Articles: I've spent a few blogs trying to come to grips with the various image types we have in Java and the performance implications of different approaches. If this interests you, you might check out some of the following blogs: BufferedImage (Part I and Part II), VolatileImage (Part I and Part II), and general image loading/creation API discussion.
• Java 2D homepage: This site has links to performance whitepapers, FAQs, and other information helpful to graphics programmers.
• http://javadesktop.org: This community site has forums, projects, blogs, and articles aimed at the Java desktop client developer.