The Java Compatibility Test Tools: JavaTest Harness

Print-friendly Version

Articles Index

The Java Compatibility Test Tools (Java CTT) are designed to help a specification lead, or other Java Community Process Program (JCP) member serving on an Expert Group, create a Technology Compatibility Kit (TCK). The JCP is an inclusive, consensus-building approach used by the international Java community to develop and evolve Java technology. If you are unfamiliar with the JCP, see the Java Community Process Program. A recent article, The Java Compatibility Test Tools: Spec Trac, described what a TCK is, introduced the tools in the Java CTT, and described, in general, how each tool helps in creating a TCK. The article then covered, in detail, how to use one of the Java CTT tools, Spec Trac.

This article covers another Java CTT tool: the JavaTest harness. The article is intended primarily for TCK providers. TCK providers need to incorporate a test harness in the TCKs they provide, that is, a test harness customized to meet the requirements of the TCK. They are also responsible for describing how that customized test harness is used. This article describes the JavaTest harness, with a focus on its flexibility -- particularly how it can be customized. The article also presents an example of the JavaTest harness in use. This is only one possible way of using the JavaTest harness, although it can serve as a model of the type of information that a TCK provider might provide in TCK user documentation. It is ultimately up to the TCK provider to describe how to use the test harness provided in the TCK.

What is the JavaTest Harness?

The JavaTest harness is a tool that manages a set of tests. A test harness is one of the components that the JCP requires for the final release of a Java technology. The other required components are:

• TCK test suite. A TCK is used to test an implementation for compliance with a Java technology specification. This is performed by running tests in the TCK test suite with the implementation on the appropriate platform.
• Documentation that describes how to use the TCK, for example, how to run the tests in the test suite.

The JavaTest harness manages the TCK-based tests. It automatically loads tests in the TCK test suite, helps configure the user's environment to run tests, runs tests, and reports test results. Let's look at an example.

Imagine that you are the specification lead for a Java technology specification that is developed in accordance with the JCP. Imagine too that the Java technology comprises a very simple subset of the J2SE APIs. Say the specification includes a number of AWT classes, as follows:

java.awt.AWTEventMulticaster
java.awt.Button
java.awt.Label
java.awt.MenuShortCut
java.awt.Polygon
java.awt.Scrollbar

and the networking class java.net.InetAddress.

As the spec lead, you're responsible for providing a TCK for the Java technology. Your TCK includes a test suite that can be used to test implementations of the technology for conformance to the specification. For example, the test suite includes button constructor tests that determine whether the constructors for the java.awt.Button class perform as specified when used with an implementation of the technology.

The TCK also includes the JavaTest harness to manage the test suite, and documentation, in the form of a User's Guide, that describes how to install and run the JavaTest harness and the test suite. After describing how to install the harness and the test suite, the User's Guide presents the following instructions.

The following steps reflect only one possible way of using the JavaTest harness. For example, starting an agent is not required in many cases. Also, the examples below occasionally reference a "Demo TCK Test Suite" and a "demotck.jar" file. The referenced names are hypothetical -- they should not be interpreted as indicating that a demonstration test suite is distributed with the JavaTest harness.

Start the JavaTest Harness

1. Make the directory where the JavaTest harness is installed the current directory.
2. At the command prompt enter:

java -jar javatest.jar

Click to enlarge

The JavaTest harness starts. It displays the test manager window and prompts you to create a work directory. The harness uses the work directory to store information about the test environment and to write test results.

Configure the Test Environment

Use the Configuration Editor to configure the information required to run the test suite.

1. Choose Configure->Configuration Editor from the Test Manager menu bar.

   Click to enlarge

   The JavaTest harness opens the Configuration Editor. The Configuration Editor consists of three panes and a menu bar.



2. Answer the questions in the Configuration Editor.

Start the Agent Monitor

Use the Agent Monitor to establish and monitor the connection to the agent. See Start the JavaTest Agent for a description of what an agent is.

1. Choose Tasks->Monitor Agent Activity from the Test Manager menu bar.

   Click to enlarge

   This adds a new Agent Monitor window to the display.
   

2. Click the Listening checkbox in the upper lefthand corner of the Agent Monitor.

   The JavaTest harness starts listening for active agents that want to connect.

Start the JavaTest Agent

The JavaTest Agent (referred to simply as the "agent") is used to run tests on a different system than the one in which the JavaTest harness runs. (However the agent can also run as a separate process in the system in which the JavaTest harness runs.) The agent and the harness communicate with each other over the network.

1. (Note: If you run the agent on the same machine as the Javatest harness, you can omit this step.) Copy the JAR files for the JavaTest harness (javatest.jar) and the TCK test suite (demotck.jar) to the system on which the agent will run.
2. Start the agent.

   Make the directory that contains the JavaTest harness and the TCK test suite the current directory and use the following command:

   java -cp javatest.jar:demotck.jar:dir1/TCK/classes \
   com.sun.javatest.agent.AgentMain -activeHost system1
   

   Replace dir1 with the root directory below which the TCK test suite classes are installed, and System1 with the network name of the system running the JavaTest harness.

   If you run the agent on the same system as the JavaTest harness, use the following command to start the agent:

   java -cp javatest.jar:demotck.jar:classes \
   com.sun.javatest.agent.AgentMain -activeHost system1
   

3. Check the JavaTest harness Agent Monitor to verify the connection. The agent should be listed in the Agent Pool.

Run Tests

1. Choose Run Tests->Start from the Test Manager menu bar. The JavaTest harness presents you with a Confirm Configuration dialog. Click OK.

   Click to enlarge

   The test suite begins to run. You should see activity in the test tree panel that indicates which tests are running. Expand the test tree folders to reveal the tests. As tests complete, the tests and their folders change color to represent their state. Click on folders and tests to see information about them in the test manager main panel.

   After the tests complete, you can also view reports in the work directory. The reports include a test summary that lists the tests and their execution result status (pass, fail, error); and detailed reports for specific tests.
   

What's Really Happening Here?

Let's put the example of the JavaTest harness in use under the microscope and see what's going on beneath the surface:

1. Start the JavaTest Harness.

   In response, the JavaTest harness looks for a property file named testsuite.jtt in the directory in which the harness is started. If it can't find this file, the harness prompts the user to identify the root directory of the test suite. At a minimum, the property file identifies the name of the test suite, any arguments it requires, and the class path for the test suite. For example, here's the testsuite.jtt file for the previous example. In addition, the harness looks for a work directory associated with the test suite. If it can't find the work directory, the harness prompts the user to create one.

   The JavaTest harness then creates a TestSuite object and passes it any additional information about the test suite that is specified in the property file. The harness uses the TestSuite object to choose a test finder. The test finder is an object that finds and reads test descriptions for the tests, and creates a TestDescription object for each test. The test tree (that is, the suite root and the tests below it) is then displayed in the test tree panel of the test manager window.

   As its name implies, a test description describes a test. More specifically it identifies important information about the test, for instance, what class to execute for the test. The JavaTest harness requires that the test description be specified as a set of name-value pairs. What the names and values are depends on the test suite. Also, the format in which the names and values are presented depends on the test finder that is used. The name-value pairs could, for example, be specified in an HTML table or in Javadoc format. For instance, here is an HTML table that contains the test description for a test in the example of the JavaTest harness in use:

   Title	Button constructor tests
   source	CtorTests.java
   executeClass	javasoft.sqe.tests.api.java.awt.Button.CtorTests
   keywords	runtime positive

 

2. Configure the Test Environment.

   The JavaTest harness requires configuration information in order to run a test suite. This information identifies the test environment data such as the name and IP address of the machine in which the test suite is to run, and test parameters such as which tests in the test suite to run. The harness uses a Configuration Editor to get the configuration information it needs. A JavaTest harness user can open the Configuration Editor explicitly, as shown in the previous example. Also, the harness automatically opens the Configuration Editor if a user requests a test run but more configuration information is needed for the test. The harness puts the configuration information in a .jti file in the work directory.

    

3. Start the Agent Monitor.

   The Agent Monitor window is used to set the JavaTest agent pool for active agents. An active agent initiates a connection with the JavaTest harness (an agent can also be passive, in which case, it waits for the harness to initiate a connection). However before an active agent can initiate a connection with the JavaTest harness, the JavaTest agent pool needs to be set. The agent pool is a list of agents that are available to run tests. When an active agent connects to the harness, the agent is added to the agent pool.

   When the Agent Monitor window opens, it displays a Port value of 1907, and a Timeout value of 180. Port 1907 is the default port used by active agents. The Timeout value indicates the number of seconds that the harness waits for an active agent to be added to the pool if no agents are currently in the pool; the default value is 180 seconds.

   As mentioned above, clicking the Listening checkbox tells the JavaTest harness to start listening for active agents that want to connect.

 

4. Start the JavaTest Agent.

   The class com.sun.javatest.agent.AgentMain starts the agent. The command line parameter -activeHost sets the host for active connections. For further information about agents, see The JavaTest Agent.

 

5. Run Tests.

   The TestSuite object creates a test script for each test. The test script is an object that runs each test based on the configuration information and the test description. The JavaTest harness creates a TestResult object for each test. When each test is complete, the test script puts the test results in the TestResult object for the test. The harness then updates the graphical user interface to indicate the execution result status of the test. As each test completes, the harness stores results in a .jtr file in the work directory -- one file for each test. The file Button_Ctor.jtr is an example of a .jtr file (in fact, it's the file that contains the detailed report that was illustrated in the Run Tests step in the example of the JavaTest harness in use).

   A test script can run a test by itself or use commands to run all or part of a test. Whether a test script runs a test by itself or uses commands depends on how the script is designed by the TCK provider. The JavaTest harness provides a set of commands that can be used in a test script. These commands can be used to do things such as execute a test in a virtual machine that is separate from the virtual machine in which the JavaTest harness or the JavaTest Agent runs. In addition, the JavaTest harness offers a way for a TCK provider to create custom commands for even more flexible test execution.

JavaTest Harness Features

The example of the JavaTest harness in use illustrates a number of important JavaTest harness features:

• It presents a graphical user interface.
• It includes a Configuration Editor that takes an interview approach to obtain information about the test configuration.
• It can run tests on a system different than the one running the JavaTest harness -- it uses an agent, the JavaTest Agent, to do that.
• It tracks the progress of a test run, and displays status dynamically.
• It captures the results of a test run in multiple reports.

Perhaps even more important than these features is the fact that the JavaTest harness is flexible. For example, the questions presented by the Configuration Editor can be customized to meet specific needs. Or the way the harness finds and runs tests can be customized. Let's examine these features and the JavaTest harness's flexibility.

Presents a Graphical User Interface

As shown in the example of the JavaTest harness in use, the JavaTest harness provides a graphical user interface that consists of various windows and menus to configure, run, and track tests.

Click to enlarge

The primary window in the graphical interface is the Test Manager window. Through menu selections in this window, a JavaTest harness user can start the configuration editor, start a test run, or view information about the test run.

The left pane of the Test Manager window is the test tree pane. It presents a tree of colored icons that indicate the status of tests in the test run. Clicking on a folder icon in the test tree provides additional information about that folder and the tests in that folder. Clicking on a test icon provides additional information about that test. The additional information for a folder or a test is presented in a folder pane or test pane, respectively. The folder pane or test pane is the right pane of the Test Manager window, and is itself comprised of multiple panes.

In addition, the test manager window presents a test progress display at the bottom of the GUI.

Click to enlarge

Another window in the GUI is the Agent Monitor window. It displays the agent pool and the agents that are currently in use. As described in the Start the Agent Monitor section of "What's Really Happening Here?", the agent pool is a list of agents that are available to run tests.

The Agent Monitor window also presents a checkbox labeled Listening, and two fields labeled Port and Timeout. Clicking the Listening checkbox directs the JavaTest harness to start listening for active agents that want to connect. The Port field displays the port used by active agents to connect to the harness. The Timeout field displays the number of seconds the harness waits for an agent to join the agent pool (if the pool is empty).

The JavaTest harness also provides a command-line interface. The command-line interface gives users the option of running the harness in batch mode, as well as taking other actions such as setting environment variables for subsequently-created test environments. The command-line interface requests are specified in command-line options, most of which are entered when a user starts the JavaTest harness. For example, the following command starts the JavaTest harness in batch mode and runs the same tests, and in the same test environment, as in the previous example of the JavaTest harness in use:

java -jar javatest.jar \
-agentPoolPort 1907 -agentPoolTimeout 180 -startAgentPool \
-batch /home/ed/JavaTestDemo/workdir/jtData/default.jti 

The -batch option starts the JavaTest harness and executes the tests in batch mode. Notice the default.jti file specified with the -batch option. This file is a configuration file. It contains information about the computing environment in which the tests are run. It is produced by the JavaTest harness in the work directory as a result of the Configuration Editor interview. In other words, before a user can run the harness in batch mode, a configuration file for the test run must already exist, that is, the Configuration Editor must have been run previously to create the configuration file.

The -agentPoolPort, -agentPoolTimeout, and -startAgentPool options are used to configure the JavaTest harness to use agents. -agentPoolPort specifies the port number for the active agent pool. This pool holds requests from active agents until the requests can be processed. If you don't specify this option, the default port for active agents, port 1907, is used. -agentPoolTimeout specifies the timeout interval for the active agent pool. If you don't specify this option, the default timeout interval, 180 seconds, is used. Notice that the port number and timeout interval are the same items of information as shown in the Agent Monitor window. -startAgentPool starts the active agent pool.

In order for the previous command to run, the user needs to start the JavaTest Agent in the same way as in the previous example of the JavaTest Harness in use.

The result of the test run in batch mode should look something like this:

test results: passed: 26 failed: 1
Report written to 
    /home/ed/JavaTestDemo/workdir/reports/report.html.
Results written to /home/ed/JavaTestDemo/workdir.
Error: Some tests did not pass

Notice that reports and test results are placed in the work directory.

Takes an Interview Approach to Obtain Test Configuration Information

The Configuration Editor takes an interview approach to obtain information about the test environment, for example, the name of the system in which the tests are run. In essence, the Configuration Editor is used by the JavaTest harness to "fill in the blanks." The harness uses the configuration information that is already available to it. If the harness needs additional information about the test environment, it prompts the user to start the Configuration Editor. A user can also start the Configuration Editor to change configuration information.

It's up to the TCK provider to build a Configuration Editor interview for the TCK. In other words, the TCK provider develops the questions for the interview and determines the order in which specific questions are asked.

The JavaTest harness provides a number of packages as part of its public API. Two of the packages, com.sun.javatest and com.sun.interview, include classes for building interviews. The classes are:

• com.sun.interview.Interview. The base class for an interview.
• com.sun.interview.Question. Provides the content, that is, the text and optional graphics, for a question in an interview. This class also provides a place to store the user's response to the question. Instances of the Question class constitute the primary elements of the Interview class. The Question class has various subclasses, each of which is intended for a particular type of question and response. For example, the subclass StringQuestion provides for a question whose response is a string.
• com.sun.javatest.InterviewParameters. Collects the information provided in the interview and makes it available to the JavaTest harness.

To create an interview, you create a class that extends the com.sun.javatest.InterviewParameters class, and then add questions to the interview. For example, here is a class, DemoTCKParameters.java, that is the base for the Configuration Editor interview illustrated in the previous example.

Notice the statement:

setFirstQuestion(qIntro);

It identifies the first question for the interview -- in this case qIntro. Later in DemoTCKParameters.java, the question is created:

private Question qIntro = new NullQuestion(this, "intro") ...

NullQuestion is a subclass of com.sun.interview.Question. The subclass provides for a question that does not have a response -- in this case the "Welcome" prompt for the Configuration Editor.

Each question nominates the next question to be asked, either as a fixed successor or based on the response to an earlier question. In the case of NullQuestion, it nominates parameter questions to be asked next:

protected Question getNext() {
           return callInterview(
                          iEnvironment, qParameters);
        }
    };
    
// Parameter questions

    private Question qParameters = 
               new NullQuestion(this, "parameters") {
        protected Question getNext() {
          return callInterview(
           iInitialFiles, getInitialFilesSuccessor());
        }
    };

An interview can contain subinterviews. This makes the interview more manageable, especially for an interview that includes a large number of questions. For example, here is a subinterview of the Configuration Editor interview in the previous example:

Click to enlarge

Click to enlarge

Here are two classes that implement the subinterview:

• DemoTCKEnvInterview.java. This is the base class for the subinterview. It extends the com.sun.interview.Interview class.
• EnvironmentInterview.java. This class extends the base class for the subinterview. It adds questions to the subinterview as instances of the com.sun.interview.Question class or its subclasses.

As is the case in DemoTCKParameters.java, DemoTCKEnvInterview.java identifies the first question for the interview with the setFirstQuestion method. In this case, the call is to qEnvName. Later in DemoTCKEnvInterview.java, the question is created:

private StringQuestion qEnvName = 
                new StringQuestion(this, "envName") ...

StringQuestion is a subclass of the com.sun.interview.Question class. It provides for a question whose response is a string. The constructor identifies the interview for the question (in this case, the current interview), and a tag for the question (here, "envName").

The following statements get the response to the question, and then set the value in the StringQuestion object.

String getName() {
        return qEnvName.getValue();
    }

    void setName(String name) {
        qEnvName.setValue(name);
    } 

After testing that the user provides a valid response to the first question, DemoTCKEnvInterview.java gets the next question:

protected Question getNext() ...

DemoTCKEnvInterview.java then goes through the same process of getting the response to the question, and setting the value in the StringQuestion object. It then returns to the main interview.

Can Run Test Cases on a Different System

One of the characteristics of the JavaTest harness's flexibility is its ability to run the test suite on a different platform than the one in which the test harness runs. This allows the harness to run the test suite on the intended platform for the implementation. For example, an implementation of the Connected Limited Device Configuration (CLDC) of Java 2 Platform, Micro Edition (J2ME) is designed for small, mobile devices such as pagers and cell phones. The JavaTest harness, by comparison, runs on a PC or workstation with J2SE. The JavaTest harness is designed so that the test suite for the CLDC implementation can run in a device such as a cell phone with J2ME, while the harness runs on a PC or workstation with J2SE.

What provides for this remote execution is an agent, that is, an application that can execute tests on a platform other than the one in which the JavaTest harness runs. The JavaTest harness makes things easy for a TCK provider by including an agent, called the JavaTest Agent. A TCK provider can choose to use the JavaTest Agent, or create a customized agent. Like all agents, the JavaTest Agent executes tests on the targeted platform. After executing the tests, the JavaTest Agent returns the test results to the harness. (Although the discussion of agents has been about the ability to execute a test suite on a platform different than the one that runs the JavaTest harness, the JavaTest Agent and the harness can, in fact, run on the same machine, although as separate processes.) The JavaTest Agent communicates with the harness using a stream connection, and supports TCP/IP and RS-232 communication protocols. Other types of serial connections can be used, such as infrared or firewire.

The part of the JavaTest harness that communicates with the JavaTest Agent is the test script. As described in What's Really Happening Here?, a test script is an object that runs a test based on the test's configuration information and the test description. A test script communicates with the JavaTest Agent through commands. These commands are part of a set of "standard commands" provided by the JavaTest harness to configure an environment for running a test. Here's an example of a test script that uses the standard command, ActiveAgentCommand to delegate a subcommand to an active JavaTest Agent -- in response, the JavaTest Agent runs the tests in javasoft.sqe.tests.api.java.awt.Button.CtorTests.

string[] args =("CLASSPATH=" + testdir + "/classes", 
           "javasoft.sqe.tests.api.java.awt.Button", 
                                       "CtorTests");
PrintWriter err = ... // create error message stream
PrintWriter out = ... // create output message stream
AgentCommand agtcmd = new ActiveAgentCommand();
Status s = agtcmd.run(args, err, out);

ActiveAgentCommand is in the package com.sun.javatest.agent. This package contains interfaces and classes pertinent to the JavaTest Agent.

Tracks the Progress of a Test Run

After a test run begins, the JavaTest harness tracks the test's progress. Specifically, the test script stores information about the test's progress in the TestResult object for the test. The harness then takes the information in the TestResult object and displays it dynamically in the graphical user interface. The JavaTest harness presents a comprehensive set of information:

In addition, the following information is available as part of a multi-pane display when a user clicks on a test in the test tree.

Captures the Results of a Test Run in Multiple Reports

After a test run is completed, the JavaTest harness creates multiple reports about the run. The harness creates a summary report of the test run. It also creates a detailed report for each test in a file named testname.jtr, where testname is the name of the test. For example, the file api/java_awt/Button/index_Ctor.jtr in the work directory contains the detailed test results for the test api/java_awt/Button/index_Ctor.

In addition, the JavaTest harness produces a variety of reports in HTML format, and puts these reports in a report subdirectory of the work directory. These reports include:

• Summary of the Configuration Editor dialog
• Environment data
• Tests that had errors
• Tests in the test suite that were excluded in the test run
• Tests that failed
• Tests that were not run
• Tests that passed

The JavaTest Harness: A Significant Tool for Managing Tests

With its graphical user interface, interactive, interview-based configuration editor, and its multiple in-progress and post-test-run reports, the JavaTest harness simplifies the testing of an implementation for compliance with a Java technology. And because it can be customized in various ways, the JavaTest harness is an attractive choice for TCK providers.

For More Information

The Java Community Process Program

The Java Community Process - Community Resources - TCK Tools & Info

About the Author

Ed Ort is a staff member of the Java Developer Connection. He has written extensively about relational database technology and programming languages.