|
Articles Index
Ed Ort
October 2001
The Java Compatibility Test Tools (Java CTT)
are designed to help a specification lead, or other Java Community
Process Program (JCP) member
serving on an Expert Group, create a Technology Compatibility Kit (TCK).
The JCP is an inclusive, consensus-building approach used by the international Java community
to develop and evolve Java technology. If you are unfamiliar with the JCP,
see the Java Community Process Program.
A recent article,
The Java Compatibility Test Tools: Spec Trac,
described what a TCK is, introduced the tools in the Java CTT, and described, in general, how
each tool helps in creating a TCK. The article then covered, in detail, how to use
one of the Java CTT tools, Spec Trac.
This article covers another Java CTT tool: the
JavaTest harness.
The article is intended primarily for TCK providers. TCK providers need to incorporate
a test harness in the TCKs they provide, that is, a test harness customized to
meet the requirements of the TCK. They are also responsible for describing how that
customized test harness is used. This article describes the JavaTest harness, with
a focus on its flexibility -- particularly how it can be customized. The article
also presents an example of the JavaTest harness in use. This is only one possible way of
using the JavaTest harness, although it can serve as a model of the type of information
that a TCK provider might provide in TCK user documentation. It is ultimately up to the
TCK provider to describe how to use the test harness provided in the TCK.
What is the JavaTest Harness?
The JavaTest harness is a tool that manages a set of tests.
A test harness is one of the components that the JCP requires for the final release
of a Java technology.
The other required components are:
- TCK test suite. A TCK is used to test an implementation for compliance with
a Java technology specification. This is performed by running tests in the TCK test suite
with the implementation on the appropriate platform.
- Documentation that describes how to use the TCK, for example, how to run the tests
in the test suite.
The JavaTest harness manages the TCK-based tests. It automatically loads tests in the
TCK test suite, helps configure the user's environment to run tests, runs tests, and
reports test results. Let's look at an example.
|
The examples in this article illustrate the use of JavaTest harness Release 3.0.1 in the
Solaris Operating Environment. JavaTest harness 3.0.1
runs on any platform that is compatible with
Java 2 Standard Edition (J2SE), version 1.3,
such as compatible Linux and Windows environments.
|
An Example of the JavaTest Harness in Use
Imagine that you are the specification lead for a Java technology specification
that is developed in accordance with the JCP. Imagine too that the Java technology comprises
a very simple subset of the J2SE APIs. Say the specification includes a number of AWT classes,
as follows:
java.awt.AWTEventMulticaster
java.awt.Button
java.awt.Label
java.awt.MenuShortCut
java.awt.Polygon
java.awt.Scrollbar
and the networking class java.net.InetAddress.
As the spec lead, you're responsible for providing a TCK for the Java technology.
Your TCK includes a test suite that can be used to test implementations
of the technology for conformance to the specification. For example, the test suite
includes button constructor tests that determine whether
the constructors for the java.awt.Button class perform as specified when
used with an implementation of the technology.
The TCK also includes the JavaTest harness to manage the test suite, and documentation,
in the form of a User's Guide, that describes how to install and run the JavaTest harness and the test suite.
After describing how to install the harness and the test suite, the User's Guide presents the
following instructions.
The following steps reflect only one possible way
of using the JavaTest harness. For example, starting an agent is not required in many cases.
Also, the examples below occasionally reference a "Demo TCK Test Suite"
and a "demotck.jar" file. The referenced names are hypothetical -- they
should not be interpreted as indicating that a demonstration test suite is distributed with the
JavaTest harness.
Start the JavaTest Harness
- Make the directory where the JavaTest harness is installed the current directory.
- At the command prompt enter:
java -jar javatest.jar
|
Click to enlarge
|
The JavaTest harness starts. It displays the test manager window and prompts you
to create a work directory. The harness uses the work directory to store
information about the test environment and to write test results.
Configure the Test Environment
Use the Configuration Editor to configure the information required to run
the test suite.
- Choose Configure->Configuration Editor from the Test Manager menu bar.
|
Click to enlarge
|
The JavaTest harness opens the Configuration Editor. The Configuration Editor consists
of three panes and a menu bar.
- Answer the questions in the Configuration Editor.
Start the Agent Monitor
Use the Agent Monitor to establish and monitor the connection to the agent.
See Start the JavaTest Agent for a description of what an agent
is.
- Choose Tasks->Monitor Agent Activity from the Test Manager menu bar.
|
Click to enlarge
|
This adds a new Agent Monitor window to the display.
- Click the Listening checkbox in the upper lefthand corner of the Agent Monitor.
The JavaTest harness starts listening for active agents that want to connect.
Start the JavaTest Agent
The JavaTest Agent (referred to simply as the "agent") is used to run tests on
a different system than the one in which the JavaTest harness runs. (However the agent can
also run as a separate process in the system in which the JavaTest harness runs.)
The agent and the harness communicate with each other over the network.
- (Note: If you run the agent on the same machine as the Javatest harness, you can omit
this step.) Copy the JAR files for the JavaTest harness (
javatest.jar)
and the TCK test suite (demotck.jar) to the system on which the agent will run.
- Start the agent.
Make the directory that contains the JavaTest harness and the TCK test suite the
current directory and use the following command:
java -cp javatest.jar:demotck.jar:dir1/TCK/classes \
com.sun.javatest.agent.AgentMain -activeHost system1
Replace dir1 with the root directory below which the TCK test suite classes are installed,
and System1 with the network name of the system running the JavaTest harness.
If you run the agent on the same system as the JavaTest harness, use the following command
to start the agent:
java -cp javatest.jar:demotck.jar:classes \
com.sun.javatest.agent.AgentMain -activeHost system1
- Check the JavaTest harness Agent Monitor to verify the connection.
The agent should be listed in the Agent Pool.
Run Tests
-
Choose Run Tests->Start from the Test Manager menu bar.
The JavaTest harness presents you with a Confirm Configuration dialog. Click OK.
|
Click to enlarge
|
The test suite begins to run. You should see activity in the test tree panel
that indicates which tests are running. Expand the test tree folders to reveal
the tests. As tests complete, the tests and their folders change color to represent
their state. Click on folders and tests to see information about them in the
test manager main panel.
After the tests complete, you can also view reports in the work directory.
The reports include a test summary that lists the tests
and their execution result status (pass, fail, error); and
detailed reports for specific tests.
What's Really Happening Here?
Let's put the example of the JavaTest harness in use under the microscope
and see what's going on beneath the surface:
- Start the JavaTest Harness.
In response, the JavaTest harness looks for a property file named testsuite.jtt
in the directory in which the harness is started. If it can't find this file, the harness prompts
the user to identify the root directory of the test suite.
At a minimum, the property file identifies
the name of the test suite, any arguments it requires, and the class path for the test suite.
For example, here's the testsuite.jtt file
for the previous example. In addition, the harness looks for a work directory associated
with the test suite. If it can't find the work directory, the harness prompts the user to create one.
The JavaTest harness then creates a TestSuite object and passes it any additional information
about the test suite that is specified in the property file. The harness uses the
TestSuite object to choose a test finder. The test finder is an object
that finds and reads test descriptions for the tests, and creates
a TestDescription object for each test. The test tree (that is, the suite root and
the tests below it) is then displayed in the test tree panel of the test manager window.
As its name implies, a test description describes a test. More specifically it
identifies important information about the test, for instance, what class
to execute for the test. The JavaTest harness requires that the test description be specified as a set of
name-value pairs. What the names and values are depends on the test suite.
Also, the format in which the names and values are presented depends on
the test finder that is used. The name-value pairs could, for example, be specified
in an HTML table or in Javadoc format. For instance, here is an HTML table that contains
the test description for a test in the example of the JavaTest harness in use:
| Title |
Button constructor tests |
source |
CtorTests.java |
| executeClass |
javasoft.sqe.tests.api.java.awt.Button.CtorTests |
| keywords |
runtime positive |
- Configure the Test Environment.
The JavaTest harness requires configuration information in order to run a test suite. This
information identifies the test environment data such as the name and IP address of the machine
in which the test suite is to run, and test parameters such as which tests in the test suite to run.
The harness uses a Configuration Editor to get the configuration information
it needs. A JavaTest harness user can open the Configuration Editor explicitly, as shown in the
previous example. Also, the harness automatically opens the Configuration Editor
if a user requests a test run but more configuration information is needed for the test.
The harness puts the configuration information in a .jti file in the work directory.
- Start the Agent Monitor.
The Agent Monitor window is used to set the JavaTest agent pool for active agents.
An active agent initiates a connection with the JavaTest harness (an agent can also be passive,
in which case, it waits for the harness to initiate a connection). However before
an active agent can initiate a connection with the JavaTest harness, the JavaTest
agent pool needs to be set. The agent pool is a list of agents that are available
to run tests. When an active agent connects to the harness, the agent is added to the agent pool.
When the Agent Monitor window opens, it displays a Port value of 1907, and a Timeout
value of 180. Port 1907 is the default port used by active agents. The Timeout
value indicates the number of seconds that the harness waits for an active agent to be added
to the pool if no agents are currently in the pool; the default value is 180 seconds.
As mentioned above, clicking the Listening checkbox tells the JavaTest harness to start listening for
active agents that want to connect.
- Start the JavaTest Agent.
The class com.sun.javatest.agent.AgentMain starts the agent. The command
line parameter -activeHost sets the host for active connections.
For further information about agents, see The JavaTest Agent.
- Run Tests.
The TestSuite object creates a test script for each test. The test script is an object
that runs each test based on the configuration information and the test description.
The JavaTest harness creates a TestResult object for each test.
When each test is complete, the test script puts the test results in the
TestResult object for the test. The harness then updates the graphical user
interface to indicate the execution result status of the test.
As each test completes, the harness stores results in a .jtr
file in the work directory -- one file for each test. The file
Button_Ctor.jtr
is an example of a .jtr file (in fact, it's the file that contains the detailed report
that was illustrated in the Run Tests step in the example of the JavaTest harness in use).
A test script can run a test by itself or use commands to run all or part of a test.
Whether a test script runs a test by itself or uses commands depends on
how the script is designed by the TCK provider. The JavaTest harness provides a set of commands
that can be used in a test script. These commands can be used to do things such as execute
a test in a virtual machine that is separate from the virtual machine in which the JavaTest harness
or the JavaTest Agent runs. In addition, the JavaTest harness offers a way for a TCK provider
to create custom commands for even more flexible test execution.
JavaTest Harness Features
The example of the JavaTest harness in use illustrates a number of
important JavaTest harness features:
- It presents a graphical user interface.
- It includes a Configuration Editor that takes an interview
approach to obtain information about the test configuration.
- It can run tests on a system different than the one running the JavaTest harness -- it uses
an agent, the JavaTest Agent, to do that.
- It tracks the progress of a test run, and displays status dynamically.
- It captures the results of a test run in multiple reports.
Perhaps even more important than these features is the fact that the JavaTest harness is
flexible. For example, the questions presented by the Configuration
Editor can be customized to meet specific needs. Or the way the harness finds and
runs tests can be customized. Let's examine these features and the JavaTest harness's flexibility.
Presents a Graphical User Interface
As shown in the example of the JavaTest harness in use,
the JavaTest harness provides a graphical user interface that consists of various windows
and menus to configure, run, and track tests.
|
|
Click to enlarge
|
The primary window in the graphical interface is the Test Manager window. Through menu
selections in this window, a JavaTest harness user can start the configuration editor,
start a test run, or view information about the test run.
The left pane of the Test Manager window is the test tree pane.
It presents a tree of colored icons that indicate the status of tests in the test run.
Clicking on a folder icon in the test tree provides additional information
about that folder and the tests in that folder. Clicking on a test icon provides additional information
about that test. The additional information for a folder or a test is
presented in a folder pane or test pane, respectively.
The folder pane or test pane is the right pane of the Test Manager window,
and is itself comprised of multiple panes.
In addition, the test manager window presents a test progress display
at the bottom of the GUI.
|
|
Click to enlarge
|
Another window in the GUI is the Agent Monitor window. It displays the agent pool
and the agents that are currently in use.
As described in the Start the Agent Monitor section of
"What's Really Happening Here?", the agent pool is a list of agents that
are available to run tests.
The Agent Monitor window also presents a checkbox labeled Listening,
and two fields labeled Port and Timeout.
Clicking the Listening checkbox directs the JavaTest harness to start listening for
active agents that want to connect. The Port field displays the port used by active agents
to connect to the harness. The Timeout field displays the number of seconds the harness
waits for an agent to join the agent pool (if the pool is empty).
The JavaTest harness also provides a command-line interface. The command-line interface
gives users the option of running the harness in batch mode, as well as taking other actions such as
setting environment variables for subsequently-created test environments.
The command-line interface requests are specified in command-line options,
most of which are entered when a user starts the JavaTest harness.
For example, the following command starts the JavaTest harness in batch mode and runs the
same tests, and in the same test environment, as in the previous example of the JavaTest harness
in use:
java -jar javatest.jar \
-agentPoolPort 1907 -agentPoolTimeout 180 -startAgentPool \
-batch /home/ed/JavaTestDemo/workdir/jtData/default.jti
The -batch option starts the JavaTest harness and executes the tests in batch mode.
Notice the default.jti file specified with the -batch option.
This file is a configuration file. It contains information about the computing environment
in which the tests are run. It is produced by the JavaTest harness in the work directory
as a result of the Configuration Editor interview.
In other words, before a user can run the harness in batch mode, a configuration
file for the test run must already exist, that is, the Configuration Editor
must have been run previously to create the configuration file.
The -agentPoolPort, -agentPoolTimeout, and
-startAgentPool options are used to configure the JavaTest harness to use agents.
-agentPoolPort specifies the port number for the active agent pool.
This pool holds requests from active agents until the requests can be processed.
If you don't specify this option, the default port for active agents, port 1907, is used.
-agentPoolTimeout specifies the timeout interval for the active agent pool.
If you don't specify this option, the default timeout interval, 180 seconds, is used.
Notice that the port number and timeout interval are the same items of information
as shown in the Agent Monitor window.
-startAgentPool starts the active agent pool.
In order for the previous command to run, the user needs to
start the JavaTest Agent in the same way as in
the previous example of the JavaTest Harness in use.
The result of the test run in batch mode should look something like this:
test results: passed: 26 failed: 1
Report written to
/home/ed/JavaTestDemo/workdir/reports/report.html.
Results written to /home/ed/JavaTestDemo/workdir.
Error: Some tests did not pass
|
Notice that reports and test results are placed in the work directory.
Takes an Interview Approach to Obtain Test Configuration Information
The Configuration Editor takes an interview approach to obtain information about the
test environment, for example, the name of the system in which the tests are run.
In essence, the Configuration Editor is used by
the JavaTest harness to "fill in the blanks." The harness uses the configuration information
that is already available to it. If the harness needs additional information about the test
environment, it prompts the user to start the Configuration Editor. A user can also start the
Configuration Editor to change configuration information.
It's up to the TCK provider to build a Configuration Editor interview for the TCK.
In other words, the TCK provider develops the questions for the interview and determines
the order in which specific questions are asked.
The JavaTest harness provides a number of packages as part of its public API. Two of the packages,
com.sun.javatest and com.sun.interview, include classes for
building interviews. The classes are:
-
com.sun.interview.Interview.
The base class for an interview.
-
com.sun.interview.Question.
Provides the content, that is, the text and optional graphics, for a question in an interview.
This class also provides a place to store the user's response to the question.
Instances of the Question class constitute the primary elements of the Interview class.
The Question class has various subclasses, each of which is intended for a particular type of
question and response. For example, the subclass StringQuestion provides for a question
whose response is a string.
-
com.sun.javatest.InterviewParameters.
Collects the information provided in the interview and makes it available to the
JavaTest harness.
To create an interview, you create a class that extends the
com.sun.javatest.InterviewParameters class, and then add
questions to the interview. For example, here is a class,
DemoTCKParameters.java,
that is the base for the Configuration Editor interview illustrated in the
previous example.
Notice the statement:
setFirstQuestion(qIntro);
It identifies the first question for the interview --
in this case qIntro. Later in DemoTCKParameters.java,
the question is created:
private Question qIntro = new NullQuestion(this, "intro") ...
NullQuestion is a subclass of com.sun.interview.Question.
The subclass provides for a question that does
not have a response -- in this case the "Welcome" prompt for the Configuration Editor.
Each question nominates the next question to be asked, either as a fixed successor
or based on the response to an earlier question. In the case of NullQuestion,
it nominates parameter questions to be asked next:
protected Question getNext() {
return callInterview(
iEnvironment, qParameters);
}
};
// Parameter questions
private Question qParameters =
new NullQuestion(this, "parameters") {
protected Question getNext() {
return callInterview(
iInitialFiles, getInitialFilesSuccessor());
}
};
|
An interview can contain subinterviews. This makes the interview more manageable, especially
for an interview that includes a large number of questions. For example, here is a subinterview
of the Configuration Editor interview in the previous example:
|
Click to enlarge
|
|
|
|
Click to enlarge
|
Here are two classes that implement the subinterview:
DemoTCKEnvInterview.java.
This is the base class for the subinterview. It extends the com.sun.interview.Interview
class.
EnvironmentInterview.java. This class extends
the base class for the subinterview. It adds questions to the subinterview as instances of the
com.sun.interview.Question class or its subclasses.
As is the case in DemoTCKParameters.java, DemoTCKEnvInterview.java
identifies the first question for the interview with the setFirstQuestion method.
In this case, the call is to qEnvName. Later in DemoTCKEnvInterview.java,
the question is created:
private StringQuestion qEnvName =
new StringQuestion(this, "envName") ...
StringQuestion is a subclass of the com.sun.interview.Question
class. It provides for a question whose response is a string. The constructor identifies
the interview for the question (in this case, the current interview), and a tag
for the question (here, "envName").
The following statements get the response to the question,
and then set the value in the StringQuestion object.
String getName() {
return qEnvName.getValue();
}
void setName(String name) {
qEnvName.setValue(name);
}
|
After testing that the user provides a valid response to the first question,
DemoTCKEnvInterview.java gets the next question:
protected Question getNext() ...
DemoTCKEnvInterview.java then goes through the same process of
getting the response to the question, and setting the value in the
StringQuestion object. It then returns to the main interview.
Can Run Test Cases on a Different System
One of the characteristics of the JavaTest harness's flexibility is its ability to run
the test suite on a different platform than the one in which the test harness runs.
This allows the harness to run the test suite on the intended platform for the implementation.
For example, an implementation of the Connected Limited Device Configuration (CLDC)
of Java 2 Platform, Micro Edition
(J2ME) is designed for small, mobile devices
such as pagers and cell phones. The JavaTest harness, by comparison, runs on a PC or
workstation with J2SE. The JavaTest harness is designed
so that the test suite for the CLDC implementation can run in a device such as a cell phone
with J2ME, while the harness runs on a PC or workstation with J2SE.
|
What provides for this remote execution is an agent, that is, an application
that can execute tests on a platform other than the one in which the JavaTest harness runs.
The JavaTest harness makes things easy for a TCK provider by including an agent, called
the JavaTest Agent. A TCK provider can choose to use the JavaTest Agent, or
create a customized agent. Like all agents, the JavaTest Agent executes tests
on the targeted platform. After executing the tests, the JavaTest Agent returns
the test results to the harness. (Although the discussion of agents has been about
the ability to execute a test suite on a platform different than the one that runs the
JavaTest harness, the JavaTest Agent and the harness can, in fact, run on the same machine,
although as separate processes.) The JavaTest Agent communicates with the harness using
a stream connection, and supports TCP/IP and RS-232 communication protocols. Other types
of serial connections can be used, such as infrared or firewire.
The part of the JavaTest harness that communicates with the JavaTest Agent is the test script.
As described in What's Really Happening Here?, a test script is an
object that runs a test based on the test's configuration information and the test description.
A test script communicates with the JavaTest Agent through commands. These commands
are part of a set of "standard commands" provided by the JavaTest harness to
configure an environment for running a test. Here's an example of a test script
that uses the standard command, ActiveAgentCommand to
delegate a subcommand to an active JavaTest Agent -- in response, the JavaTest Agent
runs the tests in javasoft.sqe.tests.api.java.awt.Button.CtorTests.
string[] args =("CLASSPATH=" + testdir + "/classes",
"javasoft.sqe.tests.api.java.awt.Button",
"CtorTests");
PrintWriter err = ... // create error message stream
PrintWriter out = ... // create output message stream
AgentCommand agtcmd = new ActiveAgentCommand();
Status s = agtcmd.run(args, err, out);
|
ActiveAgentCommand is in the package com.sun.javatest.agent.
This package contains interfaces and classes pertinent to the JavaTest Agent.
Tracks the Progress of a Test Run
After a test run begins, the JavaTest harness tracks the test's progress. Specifically, the
test script stores information about the test's progress in the TestResult
object for the test. The harness then takes the information in the TestResult
object and displays it dynamically in the graphical user interface. The JavaTest harness presents
a comprehensive set of information:
| Type of information |
Where displayed |
|
Current test, progress indicator |
Test progress display at the bottom of the GUI. The display contains two fields.
One field displays the name of the test being run. Depending on user
choice, the other field displays either a progress indicator bar or elapsed time statistics. |
|
Test status, elapsed time, memory usage, current test |
Progress Monitor dialog box.
This dialog box presents the number of tests that passed, failed, encountered errors,
or did not run; a progress indicator bar; the elapsed time of the test run and the estimated time
remaining; memory usage statistics and available memory; and the name of the test currently
being run.
|
|
Test folder status, individual test status |
Test tree pane. Presents a tree of colored icons
that indicate the status of test folders and individual tests. The colors and their status
indications are: green (passed), red (failed), blue (error encountered), white (not run),
and gray (filtered out, that is, the tests in the folder or an individual test has not been
selected for this test run).
|
In addition, the following information is available as part of a multi-pane display
when a user clicks on a test in the test tree.
| Type of information |
Where displayed |
|
Test description |
Test description pane. Presents a table of name-value pairs that
comprise the test description.
|
|
Source file contents |
File pane. Displays the contents of the source files from the
test description. A JavaTest harness user selects a file for display from a drop-down list.
|
|
Test run environment values |
Configuration pane. Displays environment values that were used
when a specific test was run.
|
|
Test run details |
Test Run Details pane. Displays detailed information about a test run,
for example, the source file for the test, the starting and ending time of the test run, and the
execution status.
|
|
Test run messages |
Test Run Messages pane. Displays messages from the test script
as well as messages produced by the test. This is normally the primary place to look
for error messages when a test fails.
|
Captures the Results of a Test Run in Multiple Reports
After a test run is completed, the JavaTest harness creates multiple reports about the run.
The harness creates a summary report of the test run.
It also creates a detailed report for each test in a file named testname.jtr,
where testname is the name of the test. For example, the file
api/java_awt/Button/index_Ctor.jtr
in the work directory contains the detailed test results for the test
api/java_awt/Button/index_Ctor.
In addition, the JavaTest harness produces a variety of reports in HTML format, and
puts these reports in a report subdirectory of the work directory. These reports
include:
The JavaTest Harness: A Significant Tool for Managing Tests
With its graphical user interface, interactive, interview-based configuration editor,
and its multiple in-progress and post-test-run reports, the JavaTest harness simplifies the
testing of an implementation for compliance with a Java technology. And because it
can be customized in various ways, the JavaTest harness is an attractive choice for TCK providers.
For More Information
The Java Community Process Program
The Java Community Process - Community Resources - TCK
Tools & Info
About the Author
Ed Ort is a staff member of the Java Developer Connection. He has written extensively
about relational database technology and programming languages.
|
Print-friendly Version
