The Java Compatibility Test Tools

Print-friendly Version

Articles Index

Recently Sun Microsystems released a set of tools called the Java Compatibility Test Tools (Java CTT). These tools are designed to help a specification lead, or other Java Community Process Program (JCP) member serving on an Expert Group, create a Technology Compatibility Kit (TCK). The JCP is an inclusive, consensus-building approach used by the international Java community to develop and evolve Java technology. If you are unfamiliar with the JCP, see the Java Community Process Program. This article gives you an overview of the Java CTT and generally describes how the tools can help JCP members create a TCK. It also focuses on one of the Java CTT tools, Spec Trac, and shows you how to use the tool. Future articles will show you how to use other Java CTT tools.

What's a Technology Compatibility Kit?

A Technology Compatibility Kit (TCK) is a suite of tests, tools, and documentation that provide a standard way of testing an implementation for compliance with a Java technology specification. It is one of the three major deliverables, along with a specification and reference implementation, that the JCP requires for the final release of a Java technology. The specification describes in detail a Java technology, for example, it describes the application programming interfaces (APIs) for the Java technology. The reference implementation (RI) is a working implementation of the Java technology to prove that the specification can be implemented. The TCK is used to test implementations for compliance with the specification.

The specification is the responsibility of an Expert Group put in place for the Java technology. During the Proposed Final Draft stage of the process, an Expert designated as the specification lead (or simply, Spec Lead) is responsible for the completion of both the RI and the TCK.

Each TCK is unique to its associated Java technology, however all TCKs must include the following:

• TCK test suite. This is the set of test cases that tests an implementation for compliance with a Java technology specification.
• Test harness. This is the set of applications and tools that manages the TCK test suite. Managing the TCK test suite includes tasks such as automatically loading and running test cases, and reporting test results.
• Documentation. This describes how to use the TCK, for example, how to use the test harness to run the TCK test suite. It also identifies the criteria for determining compliance with the specification, and how to appeal TCK tests or TCK results. The documentation is usually provided in the form of a User's Guide.

In addition, a TCK can include the description of an audit process, which might, for instance, require independent verification by a third party of the test results.

The Tools in the Java CTT

The Java CTT includes four tools:

• JavaTest Harness. As its name suggests, this is a test harness, and, in fact, is the test harness used by Sun Microsystems to manage TCK test suites. The Java CTT provides Release 3.0, the most current release of the JavaTest harness.
• Spec Trac. This tool identifies assertions made in a specification about an API. The tool also generates reports about test cases and assertions, such as, what percentage of assertions in a specification are covered by TCK test cases.
• Test Generation. This tool automates some of the steps required to build TCK test cases for use with the JavaTest harness. For example, it automatically incorporates commonly-used code segments into test case files.
• Sig Test. This tool determines if all the necessary API members are in the tested implementation, and checks that there are no additional members that illegally extend the API.

This article shows you how to use Spec Trac. Future articles will show you how to use other Java CTT tools.

Spec Trac -- A Tool for Tracking Assertions

As mentioned earlier, the Java CTT provides a test harness -- the JavaTest harness. This directly addresses one requirement of a TCK. But there are other aspects of building a TCK that the Java CTT simplifies. This is especially true in building a TCK test suite for use with the JavaTest harness. One of the Java CTT tools that plays an important role in building a TCK test suite is the Spec Trac tool. In particular, a TCK test suite developer can use Spec Trac to:

• Identify assertions in a specification
• Report on TCK test coverage of assertions

Using Spec Trac to Identify Assertions

TCK test suite developers need to analyze the documented specifications for an API to determine what needs to be tested. To do this, test suite developers identify assertions, statements that describe required characteristics of the API, in the specifications. "Required" means that any implementation of the API must also implement those characteristics. For example, the following statement appears in the JCP specification "J2ME Connected Device Configuration (JSR-36)". It describes a read method in the BufferedInputStream class.

public int read(byte[] b,
                int off,
                int len)
         throws IOException
         
Reads bytes from this byte-input stream into the 
specified byte array, starting at the given offset. 

The statement is an example of an assertion. An implementation of the Connected Device Configuration (CDC) profile must implement the read method of the BufferedInputStream class as described in the statement.

Not all assertions are testable, for example, it's not always possible to test assertions about abnormal termination. In addition, not all assertions are explicitly stated in a specification -- some are implied. Those assertions that are testable (whether explicitly stated or implied) form the basis of a TCK test suite. In other words, all the testable assertions in a specification should be tested by the TCK test suite for the technology.

The Spec Trac tool helps identify statements in a specification that are assertion candidates (it doesn't identify implied assertions). A TCK test suite developer needs to verify that the candidates are indeed assertions, and that they are testable. However because specifications can be large and complex, searching specifications for assertions can be a long and tedious process. Having a tool that builds a list of candidate assertions can save a TCK test suite developer a lot of time and effort.

Sun Microsystems recommends that the specification for a Java technology API consist of the Javadoc tag comments that are in the Java source files for the API, along with any other supporting documentation. By default, the Spec Trac tool is designed to handle specifications in this format; in fact, when you use the Spec Trac tool to identify candidate assertions, the tool runs the Javadoc tool. You can however customize the Spec Trac tool for use with other specification formats. What this means is that unless you customize it, Spec Trac expects as input source files that are acceptable to the Javadoc tool, for example, Java source files that contain Javadoc tag comments. The Spec Trac tool also expects the specification to have a tree-like structure where the specifications for a class are in a directory below the specifications for its package.

Let's use the Spec Trac tool to identify candidate assertions in the specification for the BufferedInputStream class in the java.io package of the CDC profile. The CDC specification is in Javadoc format.

Before running the Spec Trac tool, let's set up its operating environment. The Spec Trac tool requires Java 2 Platform, Standard Edition (J2SE) SDK v 1.3, and runs in any platform in which the J2SE v 1.3 SDK is available, such as the Solaris Operating Environment or the Windows NT platform.

All examples that illustrate the use of Spec Trac in this article are for the Solaris Operating Environment, with the backslash indicating continuation of a command on the next line.

Before you use the tool, you need to make sure that:

• The JDK 1.3 bin directory is specified in your path, for example:
  

  SETENV PATH /usr/jdk1.3/bin/:"$PATH"
  

• The spectrac.jar file, and the tools.jar file in the JDK 1.3 lib directory are in your class path, for example:
  

  SETENV CLASSPATH .:/usr/ctt_tools/spectrac/spectrac.jar:\
  /usr/jdk1.3/lib/tools.jar:"$CLASSPATH"
  

Now that the environment is set up, let's run the Spec Trac tool. Assume for the purposes of this example that the source file for the BufferedInputStream class is in directory usr/local/src. To get a list of candidate assertions, you use the Spec Trac tool command getassert, and specify the path to the source file in the -sourcepath option, followed by the fully-qualified name of the class:

java com.sun.tdk.spectrac.Main getassert \
-sourcepath /usr/local/src java.io.BufferedInputStream

The results are then displayed in the standard output stream.

Notice that each line of the result looks something like this:

<java.io|BufferedInputStream 99341f97> A BufferedInputStream ... 

Notice too that the result includes candidate assertions for the read method mentioned earlier, for example:

<java.io|BufferedInputStream.read(byte[],int,int) 23647791> 
Reads bytes from this byte-input stream into the ...

The information between the angle brackets is an assertion ID, comprised of a signature followed by a hexadecimal value. The signature identifies the package and the class, interface, method, or field associated with the assertion. In this example, the assertion ID <java.io|BufferedInputStream 99341f97> identifies an assertion for the BufferedInputStream class in the java.io package. The assertion ID is then followed by the assertion text, for example, A BufferedInputStream adds functionality ... Assertion IDs and assertion text are important in reporting TCK test case coverage, for example, what percentage of assertions are covered by test cases.

You can also use the getassert command to list candidate assertions for a list of classes, an entire package, a set of packages, or any combination of those. For example, the following command returns the candidate assertions for the java.io package and the javax.microedition.io package:

java com.sun.tdk.spectrac.Main getassert \
-sourcepath /usr/local/src java.io javax.microedition.io

java com.sun.tdk.spectrac.Main getassert \
-sourcepath /usr/local/src \ 
-precompilepath /usr/local/test/assertions \
java.io 

With the candidate assertions identified by the Spec Trac tool, a TCK test suite developer then verifies that the candidates are indeed assertions, and that they are testable. The test suite developer also needs to determine if any assertions need to be added (remember, the tool does not identify implied assertions). The next step is to create the test cases for the testable assertions. How to create test cases is beyond the scope of this article; for detailed information on how to create test cases for a TCK see the Java Technology Test Suite Development Guide. After a test suite developer creates the test cases, he or she can use the Test Generation tool provided in the Java Compatibility Test Tools to help turn the test cases into a TCK test suite for use with the JavaTest harness. The Test Generation tool will be covered in a future article.

Using Spec Trac to Report on TCK Test Case Coverage

After identifying the candidate assertions in a specification, and building a TCK test suite to test assertions, a TCK test suite developer can use the Spec Trac tool to report on the extent of test case coverage. For example, he or she can use the Spec Trac tool to determine which candidate assertions for a class are covered by test cases, and which are not.

In order to produce a report, the Spec Trac tool needs access to information that binds test cases to candidate assertions. A test suite developer needs to provide this binding in a file called a TCK information file. Each line in the TCK information file must be in the following format:

 
[testID] assertionID [assertion]

where testID is a test case identifier, assertionID is the assertion ID, and assertion is the text of the assertion. The test case identifier is any indicator that uniquely identifies the TCK test case. For example, if the test case is developed for use with the JavaTest harness, the test case identifier uniquely identifies a test case by conforming to the JavaTest format; this might be the name of an HTML file, such as (index.html), that contains test case descriptions, and a named reference, such as (#bisread[BufferedInputStream1001]) for a specific test case.

The assertion ID and assertion text are the same pieces of information that form each line of output produced by the getassert command. The testID and assertion fields are optional. If a test ID isn't specified, the Spec Trac tool assumes that the candidate assertion is not testable.

For example, assume that the test cases for the BufferedInputStream class are in directory /usr/local/test/java/io/BufferedInputStream. Here's what a TCK information file for the BufferedInputStream class might look like.

To report on test case coverage, use the Spec Trac tool command report. (Remember to set up the Spec Trac tool operating environment before you request reports.) In the command you specify the root directory beneath which the test cases and the TCK information file are located, the location of the assertions, and the classes and packages for which you want a report. You also specify one or more options that indicate what type of report you want. For example, the following command produces a summary report (requested by the -s option) of test case coverage for the BufferedInputStream class.

java com.sun.tdk.spectrac.Main report \
-s -workspace  /usr/local/test \
-precompilepath /usr/local/test/assertions \
java.io.BufferedInputStream

The result might look something like this:

SUMMARY COVERAGE REPORT

================================================
 API item                    | Coverage percent 
================================================
 java.io.BufferedInputStream |        98%
------------------------------------------------
                          Overall coverage: 98%
================================================

The results indicate that 98% of the candidate assertions identified in the Javadoc comments for the BufferedInputStream class are covered by test cases.

To get a list of candidate assertions that are not covered by test cases, specify the -u option:

java com.sun.tdk.spectrac.Main report \
-u -workspace  /usr/local/test \
-precompilepath /usr/local/test/assertions \
java.io.BufferedInputStream

The result might look something like this:

THE FOLLOWING ASSERTIONS ARE NOT TESTED


Assertion text:  If markpos is not -1, then all bytes 
from positions buf[markpos] through buf[pos-1] must 
remain in the buffer array (though they may be moved 
to another place in the buffer array, with suitable 
adjustments to the values of count, pos, and markpos); 
they may not be discarded unless and until the 
difference between pos and markpos exceeds marklimit.
Assertion id: java.io|BufferedInputStream.markpos cd26451b

Other options allow you to request a list of candidate assertions that are covered by test cases (-t); non-testable assertions (-n); or invalid assertions (-i) -- an assertion is invalid if it is in the TCK information file but not in the specification (perhaps because the specification was changed). For example, here is a request to list candidate assertions that are covered by test cases:

java com.sun.tdk.spectrac.Main report \
-t -workspace  /usr/local/test \
-precompilepath /usr/local/test/assertions \
java.io.BufferedInputStream

And here is the result (only the first two items in the list are shown):

THE FOLLOWING ASSERTIONS ARE TESTED

Assertion text: A BufferedInputStream adds functionality 
to another input stream-namely, the ability to buffer 
the input and to support the mark and reset methods.
Assertion id: java.io|BufferedInputStream 99341f97
Tested in the following tests: 
    /usr/local/test/java/io/BufferedInputStream/
    index.html#bis[BufferedInputStream1001]

Assertion text: When the BufferedInputStream is created, 
an internal buffer array is created.
Assertion id: java.io|BufferedInputStream 932c764c
Tested in the following tests: 
    /usr/local/test/java/io/BufferedInputStream/
    index.html#bis[BufferedInputStream1002]

Notice that in all the report examples, the test cases are in a directory that matches the fully-qualified name of the tested class. In other words, the test cases for the java.io.BufferedInputStream class are in java/io/BufferedInputStream below the root directory /usr/local/test. By default, the Spec Trac tool looks for test cases in that type of fully-qualified-name structure. It also assumes that the TCK information file is located in the same structure, has the same name as the associated package or class, and has an extension of coverage. So in these examples, the TCK information file is in the directory /usr/local/test/java/io/BufferedInputStream, and is named java.io.BufferedInputStream.coverage. You can however override the default, and direct Spec Trac to look for the test cases and TCK information file using a different scheme.

You can also couple in one command a request to build a list of candidate assertions, and report on test case coverage. For example, the following command builds a list of candidate assertions from the Javadoc specification of the BufferedInputStream class, and produces a summary report of test case coverage:

java com.sun.tdk.spectrac.Main report \
-s -workspace  /usr/local/test \
-compile \
-sourcepath /usr/local/src java.io.BufferedInputStream \
java.io.BufferedInputStream

And, as is the case for candidate assertions, you can direct a report to a file. For example, the following command lists the assertions that are covered by test cases. The list will be stored in a file in directory /usr/local/test/reports:

java com.sun.tdk.spectrac.Main report \
-t -workspace  /usr/local/test \
-precompilepath /usr/local/test/assertions \
-reportdir /usr/local/test/reports
java.io.BufferedInputStream

For More Information

The Java Community Process Program

The Java Community Process - Community Resources - TCK Tool & Info

Java Technology Test Suite Development Guide

About the Author