C H A P T E R 8

PCSL

This chapter discusses how to use the Portable Common Services Library (PCSL) in the porting process.

PCSL is the Portable Common Services Library. It is portable in the sense that it defines a set of interfaces that higher-level implementations use. It is common in that both MIDP and CLDC implementations use the interfaces defined by PCSL.

PCSL is a family of different libraries:

Networking

File system

Memory allocation

Printing

String

These libraries are largely independent of each other and can be ported separately. However, some of the implementations do depend on other parts of PCSL. For example, the PCSL file system implementations rely on the PCSL memory allocation interfaces.

A notable feature of PCSL is that multiple implementations of each porting interface are available. Each implementation resides in a module. When building PCSL, you choose one implementation module for each library. For example, PCSL has two implementations of the networking interfaces: one that implements networking using BSD sockets, and another that sends network packets over a serial port. The SoS module is useful during development and porting if a true network is not available. The SoS module is not suitable for use in production products. On the other hand, the memory allocation library has two implementation modules: a malloc-based implementation and a heap-based implementation. Either module is suitable for use in production.

Each PCSL library also contains a module consisting only of a set of stubs. This is not an implementation. Rather, it is a skeleton that you can use to create an entirely new implementation module. This is useful if none of the implementation modules provided in PCSL is a good match to the underlying platform.

When you build PCSL, be sure that the self tests are linked and then run the tests to verify that the build works. Note that the test results must be available though the output functionality described in Set Up and Configure Your Device Development Environment.

PCSL Print Library

The PCSL Print library is extremely simple and consists of a single function: pcsl_print(). This function takes a NULL-terminated string and prints it to an implementation-specific location. By itself, this function does not provide much value. However, because all debugging and error messages in the system are passed through this function, it is possible to redirect all of this output by modifying or re-implementing this one function.

The PCSL Print library has two alternate implementations: one that prints to the standard output (stdout) and one that prints to a file (file). Both implementations always flush their output so that no data is left in any internal buffers.

The stdout implementation sends output to the UNIX system standard output stream. If your platform is a UNIX operating system or a similar system (for example, Linux), use the stdout implementation. The process that invokes the Java Wireless Client software (such as a shell) can often conveniently redirect standard output without any change to the Java Wireless Client software itself.

The file implementation opens a file and sends its output to it. In this implementation, the PCSL Print library uses the file operations defined in the PCSL File library. The file implementation of PCSL Print is useful if the UNIX system style standard output stream is not available, or if it is difficult to redirect.

PCSL Memory Allocation Library

The memory allocation library provides interfaces to allocate, resize, and deallocate dynamic memory. It also has functions to duplicate strings and to get information about heap memory allocated and available.

The PCSL memory allocation interface is defined in the pcsl_memory.h header file.

Dynamic memory allocation is one of the most-used functions in any software system. Therefore, the PCSL memory allocation library is one of the first libraries that you port. The PCSL memory allocation library provides two different implementation modules. The first implementation, the malloc-based implementation, simply calls the standard C functions malloc(), calloc(), realloc(), and free(). If the underlying platform supports these functions, then the malloc-based library offers the quickest route forward.

The other implementation module, the heap-based implementation, implements its own memory allocation within a contiguous block of memory. This implementation is suitable for production, and it is sometimes preferable to the malloc-based implementation. The heap-based implementation allocates from a single contiguous range of memory and so is suitable for situations where the Java platform is limited to a specific amount of memory, or if it is desirable for memory allocated by the Java platform to be kept separate from memory allocated by other parts of the system.

In addition, the heap-based implementation provides diagnostic facilities to detect things like memory leaks and buffer overrun situations. Two modes are provided: debug and non-debug. In debug mode, C preprocessor macros are employed to provide the file name and line number from which the function was called. This information is stored inside the heap structures, so that a heap block can be examined to determine the location in the source code where it was allocated. This extra information is not present in non-debug mode.

In most PCSL libraries, the boundary between interface and implementation is established by declaring functions in C header files and providing multiple implementations of those functions in different sets of C source files. The memory allocation library defines the boundary between interface and implementation differently: instead of being based on C functions, it is based on C preprocessor macros. Suppose you use the malloc-based implementation. The main PCSL memory allocation function is named pcsl_mem_malloc(). If this is a C function, the function simply turns around and calls malloc(). However, this incurs extra function call overhead for every memory operation. To avoid this overhead, the PCSL memory allocation library defines C preprocessor macros that compile out this extra layer. Thus, in the malloc-based implementation module, pcsl_mem_malloc() is a C preprocessor macro that is defined to expand to malloc().

Given that the porting layer for the PCSL memory library is implemented through C preprocessor macros, the conventions for implementing the memory allocation interfaces are unusual. The main header file pcsl_memory.h includes the following statements:

#include <pcsl_memory_impl.h>

/* ... */

#define pcsl_mem_malloc(x)

To develop a new implementation of the PCSL memory interfaces, you must provide a header file named pcsl_memory_impl.h. In this file, you either define pcsl_mem_malloc_impl() as another preprocessor macro, or you can simply provide an ordinary extern C function declaration. Use similar techniques for the other pcsl_mem interfaces.

Porting `malloc`-Based Implementations

Consider the following reminders when porting the malloc-based implementation:

The function pcsl_mem_malloc() must allocate and return a valid heap block even if the size passed is zero. Some system malloc() calls might return NULL in this case. If this is true, a wrapper function must be added to handle this special case.

Some memory allocation implementations do not require initialization and finalization functions. However, PCSL requires them. In this case, it is sufficient to define the initialization and finalization as macros that evaluate to zero. The expressions must evaluate to zero, otherwise an error occurs.

If pcsl_mem_free() is passed a NULL pointer, it must handle it without error. The heap-based implementation module handles this internally. The malloc-based implementation includes a macro that checks for NULL before calling the underlying free() function.

If the underlying platform provides malloc() and free() but not all of calloc(), realloc(), or strdup(), it is straightforward to implement any of the last three in terms of malloc(). The heap-based implementation has code that implements these functions in terms of pcsl_mem_malloc() and can be used as an example.

The three functions pcsl_mem_get_total_heap(), pcsl_mem_get_free_heap(), and pcsl_mem_malloc_dump_memory() merely provide additional information about the state of the memory subsystem. However, because they are only used to provide information, it is adequate to define these APIs as no-ops, especially if the underlying system APIs do not provide a one-to-one mapping with these APIs.

Chunky memory consists of memory blocks that can be shrunk or expanded after they are allocated. It is used by the Java platform heap to use memory efficiently based on the current load of the VM. When an existing chunk of memory is adjusted, its starting address must remain unchanged.

The default implementation uses the Linux mmap function to implement these chunky memory functions. If the underlying platform does not have a similar mechanism, you can simulate chunky memory with regular malloc by pre-allocating the maximum block size and adjusting end pointer or size upon adjust calls. For an example, see Porting the Heap-Based Implementation.

Porting the Heap-Based Implementation

The heap-based implementation is generic, requiring only a contiguous block of memory from which it allocates. This block of memory is initialized in two ways:

If PCSL_MEMORY_USE_STATIC is defined, the memory manager uses the stack for its available memory pool. For example:

static char PcslMemory[DEFAULT_POOL_SIZE];

If PCSL_MEMORY_USE_STATIC is not defined, it calls the C function malloc() and free() to allocate and free the memory pool. If malloc() and free() are not available, you must substitute the appropriate functions. Note that you cannot use pcsl_mem_malloc() or pcsl_mem_free() here because that creates a circular dependency.

The chunked memory operations are implemented by three functions:

pcsl_mem_allocate_chunk()

pcsl_mem_adjust_chunk()

pcsl_mem_free_chunk()

These functions have generic implementations based on pcsl_mem_malloc and pcsl_mem_free. They do not require any porting.

PCSL File System Interface

The PCSL file system interface provides the persistent storage functionality required to load classes, images, and to store JAR and JAD files.

The PCSL file system hides directory hierarchies by making the file system look flat. This feature enables Java Wireless Client software to work with file systems that do not support hierarchical directories. Note that the ability to list files is required by the following interface functions:

pcsl_file_openfilelist

pcsl_file_getnextentry

pcsl_file_closefilelist

All file names are based on the 16-bit Unicode Standard. This feature allows maximum performance when the underlying platform supports the Unicode file system directly.

The PCSL file system interface is defined in the pcsl_file.h and pcsl_directory.h header files.

PCSL provides two file system implementation modules. The first implementation is based on POSIX file system APIs and is suitable for use in production. The second simulates a file system in RAM and is suitable for use during development and porting. The POSIX module supports a hierarchy of directories, while the RAM-based module supports a flat namespace.

The first porting step is to determine what file system APIs are available on the underlying platform. If POSIX APIs are available, or if the APIs available are POSIX-like, start with the POSIX implementation module and modify it as necessary for the platform. If the file system API available is very different from the POSIX APIs, it is probably necessary to create an implementation from scratch using the stubs module.

If the file system APIs are not available, or if some time is required to port to the available file system APIs, the porting effort can proceed by using the RAM-based file system implementation module. As noted previously, the RAM-based module is not suitable for production. In particular, because it is RAM-based, its files are not persistent. However, a file can be read later in the same test run and files can be preloaded into the RAM-based file system.

A PCSL file system implementation is required to implement the following functions:

pcsl_file_init()

pcsl_file_open()

pcsl_file_close()

pcsl_file_read()

pcsl_file_write()

pcsl_file_unlink()

pcsl_file_truncate()

pcsl_file_seek()

pcsl_file_sizeofopenfile()

pcsl_file_sizeof()

pcsl_file_exist()

pcsl_file_commitwrite()

pcsl_file_openfilelist()

pcsl_file_closefilelist()

pcsl_file_getfreespace()

pcsl_file_getusedspace()

pcsl_file_getnextentry()

pcsl_file_getfileseparator()

pcsl_file_getpathseparator()

pcsl_file_is_directory()

pcsl_file_mkdir()

pcsl_file_rmdir()

pcsl_file_getfreesize()

pcsl_file_gettotalsize()

pcsl_file_get_attribute()

pcsl_file_set_attribute()

pcsl_file_get_time()

See the pcsl_file.h and pcsl_directory.h header files for a complete specification of these functions.

PCSL Networking Library

The PCSL Networking Library provides portable interfaces for client-side sockets as well as for a variety of utility functions such as host name lookup. PCSL provides support for datagrams and server-side sockets.

The APIs support asynchronous operation to avoid blocking the caller during a long-running network operation. To this end, most of the networking operations are split into two parts: a start function and a finish function. The start function initiates the operation, and the finish function collects the results. A notification step must also occur between the start function and the finish function. After an operation is started, the implementation must provide some means of notifying PCSL when the operation completes and is ready for the finish function to be called. In some cases, notification originates within PCSL itself.

The start function and the finish function must each be called from a native method because when the start function initiates an operation, it might return the code PCSL_NET_WOULDBLOCK. This indicates that the calling Java programming language thread must be blocked, and that a notification context must be set up. When the operation completes, notification occurs, and the calling Java platform thread (Java thread) is awakened. When a Java thread is awakened after was blocked in a native method, the native method is restarted. This native method then calls the finish function. See the "Coding Styles for Long-running Native Methods" section in Chapter 6 of the CLDC HotSpot Implementation Porting Guide for further information about this technique.

PCSL provides the following networking operations: open, read, write, close, and gethostbyname. Each function is prefixed with the string pcsl_socket_ for socket operations and with the string pcsl_net_ for general networking utilities. The suffixes _start and _finish are used to name the start functions and finish functions. For example, the finish function for the write operation is named pcsl_socket_write_finish().

Additionally, several other operations, including operations for getting and setting socket options are available. These operations update and retrieve state synchronously, so they have no start functions or finish functions.

Note - Operations are not required to be asynchronous. The start function is not required to return PCSL_NET_WOULDBLOCK (which causes an eventual call to the finish function) if the operation can be completed immediately. For example, if pcsl_socket_read_start() is called and enough data is already available, this function can return that data directly instead of returning the code PCSL_NET_WOULDBLOCK. In this case, no notification occurs, and pcsl_socket_read_finish() is not called.

A complete specification of the PCSL network APIs is provided in PCSL_dir/src/network/pcsl_network.h.

Handles and Contexts

A couple of abstractions are added to the PCSL networking APIs to make them both portable and asynchronous. The APIs use a "handle" (a void * value) to represent an open socket. The pcsl_socket_open_start() function creates and returns a handle. The pcsl_socket_open_finish() function and the start- and finish- functions that implement the other socket operations (read, write, close, and so on) all take a handle as a parameter. The handle becomes invalid after the close operation completes. A handle can be a pointer to a block of allocated memory or it can be a value like a UNIX system file descriptor (an int). If a handle is a pointer to allocated memory, the pcsl_socket_open_start() function must allocate it and one of the pcsl_socket_close_*() functions must deallocate it when the close operation completes.

The asynchronous nature of the APIs requires you to set up information about pending operations during the start function and carry across to the finish function. This is handled through the "context" object (another (void *) value). Each start function is allowed to create and return a context object. After notification is received, the system finds the proper context object and passes it to the finish function. The context can be a pointer to a block of allocated memory. In this case, the start function allocates the block and the finish function deallocates it. If an implementation does not need a context (for example, the context might be maintained implicitly in the handle), the start function returns NULL for the context, and the finish function ignores the context parameter.

Alternative Networking Implementations

PCSL provides five alternative networking implementations. The winsock implementation is not supported; it provides a binding to Microsoft Windows networking.

The javacall implementation provides a binding from PCSL to the JavaCall porting layer. If you have already implemented the JavaCall network APIs, then the PCSL javacall implementation can be used without modification.

The javacall implementation relies on JavaCall API notifications and JavaCall API events mechanism to deliver networking and socket events from the platform to the upper layers of the MIDP implementation. For more information, please see the JavaCall API documentation.

The three remaining networking implementations are illustrated in the following diagram. The SoS implementation implements sockets over a serial line. The BSD implementation uses the BSD UNIX system socket API. The BSD implementation has two variants: a generic implementation that uses plain BSD sockets and a QTE-based implementation that wraps each socket inside a Qt object.

FIGURE 8-1 Three PCSL Networking Implementations

Diagram shows the three PCSL networking implementations

The following table shows where the technologies listed in FIGURE 8-1 are located in the Java Wireless Client software release.

TABLE 8-1 PCSL Technologies File System Locations
Implementation	Directory Location
SoS	SoS_PCSL_dir`/src/network/socket/`PCSL_dir`/src/tools/sosProxy`
BSD Common	PCSL_dir`/src/network/socket/bsd/`
`pcsl_serial_linux.c`	PCSL_dir`/src/network/serial`
BSD Sockets and QTE	PCSL_dir`/src/network/socket/bsd/qte`
BSD Generic	PCSL_dir`/src/network/socket/bsd/generic`

BSD Implementations

The implementations of the PCSL networking interfaces use the BSD socket APIs, for example, socket(), bind(), and connect(). The BSD-based implementations are suitable for production use. The BSD implementation places all sockets into non-blocking mode so that the PCSL start functions do not block the caller.

The BSD implementation has two variants: a generic one that uses socket descriptors directly, and one that uses QTE. The generic implementation relies on external code to call select() (or equivalent functionality) to determine when the socket operation is complete. The QTE implementation wraps each socket into a VMSocket object and sets up a QSocketNotifier object to handle notification of socket state changes.

If your platform uses BSD sockets and has a callback-based notification scheme, start your port with the BSD QTE implementation. If your platform uses BSD sockets but uses a select()- or poll()-based notification scheme, start your port with the BSD generic implementation.

PCSL provides an internal interface called the "notification adapter" that enables code to be shared between the BSD generic and BSD QTE implementations. This interface is defined in the header file PCSL_dir/src/network/socket/bsd/pcsl_network_na.h.

Both BSD implementations create and use BSD socket descriptors, but only the QTE implementation also creates VMSocket and QSocketNotifier objects. The notification adapter allows BSD common code to deal with sockets as abstract handles. In the BSD generic case, the handle is simply the socket descriptor, whereas in the BSD QTE case, the handle is a pointer to a VMSocket object. You might find the notification adapter interfaces useful if the platform supports BSD sockets but uses a notification scheme other than QTE.

Socket-over-Serial Implementation

The SoS implementation simulates a network connection by sending packets over a serial line. This implementation is suitable for use during development and porting and is not typically put into production.

Instead of opening a socket for each network connection, the SoS implementation multiplexes one or more network connections over a single serial port. This implementation uses a low-level serial port API internally. PCSL provides an implementation of this low-level serial API for Linux-based systems.

Because the SoS implementation does not rely on any system libraries (except serial
I/O), you have more control over the packets that are sent to the server and the way those packets are handled. This control gives you the option to send debug data (pcsl_print() output) using the SoS implementation as auxiliary packets that the server can decode and print in a console or debugging window.

To bring up SoS networking on a device with a serial port, first evaluate its serial APIs. If they are similar to Linux or the UNIX system, you can start with the existing serial port code and bring up the SoS networking on top of that. If the device's serial APIs are different from the UNIX system, create an implementation of the low-level serial API that maps the API calls to platform-specific serial port calls.

The low-level serial API is defined in PCSL_dir/src/network/serial/pcsl_network_serial.h.

A Java programming language program written for the Java SE platform is provided to act as an SoS proxy server. The SoS proxy server opens a socket connection with the other end and does all the networking operations on behalf of the actual device. Communication between the device and the proxy server is done using a method similar to RPC. The proxy server also allows SoS connections to be demultiplexed to the actual network connections. During development, it is useful to connect a device to a desktop workstation through a serial port, and then to connect the workstation to a local-area network.

Notification

To initiate a network operation, the start function is called from a native method, and the calling Java thread is blocked. At this point, it is crucial that the platform have some means of notifying PCSL when a networking operation is complete. The means for this notification is highly system specific. The alternative PCSL networking implementations handle notification differently.

The BSD generic implementation uses plain BSD socket descriptors, which have no intrinsic notification capability. Instead, this implementation requires external code to call select() or poll() with this file descriptor. For example, if the CLDC HotSpot Implementation VM is running in normal mode (as opposed to slave mode), code in the JVMSPI_CheckEvents() function checks the socket descriptor to determine when an operation completes and to awaken the Java thread that was blocked on the operation.

The BSD QTE implementation uses the QSocketNotifier mechanism, which provides a callback into PCSL code when a socket operation completes. This necessitates a call from PCSL into the rest of the system. PCSL calls the function NotifySocketStatusChanged() when a socket's state changes. This function is not provided by PCSL, but is provided externally as part of the upper layers of the MIDP implementation. This function is responsible for finding and awakening the Java programming language thread that was blocked on the socket operation.

The SoS implementation does not use notification at all. Instead, it relies on the caller to poll or to sleep for a specific period of time until the operation is presumed to be complete. As such, the SoS implementation is unsuitable for production use.

See Chapter 10 for further information on normal and slave modes.

PCSL String Library

The PCSL String Library provides portable interfaces for basic string operations.

The API defines an opaque pcsl_string type and contains functions for the creation and manipulation of pcsl_string instances.

The first porting step is to determine an internal representation of pcsl_string that is optimal for the platform. The reference implementation provides a pcsl_string definition based on UTF-16 encoding to match the internal representation of Java programming language string objects. On Microsoft Windows platforms, it is reasonable to use the UTF-16 representation because the Windows API functions support it. On platforms that do not support 16-bit characters, but do support UTF-8, it might be reasonable to select the UTF-8 internal representation to avoid repeated conversions of the same PCSL string to UTF-8.

It is also possible to have an internal PCSL string representation that caches different encodings of the same string to achieve maximum performance.

A PCSL string implementation is required to implement the following functions:

pcsl_string_is_active()

pcsl_string_initialize()

pcsl_string_finalize()

pcsl_string_length()

pcsl_string_utf16_length()

pcsl_string_utf8_length()

pcsl_string_convert_to_utf8()

pcsl_string_convert_to_utf16()

pcsl_string_convert_from_utf8()

pcsl_string_convert_from_utf16()

pcsl_string_equals()

pcsl_string_compare()

pcsl_string_cat()

pcsl_string_dup()

pcsl_string_append()

pcsl_string_append_char()

pcsl_string_append_buf()

pcsl_string_predict_size()

pcsl_string_substring()

pcsl_string_starts_with()

pcsl_string_ends_with()

pcsl_string_index_of()

pcsl_string_index_of_from()

pcsl_string_last_index_of()

pcsl_string_last_index_of_from()

pcsl_string_trim_from_end()

pcsl_string_trim()

pcsl_string_convert_to_jint()

pcsl_string_convert_from_jint()

pcsl_string_convert_to_jlong()

pcsl_string_convert_from_jlong()

pcsl_string_free()

pcsl_string_get_utf8_data()

pcsl_string_release_utf8_data()

pcsl_string_get_utf16_data()

pcsl_string_release_utf16_data()

pcsl_string_is_null()

You must also provide definitions for the following macros:

PCSL_DEFINE_ASCII_STRING_LITERAL_START_MD

PCSL_DEFINE_ASCII_STRING_LITERAL_END_MD

PCSL_STRING_LITERAL_LENGTH_MD

PCSL_DEFINE_STATIC_ASCII_STRING_LITERAL_START_MD

PCSL_DEFINE_STATIC_ASCII_STRING_LITERAL_END_MD

PCSL_STRING_NULL_INITIALIZER_MD