Skip to Content Sun Java Solaris Communities My SDN Account Join SDN

SDN Home > Products & Technologies > Java Technology > J2SE > Reference > Technical Articles and Tips >

Article

High-Level Network Programming

Print-friendly Version

HTTP-based Applications

by Qusay H. Mahmoud

November 14, 2002 -- The java.net package contains the classes and interfaces that provide both low- and high-level APIs for network programming. The low-level APIs give you direct access to network protocols, but at the expense of simplicity; you have to work with low-level TCP sockets and UDP datagrams. The high-level APIs (such as the URL, URLConnection, and HttpURLConnection classes) let you develop network applications faster, with less code to write.

A previous article, Network Programming with J2SE 1.4, described how to work with low-level sockets. The focus of this article is on how to develop HTTP-based applications using the high-level APIs of the java.net package.

This article:

Presents an overview of HTTP
Presents an overview of the high-level APIs of the java.net package
Provides code examples that demonstrate how to use the high-level APIs
Presents a useful application capable of downloading stock quotes
Demonstrates how to post data to web servers
Gives an overview of HTTP authentication and shows how to protect your network resources
Provides code examples that demonstrate how to perform HTTP authentication

Overview of HTTP

The Hypertext Transfer Protocol (HTTP) is a request-reply application protocol which supports a fixed set of methods, such as GET, POST, PUT, DELETE, and so forth. The GET method is commonly used to request resources from a web server. Here are two sample GET requests:

GET / HTTP/1.1
GET /names.html HTTP/1.1

In addition, you can use both the GET and POST methods to send data to the server. These methods, however, have different ways of sending data to the server:

GET method: The input data is sent as part of the URL.
POST method: The input data is sent as an entity body in a separate message.

Consider the following HTML form:

<form action="http://www.javacourses.com/servlets/getMarks method="GET">
Student#:
<input type=text name=number size=30>
<input type=submit name=GetMarks value=GetMarks>
</form>

This form is handled by the servlet http://www.javacourses.com/servlet/getMarks. The form uses the GET method to transfer the information. When the user enters a student number -- such as 556677 -- and clicks the GetMarks button, the form data is sent to the servlet as part of the URL. The encoded URL is: http://www.javacourses.com/servlets/getMarks?number=556677.

In the case of POST, however, input data is not sent as part of the URL; rather, it is sent as an entity body in a separate message. Therefore, the POST method is more secure, and you can send more data with it. In addition, the data doesn't have to be in plain text, as with the GET method.

Message Formats

The request message specifies the name of the method (GET or POST), the URL, the protocol version number, headers, and an optional message. The headers might contain request information and client information, such as acceptable content type, browser name, and authorization data. A reply message specifies the protocol version, response code, and reason. The response code and reason provide a report on the success or failure of carrying out the request. Some of the response codes are:

200 OK: Request succeeded. The requested resource can be found later in this message.
301 Moved Permanently: Requested resource has moved. New location is specified later in this message.
400 Bad Request: Request message is not understood by the server.
404 Not Found: Requested document is not found on this server.

More detailed information about HTTP and all response codes can be found in the HTTP 1.1 specification RFC2616.

Here is an example of a request message going from the browser to the server. The URL requested here is http://java.sun.com:

GET / HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, 
  application/vnd.ms-powerpoint, application/vnd.ms-excel, 
  application/msword, */*
Accept-Language: en-ca
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; YComp 5.0.0.0)
Host: java.sun.com
Connection: Keep-Alive
Cookie: SUN_ID=24.80.19.177:28346100732290; 
  SunONEUserId=24.80.19.177:86521021960770

And here is the server's reply message to the above request:

HTTP/1.1 200 OK
Server: Netscape-Enterprise/6.0
Date: Mon, 14 Oct 2002 15:18:04 GMT
Content-type: text/html
Connection: close

Overview of java.net High-Level APIs

The java.net package includes high-level APIs that implement some of the most commonly used protocols, such as HTTP and FTP, running on top of TCP. The two main classes that provide the abstraction needed are URL and URLConnection. Another useful class is the HttpURLConnection, which is a subclass of URLConnection, and provides support for HTTP-specific features.

A URL (Uniform Resource Locator) is an address that describes the location of a document (or more generally a resource) on the Internet. A URL has the form:

protocol://machineName:port/resource

It's important to note that the URL class is not HTTP-oriented; it has support for the ftp, https, and file protocols. As a result, all of the following URLs are valid for the URL class:

http://java.sun.com
http://localhost:8080/myApplication
http://www.yahoo.com/index.html
http://www.yahoo.com
ftp://ftp.borland.com
ftp://ftp.sun.com
https://www.scotiaonline.scotiabank.com
https://central.sun.net
file:///C:/j2sdk1.4/docs/api/index.html

When you enter a URL into a web browser, the browser issues an HTTP GET (or POST) command to retrieve (or query) the requested resource. If no resource is given, the default document (usually index.html) is retrieved.

Reading the Content of a URL

To get started, let's develop a simple application that reads from a URL directly. To give you a taste of how this can be done using low-level sockets, consider Code Sample 1. In this example, the user enters the URL of the resource on the command line. A socket is then opened to port 80 (the default HTTP server port number), and input and output streams are created. The output stream is used to send HTTP commands (such as GET) to the HTTP server, and the input stream is used to read the HTTP server replies. Note that in this example, the response headers will be read (in addition to the content of the URL).

Code Sample 1: ReadURL1.java

import java.net.*;
import java.io.*;

public class ReadURL1 {
   public static void main(String argv[]) throws Exception {
      final int HTTP_PORT = 80;
      if(argv.length != 1) {
         System.out.println("Usage: java ReadURL1 <url>");
         System.exit(0);
      }

      Socket socket = new Socket(argv[0], HTTP_PORT);
      BufferedWriter out = new BufferedWriter(new 
      		OutputStreamWriter(socket.getOutputStream()));
      BufferedReader in = new BufferedReader(
		new InputStreamReader(socket.getInputStream()));
      out.write("GET /index.html HTTP/1.0\n\n");
      out.flush();

      String line;
      StringBuffer sb = new StringBuffer();
      while((line = in.readLine()) != null) {
         sb.append(line);
      }
      out.close();
      in.close();
      System.out.println(sb.toString());
   }
}
// end code

When you run this example, note that the URL must be a domain name such as java.sun.com or an IP address. It cannot have the http:// part of the URL. To allow something like that, you'd need to parse the input to find out the protocol being used, port number, and the resource being requested. Alternatively, you can use the URL class, which provides useful methods such as getProtocol, getPort, getHost, and getFile, to name a few.

Using the URL Class

Reading content from a URL can easily be done using the URL class, as shown in Code Sample 2. This simplifies the task of reading content directly from a URL; since the response headers won't be read, you don't need to parse them. Also, the input in this example can be any valid URL of the form protocol://domainName:port/resource. The URL class parses the input and handles all the low-level dirty work.

Code Sample 2: ReadURL2.java

import java.net.*;
import java.io.*;

public class ReadURL2 {
   public static void main(String argv[]) throws Exception {
      if(argv.length != 1) {
         System.out.println("Usage: java ReadURL2 <url>");
         System.exit(0);
      }

      URL url = new URL(argv[0]);
      BufferedReader in = new BufferedReader(new InputStreamReader(
				url.openStream()));

      String line;
      StringBuffer sb = new StringBuffer();
      while ((line = in.readLine()) != null) {
         sb.append(line);
      }
      in.close();
      System.out.println(sb.toString());

   }
}

When you run ReadURL2, you'll see the HTML tags and textual content scroll in your command window, from the HTML file located at the URL you entered.

If you want to do more than just read from a URL, use openConnection to connect to the URL. This method returns a URLConnection object that can be used for communication with the URL for reading, writing, and querying it. Once a connection is made using the openConnection method, useful methods such as getContentType, getContentLength, getContentEncoding, and others can be used.

Using the URLConnection Class

To demonstrate how to use URLConnection, we'll now build an application for retrieving stock information from http://quote.yahoo.com. To retrieve information about a particular stock, the user enters the stock symbol (such as SUNW, IBM, or MOT), and the application retrieves the corresponding stock quote from the Yahoo Quote Server. The app then displays the name, price, and date of the stock symbol.

There are two ways (that I have used in my applications) to retrieve stock information from the Yahoo Quotes Server. The first one has the format:

http://quote.yahoo.com/d/quotes.csv?s=SUNW&f=slc1wop

If you enter this URL into a web browser, you will see something similar to Figure 1.

Figure 1: Retrieving stock quotes

The other way has the format:

http://finance.yahoo.com/q?s=SUNW

Figure 2: Another way of retrieving stock quotes

You can see that the first approach results in one line of text, which is much easier and faster to parse than the second one, which returns many characters with lots of advertising and formatting information. So my implementation of this stock application uses the first format, and consists of two classes: Stock.java and StockReader.java.

The Stock.java Class

This class parses a string obtained from the Yahoo Quote Server into fields (such as name of stock, price, and date). A sample implementation is shown in Code Sample 3. This utility class is used by the StockReader class in Code Sample 4.

Code Sample 3: Stock.java

public class Stock {
   private static String name, time, price;

   // Given a quote from the server,
   // retrieve the name, price, and date of the stock
   public static void parse(String data) {
      int index = data.indexOf('"');
      name = data.substring(++index,(index = data.indexOf('"', index)));
      index +=3;
      time = data.substring(index, (index = data.indexOf('-', index))-1);
      index +=5;
      price = data.substring(index, (index = data.indexOf('>', index)));
   }

   // Get the name of the stock from
   public static String getName(String record) {
      parse(record);
      return(name);
   }

   // Get the price of the stock from
   public static String getPrice(String record) {
      parse(record);
      return(price);
   }

   // Get the date of the stock
   public static String getDate(String record) {
     parse(record);;
     return time;
   }
}

The StockReader.java Class

This class is responsible for connecting to the Yahoo Quote Server and retrieving stock quotes. It uses the Stock class to parse the returned string from the quotes server. A sample implementation is shown in Code Sample 4.

Code Sample 4: StockReader.java

import java.io.*;
import java.net.*;

public class StockReader {

   public static void main(String argv[]) throws Exception {
      String quoteFormat = "&f=slc1wop";
      if (argv.length != 1) {
         System.err.println("Usage:  java StockReader <symbol>");
         System.exit(1);
      }

      URL url = new URL("http://quote.yahoo.com/d/quotes.csv?");
      URLConnection connection = url.openConnection();
      connection.setDoOutput(true);

      PrintWriter out = new PrintWriter(connection.getOutputStream());
      out.println("s=" + argv[0] + quoteFormat);
      out.close();

      BufferedReader in = new BufferedReader(
		new InputStreamReader(connection.getInputStream()));

      String line = in.readLine();

      /* DEBUG
      while ((line = in.readLine()) != null) {
         System.out.println("Got: "+ line);
      }
      */
      in.close();
      System.out.println("Name: "+ Stock.getName(line));
      System.out.println("Price: "+ Stock.getPrice(line));
      System.out.println("Date: "+ Stock.getDate(line));
   }
}

Posting Data to a Web Server

In the stock example above, the data is sent to the server as part of the URL, using the GET method. Let's now look at an example of sending data via the POST method. In this example, the CGI script (names.cgi), which lives at http://www.javacourses.com/cgi-bin, expects name and email values. As an example, if the user submits Sally McDonald and smc@yahoo.com as values for name and email, respectively, the CGI script retrieves the input, parses it, decodes the message, and returns what has been submitted to the client. Yes, this CGI script doesn't do much, but I'm using it to demonstrate how to post data to the server.

What is important to note is that when the POST method is used, the content type of the message is application/x-www-form-urlencoded. This content encoding type:

Specifies normal data encoding
Converts blanks to plus (+) signs
Converts non-alphanumeric characters to hexadecimal numbers preceded by a percentage sign (%)
Places an ampersand (&) between each name=value pair

According to this encoding scheme, the message (name=Sally McDonald and email=smc@yahoo.com) must be encoded as

name=Sally+McDonald&email=smc@yahoo.com

before it is sent to the CGI script. The CGI script receives this encoded message and decodes it. Fortunately, you don't have to encode messages by hand; you can use the java.net.URLEncoder class to encode messages as shown in the following example. Similarly, you can use java.net.URLDecoder to decode messages.

The sample implementation (using the HttpURLConnection class) in Code Sample 5 shows how to send data to a server with the POST method. As you can see:

A connection and I/O streams are opened to the CGI script
The request method is set to POST
The message is encoded using the URLEncoder.encode method (note that the URLDecoder.decode method can be used to decode messages)
The encoded message is posted to the CGI script
Server reply messages are received and printed on the console

Code Sample 5: PostExample.java

import java.io.*;
import java.net.*;

public class PostExample {
   public static void main(String[] argv) throws Exception {
      URL url = new URL("http://www.javacourses.com/cgi-bin/names.cgi");
      HttpURLConnection connection = (HttpURLConnection) url.openConnection();
      connection.setRequestMethod("POST");
      connection.setDoOutput(true);
      PrintWriter out = new PrintWriter(connection.getOutputStream());
      // encode the message
      String name = "name="+URLEncoder.encode("Qusay Mahmoud", "UTF-8");
      String email = "email="+URLEncoder.encode("qmahmoud@javacourses.com", "UTF-8");
      // send the encoded message
      out.println(name+"&"+email);
      out.close();
      BufferedReader in = new BufferedReader(
		new InputStreamReader(connection.getInputStream()));
      String line;
      while ((line = in.readLine()) != null) {
         System.out.println(line);
      }
      in.close();
   }
}

Proxies and Firewalls

If you are behind a firewall, you need to tell Java the details of your proxy server and its port number in order to access hosts outside the firewall. You do this by defining the applicable HTTP or FTP properties:

http.proxyHost (default: <none>)
http.proxyPort (default: 80 if http.proxyHost specified)
http.nonProxyHosts (default: <none>)

The http.proxyHost and http.proxyPort indicate the proxy server and port that the HTTP protocol handler will use. The http.nonProxyHosts can be set to specify the hosts which should be connected to directly (not through the proxy server). The value of the http.nonProxyHosts property can be a list of hosts, each separated by a |; it can also take a regular expression for matches; for example: *.sfbay.sun.com would match any fully qualified hostname in the sfbay domain.

ftp.proxyHost (default: <none>)
ftp.proxyPort (default: 80 if ftp.proxyHost specified)
ftp.nonProxyHosts (default: <none>)

The ftp.proxyHost and ftp.proxyPort indicate the proxy server and port that the FTP protocol handler will use. The ftp.nonProxyHosts is similar to http.nonProxyHosts and indicates the hosts that should be connected to directly and not through the proxy server.

You can set these properties when you start your application:

Prompt> java -Dhttp.proxyHost=HostName -Dhttp.proxyPort=PortNumber yourApp

HTTP Authentication

The HTTP protocol provides mechanisms to protect resources through authorization techniques. When a request is made to a protected resource, the web server responds with a 401 Unauthorized error response code. The response includes a WWW-Authenticate header specifying the authentication method and realm. Think of a realm as a database of usernames and passwords that identifies valid users of protected site resources. As an example, if you try to access a protected resource within the realm "Personal Files" of a web server, the response will be: WWW-Authenticate: Basic realm="Personal Files" (assuming that the authentication method is basic).

Authentication Methods

There are several authentication methods in use today for web applications. The most widely used are basic and digest authentication.

With basic authentication, when a user tries to access a restricted resource, the web container asks the browser to display a login dialog, for entering a username and a password. If the user enters the correct username and password, the server grants access to the resource; otherwise, an error message will be displayed after three trials. The disadvantage of this method is that the username and password are transmitted to the server using base64 encoding (that is, almost in plain text). In other words, this method of authentication is only as safe as Telnet-style username and password security.

In contrast, the digest authentication method doesn't pass the password over the network at all. Instead, a series of numbers is generated (based on the password and other information about the request), and these numbers are then hashed using MD5 (Message Digest Algorithm). The hash value (digest) is sent over the network along with other information that the server can use to verify the password. This method is obviously more secure over the network.

The form-based authentication method is similar to basic, but the server uses a custom login page written by you, instead of displaying the standard login page.

And finally, client certificate authentication uses SSL (secure socket layer) and client certification.

Protecting Resources under Tomcat

If you like, you can specify a list of users and their roles in the tomcat-users.xml file, which is in the conf directory of your TOMCAT_HOME (the directory where you installed Tomcat). By default, the file contains the definition of three users (tomcat, role1, and both). The following snippet of XML code shows the tomcat-users.xml file after I added two new users (qusay and reader):

<tomcat-users>
   <user name="tomcat" password="tomcat" roles="tomcat" />
   <user name="role1" password="tomcat" roles="role1" />
   <user name="both" password= "tomcat" roles="tomcat,role1" />
   <user name="qusay" password="guesswhat" roles="author" />
   <user name="reader" password="youguess" roles="reader" />
</tomcat-users>

The two new users I added (qusay and reader) can fill the roles author and reader, respectively. The role is important because when you create security policies, each restricted resource is associated with a role that can have access to it (as you will see later).

To start experimenting (assuming that you have Tomcat installed and configured), create a directory that corresponds to the web application you wish to create; get it ready by performing the following steps:

Inside the directory where you installed Tomcat, there is a directory called webapps. Create a directory (for example, learn) under webapps.
Create a subdirectory under the directory you created in Step 1, and call it chapter.
Inside the chapter directory, create an HTML file with contents of your choice, and name the file index.html.
Create a subdirectory under the directory created in Step 1, and call it WEB-INF.
Inside WEB-INF, create a file called web.xml with the following contents:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE web-app
    PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
    "http://java.sun.com/dtd/web-app_2_3.dtd">

<web-app>

    <description>
        Learning Web Programming
    </description>

   <security-constraint>
      <web-resource-collection>
         <web-resource-name>
            Restricted Area
         </web-resource-name>
         <url-pattern>/chapter/*</url-pattern>
      </web-resource-collection>
      <auth-constraint>
         <role-name>tomcat</role-name>
         <role-name>author</role-name>
         <role-name>reader</role-name>
      </auth-constraint>
   </security-constraint>

   <login-config>
      <auth-method>BASIC</auth-method>
      <realm-name>Authenticate yourself</realm-name>
   </login-config>

</web-app>

The web.xml Deployment Descriptor

The web.xml file is the deployment descriptor. Here I will concentrate on the security elements:

The <security-constraint>: This element restricts access to one or more resources, and it can appear multiple times in a deployment descriptor. In the above deployment descriptor, it restricts access to all resources under the chapter directory (http://localhost:8080/learn/chapter). The <security-constraint> contains the following elements:

<web-resource-collection>

<http-method>

http://localhost:8080/learn/chapter/*

chapter

<auth-constraint>>: This element specifies the user roles that can access the resource collection described above. In this application descriptor, the roles tomcat, author, and reader can access the resource above.

The <login-config>: This element is used to specify the authentication method. It contains the elements:

<auth-method>: Specifies the authentication method, with one of these values: BASIC (basic authentication), DIGEST (digest authentication), FORM (form-based authentication), or CLIENT-CERT (client certificate authentication).

<realm-name>: A descriptive name that the standard login dialog will display if the BASIC method of authentication is chosen.

Example

In the above deployment descriptor, the BASIC authentication method is being used. To try this out, start up your Tomcat server and then request the URL http://localhost:8080/learn/chapter. You will be prompted to enter a username and password to access the resource, as shown in Figure 3:

Figure 3: HTTP basic authentication

Enter one of the usernames and passwords (look at your tomcat-users.xml file) corresponding to the roles specified in the deployment descriptor (web.xml). If you enter a valid username and password, you'll be granted permission to view the resources; otherwise, you'll get three chances to retry.

To experiment with digest authentication:

Shut down your Tomcat server.
Edit your deployment descriptor file (web.xml) and replace BASIC with DIGEST.
Start your Tomcat server.
Open a new browser window.
Request the URL http://localhost:8080/learn/chapter. You will see a similar dialog box. But as you can see from Figure 4, this login dialog box is secure, since digest authentication is being used.

Figure 4: HTTP digest authentication

Behind-the Scenes-Server Replies

It's worth noting that -- when a resource is protected using the basic authentication method -- the server sends a response to the client similar to the one shown in Figure 5.

Figure 5: Server reply (basic authentication)

And, when a resource is protected using the digest authentication method, the server sends a response similar to the one shown in Figure 6:

Figure 6: Server reply (digest authentication)

Java Support for HTTP Authentication

J2SE (1.2 and above) provides native support for authentication with the Authenticator class. All you need to do is inherit from it and implement the getPasswordAuthentication method. This method gets a username and password and returns them as a PasswordAuthentication object. Once this is done, you need to register an instance of your authenticator using Authenticator.setDefault. Now, whenever you try to request a protected resource, getPasswordAuthentication will be called. The Authenticator class manages all low-level details, and the good news is that it is not restricted to HTTP, but applies to all network connections.

A sample implementation of an authenticator is shown in Code Sample 6. As you can see, the getPasswordAuthentication method will pop up a login dialog box if authentication is required.

Code Sample 6: AuthImpl.java

import java.net.*;
import java.awt.*;
import javax.swing.*;

public class AuthImpl extends Authenticator {
   protected PasswordAuthentication getPasswordAuthentication() {
      JTextField username = new JTextField();
      JTextField password = new JPasswordField();
      JPanel panel = new JPanel(new GridLayout(2,2));
      panel.add(new JLabel("User Name"));
      panel.add(username);
      panel.add(new JLabel("Password") );
      panel.add(password);
      int option = JOptionPane.showConfirmDialog(null, new Object[] {
                   "Site: "+getRequestingHost(),
                   "Realm: "+getRequestingPrompt(), panel},
                   "Enter Network Password",
                   JOptionPane.OK_CANCEL_OPTION, JOptionPane.PLAIN_MESSAGE);
      if ( option == JOptionPane.OK_OPTION ) {
         String user = username.getText();
         char pass[] = password.getText().toCharArray();
         return new PasswordAuthentication(user, pass);
      } else {
         return null;
      }
   }
}

Code Sample 7 shows a sample test drive. The first thing I do is set my authenticator implementation using Authenticator.setDefault. I don't have to parse any server replies to check if authentication is required, since the Authenticator class is smart enough to figure out if authentication is required or not.

Code Sample 7: BasicReader.java

import java.io.*;
import java.net.*;

public class BasicReader {
   public static void main(String argv[]) throws Exception {
      Authenticator.setDefault(new AuthImpl());
      if (argv.length != 1) {
         System.err.println("Usage:  java BasicReader <site>");
         System.exit(1);
      }
      URL url = new URL(argv[0]);
      URLConnection connection = url.openConnection();
      BufferedReader in = new BufferedReader(
		new InputStreamReader(connection.getInputStream()));

      String line;
      StringBuffer sb = new StringBuffer();
      while ((line = in.readLine()) != null) {
         sb.append(line);
      }
      in.close();
      System.out.println(sb.toString());
      System.exit(0);
   }
}

To test it with your Tomcat server, run it:

prompt> java BasicReader http://localhost:8080/learn/chapter

If the site you entered requires authentication (and it does), a login dialog box appears as shown in Figure 7.

Figure 7: Java handles HTTP authentication

Once you enter the correct login name and password, the server will grant access. BasicReader then reads the HTML contents of the page requested, and displays it in your command window.

Important Note: You may encounter problems with digest authentication in Tomcat 4.0. This stems from a bug in J2SE 1.4.0 and J2SE 1.4.1, which has been fixed in J2SE 1.4.2. Details about the bug are available here.

Summary

This article presented a tutorial introduction to the high-level APIs of the java.net package. These APIs allow you to create useful network applications, such as the StockReader, quickly and easily. HTTP authentication was discussed, with examples demonstrating how to "roll your own" authentication solution. Some of the advantages of using URL and URLConnection that weren't covered in this article include: automatic redirection, automatic management of persistent connections, SSL tunneling through proxy servers, and transparent handling of chunked/streamed responses. Now that you know the basics, you can explore the rest on your own.

Acknowledgments

Special thanks to Alan Bateman of Sun Microsystems, whose feedback helped me improve the article.

About the Author

Qusay H. Mahmoud provides Java consulting and training services. He has published dozens of articles on Java, and is the author of Distributed Programming with Java (Manning Publications, 1999) and Learning Wireless Java (O'Reilly, 2002).