Volume Number: 18 (2002)
Issue Number: 8
Column Tag: Palm Programming

Web Clipping

Downloading a Palm OS Web Clipping Application via a Servlet

by Andrew S. Downs

Several years ago the Palm VII introduced users to the wireless web through its built-in browser (Web Clipper, or simply Clipper). This browser supports Web Clipping Applications (WCAs), thin-clients built using Palm's supported subset of HTML. WCAs may contain pages consisting of controls, text, images and links.

A WCA is a specialized type of Palm OS database (as are other Palm OS apps). You construct a WCA using Palm's builder tool. Note that, although a hierarchy of directories and files may be used when building the WCA, inside the database everything exists in one logical directory (retaining the original file names), thus requiring unique names across all input files (both HTML and images).

An advantage to this database-oriented approach is that all of the elements necessary to display a set of pages may be assembled into one package for download and subsequent browsing. A continuous network connection is not required for viewing the pages, although it is possible (and common) to place live links on those pages.

The WCA discussed later in this article contains no live external links (see Figure 1). One benefit of this approach is that no additional web access is required when constructing or using the WCA, saving both time and money. (The Palm VII wireless network can be slow and expensive, which becomes an important factor if you need to build a WCA dynamically or access live data.) The application described here functions the same over a wireline modem, which may be more cost effective than a wireless connection.

Non-Palm VII owners: Web Clipper is also included in the Mobile Internet Kit, a software upgrade that allows other Palm devices to access the Internet.

There are several components to the system described in this article:

• A native Palm app (a .prc) that initiates the request to download the WCA. This app contains information needed to connect to a server, including URL and username.

• A Java servlet that returns the WCA from the server.

• The raw material for the WCA. In this example, it is a static text page.

• The Palm Query Application Builder (QAB) that creates the WCA. This tool is freely available for download. As of this writing the Macintosh version does not support execution via AppleScript or otherwise allow for command-line building or the inclusion of params. As always, check the Palm website for updates.

The majority of the code presented in this article is for the Palm OS client application. The core functionality is in the Connection class (C++). It sequences the calls to retrieve and display a WCA.

Connection
The methods contained here include:
   Init: drives the overall connection and download process.
   Connect: sets up portions of the http request, and hands-off to a library-specific 
      method.
   LaunchClipper: invokes WebClipper to display the WCA. Most of this method came 
      from Palm's website.
   InvokeINetLib: use the INetLib to connect to a remote URL and download data.
   const int kGetFileSize = 0;
   const int kGetFile = 1;
   const ::Char * kUser = "sample";
   const ::Char * kUrl = 
      "http://yourdomain:8080/servlet/WcaServlet";
   const ::Char * kDatabaseName = "Sample.pqa";
   const int kInBufferSize = 1024;
   const int kDefaultMsgSize = 1024;
void Connection::Init( void ) {
   // Although not an absolute requirement for a small WCA, if we know how much 
   // space the downloaded WCA will take up, our code works more efficiently if we 
   // only allocate a temp buffer of the necessary size. Since the Connect() method does 
   // not currently return anything, there is code in InvokeINetLib() that saves the size. 
   // That is not done in this method.
   Connect( kGetFileSize, kUser, kUrl );
   // Retrieve the actual WCA from the server.
   Connect( kGetFile, kUser, kUrl );
   // Display the downloaded WCA without additional user intervention.
   LaunchClipper( "file:Sample.pqa" );
}
void Connection::Connect( int op, ::Char * user, 
   ::Char * url ) {
   // This string holds the additional params in the http request.
   ::Char * buf;
   // The string representation of the operation code goes here.
   ::Char opString[ 2 ];
   
   // Allocate a buffer for our outgoing request. The default size is stored in another class.
   buf = ( ::Char * )::MemPtrNew( kDefaultMsgSize );
      
   if ( buf ) {
      // Initialize the string.
      ::MemSet( buf, kDefaultMsgSize, '\0' );
      ::MemSet( opString, 2, '\0' );
      // Set the operation code.
      ::StrIToA( opString, op );
      // Create the interesting part of the request string, formatted for an http GET 
      // method. Append the user and operation code params.
      ::StrCat( buf, "?user=" );
      ::StrCat( buf, user );
      ::StrCat( buf, "&op=" );
      ::StrCat( buf, opString );
   
      // Invoke a library-specific method to connect to the server.
      // If we are not only using InetLib then wrap this in a conditional. 
      InvokeINetLib( url, buf, op );
         
      ::MemPtrFree( buf );
   }
}
// Most of this method came from Palm's website.
::Err Connection::LaunchClipper( const ::Char * origurl ) {
   ::Err err;
   ::Char * url = 0;
   ::DmSearchStateType searchState;
   ::UInt16 cardNo;
   ::LocalID dbID;
   ::UInt16 length = ::StrLen( origurl );
   
   // Copy the URL, since the OS will free the parameter once Clipper quits.
   url = ( ::Char * )::MemPtrNew( length );
   
   if ( !url )
      return sysErrNoFreeRAM;
   ::StrCopy( url, ( const ::Char * )origurl );
   ::MemPtrSetOwner( url, 0 );
   // Locate and launch Clipper.
   err = ::DmGetNextDatabaseByTypeCreator( true, 
      &searchState, sysFileTApplication, sysFileCClipper, 
      true, &cardNo, &dbID );
   // If Clipper is not present...
   if ( err ) {
      ::FrmAlert( NoClipperAlert );
      ::MemPtrFree( url );
   }
   else {
      err = ::SysUIAppSwitch( cardNo, dbID, 
         sysAppLaunchCmdGoToURL, url );
   }
   
   return err;
}
long Connection::InvokeINetLib( ::Char *theURL, 
   ::Char *theSuffix, int selector ) {
   long retval = -1;
   static int inBufferSize = 0;
   
   // Setup buffers.
   ::Char * in = (::Char *)::MemPtrNew( kInBufferSize );
   
   ::Char * out = 
      ( ::Char * )::MemPtrNew( kDefaultMsgSize );
   // After this, we have a url in the buffer similar to:
   // http://www.yourdomain:8080/WcaServlet?user=sample& op=0
   ::StrCopy( out, theURL );
   ::StrCat( out, theSuffix );
   ::Err err;
   
   ::UInt16 libRefnum;
   // Load net library.
   err = ::SysLibFind( "INet.lib", &libRefnum );
   if ( err ) {
      ErrNonFatalDisplay( "Unable to find INetLib" );
      goto close;
   }
   ::MemHandle inetH;
   ::UInt16 indexP;
   ::INetConfigNameType config;
   
   // Other possible values include inetCfgNameCTPWireless and
   // inetCfgNameDefWireless.
   ::StrCopy( config.name, inetCfgNameCTPDefault );
   
   // Get the configuration index of the net library.
   err = ::INetLibConfigIndexFromName( libRefnum, &config,
      &indexP );
   // Open the net library.
   err = ::INetLibOpen( libRefnum, indexP, 0, NULL, 0,
      &inetH );
   // Minor adjustments to the INetLib settings.
   // Set the buffer size.
   long tempValue = kInBufferSize;
   ::INetLibSettingSet( libRefnum, inetH,
      inetSettingMaxRspSize, &tempValue, 
      sizeof( tempValue ) );
   // Disable compression.
   tempValue = ctpConvNone;
   ::INetLibSettingSet(libRefnum, inetH,
      inetSettingConvAlgorithm, &tempValue,
      sizeof(tempValue));
   ::MemHandle theSocket;
   // So we don't wait forever...
   ::Int32 timeout = ::SysTicksPerSecond() * 15;
   
   // Send our request to the URL specified in our output buffer.
   err = ::INetLibURLOpen( libRefnum, inetH, 
      ( unsigned char * )out, NULL, &theSocket, timeout,
      inetOpenURLFlagForceEncOff );
   ::UInt32 bytes = 0, tempBytes = 0;
   
   ::UInt16 status = 0;
   
   ::INetEventType event;
   bool ready = false;
   // Wait for a change in the socket's status, which will be the signal that there is a 
   // response to process.
   while ( !ready ) {
      ::INetLibGetEvent( libRefnum, inetH, &event, timeout );
   
      if ( event.eType == inetSockReadyEvent || 
            event.eType == inetSockStatusChangeEvent )
         ready = true;
   }
   ::Int32 inMaxBufferSize = kInBufferSize;
   if ( selector == kGetFile && inBufferSize != -1 )
      inMaxBufferSize = inBufferSize;
   
   ::UInt32 numBytes = kDefaultMsgSize;
   // The value of in will change as data gets read into the buffer, so save the original
   // address for later.
   ::Char * oldIn = in;
   // Read incoming data, looping while there is still data available and we have not 
   // downloaded the entire WCA (when applicable.)
   do {
      tempBytes = 0;
      if ( ( inMaxBufferSize - bytes ) < kDefaultMsgSize ) {
         numBytes = inMaxBufferSize - bytes;
      }
   
      err = INetLibSockRead( libRefnum, theSocket, in,
         numBytes, &tempBytes, timeout );
      // Advance pointer.
      in += tempBytes;
      // Increment byte count.
      bytes += tempBytes;
         
   } while ( ( tempBytes != 0 ) && 
      ( bytes < inMaxBufferSize ) && ( !err ) );
         
   // Close socket.
   err = ::INetLibSockClose( libRefnum, theSocket );
   // Restore pointer to incoming data.
   in = oldIn;
   
   if (selector == kGetFileSize && bytes > 0) {
      inBufferSize = ::StrAToI(in);
   }
   
   if (selector == kGetFile && bytes > 0) {
      // The expected WCA name should be in the stream.
      ::Char * dataP = ::StrStr( in, kDatabaseName );
      
      if ( dataP == NULL ) {
         ErrDisplay( "String not found" );
         goto close;
      }
      
      Int32 num = bytes - ( dataP - in );
      
      ::LocalID id = ::DmFindDatabase( 0, kDatabaseName );
      if ( id != 0 ) {
         err = ::DmDeleteDatabase( 0, id );
         
         if ( err != errNone )
            goto close;
      }
      // We will first write the raw data to a temporary database.
      // Check whether that database already exists (a bad thing).
      id = ::DmFindDatabase( 0, "tempSample" );
      // If we did not clean up previously, delete the temp database.
      if ( id != 0 ) {
         err = ::DmDeleteDatabase( 0, id );
         
         if ( err != errNone )
            goto close;
      }
      // Create a temp database, assigning Clipper as the owner.
      err = ::DmCreateDatabase( 0, "tempSample", 0x636c7072,
               0x70716120, true );
      // Note: from here to the end of the method some of the error checking has been 
      // relaxed in order to shorten this example. Production code should check every 
      // return value and take appropriate action.
      if ( err != errNone )
         ErrDisplay( "DmCreateDatabase() returned err !=
            ErrNone");
      // Ensure that our creation attempt succeeded.
      id = ::DmFindDatabase( 0, "tempSample" );
      if ( id == 0 )
         ErrDisplay( "DmFindDatabase() returned id == 0" );
      // Open the database for writing.
      ::DmOpenRef ref = ::DmOpenDatabase( 0, id,
         dmModeReadWrite );
      
      if (ref == 0)
         ErrDisplay( "Error opening database" );
      // Create a resource of type 'pqa '.
      ::MemHandle res = ::DmNewResource( ref, 0x70716120, 0,
         num );
      
      if ( res == NULL )
         ErrDisplay( "DmNewResource() returned NULL" );
      
      // Lock the resource for use.
      ::MemPtr ptr = ::MemHandleLock( res );
      
      if ( ptr == 0 )
         ErrDisplay( "MemHandleLock() returned 0" );
      // Write the resource into the database.
      err = ::DmWrite( ptr, 0, dataP, num );
      
      if ( err != errNone )
         ErrDisplay( "Error writing resource" );
      // Use the raw data to create the "real" WCA. Not condoned by Palm
      // for non-system databases.
      err = ::DmCreateDatabaseFromImage( ptr );
      // Unlock and free up memory.
      err = ::MemHandleUnlock( res );
      
      if ( err != 0 )
         ErrDisplay( "MemHandleUnlock() returned err != 0" );
      
      err = ::DmReleaseResource( res );
      // At this point we are so close to being done that success is likely. 
      // Still, for consistency we check result codes.
      if ( err != errNone )
         ErrDisplay( "DmReleaseResource() returned
            err != errNone" );
      err = ::DmCloseDatabase( ref );
      
      if ( err != errNone )
         ErrDisplay( "DmCloseDatabase() returned
            err != errNone" );
      // Remove the temporary database.
      err = ::DmDeleteDatabase( 0, id );
      // Locate the real database and check that we can open it for reading.
      id = ::DmFindDatabase( 0, kDatabaseName );
      if ( id != 0 ) {
         ref = ::DmOpenDatabase( 0, id, dmModeReadOnly );
         err = ::DmCloseDatabase( ref );
      }
      // Return the number of bytes read.
      retval = bytes;
   }
   
close:
   err = ::INetLibClose( libRefnum, inetH );
   // Cleanup allocated memory.
   if ( out )
      ::MemPtrFree( out );
   // Reset pointer.
   in = oldIn;
   if ( in )
      ::MemPtrFree( in );
   return retval;
}

The Web Clipping Application in this example is intentionally simple. A WCA gets created using the Query Application Builder tool, from one or more HTML and image files. This example contains static text only, contained in one source HTML file. You can extend this WCA by adding a link or button that triggers a fetch of the latest information from the server.

A starting point for a Web Clipping Application (WCA). Note the inclusion of the 
Palm-identifier in the meta tag.

<html>
   <head>
      <meta name="palmcomputingplatform" content="true">
      <title>Sample WCA</title>
   </head>
   <body>
      <h3>Courtesy of the Web Clipping sample servlet!</h3>
   </body>
</html>

The Java servlet illustrated here responds to client http requests, and can run on something as simple as Sun's servletrunner application (found in older versions of the Servlet Development Kit.) This servlet's primary task is to return the (already-built) WCA associated with a particular id.

This servlet accepts two parameters in the http request:

• the name of a user, allowing us to return WCAs tailored to specific individuals, group, etc.

• an operation code, allowing for multiple tasks to occur. This servlet can return the size of the WCA as a separate operation. This allows a client to request the WCA size first, setup a buffer to hold the actual WCA, then request the WCA itself.

WcaServlet.java
Receive http requests from clients and return a built WCA.
import java.io.*;
import java.net.*;
import javax.servlet.*;
import javax.servlet.http.*;
public class WcaServlet extends HttpServlet { 
   // Elements passed in the request string.
   static final String kOperation = "op";
   static final String kUser = "user";
   
   // Operations we handle. Passed in request string.
   static final int kGetFileSize = 0;
   static final int kGetFile = 1;
   
   // Name of actual pqa file should be the same for all users. For flexibility, we can 
   // retrieve it from different folders as needed.
   final String kPqaFilename = "Sample.pqa";
   
   // Name of base directory (relative to servlet dir) from which to build path to pqa.
   final String kPartialPath = "dev/pqa/";
   
   // Most servlets do most of their work starting from doGet() or doPost(). 
   public void doGet( HttpServletRequest   request,
      HttpServletResponse response ) 
      throws ServletException, IOException {      
      // We can handle different users. The request carries the username as a param. 
      String user = request.getParameter( kUser );
      
      // The operation of interest also gets passed as a param.
      int op = Integer.parseInt( 
    ( String )request.getParameter( kOperation ) );
            
      // The output stream is where our response will go. 
      ServletOutputStream out = response.getOutputStream();
      switch ( op ) {
         // The file size is important when the receiver needs to know how big a buffer
         // to allocate for incoming data.
         case kGetFileSize:
            File pqa = new File( kPartialPath + username
               + "/" + kPqaFilename);
            
               // Write the file size to the output stream.
               if ( pqa != null && pqa.exists() && pqa.isFile())
                  out.println( pqa.length() );
            
            break;
            
         // Return the actual pqa in the output stream. 
         case kGetFile:
             // Build the path to the file, and attempt to open the file.
            FileInputStream fis = new FileInputStream(       
               kPartialPath + user + "/" + kPqaFilename );
               
            if ( fis != null ) {
               // Open a "pipe" to get data out of the file.
               DataInputStream bis = new DataInputStream( fis );
               // Copy the pqa to the output stream. This particular implementation can 
               // be improved upon by copying more than a byte at a time.
               while ( bis.available() > 0 )
                  out.write( bis.readByte() );
               bis.close();
               fis.close();
               out.close();
            }
            break;
            
         default:
            break;
      }
   }
}

It is possible to extend this servlet to dynamically build a WCA on demand. This would allow up-to-the-minute data to be inserted into a WCA targeted at a particular user. The operation code allows for some sophisticated handling of such user requests. For example, the servlet could respond to an initial request for a WCA by spawning a thread to build that WCA dynamically. Since it takes time to perform such a build, particularly if data must be fetched from the Internet, it would be desirable to add some status codes to the process. A client can request the current status of the build, loop while it is not complete, then request the WCA itself.

There are many bells and whistles that can be incorporated into this system. For example, the Java servlet may connect to an LDAP server to validate the user and obtain user-specific configuration information. Several servlet engines are available that can run this servlet, including the servletrunner application from Sun (good for testing), and apache.org's Tomcat. Non-Palm OS devices may be supported by the servlet: the type of client requesting a file could be sent as a param in the http request.

Many WCAs consist of one or more statically coded pages. But dynamic pages often work much better if you have access to a reliable mechanism for generating those pages. Java servlets provide an easy way to gather pages, build a WCA via a command-line prompt, and then download the WCA to the Palm device.

An alternative to building the WCA using the Palm tools would be to write the database format directly. Although the format has probably remained static over the past two years, it requires time (and money) to implement such a writing mechanism, whereas the build tools can be downloaded and setup very quickly.

Combee, Ben and R. Eric Lyons, David C. Matthews and Rory Lysaght. Palm OS Web Application Developer's Guide. Syngress Publishing, Inc., 2001.

Bachmann, Glenn. Palm Programming. Sams Publishing, 1999.

Hunter, Jason and William Crawford. Java Servlet Programming. O'Reilly & Associates, Inc., 1998.

Andrew has worked with Palm OS since 1999. He wrote the Palm OS wireless client for Snippets Software. You can reach him at andrew@downs.ws.