Volume Number: 13 (1997)
Issue Number: 7
Column Tag: Plugging In

Filter It!

by Alan Weissman, TopSoft, Inc.

Join the Plug-in Revolution: Write a FilterTop Filter

In 1992 a fruitful collaboration began among a group of Mac programmers on Usenet. Fired up by the idea of pooling their expertise to develop a killer app that would incorporate many of the new features of System 7, these enthusiasts soon formally organized as TopSoft, Inc., and dubbed their chief project FilterTop. Its main purpose: to bring batch processing and pipelining from unix to the Mac, using such advanced features as Apple events, drag-and-drop and multithreading.

Four years passed before FilterTop's first full public release. A lot happened in that time. Popular programs had evolved into resource-hungry monsters that had to incorporate every conceivable feature, and this tendency was stifling both programmers and users. To let in fresh air, a contrary trend in Mac programming was emerging, toward modularity and plug-in architecture. We see this trend gaining momentum today as large applications increasingly rely on small, interchangeable plug-ins to provide much of their functionality. Adobe Photoshop is the best-known of these, but others are rapidly following suit. Well, FilterTop was ahead of the pack, and today, in an age when software bloat and creeping featuritis still dominate, FilterTop seems more innovative than ever. In fact, practically speaking, FilterTop is just a framework for plug-in modules.

FilterTop offers a distinct advantage in the way it uses plug-ins. Each module, or filter, can be linked with others in a pipeline. Then one or a batch of files can be sent down the pipe, with each filter manipulating the files in a specified way. Instead of one filter at a time operating on one file, you have many filters operating in quick succession on many files, each of which might have been created by a different application. The FilterTop framework provides the user interface and the glue that ensures everything - batches of files and assemblages of filters - works together smoothly.

FilterTop shares an exciting property with other applications that have a plug-in architecture: extensibility. A new capability may be added to those already present simply by writing a new filter and plugging it in. The filter programmer doesn't have to know a thing about the inner workings of FilterTop. Even preexisting filters provide a field for experimentation. Different permutations and combinations may provide new functionality to meet specific needs without your writing one additional line of code.

Best of all is that filters are easy to write. Most of the remainder of this article will outline the simple steps to create your own FilterTop filter. First, let's get a general overview of FilterTop's operation from the user's point of view so you can form a better idea of how a filter fits into the functioning of the whole.

The building blocks of FilterTop are filters. The user can use just one at a time or pipe together two or more. In all cases, however, they must be combined and set up by means of SuperFilters. These in turn may be saved in files called Toplets, stand alone applets that perform multiple operations on files - not by themselves, but by using Apple events to pass instructions to FilterTop.

A FilterTop Toplet is somewhat analogous to an AppleScript droplet; you drop onto its icon in the Finder other icons representing any number of files (or you can just double-click it and add files later). A Toplet is opened, causing a SuperFilter to be recreated according to choices previously made by the user. FilterTop is launched if it is not already running and sent the information that enables it to display icons for the files to be processed and for filters that represent the operations to be performed on the files. If the option to do so has been chosen, the files are automatically processed; otherwise, FilterTop waits for the processing to be initiated by the user.

To create a SuperFilter in the first place, you drag icons from a list (displayed in a floating palette) to a window. Each filter has one or more ports, indicated on the top and bottom of the icon. You use the mouse to connect the filters in custom order by dragging from one port to another. When this is done, the filters are shown connected by pictures of pipes. Thus the concept of pipelining - an abstract one in operating systems without a GUI - is graphically illustrated on the Mac. In fact, with its menus, lists, windows and icons, FilterTop is very much a Macintosh application.

Figure 1. A SuperFilter set up and ready to run.

Icons for files to be processed appear in another part of the window. Figure 1 shows an example. Say you have a group of twenty files. Each is a SimpleText file and contains a section of an article. The files are named "Part 01," "Part 02," etc. Your object is to create a single plain-text file that contains the entire article and will open in Microsoft Word. By dragging and dropping with the mouse, you create a pipeline of several filters in the SuperFilter editor window. The first assures that the files are sorted by their names; the next concatenates the text of all the files; the next changes the creator code of the output file to that of Word; the next assigns a name to the output file; and the final filter sends the result to a folder you have designated.

You click a button and witness the start of processing: the filter icons become dark as data flows through them, the pipes swell when the data flows down to the next filter, and so on. At least that is the GUI representation of what is going on behind the scenes. Bruce Bennett, an amused commentator on the Internet, called FilterTop "unix batch processing by Rube Goldberg." That is the way it looks. (It also shows that members of the Mac community have a sense of humor!) Though FilterTop may appear to be an ungainly contraption that you watch with a smile, it works, and with surprising speed and efficiency.

Writing a filter to perform a given function is easier by an order of magnitude than creating a stand-alone application. FilterTop handles the tough stuff. It coordinates the operations of the various filters and assembles the files passed to it by the user. It passes to your filter the data to be processed. It allows you to retrieve the user's preferences as well as a certain amount of data to be used in performing the operation, such as a string to be searched for. When you are through processing - "filtering" - the data, the FilterTop engine receives the results, which it passes on to the next filter or saves to disk as a file.

Thus, you the filter programmer need be concerned only with the specific task you want your filter to accomplish. This may be simple, such as counting the characters in an input file, or much more advanced, such as applying a sophisticated compression algorithm or translating from one graphics format to another. Professional developers can find intriguing potential in FilterTop filters, yet programming one is simple enough for hobbyists.

Getting Started

For your filter to work properly you are required to follow a strict set of guidelines. This may seem constraining at first but this system has its advantages. First of all, FilterTop provides programmers with numerous aids to help you comply with its restrictions and get your code working. Secondly, once you set up everything correctly for interaction with the FilterTop engine, you are left with a great deal of latitude in what you can do in the body of your code. In addition, you will find that, once you are comfortable using the framework FilterTop provides, you are freed from the need to spend time on the details of pre- and post-processing, including handling most of the user interface and all but the simplest aspects of error trapping and reporting.

The Basic Steps

Here are the five basic steps to create a fully functioning FilterTop filter:

• Receive messages from the engine.
• Retrieve data from the engine.
• Process the data.
• Return the results to the engine.
• Send a return code back to the engine indicating the outcome of the operation.

There are also a few things to be done with resources at different stages. At this time support is provided for writing filters in "C," using either Metrowerks CodeWarrior or THINK C. Additional support may be added in the future. (It should also be mentioned that FilterTop is not currently PPC native; this will probably change with the next major revision.)

Receive Messages

To receive messages from FilterTop, you must #include "FTFilter.h", a header file that defines these messages. FTFilter.h, supplied by TopSoft with its other programming tools, also defines the data structures and functions that FilterTop uses and with which you must become familiar.

In fact, messages and everything else transmitted to you by FilterTop come through a parameter block, a pointer to which is passed to your filters main() function when its code is executed. In this respect as well as in others, as we'll see, a FilterTop filter resembles a HyperCard XCMD. If you have any familiarity with the latter you can easily understand the former. (Photoshop filters work in the same general way; if you can write one of those, a FilterTop filter is a cinch.) The structure of this parameter block is worth studying; once you get the hang of how it works you will have gone a good way toward comprehending the interface between your own code and FilterTop.

typedef struct
{
   FilterMessage   serviceRequested;        // essential
   long            filterShellVersion;      // ignore
   Callbacks       *cb;                     // essential
   DialogPtr       dlg;                     // rarely used
   EventRecord     *event;                  // rarely used
   Handle          filterGlobalsH;          // useful
   Handle          filterConfigH;           // rarely used
   long            data;                    // useful

}   PBCallToFilter, *PBCallToFilterPtr;

Understanding the FilterTop parameter block is easier than it at first appears, because it is essential to deal with only two of its elements. Two others are also useful at times, and the rest are either reserved by FilterTop or used only with certain special types of filters. filterShellVersion is reserved by FilterTop. filterConfigH, dlg and *event are used in filters only if the filters must interact with the user through their own self-created dialog. This is rarely if ever necessary since FilterTop also provides a remarkably complete means of putting up such a dialog and retrieving preferences automatically, through a mechanism called AutoConfig. More about AutoConfig later.

All filters are reentrant; that is, if a filter has been called, it may be called a second time (and a third, etc.) before the first instance returns. This imposes the restriction that filters may not have global variables (otherwise, more than one instance of a filter could struggle over the globals). It is permissible, however, for memory allocated on the heap to be used as though it were global, and the member filterGlobalsH is provided as a convenient way of accessing a block of these "globals." The member data is similar to the refCon provided in Toolbox window and control records. Basically, you can put any four-byte value you want there for convenient later retrieval.

The absolutely essential members of the parameter block, however, are the first and the third. (We will come back to the third a little later on.) serviceRequested is a variable that you must examine immediately whenever your filter is called. It is standard practice simply to set up a switch statement to handle the messages in serviceRequested, and in most filters the main function will consist of little more than this switch statement.

pascal FilterResult main( PBCallToFilter *pb )
{
   FilterResult   rc;

   ENTER_FILTER;

   switch (pb->serviceRequested) 
   {
      case msgOpenFilter:
         rc = filterMsgNotHandled;
         break;
      case msgFilterData:
         rc = MyFilterFunction( pb );
         break;
      case msgCloseFilter:
         rc = filterMsgNotHandled;
         break;
      default:
         rc = filterMsgNotHandled;
   }
   EXIT_FILTER(rc);
}

(ENTER_FILTER and EXIT_FILTER are macros that enable your filter to use the A4 register to handle static data that the compiler treats as globals.) msgOpenFilter and msgCloseFilter are occasionally useful if you want to set up some structure or perform a particular action only once at the beginning of a filtration session and not every time data is sent to your filter. There are other messages that you must know for special types of filters. In most cases, however, the only message you must act on is msgFilterData, which simply is the go-ahead that signals you to perform the action that your filter is supposed to perform.

The variable rc, of type FilterResult, receives a return code. Normally it will be filterOK, which indicates that your filter successfully did what it is supposed to do, or filterMsgNotHandled, as seen here (which simply indicates that you take no action on this particular message). On occasion you might also return filterAborted, in case you were informed by the engine that the user has aborted the filtration, or filterProcessingError, to inform the engine (or confirm that you have been informed) that an unrecoverable processing error occurred. Either of these codes, or filterOK, might be assigned to rc through MyFilterFunction in the example above, depending on what happened during filtration. That is all you must know about the FilterTop messages.

Retrieve the Data

The second member of the FilterTop parameter block, *cb, is extremely important. It is, simply, a pointer to another structure, this one consisting of (at this writing) twenty-two pointers to functions. These functions are the FilterTop callbacks, a term that, again, you will be familiar with if you have written any HyperCard XCMDs. The callbacks are functions that FilterTop provides for interacting with it; the engine passes you pointers to the callbacks, and you call back to the engine by simply dereferencing one of the pointers, passing the parameters required by that particular routine.

static FilterResult MyFilterFunction( PBCallToFilter *pb )
{
   FTErr   err, writeErr;
   Stream   inputStream, outputStream;
   FInfo   fndrinfo;
   FSSpec   suggested_spec;

   err = pb->cb->Open(input1, &inputStream, &fndrinfo, 
                                             &suggested_spec);
   HANDLE_ERR(err);

Note the double use of the pointer-to-member operator to call Open, a pointer to which is passed as part of cb, the Callbacks structure, which in turn is pointed to by pb, the parameter-block pointer. The Open callback will become very familiar to you, as it is used to open both input and output streams.

FilterTop is built around the model of streams. This concept should be familiar to anyone who programs in C. Streams, which are simply sequences of bytes, interpose a layer of abstraction between your filter and the files input to it. Instead of handling the details of file I/O, you simply open, read from, write to and close streams.

After opening an input stream, we must open an output stream:

err = pb->cb->Open( output1, &outputStream, &fndrinfo, 

&suggested_spec );
   HANDLE_ERR(err);

input1 and output1 are constants representing pipes from which to read, or to which to write, the stream in question. &inputStream and &outputStream are pointers to variables of type Stream; the FilterTop engine stores in these variables stream identifiers that you need to pass back to it whenever you access the streams. The next two parameters similarly receive Finder information (FInfo) and a file-specification record (FSSpec) associated with the stream. Usually these have been derived from a file on disk that was the origin of the stream. But since streams do not necessarily correspond to files on disk (they may have been created in a previous filter, for example), this information should not be considered as necessarily bound to a file. It is there so that if needed the engine can use it to create a new file on disk at the end of the filtration process. In most cases, you just pass along what you receive, although specialized filters may alter this information.

HANDLE_ERR is a macro that represents a simple error-handling mechanism, defined as:

#define HANDLE_ERR( err ) if( err != ftOK &&   \
                            err != ftEndOfData )   \
                            if( err == ftAbort )   \
                           return filterAborted;   \
                       else                        \
                         return filterProcessingError;

Process the Data

Now you are ready to have your filter perform its filtration.

#define BUFSIZE 1024

long bufsize = BUFSIZE;
char buffer[BUFSIZE];

while( err != ftEndOfData )
{
   err = pb->cb->Read( inputStream, buffer, &bufsize );
   HANDLE_ERR( err );

Use Read to obtain a convenient number of bytes from the input stream into a buffer that you have set up. This buffer may hold anywhere from a single byte to several megabytes. The latter is an extreme case. If you want to create a large buffer, you can allocate memory on FilterTop's heap. This may be necessary with certain kinds of filters. But whenever possible use a small buffer on the stack and process your data in small chunks, writing the processed data to the output stream as you go. This enables the data to be passed to the next filter immediately for efficient pipelining. Also remember that available memory must be shared among all currently operating filters, and there may be quite a few of them. FilterTop is multithreaded, and several threads may be active at the same time.

/*
Our filter performs a very simple operation: converting all plain 
lowercase characters to uppercase; we loop until all characters in 
the input stream have been read, checked, converted if necessary 
and written to the output stream.
*/
   long i;

   for( i = 0; i < bufsize; ++i )
   {
      if( buffer[i] >= 'a' && buffer[i] <= 'z' )
         buffer[i] -= 32;
   }
   if( bufsize > 0 )
   {
      writeErr = pb->cb->Write(outputStream, buffer, bufsize );
      HANDLE_ERR( writeErr );
   }
} // end while ( err != ftEndOfData )

When the Read callback returns the code ftEndOfData, that is the signal to end the while loop, as all the bytes have been read from the input stream. The FInfo and FSSpec were already copied to the output stream when it was opened. There remain only the resources associated with the stream to be copied to the output stream.

err = pb->cb->CopyResources( inputStream, outputStream );
HANDLE_ERR( err );

CopyResources simply copies all the resources at once, and very quickly; if there are no resources it does nothing, so it can't hurt to use this callback if you are not sure about the resource forks of the files you are filtering. FilterTop also provides callbacks for copying individual resources if you should want to do so.

Return Your Results

Now we close both the input stream and the output stream:

err = pb->cb->Close( inputStream );
HANDLE_ERR( err );
err = pb->cb->Close( outputStream );
HANDLE_ERR( err );

Notice that we always check for errors. This not only allows us to return control gracefully to the FilterTop engine in case a serious error has been detected (which we can do also if we encountered such an error in our own code; FilterTop offers yet other, more sophisticated means of error reporting that you should look into), it also gives you a way to find out if the user has decided to abort the operation. If the engine returns the code ftAbort from any of its callbacks, that is the signal to clean up and immediately return control to FilterTop. With a simple filter like this one, the macro HANDLE_ERR is all you need if you receive this message. All it does is return control to FilterTop (with the appropriate return code), and that is sufficient. (Though just barely. Unless your filter is of the simplest kind, you should consider using the ReportError callback before returning filterProcessingError. There you can pass a message string to the engine indicating the type of error that occurred. The engine then displays this message to the user.) More complicated filters may perform one or two cleanup operations. If any memory was allocated on the heap, it should be disposed of. In most cases, that is all you must worry about.

Send a Return Code

And, finally, since if we have reached this point all has gone satisfactorily, return to FilterTop (through our main function) with the code filterOK.

   return filterOK;
}

Always remember to return a return code whenever you hand control back to FilterTop. If you are responding to close messages, you also must clean up when you receive one. You might, for example, have allocated some quasi-global memory when you were opened; naturally, you must free this when told to close. The FilterTop parameter block includes the field filterGlobalsH as a convenient way of holding and managing a handle to a block on the heap; it does not have to hold "globals" but can be used any way you want.

AutoConfig

Now (continuing with our example) suppose you decide that the user should have the option of deciding whether to convert uppercase to lowercase text or the reverse. You want to display a dialog box giving the user the option to check "Uppercase to Lowercase." In a full-scale application this is relatively simple. Even so, you have to deal with setting up the dialog and trapping events to find out what the user did in the dialog. You could instead add a menu item, but here you are a stand-alone code module, and, running in the context of FilterTop, you cannot provide your own menu item or even trap events, or you might interfere with FilterTop's own operation.

Not to worry. FilterTop provides you, the filter programmer, with all the machinery to retrieve user preferences without having to be concerned about displaying dialogs (except in ResEdit), creating menu items, or handling a single event. This machinery is called AutoConfig. With AutoConfig, instead of devoting twenty-five percent of your code to creating and disposing of dialogs and managing event loops, you can handle the whole business of filter configuration and interacting with the user by a few minutes' work with ResEdit or another resource editor, and just a few extra lines of code.

The FilterTop developers' kit provides ResEdit templates for the two special resources required, 'fCFG' and 'fVLD'. In addition, you must create one each of the usual 'DLOG' and 'DITL' resources. FilterTop expects all these resources to have the ID of 128. There are three standard items that your dialog must have to be consistent with all filter-configuration dialog boxes. These are the three buttons that the user clicks to "Use Now" (the default, always item 1), "Cancel" (item 2) or "Make Default" (item 3), which tells FilterTop to save the current configuration as the default setup whenever this filter is used.

Figure 2. The AutoConfig dialog as seen in ResEdit.

We will add a single item of our own: a checkbox (item 4) with the text "Uppercase to Lowercase". (Figure 2 shows the result of setting up the 'DLOG' and 'DITL' resources.) By convention FilterTop passes on to the filter code the Boolean value of a checkbox with a single character (not the actual number), '1' for "true" and '0' for "false." This result is linked with the internal name of the item. This name (which has nothing to do with the name of any resource) is assigned by you in the 'fCFG' resource (Figure 3) and must be a four-character name of type OSType - the same kind of name used for resource types, file types and creator codes, among other things. We will assign the name 'UtoL'. A default value can be supplied, which we will make '0'. (We don't actually use the single quotation marks in the resource.)

Figure 3. The 'fCFG' resource.

The values in one more resource must be filled in to set up our AutoConfig dialog: those in the 'fVLD' resource (Figure 4). There are several fields to be completed in the template. Values such as "Group," "Class," etc., may be used to link dialog items in complex ways. Also, for dialogs with edit-text items, minimum and maximum values and lengths, as well as a set of valid characters, may be provided. All these options are explained in the filter programmer's documentation. For our simple example, as for all simple checkboxes unlinked to others, we can enter either zero or nothing at all in all the fields. Also be sure to enter the number of the dialog item. (Do not create items in the 'fVLD' resource for the three required button items 1 to 3; these are handled automatically.)

Figure 4. The 'fVLD' resource.

This done, modifying our code to retrieve the user's configuration preference requires just a few lines, as follows:

char configChar;
long configLen = sizeof( char );
   
err = pb->cb->CfgFindData('UtoL', &configChar, &configLen);
HANDLE_ERR(err);

Then we modify our loop to extend our functionality based on the additional preference:

   if( configChar == '1' ) // Uppercase to lowercase
   {
      for( i = 0; i < bufsize; ++i )
      {
         if( buffer[i] >= 'A' && buffer[i] <= 'Z' )
            buffer[i] += 32;
      }
   }
   else // Lowercase to uppercase
   {
      for( i = 0; i < bufsize; ++i )
      {
         if( buffer[i] >= 'a' && buffer[i] <= 'z' )
            buffer[i] -= 32;
      }
   }

All this takes longer to explain than to do! (Note: There already exists a "Change Case" filter, written several years ago by TopSoft's founder, Stephen Jovanovic. My example code is slightly different for demonstration purposes but does pretty much the same thing.)

The 'fINF' resource

At this point we almost have a fully functioning filter. We need only to register some information with FilterTop so it will know how many ports we have and what names should be attached to them, and a few other items. This is done in the 'fINF' resource (again, ID 128), which must be part of all FilterTop filters.

Figure 5. The 'fINF' resource.

Let's quickly run through the 'fINF' items. (See Figure 5.) The item "Filter Version" is your own three-digit number. There is an imaginary decimal point between the first and second digits. "Min FilterTop Version" at this time should just be "100." In the future there conceivably could be filters that will run with FilterTop version 2.0 but not earlier, for example. "Cur FilterTop Version" is also now "100," but will naturally change. "Stack Memory Required" must be filled in only if you determine that you will need an unusual amount of stack space. In most cases just leave this at "0" for the default. (It can't hurt to put in an estimate; future versions of FilterTop may require one.) "Config Possible" should be true if you use AutoConfig or are self-configuring and false if not. ("Self-configuration" is an option that admittedly has not been well supported; this is understandable, however, since AutoConfig has so far provided for all configuration needs anybody has had.) "Config Required" is not currently supported. Set this to false. "AutoConfig" is self-explanatory. "Uses Error Port": FilterTop provides the option of having a special output port for reporting of non-fatal errors - set this to true if you use this option. "Filter Name," "Author," "Copyright" and "Description" are all self-explanatory. There should, finally, be an input field for each of your inputs (currently only one is supported) and an output field for each output. What comes in and what goes out of these ports should also be briefly described.

Now, with these resources in your projects resource file, you are ready to compile your filter. Compile it as a "Code Resource" with a resource type of 'FILT' and ID of 128. Assign the resulting resource file a type code of 'FILT' and creator code of 'fTOP'. Depending on your code and your development environment, you may have to include one or more libraries in your project. Check the documentation if necessary.

That's it! You now have a FilterTop filter that will be fully recognized by FilterTop, can be added to any pipeline and will work harmoniously with all of the existing filters. Just drag it into the appropriate subfolder in the "Filters" folder. Again, all of the above steps take longer to describe than to do. With practice, all of this will become second nature as you build up a collection of useful filters, each easy to write because FilterTop handles all of the drudgery for you. Now you can devote all of your creativity to that translation or encryption utility (or whatever) that you've been hankering to write. Feel free to submit your work to TopSoft. It may just be added (with your permission of course) to the growing collection of free filters available at the TopSoft FTP site.

The FilterTop area of the TopSoft FTP site is located at ftp://ftp.topsoft.org/Visitors/FilterTop/. There you will find the complete FilterTop application (it's free!) as well as various filters not included in the main package. Also of special interest are the archive of filter source code (see what others have done) and the Filter Writer's Guide, which provides much more information about writing filters than I have been able to give here, including guidelines for creating your own custom icons for your filters and for writing filters that have more specialized requirements than our simple example. You can also easily create help documents that are modular like filters and are automatically added to FilterTop's elaborate About box. Also check out the Web site at http://www.topsoft.org. Information about anything you can't find at the FTP or Web site may be obtained directly from Tony Jacobs, the President of TopSoft, at info@topsoft.org.

Join the plug-in revolution now!

Thanks to John Tsombakos for his helpful feedback, which resulted in a number of improvements in this article; to Tony Jacobs for his encouragement and for urging me to write the article in the first place; and to all members of the FilterTop development team, past and present, without whose phenomenal efforts none of this would have been possible.

Alan Weissman believes that the tinkering of hobbyists is as important as the effort of professional developers in keeping the Mac an exciting platform. When not doing his own tinkering, he may be found working in Macintosh tech support, reading detective novels or listening to old jazz recordings, among multitudinous other activities. He may be reached at alanw@bway.net.