Volume Number: 13 (1997)
Issue Number: 3
Column Tag: Tools Of The Trade

MCL in the Motorola Chip Factory

By Stephen L. Hain, Digitool, Inc.

Creating a "fab" solution in a web browser using Macintosh Common Lisp and MIT's CL-HTTP server

Macintosh Common Lisp (MCL) is in active use in projects at several universities and corporations worldwide. One of these is Orchestra, a Motorola project to develop automated tools for scheduling wafer fabs (factories in which semiconductor wafers are fabricated) in their Semiconductor Products Sector.

When the Orchestra implementors decided to improve the user interface for the "Fab Browser" part of the system by putting it on the web, they found a simple and elegant solution using only a hundred additional lines of MCL code and CL-HTTP [Mallery 94], a Lisp program from MIT which I describe below. A description of that solution, based on a presentation by Byron Davies at the 1996 Dynamic Objects Workshop in Boston, appears later in the article.

Lisp differs from traditional languages such as C and Pascal in several ways. Lisp variables can hold values of any type, and there are quite a few available: strings, symbols, lists, vectors, structures, hash tables, integers, floats, ratios, and packages to name but a few. Lisp functions can be created and used without requiring an edit-compile-link cycle; they can be generated by Lisp code and passed as arguments to other functions; and they can "close over" data values in sophisticated ways. A macro facility allows for compile-time manipulation and generation of Lisp expressions, and a package system segregates program symbols (variable and function names) into separate namespaces to ease the integration of programs and development of large systems. There are powerful output formatting functions, a stream mechanism, and comprehensive mathematics functions and data types, including bignums (unlimited-size integers), complex numbers, and ratios. There are high-level control structures for dealing with non-local exits, and an object-oriented exception mechanism. And of course, there is CLOS (Common Lisp Object System), an extensible object-oriented framework that's particularly useful for implementing other object-oriented frameworks, if you're into that sort of thing. CLOS objects are instances of classes, and they are operated on by functions called methods.

What's "common" about Common Lisp is that it's a widely-used standard for the language, and code or portions of code that are compliant to the standard are able to run on platforms implementing a compliant Common Lisp. CL-HTTP is an example of a large Common Lisp program which runs on many platforms.

MCL is the Macintosh implementation of Common Lisp. MCL was originally developed at Coral Software, then became an Apple product when that company was acquired by Apple, and is now a product of Digitool, Inc. Over the last decade or so it has kept evolving - the current version, MCL 4.0, generates native PowerPC machine code. MCL was used at Apple as the host environment for the Apple Dylan and SK8 projects.

Besides being true to the standard embodied in [Steele 90] (commonly referred to as CLtL2), MCL provides a rich development environment based on the Macintosh user interface, and several useful extensions to the standard. Typically, the user interacts with MCL through one or more listener windows, specialized text editors that evaluate Lisp forms that are typed or pasted in, display their results, and maintain a history of the interactions. Files are edited and displayed in Fred windows (Fred Resembles Emacs Deliberately!) and, like other Emacs text editors, Fred is extensible and programmable.

MCL supports both low-level and high-level Toolbox interfaces. You can access the Toolbox most directly using the supplied interfaces, which define the constants, records, and routine ("trap") calls we've all come to know and love. Object files and shared libraries can be dynamically linked to an MCL program, and the interfaces to them are defined in a similar manner. In general, it's better to use the high-level interfaces to access the Toolbox. These interfaces include the menu, window, and dialog classes and methods, the file system functions and file streams, as well as a host of others (QuickDraw, MacTCP, serial streams, etc.) With these, a bug in your code will likely lead to a Lisp error instead of a crash into the debugger.

Built-in development tools include a window-based graphical data inspector; a stack-backtrace window which displays program execution frames, allows frame values to be changed, and let's you restart execution from any frame; facilities for quickly finding the definition(s) or users of a selected variable, function, or method; a programmable trace facility for tracing function calls, also accessible via dialog; an advise facility for modifying the behavior of functions; and a step facility for single-stepping through Lisp expressions.

The MCL development environment makes use of multiple processes (threads), as can MCL programs. Processes allow different pieces of Lisp code to run concurrently. Details of MCL's processes can be found in the MCL Reference Manual. Some of the many additional library and example modules that come with MCL are a graphical class grapher, WOOD (an object-oriented database), and even a turtle-graphics package.

Figure 1. MCL Listener, Get Info, and Inspector windows.

Figures 1 and 2 show several different MCL tools and windows in action. In figure 1 there is a listener, currently in a debugging break loop; a Get Info window displaying all of the callers of the window-close function; and several Inspector windows displaying information about the installed menus and menu items.

In Figure 2 there is a stack-backtrace window (associated with the listener process that is in the break loop); a Processes window displaying the status of all processes; and a Fred window displaying one of the example files (in two viewing panes).

Figure 2. MCL Stack Backtrace, Processes, and multi-paned Fred windows.

There is not enough room here to go into all of the features and tools in the MCL development environment; more information on them can be found in the "Getting Started with Macintosh Common Lisp" manual (available on-line at <http://www.digitool.com/mcl-getting-started.html>) and in [Parker 95].

The MCL program environment consists of a standard Macintosh heap, containing pointers, handles, resources, and the like, and a Lisp heap, containing Lisp data, including the functions of the development system itself and any user code. In addition, MCL 4.0 on the PowerPC makes use of shared libraries to separate various parts of the program. Once code is loaded into MCL, a new image (MCL application) can be created with a simple Lisp function call (see listing 1).

; load all source files in a folder:
(loop for file in (directory "HD:sources:*.lisp") do (load file))
; save out a new image:
(save-application "MCL + My Stuff")

Forms and functions "evaluated" from a Listener or Fred window are usually first compiled automatically by an incremental compiler, and then run as native code. Lisp source files can be compiled into object files which then load very quickly into MCL. There are two garbage collectors to pick from in MCL: the "mark/sweep" collector, which runs when memory has filled up, displaying a GC cursor as it pauses the program to do its job; and the ephemeral garbage collector, which operates in the background, cleaning up unused data as a program runs, rarely (if ever) causing a pause. Every listener window you create has its own process, so you can start some Lisp code running and continue to work with the MCL environment. There is a separate process running that handles events (mouse clicks, menu selections, key presses, Apple Events, etc.); should an error occur in that process - for example, in a function you've defined that is called from a menu item - all is not lost! A new, stand-in event process is created so that life can go on while you debug the problem. The latest versions of MCL can even grow the stacks of processes that use them up.

There are many different reasons why Lisp or MCL is ideal for some tasks; here are a few of my favorites:

• The dynamic environment. You can add and change functions quickly, and have access to the program you're working on while you're in the development environment. Or you can think of it as having access to the entire development environment while your program is running. Might you wonder if changing a program while it's running isn't asking for a trip to MacsBug? Well...

• Safety. If you stick to the high-level interfaces and supplied mechanisms, you're less likely to crash than to cause Lisp errors, which are handled more gracefully. Because Lisp is weakly typed (variables can hold values of any type), it spends some of its time checking for valid data before passing it down to the inflexible operating system routines below. (Declarations can be added to a Lisp program to speed it up by bypassing some of the type-checking, trading some safety for increased speed.)

• Elegance. Whatever you're programming style, you can't beat Lisp (well, I can't) for giving you so many facilities for writing clear and well-organized code. Macros, packages, and CLOS come into play here, as well as the exception mechanism and a wealth of control structures. (See listing 2)

; Define a macro named with-locked-handle.
(defmacro with-locked-handle (handle lisp-form)
  `(progn                   ; just groups the following forms together
     (lock-handle ,handle)  ; lock the handle
     (unwind-protect        ; ensure that unlock-handle is called, even if
                            ; lisp-form "throws out" to a higher frame
       ,lisp-form           ; this "expands" to the code in the macro call
       (unlock-handle ,handle))))

; Define a function that uses the with-locked-handle macro.
; (Assume that dereference dereferences a handle to obtain a pointer, and
; that deposit-bytes deposits bytes from a list into memory.)
(defun fill-handle (a-handle list-of-bytes)
  (with-locked-handle a-handle
    (deposit-bytes list-of-bytes (dereference a-handle))))

• Ease of debugging. The combination of the Inspector, function tracing and advising, Lisp breakpoints and break loops (Listeners created to allow debugging "paused" code), and having errors caught by Lisp instead of a low-level debugger can speed the debugging process considerably. Say you want to advise a function to write its arguments and result to a log file every time it's called. A lot of work? Not really - see listing 3. When you're happy with the operation of the function, you unadvise it.

; Make hairy-function log its arguments and result to a file.
; with-open-file is a macro - it takes some "keyword arguments" to guide it.
; loop is also a macro, that parses an English-like form and generates tight Lisp code.
; :if-does-not-exist, :create and other colon-prefixed symbols are "keywords" - symbols that
; evaluate to themselves (unlike variables, which evaluate to the values they are bound to)
(advise hairy-function
        (with-open-file (file-stream "home:log"
                         :if-does-not-exist :create
                         :if-exists :append
                         :direction :output)
          (loop for argument in arglist do
                (print argument file-stream))
          (let ((result (:do-it)))     
; this is where hairy-function actually gets called
            (print result file-stream)
            result))
        :when :around)

A concise description of the "Fab Browser" component of Mototola's Orchestra system appears in [Davies 1996]:

" the Fab Browser is a textual outline browser, implemented within MCL's Fred version of Emacs. In the browser, an object is displayed as lines of text, each representing a component or attribute of the project. Some of the text lines represent other objects, which may in turn be expanded into multiline displays. expansions happen very quickly because tens of megabytes of manufacturing data are cached in a network of CLOS objects within the browser application.

Because of the large amount of data cached within it, the browser application runs in 70 to 100+ megabytes of memory, depending on the amount of historical or projected data. This was a significant roadblock to widespread use of the browser, both because of the amount of RAM needed to run such applications efficiently and because of problems of keeping large applications up to date on multiple desktops, some connected by modem lines. The situation demanded a client-server solution, but resources were not available to create either a customized server or a customized client.

The goals for the web implementation were to (1) provide access to the same data, (2) preserve the look and feel, which had already demonstrated its effectiveness, (3) preserve performance, and (4) eliminate dependence on a heavyweight, expensive user platform in favor of lightweight, inexpensive client software."

Figure 3 shows a Fab Browser window using the original, Fred-based user interface.

Figure 3. A Fred-based Fab Browser window.

Before getting into the implementation of the solution to the problem, a little bit about CL-HTTP is in order.

CL-HTTP is a full-featured HTTP server developed at the MIT Artificial Intelligence Laboratory. It is written in Common Lisp, is available for several Lisp platforms, comes with full source code, and is freely available. CL-HTTP is, according to the on-line documentation for the server,

" implemented in Common LISP in order to facilitate exploratory programming in the interactive hypermedia domain and to provide access to complex research systems over the Web. The general motivation for developing this server was to provide a computational tool that would strengthen the link between the artificial intelligence researchers and the distributed hypermedia community. As the amount of information available over the World-Wide Web grows, it will become increasingly necessary to deploy intelligent methods to store, retrieve, analyze, filter, and present information. At the same time, high-productivity programming tools employed by AI researchers will become increasingly relevant for testing new ideas in servers before incorporating them into standards-based clients. A Common LISP HTTP server provides a bridge that allows AI researchers to plug their systems into the WWW as it affords developers of distributed hypermedia standards a vehicle through which they can.

As the Orchestra project demonstrates, it's not only AI researchers who stand to benefit from adding CL-HTTP server capabilities to their software. A CL-HTTP server is basically a Lisp program, running on a (usually) networked computer, that "exports" certain URLs, making them visible to any machine on the network that is authorized to access them. CL-HTTP listens for network requests for these URLs, processes them, and returns HTML pages to the requesting clients. The exporting and processing of the URLs is where other programs hook into CL-HTTP server, as discussed below.

The Orchestra program consists of over 80,000 lines of Lisp code. CL-HTTP, while not quite as large, is still a significant program. Using MCL, the process of making these two systems work together was as simple as loading them, adding a few functions and calls, and modifying a few Orchestra routines. Due to the interactive nature of MCL, it was not necessary to recompile anything more than the few files that had to be modified.

CL-HTTP supports many types of URLs, including one type that causes a Lisp "response function" to be called when that URL is requested from a remote client. CL-HTTP passes the response function an object representing the URL, and a stream object to which the function sends its "response" (a stream of HTML text).

Listing 4 shows the Lisp form that the Orchestra Fab Browser uses to instruct CL-HTTP to export the top-level URL for the system.

(export-url #u"/fm.html"
            :html-computed
            :response-function #'write-extended-factory-model
            :documentation "Top level access to fab model"
            :keywords ‘(:cl-http :fab-model))

The #u"/fm.html" syntax will expand into a CLOS object representing a URL that looks something like this: http://some.place.com/fm.html (depending on the host machine's network name, of course). The :html-computed value specifies to generate the kind of URL that is associated with a response function, which itself is then specified as write-extended-factory-model (#'foo is a Lispy way of saying "the function named foo"). The other parameters are optional CL-HTTP "housekeeping" information.

The write-extended-factory-model response function is in Listing 5.

(defmethod write-extended-factory-model 
((url http-computed-url) stream)
  (with-successful-response (stream :html)
    (let ((title "Wafer Fab Browser - Top Level"))
      (with-html-document (:stream stream)
        (with-document-preamble (:stream stream)
          (declare-base-reference url :stream stream)
          (declare-title title :stream stream))
        (ns1.1:with-document-body 
 (:background :white :stream stream)
          (with-section-heading (title :stream stream)
            (image-line :stream stream)
            (let ((*browser-conn* (find-browser-conn))
                  (*output-stream* stream))
              (declare 
 (special *browser-conn* *output-stream*))
              (with-rendition (:teletype :stream stream)
                (orch:factory-browser-initial-contents :mos6))
              (fresh-line stream)
              (image-line stream)
              (wfb-signature stream))))))))

The above Lisp form defines a method (a function specific to a certain class or set of classes) called write-extended-factory-model, which will be called with two arguments: url, which is an instance of the class http-computed-url (a class supplied by CL-HTTP); and stream, which is not specified to be of any particular type.

There could be other methods of the same name, taking arguments which are of different classes; it is the CLOS object system which brokers which method is actually invoked when a call to write-extended-factory-model is made.

with-successful-response, with-html-document, with-document-preamble, with-section-heading, and with-rendition are CL-HTTP macros used with code that generates HTML. Some of these macros use keyword arguments (e.g., :expires, :stream) to determine exactly what code to generate. Note that the name of the ns1.1:with-document-body macro is a symbol from the ns1.1 package: there may be other macros named with-document-body defined in other packages. The other macros don't have such package qualifiers because they have presumably been made accessible to the current package by means not shown here.

The stream variable that is so prevalent above holds a Lisp object, supplied by the CL-HTTP system, that directs characters to a buffer that will be output by low-level networking code to a client browser. Instead of the stream value being passed down to orch:factory-browser-initial-contents, the "special" (globally accessible) variable *output-stream* is bound (set) to that value, so that other parts of the system will be able to direct their output there.

declare-base-reference, declare-title, image-line are CL-HTTP functions that generate HTML text. Note that in general, the implementor does not have to be too concerned with whether a lisp form denotes a macro or function call.

find-browser-conn, orch:factory-browser-initial-contents, and wfb-signature are Fab Browser functions. The two variables *browser-conn* and *output-stream* are bound to (i.e., given) values in the enclosing let form, and because they are "declared special," the functions called within that let form will have access to their values.

From the above code, it's apparent that the function orch:factory-browser-initial-contents is being used to generate the body of the HTML page presented to the client. The function must have previously been used to display a page of text in a Fred buffer, where clicking the lines of that text would elicit changes in the displayed text or other actions. Now, its job is to output a page of HTML text that contains hypertext links. Each link will need to have associated with it another exported URL, and each of those URLs would in turn need to have a response function which would be called by CL-HTTP when the link was clicked, to display a new page or take some other action. Hmm, that could turn into quite a lot of URLs and response functions! Fortunately, CL-HTTP has a special kind of URL, called a search URL, which allows one exported URL to elicit a range of responses from a number of different hypertext links. Without going into too much detail on that mechanism (provided by [Mallery 1994]), we can show how such a URL would be used for the Fab Browser (see listing 6).

(export-url #u"/expand?"
            :search
            :response-function #'respond-to-expand
            :documentation "Expand a fab object.")

Specifying #u"/expand?" and :search indicates that the function named respond-to-expand should be called whenever a URL that looks like the following is requested (where text following the ? can vary).

http://some.place.com/expand?somekey=value&anotherkey=othervalue

So, you can pepper an HTML page with such URLs, each having different keys and/or values, and when one is clicked, CL-HTTP arranges to call the proper response function, passing it that key/value information. The respond-to-expand is shown in listing 7.

(defmethod respond-to-expand ((url http-search) stream)
  (with-slots (search-keys) url
    (with-successful-response (stream :html)
      (with-html-document (:stream stream)
        (let ((title "Wafer Fab Browser"))
          (with-document-preamble (:stream stream)
            (declare-base-reference url :stream stream)
            (declare-title title :stream stream))
          (ns1.1:with-document-body // (:background :white :stream stream)
            (let* ((browser-conn-id // (read-from-string (first search-keys)))
                   (browser-conn // (find-browser-conn browser-conn-id))
                   (object-id // (read-from-string (second search-keys)))
                   (object // (browser-conn-lookup object-id browser-conn))
                   (*browser-conn* browser-conn)
                   (*output-stream* stream))
             (declare (special *browser-conn*))
             (with-section-heading (title :stream stream)
               (image-line :stream stream)
            (orch:expand object // (orch:plan-customer orch::*factory*) 0)))
           (image-line :stream stream)
           (wfb-signature stream)))))))

Since this method outputs an HTML page like the other method above it, they look pretty similar. Note that this method is defined for a different kind of URL (an instance of the http-search class). Also note that CL-HTTP uses the url object to stash the keys and values from the request - with-slots is a Common Lisp form that extracts a value (search-keys in this case) from a slot of a CLOS object (url in this case). CL-HTTP arranges for the search-keys slot to hold a list of the keys and values, as strings. Orchestra decodes them (read-from-string produces a Lisp object given a string, for example the string "259" would produce - you guessed it - the integer 259) then uses them to access internal database objects (via find-browser-conn, browser-conn-lookup) and finally, it passes those objects down to a function which will display the proper HTML page (orch::expand). That page, in turn, may contain other hypertext links, associated with the exported search URL or other exports that CL-HTTP was instructed to create.

The above discussion demonstrates a few points about Lisp & CL-HTTP:

• Using CL-HTTP macros to hide the details of HTML code generation;

• Using packages to ensure that large programs (here, Orchestra and CL-HTTP) can be loaded together and communicate with each other;

• Adding a small amount of code (the above two methods) and changing a small amount of code (the two functions in the ORCH package) can essentially create an entire new user interface for a Lisp-based program;

And the result? From Davies' paper:

" we ported the outline browser to CL-HTTP in about one person-week. The goals for the web implementation were achieved for the most part. Fab data is now accessible through a web browser to anyone in the wafer fab with a computer, whether Mac, PC, or UNIX.

Figure 4 shows a number of Fab Browser windows using the new, web-based user interface.

Figure 4. Web-based Fab Browser windows.

Motorola's Orchestra project will continue to evolve, and CL-HTTP has opened up new avenues of functionality for it. Besides the Fab Browser, the other parts of the system (fab model extraction software, historical data browser, scheduling system, and execution system) will find ways to benefit from the new, web-based user interface. The manufacturing managers and associates who use the system will continue to see improvements in their ability to access and manipulate the data they need.

MCL is also evolving. Digitool is improving its compilers and development environment; is developing a high-level interface to Open Transport (to be available early in 1997); and has projects under way involving QuickDraw 3D, QuickDraw GX, and OpenDoc. Soon, PowerPC MCL developers will be able to develop smaller stand-alone applications, and we are exploring ways in which to use the shared library mechanism to give users more delivery options for their programs.

I'd like to thank Byron Davies of Motorola and Victoria Bryan Davies of Industrial Lisp and Magic for permission to use their paper as the basis for this article, and John Mallery for his work on CL-HTTP.

Davies, Byron and Davies, Victoria Bryan. "Patching onto the Web: An Emerging CL-HTTP Success story". 1996 Dynamic Object Workshop of the Association for Lisp Users, Boston,1996:. <http://www.ai.mit.edu/ord/alu/dow96/DOW3/2.SUB9-3.html>.

Mallery, John C. "A Common Lisp Hypermedia Server". In Proceedings of The First National Conference on the World-Wide Web, Geneva: CERN, May 25, 1994.

<http://www.ai.mit.edu/projects/iiip/doc/cl-http/home-page.html>.

Parker, Rich. "Macintosh Common Lisp 3.0". MacTech Magazine (formerly MacTutor) 11:9 (September 1995), pp. 44-52.

Steele, Guy L., Jr., Common Lisp the Language. 2nd edn.

Digital Press, 1990. <http://www.cs.cmu.edu/Web/Groups/AI/html/cltl/cltl2.html>.