Volume Number: 13 (1997)
Issue Number: 8
Column Tag: Rhapsody

Rhapsody Brain Dump

by Michael Rutman, independent consultant

What did Apple have to say about Rhapsody at WWDC??

Once a year, Apple hosts the World Wide Developer Conference. Developers from all over the globe come to Cupertino to learn what Apple plans for the next year. In the past, Apple wanted to show us their technologies as early as possible to get our feedback as well as help us prepare our products for the new technologies. Unfortunately, after the developer feedback and market realities have settled in, what Apple showed doesn't always make it into the marketplace. This has left the developer community a bit jaded towards Apple telling us about new technologies at WWDC.

This year, however, was different. Apple had only one main message, and that message is Rhapsody. Last December, Apple purchased NeXT, put everything else on hold, and has been working day and night to integrate their new technology with the existing Macintosh OS that we have all come to love. The teams were merged in the first week of February, so Apple was really only able to talk about what they have done in the last 3 months.

What did Apple have to show for only 3 months work? They showed a lot. They showed Marathon running on a diskless Macintosh computers booted over a network. They showed OPENSTEP with the start of a Macintosh look and feel. They talked about what will continue to work and what will break, and for the most part, everything will continue to work, as long as the software is not patching low level routines, or directly messing with hardware.

However, Apple also projected a huge reality distortion field. Even though they had been working on this for only 2 months, the level of excitement at Apple had people believing. They showed things running, and developers, including me, were willing to believe that anything's possible. You should remember that after WWDC developers sometimes don't think as clearly as they should.

Rhapsody is the mature NEXTSTEP foundation -- Mach, OPENSTEP, BSD UNIX, and Distribute Objects -- mixed with the industry leading Apple technology: QuickTime Media Layer, ColorSync, and AppleScript. Rhapsody will also include other cutting edge technologies, such as EOF, WebObjects, and Java. Rhapsody is still developing, so more technologies probably will be added over the next year.

How does Apple plan on putting all this together? Figure 1 shows the Apple Roadmap to Rhapsody, but what does it mean? The bottom box represents the hardware. That currently can be either PPC or Intel chips. For the vast majority of programmers, the hardware chip is completely irrelevant because the Core OS layer above the hardware entirely covers the hardware.

Figure 1. The Apple Roadmap to Rhapsody from the DevWorld web site.

The Core OS layer consists of Mach, a BSD Implementation, and a plugin driver architecture. Mach is often confused with unix, but in reality is a micro-kernel. The Mach code consists of a mere 35 thousand lines of code out of over 6 million lines in Rhapsody. The Mach micro-kernel controls the Virtual Memory, Tasks, Threads, multiple processors, and inter-application communication.

The BSD implementation is a fully compliant unix with Posix extensions. However, many Macintosh users have come to fear and loath unix. Well, maybe not fear and loath, but few Macintosh users embrace unix, and don't want to learn unix tools to get their jobs done. Fortunately, Apple recognizes this and has hidden the unix layer near the bottom. Most users will never see the unix layer, and better yet, most developers can ignore the unix layer. Best of all, if you like unix, it's there and is accessible.

The plugin driver architecture was created because of OPENSTEP for Intel. In the Intel market, NeXT had to be able to create new drivers quickly. Every month, there were new drivers to port, and few of the hardware manufacturers would standardize. The Rhapsody team is working very hard on this. In addition, because this is a Mac, Rhapsody is also promising true plug-and-play.

For most people, users and developers, the entire Core OS is hidden behind the next layer, which is either the Blue box or the Yellow Box.

Neither Apple, nor the developers, are prepared to abandon the thousands of applications already created. Many of these are legacy applications, which will continue to be used for many years. Apple is creating what they call a compatability layer, code named, "Blue Box".

The compatability layer consists of the MacOS, the Finder, ToolBox, extensions, whatever currently runs on your Macintosh. Apple plan to take all of this code, and turn it into an OPENSTEP application. This application, running under Rhapsody, will run almost all existing Macintosh Software.

How can Apple do this? Isn't there a huge difference between an OS and an application? Actually, there isn't. Both are written the same way, both are launched the same way, and both can do the same things. An OS is used to launch other applications, but many applications also do this with a plug-in architecture. There is very little difference between how Netscape uses helper applications and an OS launching an Application.

Apple is planning on presenting users with 3 ways of running the Blue Box: a single window for all Blue apps, Blue apps using all of the screens, and dual boot. As each user's needs are different, Apple is looking at a variety of solutions to best serve their users.

The single window mode is best for those users that quickly migrate to Rhapsody, but want to continue running a few older applications. These older applications, called Blue Apps, all appear in a single Rhapsody window with the Blue Box menu bar. If two Blue Apps are running, their windows appear in the same Blue Box window. Rhapsody applications continue to look and act like normal Macintosh applications.

The second mode, where the Blue Box takes over the screen, looks like a traditional MacOS 8 machine, but has Rhapsody running in the background. Rhapsody applications are not able to interact directly with the user, but background processes continue to provide services. A user is able to quickly switch between these two modes.

The last mode is a dual boot mode. No matter how hard Apple tries, some applications just aren't going to work in a compatability mode. Some developers have programmed directly to the hardware, and their applications expect there to be hardware, not a compatability layer. Despite there being very few of these applications, some users will want to continue using them. To allow these users to have access to Rhapsody and MacOS, Apple has a dual boot mode. At boot time, users can choose to boot into Rhapsody and run the Blue Box, or they can choose to boot straight into MacOS 8. With a reboot, users will be able to run their incompatable applications, then reboot back to Rhapsody as they need.

The Yellow Box has two parts, the Java VM and OPENSTEP. The Java VM has been talked about in just about every trade magazine for the last year, but OPENSTEP, despite being around for almost a decade, is still relatively new to most Macintosh programmers.

Apple's Java VM is still being defined, but so is the Java language. Apple has fallen behind on the Java curve, but is working to catch up. Apple is promising JDK 1.1 for the Java VM, full implementation of AWT, and 100% pure Java libraries. Apple is also providing the cross-platform, web oriented Java capabilities, and Apple is looking to integrate Java into the system as tightly as any other language. Java under Rhapsody can run QuickTime3D, send Apple events, use native code... whatever.

OPENSTEP is the main part of the Yellow Box. OPENSTEP is based on NEXTSTEP. It has existed in one form or another for almost a decade, but in the last 3 years, NeXT has fleshed it out with Sun and HP to provide a full-featured framework for developing applications.

OPENSTEP, from NeXT, was a second generation object oriented framework. Originally, NeXT provided developers with, most people will agree, the most powerful framework for developing applications ever seen. Study after study showed how development under OPENSTEP would go 3 to 10 times as fast as under either Macintosh or Windows. NeXT, however, felt that it wasn't good enough, and they sat down and redid it.

They took everything that developers liked, and expanded them. They then fixed most of the problems developers had been complaining about for years. They then addressed the issues developers were going to have in the future, but hadn't gotten around to yet. Now, Apple owns it, and is adding more technology to make it even better.

Even though this sounds like a lot of marketing speak, Apple was prepared to show off their power. They started with a demo where a marketing person made a version of SimpleText without writing a single line of code. They then had engineers come up and talk about how to implement various technologies. If there was one unifying theme, it was simplicity and rapid development.

Even without OPENSTEP's rich environment, Apple has led the market in certain areas. Apple is the leader in QuickTime, and their scripting environment is second to none. Furthermore, Apple defined ease-of-use, and while NEXTSTEP is the easiest to use unix, nothing is as pleasant as the Macintosh user's experience.

Apple has already ported the QuickTime Media Layer, ColorSync, GX Typography, and V-Twin to the Yellow Box. Apple is also creating more widgets, such as the TabView and OutlineView. The OutlineView is the Finder View, and by making this public, developers will be able to make their applications look and feel more like the Finder, and each other.

Apple is also working on porting AppleScript by integrated it into the frameworks. As any application developer will tell you, implementing a full-featured AppleScript environment can be as much work as all other features combined. While that's an exaggeration for most applications, it's still very difficult. By putting AppleScript into the framework, the hope is to make supporting AppleScript as easy as anything else under OPENSTEP.

OPENSTEP is made up of a number of different kits. The original kit -- the AppKit -- handles all UI features such as windows, views, menus, and controls. The AppKit has been expanded and revamped over the years to provide more features and to be easier to use.

NeXT also created a MusicKit to help developers create music applications with full support for MIDI and score files. Unfortunately, very few applications based on the MusicKit were ever created, and NeXT eventually unbundled the source and released it, with full sources, on the Internet.

Later, NeXT realized that connecting to databases was both important and hard. Every database has a different API, and each version of each database is slightly different. Furthermore, every database API is hard to use. NeXT's first attempt was the dbKit. The dbKit wrapped an object oriented framework around any database. A retrieval returned a list of objects containing the data requested. Complex searches and joins were, likewise, done through an object-oriented approach.

Unfortunately, NEXTSTEP, at the time, was not powerful enough, and dbKit had serious design flaws. NeXT responded by dropping dbKit, creating a more powerful framework called the FoundationKit, and re-writing the database kit as the Enterprise Objects Framework (EOF). Even though EOF has a different API, and is more powerful, the basic concepts behind it are the same as in dbKit.

The FoundationKit dealt with two major difficiencies in the original NEXTSTEP, garbage collection and the ability to encapsulate arbitrary data in objects.

Automatic garbage collection was needed to manage the deallocation of objects returned by functions. Each time an object was returned, there was always the question of which object is responsible for freeing it. If the receiver didn't free it when it was supposed to, memory leaked. If the receiver freed it when it wasn't supposed to, the second free method would cause a crash. dbKit passed a large number of objects through its layers, and at each layer, there was potential for leaking. Automatic garbage collection handles this problem by letting objects hold and release objects, then freeing them at the next iteration of the event loop.

Objects that encapsulate data are important as databases return numbers and strings. Dealing with the return values as either strings or data can lead to confusion. The same call can return a string one time, a number the next time, and a list of strings and numbers the third time. Worse, memory allocation for the storage has to be freed, and the arbitration of memory freeing is a nightmare.

NeXT dealt with this by creating a data object. Any number or string can be placed inside an NSData object. Automatic conversions between types allow programmers to abstract the return type and deal with the data as needed. Furthermore, the garbage collection handles memory allocaion and freeing.

Apple has evaluated user experiences with the Macintosh, and has found that diversity is good. Some users want a "modern" look with 3D graphics and soft grey-scales, others want a colorful experience, and some users with site imparements need a black and white, high-contrast environment. Apple is giving us the Appearance Manager so all users will be satisfied.

In theory, the Appearance Manager allows a user to specify how they want their applications to look, and all applications written correctly (and the vast majority are) will adopt the specified look. Currently, Apple supports only one appearance, and that one is the "modern" look, but Apple promises to help third party developers create as many appearances as they want.

No matter what the appearance, though, and no mater what system is being run (Java, MacOS 8, Rhapsody, Blue Box, etc.), the User Interface Layer will call down to the appropriate layer underneath it. An OPENSTEP Window will know to call the OPENSTEP layer and a Java Window will call the Java VM, even though, for the user, the two windows look identical.

For the developer, this is great news. No longer will developers have to worry about the nit-picky UI issues, such as using flat gray vs. dithered gray, and getting light sources correct. As long as the guidelines are followed, the system will put the correct appearance in the application.

Applications can communicate through Apple events and TCP/IP. Applications can transfer data through the file system and copy and paste. Rhapsody applications can also talk to each other through Distributed Objects and mach ports. Distributed Objects are a way of sharing objects between processes on the same machine or on different machines. Mach ports are the low-level mechanism mach uses for communicating.

Programming at the mach port level is not difficult, but it is not as straight-forward as most OPENSTEP programming. To make communication easier, OPENSTEP hides the mach messaging. One of the ways of hiding the mach messaging is by putting a standard BSD unix on top of it. In reality, the BSD layer is no easier to program, but it has the advantage of thousands of programs already support it.

The other, and better, method is the Distributed Objects (DO) mechanism. DO is used to transfer objects from one process to another. The receiving process can be on the same machine as the sender, or anywhere on the same network. Once connected, it is irrelevant to the programmer where the receiver or sender are. Unlike Java, where the entire object, including byte code, is sent, in DO only a proxy to the object is sent. Calls to that object then result in a call across a mach port. Calls to local objects and remote objects are identical, and from the application developers view, their is no difference between a local object and a remote object. These mach messages are quite fast.

The Macintosh uses HFS, and OPENSTEP uses a unix file system. Needless to say, there will have to be a compatability layer between them. Apple has developed a multi-layer scheme to allow access to any file system from any layer.

Apple has three goals with their file system: compatability, performance, and expandibility. Compatability is more than just supporting existing Macintosh applications running with an HFS volume. Compatability issues now involve making old Macintosh applications work with new file systems as well as making new applications work with the old HFS volumes.

Performance issues are always important, and they always will be, but it's good to see Apple recognizing the issue and making it a high priority.

Expandability is important for the future. In the past, the Macintosh has suffered from lack of expandability in the file system. Already, developers have had to deal with HFS versus MFS, using File Specs versus using the old style file operators, adding hard drives, AppleShare, and so on. Each time, there has been a new API programmers have had to learn. In the new system, developers will write their code once, and no matter what changes at a lower layer, their code will still work.

To achieve these goals, Apple has gone with a multi-layer file system. At the bottom is the native file system. The new file system supports HFS, NFS, UFS (Universal File System), HFS+, and allows for new file systems created later. To accomplish this, all the native file systems plug into the Virtual File System (vfs). The vfs layer knows how to convert file information for the different volume formats and the different API's. The vfs layer works with a stack of modules. Each module is called in turn, and developers that used to patch the file system only need to add a module to the stack.

Above the vfs is either the Posix+ layer, or an HFS/HFS+ layer. For Blue box applications, the HFS/HFS+ layer is the top level of the file system, and all Blue Box applications will access this layer. Even though it is an HFS interface, other file systems can sit under the vfs and be accessed.

Rhapsody applications, have a variety of interfaces for accessing the Posix+ layer. For support of unix applications, there is the Posix layer. OPENSTEP applications, however, have abstracted the concept of reading files into the framework. In a nutshell, tell an object to save itself, and it does. Tell the object to restore from a file and the object unarchives itself, taking into consideration version numbers for individual objects.

One concern Macintosh developers have is their file system metadata. The metadata, on the Macintosh, currently consists of Finder info, but there are many plans to expand metadata in the future. Apple recognizes this, and is looking to provide a flexible metadata scheme that is compatible with existing API calls. So far, Apple expects to support metadata for all file system formats.

Furthermore, Apple recognizes the need to get access to the File System at all levels and Apple plans to release API's for accessing all levels of the File System. In general, it will always be best to access the File System at the highest level possible, but for some developers, there will always be a need to go one level lower.

Under the old Macintosh, files were specified by a directory ID, volume ID, and a file name. Case sensitivity never mattered. Under the new system, files are accessed by path names, and there is an optional case sensitivity. In general, Apple recognizes that using pathnames can lead to problems, such as broken paths caused by folders being renamed, but to support all existing standards, pathnames will have to exist.

The optional case sensitivity is an interesting idea. Two files with the same name but different cases can exist under most file systems. Under HFS, where the case is irrelevant, this is not possible, but under NFS or UFS, it is both likely and common. Under Rhapsody, when a file is specified, first a file of the exact same case is looked for, if it is found, then that file is returned. If it is not found, then the vfs will look for any other file with the same name and path, but different cases.

Rhapsody, at the high level, is also providing some abstractions for file systems. As already mentioned, developers can read and write objects as files. In addition, there is an NSFileHandle object for handling file descriptors. Finally, there is a new file management abstraction based on URL's. Apple is excited about using URL style files, but they did not have a lot to talk about it.

Reading an object from a file is pretty simple. Here is an example:

contents =
NSString.stringWithContentsOfFile(@"/Library/README");

This is the new syntax to Objective-C, and currently no compiler supports this syntax, but soon they will. The '@' say;s to create an NSString object containing the string following it. Garbage Collection will clean it up later. NSString is a class, the '.' is the new way to call methods, and stringWithContentsOfFile is a static method. Basically, this one line of code will find, open, load, and close the file. As expected, there is a command for saving an NSString to a file. Many OPENSTEP objects have methods to facilitate file system accesses.

Finally, because Apple wants to have full and strong Java support, they are implementing a Java like mechanism for doing asynchronous file operations. As this was still in prelimary stages, Apple didn't have much to show about it.

Apple is going to provide three mechanisms for dealing with hard drives, and they will vary as far as compatability versus convenience. There will be Blue Box only drives, Rhapsody files that look like Blue Box drives, and drives shared between Rhapsody and the Blue Box.

Blue Box only drives will be compatable with almost all existing software. There may still be a few applications that break, but Apple is going for maximum compatability. Unfortunately, any data saved by a Blue Box application on a Blue Box only drive will not be accessable from a Rhapsody application. As far as user experience goes, this mode is only for the last ditch, have to get this one piece of software I own running, type of users.

Drives shared between Rhapsody and the Blue Box are most convenient, but patches to the file system will break. A lot of software out there patches the file system. Disk doublers, some compression software packages, MPW, all patch the file system, and none of these will work. However, most productivity software packages will work just fine, and being able to save a file from the Blue Box and open it in another application under Rhapsody will offer users the best environment.

The final mode, the Rhapsody file that looks like a Blue Box volume offers some advantages over the other modes. The hard drive shows up under Rhapsody, but a file can be turned into a Blue Box volume. The Blue Box will treat the file as a volume, and write to it like a volume. Rhapsody applications can access it at the raw volume level, but won't have easy access to the files in it. This would be the route for a user that needs a Blue Box only volume for an application, but does not want to give up an entire hard drive.

One last thing to talk about is HFS+. HFS, the file system Apple has used for many years, has several problems. The biggest is the large allocation block size. With today's one and two gigabyte drives, users are finding trivial files take up 44k. Furthermore, HFS files are limited in size. Apple has rewritten HFS to provide small allocation blocks, large files, and direct unicode support, and they have added extensions for Rhapsody, such as permissions, hard links, and better date support. All this, and HFS+ is still compatable with HFS.

Rhapsody applications, being based on mach will see and use virtual memory at all times. Blue Box applications, on the other hand, will never see the VM, even though the Blue Box will use it. Gestalt will even go so far as to say VM is not running. There will be a new API for requesting held memory at a later time, but existing applications will have to be updated to use this. Meanwhile, this will cause some problems for certain applications. Apple's solution is to provide an option for starting the Blue Box to run using only physical memory and never swap any of its memory.

Under normal use, though, the memory map will have a 1 gigabyte virtual space. Running applications will think there is a full gigabyte of RAM in the machine. Each application can set its partition to 100 megabytes and use all the memory it needs, or the memory partition can be left alone and hundreds of applications can be run. Applications will allocate their memory from the core OS VM, not from temp memory.

Furthermore, guard pages will be added around application stacks and heaps. No more walking off the stack and trashing memory. Unfortunately, this will break a few apps that try to walk the heaps, but for the extra stability, it is well worth it. Once again, if users have an incompatable application they need to run, they can turn off the new memory map.

In a shared Blue Box/Rhapsody world, there are going to be device conflicts. Because Apple is moving to Rhapsody, and because the Rhapsody's driver model is cleaner, Apple is going to favor Rhapsody drivers over Blue Box drivers. Where Rhapsody is set to ignore a particular device, then the Blue Box can access it directly, otherwise, the Blue Box will use Rhapsody accessing the devices.

In the Blue Box, there are no native device drivers, no ndrv, no name registry, no SCSI access to devices Rhapsody is accessing, no VIA access, no SCC access, no ADB support for devices in use by Rhapsody, and so on. In other words, if the core OS can handle it, the Blue Box can't touch it.

There will be some support for native drivers in the core OS, and DRVR's that do not touch the hardware will continue to be supported. Furthermore, serial ports accessed through OpenTransport or the serial driver will continue to work.

Apple knows how important the Internet is, and how important both existing applications and standard unix applications are. Unfortunately, Apple and unix have followed very different models for communicating over TCP. To further complicate things, the Blue Box does not really know about Rhapsody, so anything Rhapsody does to the TCP stack has to have minimal impact on the Blue Box. In a nutshell, there has to be two TCP stacks running.

Within the Blue Box, OpenTransport will continue to be available, including the OpenTransprot classic AppleTalk compatibility services, however, OpenTransport will require new low-level driver modules.

Again, Apple has come up with a flexible solution to a hard problem. Users can decide to either share an IP address between Rhapsody and the Blue Box, or give the two different IP addresses. In reality, Rhapsody supports multiple IP addresses, so this is not a difficult choice.

The single IP address is best for machines that connect over PPP and are assigned a single address. Sharing the IP address will have limitations as a user can run into port conflicts, but overall, that is an easy problem to get around.

Either way, packets are multiplexed at the driver level and sent up different stacks depending on the port in use. Apple will be working very diligently on performance issues.

OPENSTEP provides a powerful, and easy to use localization system. Before talking about how to localize under OPENSTEP, a refresher in Macintosh localization is in order. If there is a routine that needs to put a string in a dialog, getting the string would be along the lines of:

{
	Str255	aString;
	GetIndString(aString, kStringResourceID, kThisMessageID);
	if (aString[0] == 0)
		BlockMove( "\pThis is my message", aString, 19);
	.
	.
	.
}

After doing this a few hundred times, developers start writing special routines just to make the source code readable. Of course, the programmer better remember to add every string to a resource file.

Under OPENSTEP, NeXT simplified this into one call: NXLocalizedString(const char *key, const char *value, comment). The call to NXLocalizedString looks for an entry for key and uses the value returned. If the key is not localized, then NXLocalizedString will return the value. The comment field is used for automatically generating localization files.

OPENSTEP has a command line program called genstrings that runs on specified source files to create a localization file. The file will have the format "key"="value"; /*comment*/. The comment helps localization experts generate correct keys. Duplicate key/value pairs are merged. Most OPENSTEP developers use the same value for both key and value in their native language. This makes the key easier to generate. No matter how many NXLocalizedString( "Yes", "Yes", "This is a Yes"); there are, there will be only one "Yes"="Yes"; line in the localization file. Then, when ported to another language, there will be only one entry to modify.

The localization file is actually a dump of an NSStringTable object. Once loaded, the localization file can be treated like any other NSStringTable. In addition, NSStringTable objects can be created on the fly and used by routines similar to NXLocalizedString.

The localization file works in concert with the nib files. Nib files are used by NeXT's Interface Builder application to store User Interfaces. Like the strings embedded in the code, the User Interface will need to be modified for each language. Each language's nib files with that languages localization file is placed in a subdirectory called <language>.lproj. Each lproj is placed inside of the application wrapper. An application wrapper is actually a folder, but appears as a single file to users. Double clicking an application wrapper will launch the application, not open the folder. As each language has a different name, many lprojs can be placed in the application wrapper.

On a Macintosh, each binary can have only one resource fork, and thus, each binary can only be localized for one language. Under OPENSTEP, multiple lprojs allow a single binary to support multiple languages, saving a great deal of space for places that need multiple language support of their applications.

Furthermore, since each application can have multiple languages, the users can set the preference for which order to look for languages. If a user has his preferences set to French, German, English, Japanese, the system will first look for a French.lproj, then a German.lproj, and so on until it finds a language the application supports.

Unfortunately, despite all the additional information Apple had to offer, there was no where near enough time for me to see it all, nor was there time for me to get it ready by the print deadline. As it is, I have to thank the editors for already stretching their deadlines.

Some things to tantalize you with, though, are a Text system ready made for Unicode, multiple views, custom input for multi-byte languages, and glyph support; printing support that will make you drool; NSString objects for quick and powerful manipulations of strings; WebObjects; full Java support; the most powerful imaging system this side of a high end printer; and the list goes on.

Apple has promised a lot in the past that they have been unable to deliver. NeXT has been unable to sell anything they've delivered in the past. Put them together, and if their strengths compliment each other, they should be on to something. This could be the match that turns Apple around. After a week of listening to Apple, I believe again.

Michael Rutman, moose@manicmoose.com is a software developer with experience developing for several platforms, including Macintosh, NeXTSTEP, Newton, Pilot, and Windows NT. While working at Software Ventures, he lead the development of Snatcher and MicroPhone Pro for NEXTSTEP. He also worked on the MicroPhone Pro for Macintosh product line. He now works as an independent consultant on a variety of projects including encryption, compilers, web based add-rotation software, and ship stevedoring.