Implementing SMTP with PowerPlant

Create a simple Internet mail sender using PowerPlant’s network classes

By Christopher Haupt

Introduction

Since the Internet’s explosive growth in the early 1990’s, a variety of new and interesting tools have been developed to explore its resources. Without doubt, the greatest attention has been placed on the World Wide Web; browser battles are a constant focus of the media. Many new users want to be able to cruise about, exploring the vast resources open to them. This attention momentarily diverts people from the fact that the most frequently used tools on the Internet are file transfer (FTP) and electronic mail (often SMTP/POP) programs.

If you consider the experiences you have today on the Web, you will note the solitary nature of that activity. Only now are sites becoming aware of the community building abilities of this new media. By and large, people want to communicate with each other, and not just enter a ghost town of information.

Over the past few years, my company has focused on this community building aspect of net use. In particular, we are interested in building tools that allow kids to meet one another and form relationships. In the beginning, we hope to encourage these relationships through forming “pen-pals”, or email friends. To this end, we are exploring the implementation of simple Internet mail enabled tools. From this research I present this introduction to the primary mail sending protocol, the Simple Mail Transport Protocol (RFC821), and explore a simple implementation of this protocol using Metrowerks’ PowerPlant network class library.

This article does not provide an introduction to TCP/IP development, but rather the higher level mail protocol. Several good references exist on TCP/IP, of which I recommend (Stevens 94), (Comer 91), and (Tannenbaum 81) as useful additions to your library.

The code presented in this article meets several simple requirements. It implements SMTP using the current PowerPlant network classes. It provides the ability to send “one-shot” email messages-it doesn’t store messages in a mailbox. Additionally, it will not receive messages, although implementing a mail receiver using the Post Office Protocol (RFC1725) can be done using this code as a guide. By using the latest PowerPlant, we also get the ability to dynamically switch between MacTCP and OpenTransport without additional code.

the Simple Mail Transport Protocol (SMTP)

The Simple Mail Transport Protocol was designed to be an easily implemented, reliable mechanism for moving messages from one trusted host to another. This article includes an overview of the protocol, but the definitive specification is (RFC821). (Stevens 94) is an excellent treatment of this material.

SMTP is specified independent of a transport service (it can run over any type of network with a reliable transport layer). This paper describes an SMTP implementation using TCP, which is the most common transport medium in use today for SMTP on microcomputers. SMTP is assigned to the permanent TCP port 25.

The SMTP specification describes a lock-step protocol in which the sender and the receiver transmit specifically formatted ASCII messages to one another, awaiting a response before continuing. At a high level, the SMTP architecture can be described by a simple finite state machine which contains three main states (see Figure 1). The machinery is either checking reply codes, sending new commands, or sending the message contents. In smart implementations, an error code does not necessarily force a disconnect, although it is illustrated and implemented this way in this article.

Figure 1. SMTP Finite State Machine

SMTP defines a small required command set, with several optional commands included for convenience purposes. Table 1 shows the minimal set required for an SMTP sending client.

HELO      - Initial State Identification
MAIL- Mail Sender Reverse Path
RCPT      - One Recipient’s Forward Path
DATA      - Mail Message Text State
RSET      - Abort Transaction and Reset all buffers
NOOP     - No Operation
QUIT- Commit Message and Close Channel

Table 1. Minimum SMTP Command Set

Commands may have zero or more parameters. Commands and their parameters are issued as ASCII plain text strings. A command is terminated with a carriage-return, line-feed (<CRLF>) pair. Commands do not span lines. The termination pair completes the command line when it is encountered. For instance, the command to identify the sender of a mail message would be sent as MAIL FROM:<cfh@cyberpuppy.com><CRLF>.

Acknowledgment messages are formed by a three digit return code, followed by optional text. The three digits represent error and success codes. A typical success message would appear as 250 Requested mail action okay<CRLF>. Note that, within an acknowledgment message, only the first three digits are significant. The textual portion of the reply messages is for human understanding and can contain any text. Messages are grouped by meaning by using the first digit as a key. Messages beginning with a “2” are success messages, “3”’s are error codes, etc.

Normally, the acknowledging process will send one reply message per command. Each reply is terminated with the standard <CRLF> token. It is possible that more than one acknowledgment message may be sent, and this is not prohibited by the protocol specification. You should consider that some servers may generate more than one line of response and handle that case accordingly-this occurs most frequently with message 220, the service ready message transmitted on startup from the receiver when the sender initiates a connection. If you aren’t careful, this can throw your state machine off. The SMTP specification states that multiline responses should include a hyphen (“-”) immediately following the result code of each intermediate status code. The final result line is formatted normally, without the hyphen.

A typical SMTP session can be characterized as shown in Figure 2 and described here. The sender ([S]) opens a two-way channel to the receiver ([R]). The receiver can be the final destination or an intermediate node, described in the message’s path explicitly, or implicitly by network routing tables. At connect time, both hosts are in the Initial state. [R] sends an acknowledgment that the channel is open. [S] sends a HELO message, identifying itself to the receiver. Note that authentication is not required, so it is very easy to spoof sender IP addresses; SMTP is not a secure messaging protocol. [R] sends back a success or error message, possibly denying access to the sender. If the HELO was successful, both sides are now in the Send Command/Response state. [S] sends a MAIL command describing the sending party’s fully qualified reverse path. [R] acknowledges the successful receipt of the path and clears all of its transaction buffers. [S] sends one or more RCPT commands describing the forward path of recipients of the mail message, one recipient per line. [R] accepts or rejects each address.

Figure 2. Typical SMTP Transaction Flow

[S] now sends a DATA command, instructing the receiver that all following data is the actual mail message, thereby putting the transaction in the Send Data state. Transmission of the message text completes when the end-of-message (EOM) sequence is sent (a <CRLF>.<CRLF> triplet, which looks like a period alone on a line). This raises the question, “what if the message contains the EOM sequence?” Data transparency is achieved by stuffing any instance of the EOM sequence occurring within the body of a message with a period “.” character prefix. The receiver checks each line for a leading period and removes it before buffering the data. Only when [R] detects the “real”, tailing EOM, does it send an acknowledgment.

[S] sends a QUIT command to place the transaction in the Commit state. [R] acknowledges the command and closes the channel. It then delivers the message to the recipients’ mailboxes or forwards the message on to the next server in the recipients’ forward paths.

You will note that the SMTP protocol does not handle any of the fields you would associate with a standard mail message (fields such as Subject:, Reply-To:, etc.). These fields, which make up a message that conforms to (RFC822), are built and parsed by the mail handling agent on either end of the SMTP transaction. SMTP treats the mail message in an opaque manner, sending the headers and message body all at once during the Send Data state. The SMTP code only peeks at the message to ascertain EOM transparency conditions. SMTP places the path information generated in a transaction at the front of the message contents. This area is often called the envelope.

Implementing SMTP in PowerPlant

To implement a simple SMTP client for the PigMail project, I chose to create a rudimentary mail editor and tie it to the SMTP code by using a LSingleDoc derived class and its associated window member. The implementation started out using a threaded approach. Before long, debugging of the PowerPlant network classes in the older versions of CodeWarrior bogged the project down. I muttered “Keep It Simple Stupid” to myself a couple of times and created the very simple, event-loop based asynchronous version which is presented here.

A threaded implementation is actually not much more difficult to construct, but it does add some complexity to the discussion at this introductory level. Because SMTP is a simple problem domain, it is a great opportunity to experiment with threading. You could implement the entire SMTP state machine as one thread, to which you hand off all data and let it rip. Or, you could be creative and implement a two thread approach and play with the Producer/Consumer model of cooperative processes (Silberschatz 92). I tried both, and while they work fine, they violated my KISS requirement. The most important thing I learned with these experiments was the danger of mixing threads which operate with different PowerPlant drawing contexts; talk about major view foci problems!

The simple mail sender class displays a window which contains a number of text edit fields: SMTP host, sender address, destination address, subject, and message body. It also contains a button to send the message when done and a status field. Figure 3 shows a picture of the simple interface.

Figure 3. Simple Mail Editor

In the event-loop/asynchronous handling implementation below, I began by creating a LSingleDoc, LAsyncClient class similar to the one shown in Listing 1. To increase your understanding in the following walk-through, you may want to refer to the sample source code that is supplied with this article’s distribution.

Listing 1: SMTPSenderDoc.h

SMTPSenderDoc
This class implements all of the machinery necessary to experiment with the SMTP protocol using the PowerPlant 
network classes (in async/eventloop mode)

class SMTPSenderDoc :   public LSingleDoc,
 public LListener,
 public LAsyncClient 
{
public:
 SMTPSenderDoc(LCommander *inSuper,
 const LStr255 &inTo = “”,
 const LStr255 &inSubj = “”);
 virtual~SMTPSenderDoc();
 
 virtual  void   ListenToMessage(MessageT inMessage,
 void*ioParam);
 virtual  void   Connect();
 virtual  void   Disconnect();
 virtual  BooleanIsIdle();
 virtual  BooleanAllowSubRemoval(LCommander* inSub);
 
protected:
 virtual  void   HandleAsyncMessage(const LAsyncMessage&
 inMessage);
 void   BuildSessionWindow(void);
 void   SendMailMessage(void);
 void   RunMachine(char *inDataBuffer, 
 Uint32 inDataSize);
 void   SendHELO(Boolean inUseShort = false);
 void   SendQUIT(void);
 void   SendRSET(void);
 void   SendNOOP(void);
 void   SendMAIL(void);
 void   SendRCPT(LStr255 &inRecipient);
 BooleanSendNext(void);
 void   SendDATA(void);
 void   SendBody(void);
 void   SendHeader(void);
 BooleanParseReply(char *inBuffer, Uint32 inBufferLen,
 Uint32& inPos);

 MailPreferenceTypeH mMailPrefs;
 LStr255mTo;
 LStr255mSubject;
 Handle mBody;
 LEndpoint* mEndpoint;
 LCaption*mStatusPane;
 Int32  mMachineState;
 Int32  mLastCode;
 Int32  mCurPos;
 Int32  mMachineReplyState;
 char   statusBuffer[8];
};

The constructor initializes all member data, and calls the ::BuildSessionWindow() member function to create the interface. The To: and Subject: fields are filled with optional data supplied by the caller of the constructor.

At this point, control rests within the standard PowerPlant event mechanism, and the user can interact with the editor. When her message is done, she presses the Send button, and away we go.

The SMTPSenderDoc class receives the button message via its ::ListenToMessage() method. Here we call the ::SendMailMessage() method. ::SendMailMessage() extracts the data from the UI and initiates a connection.

The SMTPSenderDoc::Connect() method makes use of a wonderful PowerPlant object called the UNetworkFactory. This object allows you to use the best transport mechanism installed at run time. It will automatically switch between OpenTransport and “Classic Networking” (a.k.a. MacTCP) depending upon which is active at the time the UNetworkFactory is called. We create an asynchronous endpoint object that uses the event-loop to receive incoming asynchronous messages. An endpoint is simply an object that represents one-half of the communication link.

After creating the endpoint, we bind it to a network address. We specify the address information for the originating host. When the ::Bind() operation completes, we connect to the SMTP server host. Listing 2 shows the connection sequence.

Listing 2: Connect Method

SMTPSenderDoc::Connect()
The connect method requests an asynchronous endpoint from the network factory which automatically will 
select the correct networking mechanism (OpenTransport or MacTCP).  We then try to Bind to a local address.

void SMTPSenderDoc::Connect()
{
 mEndpoint = 
 UNetworkFactory::CreateTCPEndpoint(
 UNetworkFactory::Asynchronous(this));
 ThrowIfNil_(mEndpoint);
    // Initialization: Bind to any local port
 LInternetIPAddress address(0, 0);
 mEndpoint->Bind(address); // when this completes, 
    // finish making connect
    // in HandleAsyncMessage
}

The asynchronous networking mechanism in PowerPlant is very easy to use. When network commands complete, or incoming messages are received, the networking classes call your LAsyncClient object’s ::HandleAsyncMessage() method. Here you can crack the incoming message and dispatch to your various handlers. Listing 3 shows how simple the ::HandleAsyncMessage() dispatch mechanism can be.

When we are establishing the initial connection, as soon as we are notified that the connection is created, we set our endpoint to be in auto-receive mode. This endpoint mode automatically issues a receive command on your connection, thereby catching all data that is sent to your client without needing to explicitly issue receive commands.

In the SMTPSenderDoc code, whenever we get something from the SMTP server, we send that in to our SMTP state machine (the ::RunMachine() method). ::RunMachine() alternates between parsing incoming messages for their response codes and sending the next appropriate SMTP command.

Listing 3: HandleAsyncMessage Method

SMTPSenderDoc::HandleAsyncMessage()
This method is called by the notifier mechanism of PowerPlant whenever we receive an asynchronous response 
to one of our earlier requests.  We will dispatch according to the message type.  This is the meat of async 
messaging in PP and is really simple! Note that there isn’t a lot of heavy error checking in this sample.

void SMTPSenderDoc::HandleAsyncMessage(const LAsyncMessage&inMessage)
{
 switch (inMessage.GetMessageType()) {
 case T_DISCONNECT:
 case T_ORDREL:
 mEndpoint->AcceptDisconnect();
    // fall through as the connection is closed 
    // at the other end and we won’t necessarily
    // get the DISCONNECTCOMPLETE when we issue an 
    // AcceptDisconnect instead of a Disconnect
 case T_DISCONNECTCOMPLETE:
 delete this;
 break;
 case T_BINDCOMPLETE:
 if (noErr == inMessage.GetResultCode())
 {
 LInternetDNSAddress remoteAddress(mSMTPHost, kSMTPPort);
 mEndpoint->Connect(remoteAddress);
 }
 break;
 case T_CONNECT:
 if (inMessage.GetResultCode() == noErr)
 mEndpoint->AutoReceive();
 break;
 case T_DATA:
 case T_EXDATA:
 LDataArrived* data = (LDataArrived*) &inMessage;
 if (data->GetDataSize())
 RunMachine((char *) data->GetDataBuffer(),
 data->GetDataSize());
 break;
 }
}

Listing 4 shows part of the ::RunMachine() method, which is an example implementation of the SMTP state machine described above. This implementation is a little unusual, and probably a little less clear, because its external switch statement jumps between result codes, while the inner conditionals branch on the actual state of the system. This code folds the alternating Reply/Send states together. Most of the time, the machine will be receiving state code 250 (success) and staying within the first case. Other status cases cover initialization, rundown, and error conditions.

Listing 4: RunMachine segment

SMTPSenderDoc::RunMachine()
RunMachine is the state machine for the SMTP protocol.  In this implementation, our outer switch detects 
the last response of the SMTP transaction while the inner switches select the proper next state sequence. 
 This Listing is a subsection of the entire method.

void SMTPSenderDoc::RunMachine(char *inDataBuffer, Uint32 inDataSize)
{
 Uint32 thePosition = 0;
 Boolean done = false;
 
    // this mechanism provides an intermediate buffering in 
    // case all of the reply hasn’t arrived yet, we wait until
    // the line terminator is received before parsing
 while (!done && thePosition < inDataSize) 
 if (ParseReply(inDataBuffer, inDataSize, thePosition))
 {
 switch (mLastCode) {
 case 251:// ok, but non-local user
 case 250:// success
 switch (mMachineState) {
 case eGreetingLong: // HELO was successful
 case eGreetingShort:
 SendMAIL();
 mMachineState = eMailSender;
 break;
 case eMailSender: // MAIL was successful
 if (SendNext())
 mMachineState = eMailDestination;
 else
 mMachineState = eMailInitiateData;
 break;
 case eMailDestination: // RCPT worked, more?
 if (!SendNext())
 mMachineState = eMailInitiateData;
 break;
 case eMailInitiateData:  //recipients accepted
 SendDATA();
 mMachineState = eMailBody;
 break;
 case eQuitting: // data accepted
 SendQUIT();
 mMachineState = eDisconnecting;
 break;
 case eDisconnecting: // n/a
 break;
 default:
 Assert_(false); // this is for debugging
 break;
 }
 break;
/*
...error cases and intermediate data case removed...see sample source
*/       
 }
 }
}

::ParseReply() collects the return information from the server and breaks out the result code. It discards the extra textual information. The Send methods simply format the corresponding commands with any parameters and push them out the endpoint. Note that the ::SendNext() method actually parses the To: field’s data to allow for more than one destination address. In this way, the user can specify a comma delimited list of mail addresses. SMTP allows one forward path per RCPT command, so we have to cycle through the n addresses sequentially and send multiple RCPT commands if more than one recipient is specified.

When we get to the Send Data state, we begin by formatting and sending a (RFC822) header. The header minimally includes return path information, subject, recipient address information, and a properly formatted date field. Other fields are optionally appended to the header. The client then sends each line of the message body, testing each line for instances of the EOM sequence and properly byte-stuffs those lines.

At the end of the message body, an EOM is sent. Assuming that the server has accepted the transaction up to this point, we send a QUIT command, which commits the transaction.

The QUIT causes the SMTP server to send an acknowledgment and close the TCP channel from that end. The LAsyncClient object receives a T_ORDREL message requesting an orderly shutdown of the endpoint. The endpoint accepts the disconnect request and then deletes itself.

Assuming that no error messages were encountered, we just sent an Internet mail message via SMTP!

Implementation Problems

During this exercise, I encountered several gotchas with PowerPlant. Here I will try to explain them. Note that some of these issues are expected to be fixed by the PowerPlant release for CodeWarrior 10. The bugs and problems have been reported to Metrowerks and are described here in case you are using an older version of PowerPlant.

Endpoints in their current implementation are tricky beasts. One problem with the asynchronous model is that you can get unusual dependencies that are not normally expected. One of the great current mysteries of the PowerPlant networking classes is when to properly destroy an endpoint. The asynchronous messages which tell an LAsyncClient what is happening are allocated out of a pool of memory created by the LEndpoint’s Notifier object (most of the time. Actually there was a “bug” in that some LNetMessage objects were created from the endpoint’s pool in CW8.) The notifier is destroyed by the endpoint when the endpoint is destroyed. Unfortunately, if you delete your endpoint from within ::HandleAsyncMessage() when you receive a message-such as T-DISCONNECTCOMPLETE-you will kill the memory pool from which the message is currently allocated. This causes problems when the message call stack pops back to the messages originating method, and then tries to delete itself again Boom!

The destructor in the sample code defers the deletion of the LEndpoint object. In the example, a thread is spawned to handle the deletion. This is clearly a work around, and an official solution may exist in a future version of PowerPlant.

A second problem can occur due to overflow problems on sending data. The current endpoint implementations of the Send method do not notify you if the outgoing data has caused an overflow condition. This can happen if you are relying on the auto-send mechanism which copies your data to an intermediate buffer. If you are generating data to go out more quickly that it can be sent, or if you try to send chunks larger than the pool can accommodate, the send will fail silently. There is the beginning of a mechanism to implement a notification of this error, but it is not complete in the CodeWarrior 9 release. You will need to apply a work around in your own code and/or modify the class to patch this up. In my sample code, I simply return the size of the buffer sent or the result code by changing the Send methods to make the data size parameter a reference copy (e.g. void LMacTCPEndpoint::Send( void* inData, Uint32& ioDataSize, LNotifier* inNotifier)).

Another significant gotcha is encountered when you implement a scheme in which you take very small pieces of data out of the incoming data stream. By making repeated calls to the endpoint’s Receive method with a buffer size of one-or other small sizes-you will quickly run out of pool space. Or so it will seem. On closer examination, you will note that there is sufficient space within the pool, but it is impossible to allocate space for your receive request. This fragmentation of the endpoint pools can be avoided by using all data immediately when you get a T-DATA or T-EXDATA message.

If you must receive data in extremely small pieces, you can implement a more sophisticated free block coalescing algorithm, or some kind of intermediate buffering scheme. Using all of the data immediately appears to be the most efficient mechanism at this time. This problem is expected to be fixed in CodeWarrior 10.

A final problem with the networking classes is a collection of memory leaks. There are some instances of exceptions being raised before deleting memory allocated from a pool. See LMacTCPEndpoint::Receive() for an example. These leaks are being cleaned up in the CodeWarrior 9 and 10 releases.

Conclusion

The goal of this article has been to get you started on your own experiments with the networking classes of PowerPlant in general, and SMTP in particular. With the provided code, you should be able to add mail sending via SMTP to your project in short order. Internet programming is not as mysterious as you may think, and armed with this simple example, you can tackle other interesting Internet protocols.

As the Internet’s popularity grows, your task of writing classes like this one will become much easier. You can be sure that class libraries like PowerPlant will have the functionality described here as standard features in the not too distant future.

Bibliography and References

Comer, Douglas E.. Internetworking with TCP/IP Volume I: Principles, Protocols, and Architecture. Prentice Hall, Englewood Cliffs, New Jersey. 1991.

Crocker, D.H. “Standard For The Format of ARPA Internet Text Messages,” ARPANET Request for Comments, No. 821. SRI International: Menlo Park. August 1982.

Postel, J.B. “Simple Mail Transfer Protocol,” ARPANET Request for Comments, No. 822. SRI International: Menlo Park. August 1982.

Myers, J.. “Post Office Protocol - Version 3,” ARPANET Request for Comments, No. 1725. SRI International: Menlo Park. November 1994.

Silberschatz, Abraham, Peterson, James C., et. al.. Operating System Concepts, 3rd Edition. Addison-Wesley Publishing Company, Reading, Mass.. 1991.

Stevens, W. Richard. TCP/IP Illustrated, Volume 1: The Protocols. Addison-Wesley Publishing Company, Reading, Mass.. 1994.

Tannenbaum, Andrew. Computer Networks. Prentice-Hall, Inc., Englewood Cliffs, New Jersey. 1981.