WHAT'S IN A PROTOCOL? By Greg Gilley (Reprinted from the Austin TI Professional Computer Users Group Newsletter, February, 1987) A file transfer by any other name would take as long This is the first of two parts on microcomputer file transfer protocols. I will attempt to give an overview of the protocols and some of their good and bad points. This part will cover XMODEM and its derivatives: MODEM7, TELINK, and YMODEM. The second part will cover KERMIT, ZMODEM AND WXMODEM. I have been meaning to write something for the newsletter for some time now. I just never could come up with a topic that I felt would be interesting to the readership. Someone finally suggested that I consider file transfer protocols and that sounded like a good idea. XMODEM XMODEM is the granddaddy of the microcomputer file transfer protocols. It was invented by Ward Christensen, the same person who created microcomputer bulletin boards. XMODEM was originally developed on a CP/M based micro-computer in 1977. Many of its characteristics are a direct consequence of that operating system and early microcomputer hardware. XMODEM transmit data 128 bytes at a time (the size of a disk sector under CP/M) along with a block number and a one byte checksum for error detection. It has no provision for passing the file name between the machines; originator and receiver must both specify the name. Since CP/M did not support time and date stamping of files, no provision was made to transfer the file's date and time. CP/M only provided system calls to do I/O a sector at a time; therefore, all file sizes were a multiple of 128 with a CTRL/Z character to mark the end of a text file. Because of this XMODEM always rounds the size of the files being transferred to the next 128 byte boundary. This results in up to 127 garbage bytes added to the file. This can wreak havoc with files on other systems that do not conform to this convention. XMODEM is essentially a half-duplex protocol. It assumes that the computers cannot send and receive data at the same time. Most of the microcomputers around when it was invented could only do polling to see if they could send a character or to find out if one had been received. Most of today's computers use interrupts to stop what the computer is doing, send or receive the next character, and then resume the processing. With polling, the computer must continually ask the I/O device if it is ready to send or receive a character. Another reason for the half-duplex operation was that most of the older machines could not receive data and write to the disk at the same time. If data was sent continuously, all that arrived while the program was writing the last block to disk would be lost. XMODEM will send a block of data and wait for the receiver to acknowledge receipt of the block. This gives the receiver a chance to write the data to disk before more arrives. If the receiver does not receive the block correctly, it NAKs the block and the sender will retransmit it. Because of its half-duplex nature, XMODEM is susceptible to and in fact causes problems with delays over packet switching networks such as TYMNET, TELENET and those used by CompuServe. It can usually be tweaked so that it can live with network delays, but transfers are excruciatingly slow. Some other problems with XMODEM are that it only uses an 8 bit checksum for error detection. This means that we could expect 1 out of every 256 packets with errors to have a valid checksum and go undetected. This was corrected with XMODEM/CRC which uses a 16 bit CRC (cyclic redundancy check) to detect transmission errors. Using a 16 bit CCITT standard polynomial equation, CRC error detection is theoretically accurate 99.9965 percent of the time. Another problem with XMODEM is that it is easily corrupted because of its simplicity. Phone lines noise or extra delay can causes transfers to abort. So, with all of these problems, why is XMODEM so popular? Because it is easy to implement and because it is public domain with many sample implementations readily available. Without XMODEM, the current explosion in microcomputer telecommunications would have been much longer in coming. MODEM7 MODEM7 is a simple extension of XMODEM. One problem with XMODEM is that the file name must be specified by both the sender and receiver of the file. XMODEM can also transfer only one file at a time. MODEM7 corrects this problem by passing the file name from the sender to the receiver before sending each file. This also allows wildcard transfers to occur such as "*.EXE" which would send all files with an EXE extension. However, the file name is transferred one character at a time and its transfer is unreliable. Since MODEM7 also originated on CP/M systems, it still did not transfer the file date and time. It suffers from all of the same problems mentioned for XMODEM. MODEM7 is popular only on CP/M based machines. TELINK The TELINK protocol was created by Tom Jennings, author of the FIDO bulletin board system. It is an extension of MODEM7. In addition to the file name, TELINK also provides for the transfer of the file's date and time. The first block transferred after the name contains the file size in bytes and the file date and time. Because it transfers the file's exact size, the receiving computer can truncate any garbage bytes that the protocol appends to the last block. It also allows maintaining the file's original date and time. A problem with TELINK is that it still uses MODEM7's convention for passing the file name. Another is that the file date and time is in an MS-DOS specific format and does not take the different time zones into account. The format could be transferred to another machine's format, but it would be awkward and transfers between time zones would still be wrong. TELINK is, in general, not a widely supported protocol. FIDO bulletin boards and MINITEL, a comm program also by Tom Jennings, support it as well as TIgger and TERM. YMODEM YMODEM is an XMODEM extension created by Chuck Forsberg. It allows batch transfers and also supports transferring the file's exact size, date and time. It does not use the MODEM7 convention for transferring the file name, but instead puts the file's name and other data in the first block to be transferred. The date and time are stored as a 32 bit quantity signifying the number of seconds since Jan 1, 1970 in GMT (Greenwich Mean Time). This allows receivers in different time zones to adjust the file's date and time correctly. YMODEM also provides an extension to the block size of XMODEM. YMODEM allows transfers to use 1K blocks (1024 bytes) for data. This reduces the number of times the sender has to wait for the receiver to respond to a block. This help reduce the overall transfer time. On noisy phone lines, however, YMODEM can actually increase the time of the transfer because the blocks take longer to retransmit. It is especially vulnerable at lower baud rates where the probability of noise occurring during the transfer of a block is higher. YMODEM also has a non-batch mode where it does not transfer the file name, date and time, but does allow using 1K blocks to transfer the data. YMODEM is gaining popularity, but unfortunately most implementation are non-batch and therefore lose most of the benefits. CONCLUSION XMODEM helped to lay the foundation for the microcomputer telecommunication revolution. It is dated, however, and should be replaced. MODEM7, TELINK, and YMODEM while correcting some of the problems with XMODEM, still retain many of its fundamental characteristics and are not a long term solution. Next time, I'll talk about some of the possible successors to XMODEM: KERMIT, WYMODEM AND ZMODEM. TTFN. <<<<<<<<<<>>>>>>>>>> WHAT'S IN A PROTOCOL? THE SEQUEL By Greg Gilley (The second installment of the file transfer article which appeared in the March issue of The TImes. Reprinted from Austin TI Pro. Computer Users Group Newsletter, March '87.) Last time, I talked about XMODEM, MODEM7, TELINK and YMODEM. This time I'll try to give you some background in KERMIT, WXMODEM, and ZMODEM. There are a number of other protocols available, but most are proprietary and really aren't worth discussing. Now that the social amenities are out of the way, lets jump in with both feet. WXMODEM - WXMODEM was developed by Peter Boswell to try to overcome problems caused with XMODEM by network delays. It uses the "sliding window" to reduce network delays. This means that instead of waiting for each packet to be acknowledged by the receiver, up to four packets may be outstanding before an acknowledgement is required. This allows time for the normal network delays that would occur for the "ACK" packets. The ACK packets include a window index to tell which packet is being ACKed. (Note: ACK stands for "acknowledged", NAK for "not acknowledged". ed.) The data packets are normal XMODEM packets with the CRC (cyclic redundancy check) error checking extension. The characters XON, XOFF, SYN and DLE are quoted to allow XON/XOFF flow control so the network can keep up with the sender and receiver. It does require an 8-bit data channel to operate. WXMODEM does help reduce network delays, but is susceptible to noise. It also does not support batch (wildcard) transfers, exact file size, or file date and time. This is unjustifiable in a recent protocol definition. WXMODEM is sure to be a short lived protocol. KERMIT - The KERMIT file transfer protocol was originally developed at Columbia University by Franca da Cruz and Bill Catchings. It was designed to handle a specific problem, transferring files between microcomputers and mainframes. It does a good job of solving the problem, even with brain damaged IBM front ends that run only at half-duplex. It is, necessarily, very complex. It provides for transferring 8-bit characters across serial lines that support only 7-bit data transfers. It does this by using a quote character to precede each character that has its 8th bit set. Some quick figuring should tell you that a file with all characters greater than (ASCII) 127 would double in size. Control characters (less than 32: ie. backspace, carriage return, linefeed, etc.) are also quoted to keep the network or operating system from eating them. KERMIT has three error checking types: 6-bit checksum, 12-bit checksum, and 16-bit CRC. The 6-bit checksum is the normal default for file transfers, but use of the CRC is recommended to reduce the likelihood of undetected errors. KERMIT knows the difference between binary data files (like executables) and text files. There is a standard format for transmitting text files which allows conversions to be performed between systems. A DEC VAX, for instance, uses a zero to terminate lines; UNIX uses a line feed; MSDOS, a carriage return and a line feed. KERMIT makes transfers between these systems much simpler. It also has provisions for supporting many different types of file attributes such as fixed length records or encoding in EBCDIC (a character encoding scheme like ASCII that was used for punched cards). It supports exact file size, file date and time transmission, and wildcard transfers. KERMIT does have some drawbacks, however. KERMIT data packets are restricted to a maximum of 95 characters. This small packet size combined with the character quoting means that KERMIT has a lot of overhead to slow the transfer down. Since every packet must be acknowledged by the receiver before the next one can be transmitted, KERMIT is unbearably slow over a network. One way to reduce the network delays is the addition of "sliding windows." Sliding windows simply means that there is a "window" of up to 32 packets which may be transmitted before an acknowledgement is required. This means that ACKs which would be delayed by the network aren't waited for, but are processed as they arrive. This means the sender can continuously transmit packets and the speed of transfer increases accordingly. Another extension that helps to reduce overhead is to transmit characters with their 8th bit set unquoted on 8-bit data channels. Since most microcomputer connections are with 8-bit data channels, this (with sliding windows) makes KERMIT more viable as a general purpose file transfer protocol. Another proposed extension would allow data packets to be longer than 94 bytes. This would allow throughput increases for those systems which cannot support the complexity of the sliding window mechanism. KERMIT has another nice feature. It has a server mode where a client (that's you) issues commands, such as file transfers, and the server executes them. This means the file names are typed in only once. ZMODEM - ZMODEM was developed by Chuck Forsberg, the inventor of YMODEM. It really has no relation to XMODEM other than the last five characters of its name. ZMODEM was developed at the request of some network services to help reduce delays inherent in file transfers. ZMODEM is based on ideas derived from X.PC and KERMIT. It allows continuous transfers with ACKs/NAKs occurring only when an error is detected. This permits ZMODEM to transfer data without network delays at almost the full speed of the modem. It supports variable block sizes to allow it to adjust to the speed of the connections. It uses larger blocks at higher speeds to reduce overhead. ZMODEM has a lot of features which means that it is a fairly complex protocol. It supports wildcard transfers, exact file size and file date and time. It can perform selective updating, transferring only those files with more recent dates than those on the receiving system. It also supports restarting transfers that are aborted in the middle. It can perform remote command execution and transfers the results back to the sending system. This works best in an environment like UNIX or XENIX, but could also be used on a DOS system with appropriate software. ZMODEM performs error checking with a 16-bit CRC. A recent addition to the protocol allows the use of 32-bit CRC checking for transfers that require the highest integrity. CONCLUSION - Before I wrap all of this up, let me confuse the issue a little bit more by talking about a couple of other protocols. One that you might be familiar with is the protocol used by CompuServe: CIS-B. This is a proprietary protocol, although they have recently released example source code implementation of the protocol. I guess they realized their fortunes were being made from people who transferred files, and not from selling their VIDTEX software packages. It is developed specifically to reduce problems with transfers over networks by using packets up to 1024 bytes in length and ignoring delays between characters within a packet. The CompuServe host transfers the name to the sending or receiving system which means that the name needs to be specified in only one place. It also transfers files with the exact size, although date and time stamping are not supported. For transfers on CompuServe, I would recommend using CIS-B. It is much more reliable and faster than doing XMODEM transfers with CompuServe. Something else I should mention is X.PC. X.PC is a microcomputer implementation of X.25, a mainframe networking protocol. It supports multiple simultaneous "channels." This means that on systems that support X.PC (like TYMNET, the creator of X.PC) you can be connected to more than one system at a time. X.PC is not a file transfer protocol, it is a data transport layer. It is a means of getting data from one place to another. It handles error correction, and retransmission of error packets. All you do is feed it data and it makes sure it arrives at the other end. X.PC is slowly gaining a little popularity, but it still has a long way to go. It is a very complex protocol, and, although TYMNET supplies C source code for a copying fee, is still a significant effort to implement, especially on non-standard (read TIPC) hardware. It may gain more popularity in the future if someone comes up with a microcomputer bulletin board that supports it. It will require a file transfer protocol to operate in conjunction with it. ZMODEM would make an excellent candidate. Another data transport layer is MNP. MNP operates in much the same manner as X.PC: Feed it data and it gets it where you want it, error free. It is, however, completely proprietary. The only thing going for it is that the people who created MNP have sold it to modem manufacturers and they have implemented it in their modems. This means that if you connect two MNP modems together (must be one on each end of the connection), then you have essentially an error free connection. Very nice for those of us with noisy phone lines. It is, however, not a file transfer protocol, and would require one to work in conjunction with it. Once again, ZMODEM comes to mind. "So, what does all this mean!?!" you may scream! Well, let me drag things out a little further by adding that you still can't transfer data any faster than the modem can. The absolute minimum time to transfer one megabyte of data at 2400 baud is about 70 minutes, at 9600 baud this is reduced to about 18 minutes. Although some file transfer protocols help approach the theoretical limits of the transfer rates, they won't do as much to help as data compression. Data compression will reduce the transfer time by reducing the number of bytes that need to be transmitted. ARC, SQ (CP/M squeeze), and ZOO will do more for reducing transfer times than any of the protocols. In fact, some of the so-called high speed modems simply have a 68000 in them doing compression like one of these programs. Try running an ARCed file through one of them and see what the transfer time really is. In conclusion (), my OPINION about all of these file transfer protocols is this: For micro to mainframe transfers, transfers between systems with strange file formats, or transfers over 7-bit data channels, the winner is KERMIT. The sliding windows extension makes this protocol one that should survive for some time to come. For best all-around micro to micro transmission: ZMODEM. This is a very robust protocol and seems to work well even under noisy environments. It has hooks for extendibility and I see it going no place but up. What does the future hold? I know better than to try to predict anything that has to do with the computer industry. Those "industry analysts" who get paid for throwing darts aren't doing well, and I'm sure not going to join them. I hope I have given you enough to go on to draw your own conclusions. I know I haven't been completely objective, but I have tried to give the reasons that I wasn't. In case you were wondering, the next release of TERM should have support for CIS-B, ZMODEM, and KERMIT with sliding windows. Nothing like covering all bases. <<<<<<<<<<<<<<<>>>>>>>>>>>>>>>