PC Magazine Communications Software Testing Irregularities The April 30 1991 issue of PC Magazine features reviews of 17 async communications programs that allegedly support ZMODEM and 16550A FIFO UARTs. This issue has prompted a number of messages questioning the published test results and the procedures used to generate them. PC Magazine sent files from a 25 MHz 386 computer to an 8 MHz IBM PC/AT upgraded with the 16550A UART chip. Speeds to 115200 bps were used with hardware handshaking. Software handshaking is normally used with high speed direct computer to computer connections, but this was disabled. PC Magazine reported that Professional-YAM generated so many errors and retries at 115k that throughput fell sharply. Randol N. Tigrett, PC Magazine LAN Labs Project Leader, claims both he and Sara Parker attempted to contact Omen Technology about these errors. While Omen had numerous contacts with PC Magazine at the time concerning lists of features, there is no recall of being contacted about technical problems and/or error messages during the performance tests. None of the FAX messages Omen received from PC Magazine mentioned any technical problems, and no electronic mail messages were received. Scott McGinnis, author of COM-AND, another program that suffered from improper PC Magazine test conditions, was not contacted either. PC Magazine mentioned in a sidebar that Professional-YAM with software flow control transferred files faster than all other programs tested. PC Magazine has refused to disclose these performance figures, either in the original article or in response to repeated interrogatives to the test director on PC Magazine's CompuServe bulletin board. PC Magazine's refusal to disclose these figures is troubling: was this information covered up because the numbers would have raised questions about the test procedures, or because the information would have refuted PC Magazine's speed rankings? PC Magazine's refusal to divulge the text of the error messages and other vital information about the tests forced me to prepare this rebuttal. My tests with a variety of configurations (shown below) strongly suggest that failure to issue a "handshake on" command was the one and only cause of Professional-YAM's failure to replicate its first-place performance with hardware flow control. Pro-YAM Tests (not shown here) show no speed difference between hardware and software flow control in direct connect tests at 115 kbps. The cause of Professional-YAM's substandard performance was improper setup. In my tests, Professional-YAM operated without error at 115 kbps, outperforming ProComm Plus 2.0 by at least 28 per cent. I invite readers to repeat these tests themselves. Copies of the test files are available on TeleGodzilla. Copies Omen's ZCOMM shareware communications program may be used for these tests. The ProComm Plus 2.0 transfer failures result from a ProComm bug that disables input on 16550A/AF chips. This bug appeared in a variety of terminal emulation and file transfer tests on a variety of machines. Reports of similar problems at speeds as low as 2400 bps have appeared on bulletin boards. The purpose of this paper is not to single out ProComm for crticism. ProComm Plus 2.0 was tested here because PC Magazine's flawed tests ranked it the fastest. An IBM AT modified for 8 MHz clock speed is the only configuration I have tested that has not exhibited problems with ProComm Plus 2.0 disabling 16550A serial input. Just replacing the AT's CPU clock crystal with the stock IBM part causes problems. PC Magazine has not explained their choice of such an antique as the sole receiving test machine for tests at speeds at least three times faster than today's fastest available dialup modem. The tests are, however, relevant to direct connect transfers between PC's. (Simple cables and software flow control are the norm for this application.) To be useful in such applications, programs should be able to transfer entire directory trees with a single command and maintain the most recent revision of a file on both machines. Unfortunately, no mention is made of which programs have these essential ZMODEM features. A further disappointment is the failure of PC Magazine to test ZMODEM compression with their "compressible file". The results show, as the reader can easily verify, that ZMODEM compression is quite effective on suitable files. I was also disappointed that no mention was made of the programs that violate the ZMODEM protocol description in one or more respects. The sine qua non of a file transfer protocol for async dial-up applications is its performance and integrity under noisy line conditions. How long must we wait for this critical facet to be tested? Since PC Magazine's prose indicates raw speed is a prime determinant of a program's quality, it is incumbent on the reviewers to get their facts right. As the developer of the ZMODEM protocol, and as the author of a program that was not given a chance to perform up to standard, I request that PC Magazine repeat these tests under suitable supervision. If the results or performance rankings of correctly executed tests are different from the published values and graphics, I request that corrected test results, graphics, and associated review comments be properly publicized and published without undue delay. Chuck Forsberg, Omen Technology INC April 22, 1991 ---------------------------------------------------------------------------- The configurations described here duplicate PC Magazine's published test configurations as closely as possible. Source file: b17mh.gif, 356542 bytes, available on CompuServe and TeleGodzilla (503-621-3746 24/96). A similarly sized ZIPfile may be used without affecting the results. Transfers used b17mh.gif except as noted. Compressible file: RTTYPIX, 343851 bytes, a concatenation of the 30 line printer prcture (RTTY art) files in my collection. The source files chosen were long enough to allow accurate manual timing with a stopwatch. They were stored on ramdisk. Transfers completed without errors except as noted. Sending machine: Micronics 33 MHz, 128k cache or Intel 386 ISBC 18 MHz Receiving Machine: IBM 5170 PC-AT s/n00212305170 modified for 8 MHz, replacement HD and controller (Coretest 2.7 performance index: 1.890), CGA clone, Hayes ESP board. Cabling: Special null modem connection with TR/DCD and RTS/CTS crossovers Commands: speed 115200 handshake on sz -ym d:b17mh.gif -or- speed 115200 handshake on sz -yZ d:rttypix speed 115200 handshake on t (ProComm set for ZMODEM auto d/l, crash recovery off) Professional-YAM to Professional-YAM: @115kbps 50 50 50 71 kbps average Professional-YAM to Professional-YAM 17.62: @115kbps 53 53 67 kbps Professional-YAM to DSZ.EXE pD16384 @115 kbps 50 50 50 71 kbps av Professional-YAM to DSZ.COM @115 kbps 82 82 83 43 kbps av Professional-YAM to ProComm Plus 2.0: @115kbps 64 64 64 56 kbps av As Above, to IBM PC-AT at 6 MHz (stock IBM crystal), Hayes ESP board. Professional-YAM to Professional-YAM: @115kbps 67 67 67 53 kbps average Compressible file: 38 38 90.5 kbps ZMODEM Compressible file: 80 43 kbps Kermit Professional-YAM to Professional-YAM 17.62: @115kbps 74 71 72 49 kbps av Professional-YAM to DSZ.COM @115 kbps 112 112 32 kbps av Professional-YAM to ProComm Plus 2.0: @115kbps FAILED FAILED Compressible file: FAILED Sending to Everex System 3000 386 16 MHz 64k cache, 16550A Professional-YAM to Professional-YAM @115kbps 35 35 102 kbps Compressible file: 19 19 19 181 kbps ZMODEM Compressible file: 48 72 kbps Kermit Professional-YAM to Professional-YAM 17.62: @115kbps 48 48 74 kbps Professional-YAM to Professional-YAM 17.62: @38.4 kbps 110 110 3241 cps Professional-YAM to ProComm Plus 2.0: @115kbps 56 55 55 64 kbps Compressible file FAILED FAIL ZMODEM Compressible file 53 65 kbps Kermit Professional-YAM to ProComm Plus 2.0: @38.4 kbps 120 2971 cps ********************************************************* * NOTE: A 16550A/16550AF must be used!! * *********************************************************