This message is being posted in the following Compuserve forums/sections: OS2USER: Sections 01,15 PSPBETA: Section 16 WINNT: Section 02 CANOPUS: Sections 06,12 PURPOSE: The purposes of this message are: 1. Promote the sharing of the information in WINBN2.TXT. WINBN2.TXT is a file available for downloading from Compuserve Forum OS2USER Section 01: OS/2 Public Image. WINBN2.TXT includes the first *multitasking benchmarks* comparing the March 1993 beta releases of OS/2 2.1 and Windows NT. The benchmarks were carried out by Timothy Sipples, author of the OS/2 Frequently Asked Questions (FAQ) list. THE MULTITASKING BENCHMARKS SHOW WINDOWS NT's PERFORMANCE RATINGS TO BE ABOUT 75 PERCENT OF OS/2 2.1's PERFORMANCE RATINGS. 2. Show a method by which multitasking performance of OS/2 2.1 and Windows NT can be benchmarked and compared. This should be of value to both users and the trade press. 3. Share analysis and interpretations of the benchmarking results that raise open issues for future benchmarking such as: a. What is the relative efficiency of message passing kernel operating systems - which are highly portable - (e.g. Windows NT; Mach 3 derivatives such as IBM Workplace OS, Taligent Pink and future microkernel versions of OSF Unix and USL System V.4.2 Unix) when compared to monolithic kernel operating systems - which are less portable or non-portable - (e.g. OS/2 2.X, Microsoft Windows 4.0, (aka Chicago) and current monolithic kernel versions of Unix)? b. How appropriate is it to run message passing kernel operating systems on Intel X86 microprocessors as compared to running these same operating systems on the new generation of RISC microprocessors? c. How does the performance of a particular message passing kernel operating system or a monolithic kernel operating system scale as more and more processors are added to symmetric multiprocessor (SMP) configurations? BENCHMARK ANALYSIS The hardware used for these benchmarks was an IBM PS/2 Model 56 with 486SLC2-20/40 microprocessor, 12 MB RAM, 160 MB SCSI hard disk (16msec)and integrated 16 bit VGA display adapter. The use of the VGA display adapter should minimize biases toward either operating system during benchmarking since both OS/2 2.1 and Windows NT should have well tested device drivers for VGA. The multitasking benchmark involves four components: 1. The first component of the benchmark is the singletasking performance of each operating system as measured by WinTach, a Win16 benchmark. Higher number means better performance. Numbers are expressed as a multiple over a baseline system. The results were: Component OS/2 Full Screen Win NT Full Screen Ratio Word Processing 2.23 1.89 0.85 CAD/Draw 1.84 3.11 1.69 Spreadsheet 2.08 1.9 0.91 Paint 2.33 1.98 0.85 Overall RPM 2.12 2.22 1.05 The ratio shows the rating for Windows NT divided by the rating for OS/2 2.1. Analysis of component 1: The word processing, spreadsheet and paint components have better performance under OS/2 2.1. The CAD/Draw component has better performance under Windows NT. Because of the weight on CAD/Draw in determining the composite rating, Windows NT has a higher composite rating. Windows NT has three radio button choices for determining the balance of resources applied to foreground task versus background task. These radio buttons are found by selecting the sequence: Control Panel -> System -> Tasking. The choices are: o Best Foreground Application Response Time (referred to as "Best"). o Foreground Application More Responsive Than Background Application. o Foreground and Background Applications Equally Responsive (referred to as "Equal"). The benchmark document does not specify which value was selected for the WinTach singletasking test, however it was probably "Best." 2. The second component of the benchmark is singletasking of a batch file COUNTER.BAT. The batch file reads: @echo on cls c: cd\ echo A > counter.fil :loop type temp.txt > temp1.txt xcopy temp1.txt temp.txt del temp1.txt echo A >> counter.fil goto loop The file temp.txt was constructed using the following command: dir d:\ /s > temp.txt The OS/2 2.1 beta code was the only software installed on the d: drive. On each cycle of the loop, the length of COUNTER.FIL increases by 4 bytes (one byte each for the letter A, a space, a carriage return and a line feed). COUNTER.FIL Length OS/2 2.1 Win NT Full Screen DOS Sess. 236 bytes 164 bytes Windowed DOS Session 164 bytes 164 bytes Analysis of component 2: OS/2 outperforms Windows NT using a Full Screen DOS Session on both systems and the two operating systems have equal performance when using Windowed DOS sessions. 3. The third component of the benchmark is running COUNTER.BAT in the background and WinTach in the foreground. In this run, Windows NT tasking is set to "Best Foreground Application Response Time," i.e. "Best." This is performed on both systems by the following procedure. WinTach was loaded and made ready to go. COUNTER.BAT was started, then WinTach was started. As soon as WinTach displayed the final screen (reporting the benchmark figures), a switch was made to the COUNTER.BAT session and it was immediately terminated. The author of the benchmarks estimates that this method can produce errors in the range of plus or minus four iterations (16 bytes). The author of the benchmarks used this method with COUNTER.BAT in Full Screen and Windowed DOS sessions. The results were quite close. For the sake of brevity, I am showing only the COUNTER.BAT Full Screen DOS session data. Component OS/2 2.1 Full Screen Win NT Full Screen Ratio Word Processing 1.48 1.88 1.27 CAD/Draw 1.24 3.07 2.41 Spreadsheet 1.47 1.88 1.28 Paint 1.62 1.96 1.21 Overall RPM 1.45 2.20 1.52 COUNTER.BAT Bytes 388 20 0.05 Analysis of component 3: These benchmarks show that WinTach performance under OS/2 2.1 loads down when multitasking. On the other hand, using the "Best" setting, the background task gets little or no resources. In other words, in the "Best" setting, Windows NT seems to be a singletasking operating system. 4. The fourth and final component of the benchmark is running COUNTER.BAT in the background and WinTach in the foreground. In this run, Windows NT tasking is set to "Foreground and Background Applications Equally Responsive," i.e. "Equal." The sequence of events is exactly as described in component 3 above. Component OS/2 2.1 Full Screen Win NT Full Screen Ratio Word Processing 1.48 0.73 0.49 CAD/Draw 1.24 1.65 1.33 Spreadsheet 1.47 1.01 0.69 Paint 1.62 0.82 0.51 Overall RPM 1.45 1.05 0.72 COUNTER.BAT Bytes 388 308 0.79 Analysis of component 4: This component shows "Foreground and Background Applications Equally Responsive" (i.e. ratios of 0.72 and 0.79 compared to OS/2 2.1) just as the Windows NT "Equal" radio button states. HOWEVER, THE PERFORMANCE RATINGS FOR WINDOWS NT ARE ABOUT 75 PERCENT OF THE PERFORMANCE RATINGS FOR OS/2 2.1. 1. The first interpretation from these benchmarks that I propose for discussion is that, in this test, Windows NT has about 75 percent of the performance of OS/2 2.1. In other words, tradeoffs exist between benefits and costs of various operating systems. The portability of Windows NT in comparison to OS/2 2.1 is a benefit. The cost of this benefit is the reduced performance of Windows NT in comparison to OS/2 2.1. This seems to be in agreement with information that I previously received on the Fidonet Operating System Debate Conference from someone named Thomas McWilliams. He wrote: "There is no way NT is going to outperform OS/2 on current Intel hardware. NT is a message passing kernel that relies on separate servers for almost all services. Because of the complexity of the data structures that have to be maintained, NT has to manipulate many more data structures in order to perform a comparable task than would be necessary under a more traditional kernel such as OS/2." "Because of the dearth of registers on the Intel CPU, these data pointers must continually be re-loaded from core. This is a tremendous overhead. This is not an NT fault per se, but rather a fault of the hardware/software mismatch between Intel X86 architecture and a complex message passing kernel. By the same token, OS/2--Mach will not perform as well as the current OS/2 monolithic kernel on a given Intel X86 machine." 2. The second interpretation from these benchmarks that I propose for discussion is that the performance difference confirms Thomas McWilliams' statement. 3. The third interpretation from these benchmarks that I propose for discussion is that heavier loads most likely will show a larger performance difference between Windows NT and OS/2 2.1. This is simply extending Thomas McWilliams' ideas and asking "what happens as a single processor (especially one with few registers like an X86) has to handle more and more tasks and a wider and wider variety of tasks?" This third interpretation suggests that careful benchmarking under real load conditions must be done when evaluating operating systems on various platforms. 4. The fourth interpretation from these benchmarks that I propose for discussion is that users should expect that it is better to run message passing kernel operating systems on RISC microprocessor platforms than on Intel X86 platforms. Therefore users who are considering Windows NT and other message passing kernel operating systems will have to consider purchasing new and different hardware. This will probably lengthen project schedules and add new forms of project risk. 5. The fifth interpretation that I propose for discussion is that message passing kernel operating systems on SMP hardware may not be the "magic bullet" that the trade press has suggested. I have asked systems engineers at both IBM and NCR about the relative performance contribution of the processors in an N+1 processor SMP system. They agree that the 2nd through N+1st processors achieve at most 80 percent of the performance of the first processor. What is the implication for the relative performance of a single processor OS/2 2.1 system (scale performance so this counts as 1 unit) and an SMP Intel X86 system under Windows NT? As measured in the benchmark, the first processor of the SMP Intel X86 system under Windows NT is 0.75 units of performance (related to OS/2 2.1). Each of the 2nd through N+1st processors contribute 0.8 * 0.75 =0.60 units of performance. Therefore a two processor SMP Intel X86 system under Windows NT is 0.75 + 0.6 = 1.35 units of performance. Furthermore, a three processor SMP Intel X86 system under Windows NT is 1.35 + 0.6 = 1.95 units of performance. This analysis concludes that a three processor SMP Intel X86 Windows NT system is required to double the performance of a one processor OS/2 2.1 system. At the Windows NT launch event on May 24, 1993 I asked an Intel employee how long it takes Intel to double the performance of the X86 processor that it sells for a given price. The reply was that Intel doubles the performance of its chips at a given price in 18 months. Therefore, if time is on the side of the project manager, it is possible to wait for processor improvements and to run OS/2 2.1 or some other monolithic kernel operating system. This will be less expensive than buying an SMP system. This also suggests that the real market for Windows NT SMP systems is configurations with 6 or more processors. This would give 4 times the performance of a single processor monolithic kernel operating system. 6. The sixth interpretation that I propose for discussion is that users who believe that they need SMP performance should benchmark SMP versions of monolithic kernel operating systems. I believe that examples of such operating systems include NCR SMP Unix, Sequent Dynix and the SMP extensions to OS/2 2.X that IBM demonstrated at Comdex. The IBM extensions will enter beta test later this year. According to a message from Steve Woodard of IBM, the current expectation is that these extensions will be able to support 16 microprocessors. Thank you for reading this message. Best wishes. Jonathan Handler