Benchmarking "Round 2" May 17, 1993 by Timothy F. Sipples Internet: sip1@kimbark.uchicago.edu Windows NT v. OS/2 2.1 (March public beta releases) Some months ago I spent some time comparing the then current beta releases of OS/2 2.1 and Windows NT 3.1. Testing was conducted in a fairly limited fashion using an older release of the Ziff-Davis WinBench benchmarking program. In this comparison, I have used a wide range of Windows benchmarking programs, not just WinBench. I have focused on Windows benchmarks to be, if anything, more fair to Windows NT. The hardware used for these comparisons is an IBM PS/2 Model 56 with 486SLC-20/40 CPU, 12 MB of RAM, and 160 MB SCSI hard disk drive (a fairly good baseline system, comparable to the typical systems found on desktops in businesses). I did not conduct tests with 8 MB of RAM (to be more than fair to Windows NT). This system was equipped with the standard IBM 16-bit onboard VGA display hardware. A standard IBM PS/2 mouse was used, and it was not moved while benchmarks were running. The hard disk drive was partitioned into two drives, C and D, both FAT, and OS/2's Boot Manager. NT was installed on Drive C (about 90 MB), and OS/2 was installed on Drive D (about 60 MB). Both operating systems were installed with all default settings (unless otherwise noted below). All benchmarking programs were installed on Drive C. The following benchmarking programs were used: Dhrystone for Windows (Version 1.1 by Richard Levey, Visual BASIC); WinTach 1.0 (Texas Instruments); WinSpeed 1.0 (Chris Hewitt); Ziff-Davis Labs WinBench 3.11. For the multitasking tests, a batch file, COUNTER.BAT, was used. The batch file reads: @echo on cls c: cd\ echo A > counter.fil :loop type temp.txt > temp1.txt xcopy temp1.txt temp.txt del temp1.txt echo A >> counter.fil goto loop The file temp.txt was constructed using the following command: dir d:\ /s > temp.txt OS/2 supports two modes for Windows compatibility. Full screen mode is most closely analogous to Windows itself. The Program Manager, and one or more Windows applications, run in a full screen Win-OS/2 (Windows) desktop. Windows applications can also run "seamlessly," on the OS/2 Workplace Shell desktop. Both modes of operation were tested. When running seamlessly, applications were started from the Program Manager (to most closely approximate what Windows NT does, and to be more than fair to NT). OS/2 also supports both standard and enhanced mode Windows compatibility. Tests were conducted under the default standard mode. Comparisons were made with enhanced mode, however, and no measurable differences in performance were detected. No patches were applied to either operating system. The original, public beta distributions were compared. (The retail release of OS/2 2.1 was not used.) Neither beta release could successfully complete the Windows User Magazine benchmark suite. OS/2 2.1 beta could not execute the WinTach benchmark in "seamless" Win-OS/2 mode. With WinSpeed, which provides a continuous readout of values, the value was recorded six times, once every five seconds. These values were then averaged, and the average is reported below. Note that neither company recommends using these beta releases for performance benchmarking. However, the results may be interesting, and they should provide some help in benchmarking the final releases. Here are the results. Dhrystone for Windows Higher number means better performance. Linear scale. Windows NT: 3776 CPUStones/Second OS/2 Full Screen: 15210 OS/2 Seamless: 15037 Here OS/2 outperformed Windows NT by a factor of four. The results are certainly surprising, since this benchmark is written in Visual Basic (a Microsoft product). So surprising, in fact, that the benchmark was rerun to confirm these results. The numbers hold. WinTach Higher number means better performance. Numbers are expressed as a multiple over a baseline system. Component OS/2 Full Screen Win NT Full Screen Word Processing 2.23 1.89 CAD/Draw 1.84 3.11 Spreadsheet 2.08 1.9 Paint 2.33 1.98 Overall RPM 2.12 2.22 These results are somewhat mixed. WinTach weights the CAD/Draw category heavily enough to give NT the slight nod in the Overall RPM figure. However, OS/2 bested NT in all the other categories comprising the WinTach overall figure. Multitasking Tests with WinTach The batch file (COUNTER.BAT) creates a file, COUNTER.FIL, which measures the number of iterations completed. The number of bytes in the file is equal to the number of iterations multiplied by four (the letter A, a space, carriage return, and line feed are recorded for each iteration). Byte size values for COUNTER.FIL are reported below. First COUNTER.BAT was executed by itself, without another task running. It was run for 60 seconds. COUNTER.FIL Length OS/2 Win NT Full Screen DOS Sess. 236 bytes 164 bytes Windowed DOS Session 164 bytes 164 bytes Thus, with COUNTER.BAT itself, OS/2 has the edge, since it bested NT in one mode by about 40% and matched NT in the other. Then, in the following tests, WinTach was loaded and made ready to go. COUNTER.BAT was started, then WinTach was started. As soon as WinTach displayed the final screen (reporting the benchmark figures), a switch was made to the COUNTER.BAT session and it was immediately terminated. I estimate that this method can produce errors in the range of plus or minus four iterations (16 bytes). "Full Screen" and "Windowed," below, refer to COUNTER.BAT. (Windowed DOS sessions were maximized.) Component OS/2 2.1 Full Screen Win NT Full Screen ("Best") Word Processing 1.48 1.88 CAD/Draw 1.24 3.07 Spreadsheet 1.47 1.88 Paint 1.62 1.96 Overall RPM 1.45 2.20 COUNTER.FIL Bytes 388 20 Component OS/2 2.1 Windowed Win NT Windowed ("Best") Word Processing 1.50 1.85 CAD/Draw 1.25 3.12 Spreadsheet 1.43 1.86 Paint 1.58 1.75 Overall RPM 1.44 2.15 COUNTER.FIL Bytes 432 16 Thus, in the default setting, NT is tuned as essentially a single tasking operating system, which is curious given that Microsoft proposes NT for server environments. Both OS/2 and NT provide some control over the priority given to foreground and background tasks. OS/2 provides the CONFIG.SYS parameters MAXWAIT, TIMESLICE, and PRIORITY. NT provides a Tasking control in the System section of its Control Panel. This tasking control provides three options: one providing the "best" foreground response time (as shown above), one providing a moderate foreground boost, and one providing an equal responsiveness to foreground and background tasks. These settings appear to be roughly analogous to OS/2's PRIORITY=DYNAMIC and MAXWAIT as some high value, PRIORITY=DYNAMIC and MAXWAIT as some moderate value, and PRIORITY=ABSOLUTE, respectively. In practice, however, there was no measurable difference between NT's first two Tasking settings. The third, however, did have a measurable impact, and those numbers are represented below (alongside the same OS/2 numbers): Component OS/2 2.1 Full Screen Win NT Full Screen ("Equal") Word Processing 1.48 0.73 CAD/Draw 1.24 1.65 Spreadsheet 1.47 1.01 Paint 1.62 0.82 Overall RPM 1.45 1.05 COUNTER.FIL Bytes 388 308 Component OS/2 2.1 Windowed Win NT Windowed ("Equal") Word Processing 1.50 0.71 CAD/Draw 1.25 1.82 Spreadsheet 1.43 1.02 Paint 1.58 0.83 Overall RPM 1.44 1.10 COUNTER.FIL Bytes 432 308 These numbers seem to indicate that OS/2 is the better multitasker of the two. WinBench Ziff-Davis Labs WinBench 3.11 (running alone) gives the following results: WinBench 3.11 OS/2 Full Screen OS/2 "Seamless" NT BITBLT Alignment Source Aligned Destination Aligned AA 32x32 3586101 pixels/sec 2397817 2738061 AA 64x64 5270784 4492703 4713878 AA 128x128 6016760 5788669 5867332 AA 256x256 6193597 6325468 6216854 Source Aligned Destination Not Aligned AN 32x32 198657 520954 628517 AN 64x64 793925 790632 922501 AN 128x128 1019222 1008052 1132575 AN 256x256 1109711 1149754 1251744 Source Not Aligned Destination Aligned NA 32x32 546133 503832 802738 NA 64x64 806287 780917 1103764 NA 128x128 1009411 998264 1251744 NA 256x256 1103764 1143825 1334551 Source Not Aligned Destination Not Aligned NN 32x32 522095 481705 595782 NN 64x64 777875 753017 857926 NN 128x128 991280 980527 1103764 NN 256x256 1088556 1127281 1230001 BITBLT OVERLAP Horizontal Overlap 5212353 4567712 4980364 Vertical Overlap 1180743 1148397 2278712 BITBLT ROPS Screen to Screen SS SRCCOPY 1808738 1756871 3092727 SS SRCPAINT 704140 699178 155191 SS SRCAND 705113 699589 158601 SS SRCINVERT 703723 699589 160364 SS SRCERASE 669936 666934 158601 SS NOTSRCCOPY 585254 892411 161560 SS NOTSRCERASE 684310 680659 156877 SS MERGECOPY 847090 835259 118301 SS MERGEPAINT 691062 377497 156877 SS PATCOPY 6439612 5035005 5790108 SS PATPAINT 632510 627634 114545 SS PATINVERT 1002273 876159 3608182 SS DSTINVERT 5286740 4607851 4758042 SS BLACKNESS 11570966 8069324 9334209 SS WHITENESS 11552512 8056478 9438669 Memory to Screen MS SRCCOPY 2831629 2566164 1427413 MS SRCPAINT 892244 863509 643042 MS SRCAND 892411 863850 649473 MS SRCINVERT 891409 863509 643042 MS SRCERASE 853382 827816 627510 MS NOTSRCCOPY 1319378 1257007 853548 MS NOTSRCERASE 856059 829704 620762 MS MERGECOPY 1193909 1134432 147273 MS MERGEPAINT 891576 863167 643042 MS PATCOPY 6444713 5055045 5788951 MS PATPAINT 818811 790953 140124 MS PATINVERT 1006378 876804 3647832 MS DSTINVERT 5312085 4608758 4785709 MS BLACKNESS 11685140 8072542 9412648 MS WHITENESS 11659532 8040477 9436783 Memory to Memory MM SRCCOPY 3109283 3262830 713849 MM SRCPAINT 2309796 2384477 291570 MM SRCAND 2309796 2383091 303847 MM SRCINVERT 2308005 2381400 300682 MM SRCERASE 2065594 2117427 297582 MM NOTSRCCOPY 2405455 2479656 307079 MM NOTSRCERASE 2184627 2241936 291570 MM MERGECOPY 2024603 2033780 181925 MM MERGEPAINT 2180341 2244997 297582 MM PATCOPY 3473270 3553621 6978462 MM PATPAINT 1768100 1753159 173193 MM PATINVERT 2568680 2598234 225511 MM DSTINVERT 3284043 3405168 8802378 MM BLACKNESS 4589354 4925094 8169604 MM WHITENESS 4544058 4842149 7922312 SCREEN/Memory BITBLT (64x64 mono) Memory to Screen Blt 6344240 5062559 4222680 Screen to Screen Blt 807094 808368 963009 IMAGE BLT 5116124 5286195 2852147 DRAW ARCS Arc Major Axis X 27885 27685 71058 Arc Major Axis Y 26945 26997 63789 FILL ELLIPSES Ellipse Major Axis X 901457 1126338 5352360 Ellipse Major Axis Y 1592549 941584 4911289 DRAW SINGLE LINES Single Horizontal 2937451 576868 1347988 Single Vertical 1003996 826254 664390 Single Diagonal 341798 325243 421678 DRAW POLYLINES Poly Horizontal 4923690 5015618 10969468 Poly Vertical 743174 769719 763516 Poly Diagonal 360359 374840 599669 Poly Mixed 671477 696494 997168 DRAW LINES IN ALL DIRECTIONS Long Lines 329359 441831 437931 DRAW WIDE LINES Line Width 3 68754 20502 156431 Line Width 5 103125 49568 225218 Line Width 10 209436 104368 446543 Line Width 20 329179 162942 695987 POLYGON ALTERNATE FILL 1141140 498741 3074339 POLYGON WINDING FILL 18.9 polygons/sec 9.58 59.3 DRAW & FILL RECTANGLES Square Rectangles 20923318 pixels/sec 14864968 18376880 Rounded Rectangles 2370741 1208339 5201881 RANDOM RECTANGLES 3058754 3331129 3653264 FILL WITH PATTERNS Black Pattern 27982547 28475847 29049866 12% Grey Pattern 27786828 28382375 28737501 25% Grey Pattern 27786828 28442786 28870545 37% Grey Pattern 27797337 28431783 28786375 50% Grey Pattern 27987875 28514514 29049866 62% Grey Pattern 27776327 28431783 28786375 75% Grey Pattern 27982547 28497929 29695418 87% Grey Pattern 27797337 28426285 28737501 White Pattern 27961255 28470331 29083142 Hatch Patterns 27823646 28453798 29049866 DISPLAY VARIOUS FONTS Display 16 pt System 3070623 3222089 3656036 Display 21 pt New Tms Rmn 3748763 3551673 3790435 Display 27 pt New Tms Rmn 4596578 4369888 5130836 Display 20 pt Arial 3549849 3301090 3793064 Display 24 pt Arial 4513947 4261576 4609698 SCROLL VARIOUS FONTS Scroll 16 pt System 170721 163487 137478 Scroll 21 pt New Tms Rmn 184579 182071 152966 Scroll 27 pt New Tms Rmn 273838 270682 295948 Scroll 20 pt Arial 165588 161938 205146 Scroll 24 pt Arial 208042 194965 197907 EXTTEXTOUT VARIOUS FONTS ExtTextOut 16 pt System 5806584 4789165 3550229 ExtTextOut 21 pt New Tms Rmn 3804645 3578639 3630130 ExtTextOut 27 pt New Tms Rmn 4675541 4405458 4822315 ExtTextOut 20 pt Arial 3613631 3330634 3539799 ExtTextOut 24 pt Arial 4609851 4293893 4408150 DISPLAY TEXT - BACKGROUND MIXED Transparent Text 4022359 2759387 2885682 Opaque Text 5797364 4958974 3599412 GREYED TEXT 363374 332734 74050 MIXED FONTS 814001 670476 792257 STRETCH BLT Stretch 32x32 15476 14337 13194 Stretch 48x48 105326 106790 26551 Stretch 80x80 189274 194037 60711 Stretch 96x96 224890 231093 77937 Stretch 192x192 361294 368640 187922 Stretch 32x48 75090 75956 17638 Stretch 48x96 158988 162540 42601 Stretch 48x32 83980 84768 18481 Stretch 108x96 229762 236667 85042 Stretch 21x21 29389 29444 6014 Stretch 105x105 235913 243020 87154 IMAGE DATA 3171841 2220601 831675 DIALOG BOXES 6.40 dialogs/sec 2.58 3.54 RUBBER BANDING 207367 pixels/sec 153400 153722 SCROLL TEXT IN WINDOW Byte Aligned Scroll 1415 characters/sec 1392 1166 Not Aligned Scroll 1086 characters/sec 1081 1074 ERASE WINDOW 24800036 pixels/sec 24936026 25571696 RECTANGULAR CLIPPING OPERATIONS BitBlt Clipping 2252 operations/sec 1883 1136 Line Clipping 4221 2888 2138 Text Clipping 3030 2129 175 Apparent Disk Tests Sequential Read 200 194402 bytes/sec 237381 157474 Sequential Write 200 213324 212221 239774 Random Read 200 6522 6117 19387 Random Write 200 4764 4538 25300 Sequential Read 512 405269 415137 186786 Sequential Write 512 143873 466135 283399 Random Read 512 16712 3262 76875 Random Write 512 23684 24251 92369 Sequential Read 2048 476483 489854 204546 Sequential Write 2048 401081 385026 311556 Random Read 2048 16075 18644 300339 Random Write 2048 186219 185645 374417 Sequential Read 4096 79227 72025 233767 Sequential Write 4096 451251 464863 459045 Random Read 4096 118377 120405 403719 Random Write 4096 90837 84752 1371676 OVERALL Graphics WINMARK 4042864 2762427 4281517 Apparent Disk WINMARK 26688 16832 95309 Note that WinBench has received some bad press lately. (It is claimed that it is far too easy to optimize to WinBench to inflate the results.) The WinBench numbers resemble those of WinTach in that OS/2 seems to be outperforming NT in more categories, but the weighting system gives NT a slight edge in the overall index. In this set of benchmarks I did not attempt to run comprehensive disk performance tests. OS/2 has two file systems (FAT and HPFS), and NT has three (FAT, HPFS, and NTFS). A comprehensive test would report figures for all three file systems. However, the WinBench disk figures are reported for completeness. One interesting result coming out of these figures is that OS/2's seamless Win-OS/2 mode is performing much better than in the previous beta release. It is no longer the case that full screen mode outperforms "seamless" mode. In some cases, full screen may be a bit slower. It is somewhat puzzling that NT performs reasonably well with the two most popular Windows benchmarks, WinBench and WinTach, yet does not post good results with two lesser known benchmarks, Dhrystone for Windows and WinSpeed. There are several possible explanations for this phenomenon. One possibility is that NT's programmers optimize to these benchmarks. That need not be a sinister act; these benchmarks can help determine an operating system's weak points, and that information can be incorporated into the development process. It does, however, point out the need to perform rigorous, "real world" benchmarks. It is also generally a bad idea to rely on a particular benchmark's index number. It is tempting to try and express an operating system's performance in terms of a single index number, but the weighting used to determine that number may have little or no resemblance to the kind of work the operating system will be asked to perform in a particular environment. The multitasking benchmarks are rather disturbing. It is possible that the default priority setting was chosen with developers in mind, since NT is still in beta. However, to deploy NT on a server this default must be changed. If an administrator runs a task at the console with the current default setting, the entire system will grind to a virtual halt. All the benchmarking programs used are available via Internet anonymous ftp from ftp.cica.indiana.edu. T.F.S. Addendum: WinSpeed Results WinSpeed results are cited but not enumerated. Here are the results produced by that benchmark (higher number indicates greater performance). WinSpeed Average: 172 NT 190 OS/2 Full Screen 190 OS/2 "Seamless" Video 70 130 94 Disk 125 119 114 Again, I cannot guarantee the accuracy of any disk I/O benchmarks (see the original note in the benchmarking document), but they are reported for completeness.