Testing Protocol.

Hardware used:

A 50 MHz 486 clone with 8 Mbytes of RAM, one 5.25" 1.2 Mbyte and one
3.5" 1.44 Mbyte floppy disk drives, one 311 Mbyte and one 440 Mbyte
hard disks, running under DR-DOS 6.0 (patched to be able to run
Windows 3.1), QEMM 6.01, and 4DOS 4.02. The virus collection occupies
one 260 Mbyte partition on the second hard disk. The scanners are on a
second, 180 Mbyte partition on the same hard disk.

Testing the scanners on file infecting viruses.

The viruses are stored in a huge subdirectory tree, the hierarchical
structure of which reflects the CARO virus naming scheme, with the
samples of each virus stored in the leaf directories of the tree. A
virus can be (and usually is) represented by more than one replicant,
although the different viruses are not represented by one and the same
number of replicants. All replicants that contain one and the same
virus, are stored in one and the same directory. If two files are in
two different directories, this means that they contain two different
viruses. All efforts have been made to ensure that the samples used
during the test are natural replicants of working viruses - no Germs,
or Corrupted files, or Intended viruses. Nevertheless, it is possible
that we have made some mistakes in this aspect (although we believe
that there are much fewer such mistakes then in other tests we have
seen). If somebody notices any mistakes of this kind, we shall
appreciate being told about them.

Each scanner is run on this directory tree and the resulting report
file is preprocessed. The preprocessing is done with a set of batch
files, some Unix utiltities ported to DOS (sort, join, cut, paste,
awk), and a set of awk scripts. The scripts are available in the
archive SCRIPTS.ZIP. The preprocessed report contains three columns.
The first column contains the directory and the file name of each file
containing a virus. The second column contains the full standard CARO
virus name of the virus contained in the file. The third column
contains information of how the particular scanner report that virus,
or blank if the scanner does not detect a virus in that particular
file.

Not the whole output of the scanner is contained in the third column,
because this output often tends to be too verbose. We have put there
only the distilled information that we have judged important for that
particular scanner. If we have missed some important information, we
shall appreciate being told about it.

Testing the scanners on boot sector infecting viruses.

The boot sector viruses are kept in a similar subdirectory tree, as
files, containing the images of the infected boot sectors. For the
purposes of the test, we used a program, called SimBoot, developped by
Dmitry Gryaznov. This program is still under development and is not
available to the general public, but we will make it available to
those producers of the scanners, who have reasons to suspect that the
program has unfairly interferred with their product and has not
allowed it to be tested properly.

The program takes a file, the first 512 bytes of which are supposed to
contain the first sector of a boot sector virus. It then emulates a
blank, formatted floppy disk in drive A:, the boot sector of which is
replaced by the image in the file. If the file is smaller than 512
bytes, it is padded with zeroes. If the image contains a valid
diskette BPB which indicates a particular diskette size, a diskette
with that particular size is emulated. If a valid BPB is not found, a
360 Kb diskette is emulated. Currently only the first sector of the
boot sector virus is put on the emulated diskette. The program SimBoot
is able to handle complete viruses, consisting of several sectors, but
this requires that the file image of the virus conforms to a
particular format. We did not have the time to prepare all our boot
sector viruses in this way, although we are considering to do this in
the future. One major flaw of this approach is that hard disk, and
respectively MBRs are not emulated. The testing of a virus which
infects only MBRs (e.g., Tequila) but not boot sectors of floppy
disks, is still done by putting an image of the infected MBR on the
boot sector of the simulated diskette. We understand that this is not
very correct - a scanner may refuse to look for a particular virus on
a diskette boot sector, if it knows that this particular virus just
cannot be there. The author of SimBoot is considering to improve it in
the future, in order to make it able to simulate hard disks too.

Once SimBoot creates the simulated infected diskette, it runs the
scanner to be tested, as specified in the configuration file for this
scanner. (The configuration files are available in the archive
SCRIPTS.ZIP.) The scanner is supposed to scan the diskette (SimBoot
intercepts all INT 13h requests to drive A: and redirects them to
access the simulated diskette), report its status in the report file,
and prompt the user to insert the next diskette to be scanned.

SimBoot intercepts the prompt and simulates user input from the
keyboard. Both the prompt and the required user input are specified in
the configuration file for each scanner. SimBoot is able to handle
scanners that write their prompts directly to the video RAM. It is
also able to handle scanners that poll directly the keyboard when
waiting for user input, instead of using the BIOS. Unfortunately, this
capability does not work under QEMM. SimBoot is able even to simulate
changing the status of the floppy drive from Closed to Open and then
again to Closed, in order to handle those scanners which poll the
DiskChanged line, in order to figure out when the user has put a new
diskette.

The resulting report of each scanner is further preprocessed with a
similar set of batch files and awk scripts as the report of the file
virus scanning.

Creating the final summary of the results.

The files containing the preprocessed information mentioned above are
huge. They are not included in this archive. Instead, they were made
available via anonymous ftp from our site. The name of the site is
ftp.informatik.uni-hamburg.de, the IP address is 134.100.4.42, the
name of the directory containing the results is /pub/virus/texts/tests,
and the preprocessed report for each scanner (separately for file and
for boot sector infecting viruses) is storred in an archive, the name
of which suggests the name and the version number of the scanner
tested.

Those results were further proprocessed with several awk scripts, in
order to obtain the data for the summaries listed in the file
RESULTS.TXT.