TESTMAN User's Manual

--------------------------------------------------------

TESTMAN 1.1 - copyright 1996 by Jim Langseth

To register your copy of TESTMAN send correspondence to:

Jim Langseth
PO Box 44078
Eden Prairie, MN 55344

Single User:			$25
10 User Site License:	$200

Comments may be emailed to: jlangseth1@aol.com

This document describes the use of TESTMAN with various
Microsoft products including Windows NT, Visual C++, and
MFC.  These products are trademarks of Microsoft
Corporation.

Neither the registered version of TESTMAN nor the
unregistered version are warrantied in any way.  The
author cannot be held responsible for any consequence
related to the use of this software package.

--------------------------------------------------------

TESTMAN processes a text file containing test case
descriptions and generates a single source file that
when compiled exercise your C/C++ code and check
conditions you specify.  Failures are recorded and
are listed in the output of the executable.

The purpose of this utility is to allow a C/C++ developer
to apply various data and sequence scenarios to software
modules that produce quantifiable results.  This utility
is geared toward non-GUI modules and classes; other
software packages are available to test GUI code.  

The key advantages to the TESTMAN approach are:

	o Ease of testcase definition
	o Batch style testing

After defining a series of testcases, the driver may
be compiled and linked to an executable.  Running
the executable will apply the testcases and summarize
the success or failure.  Whenever your code changes,
simply recompile, link, and execute to verify the
integrity of your work.

TESTMAN's batch oriented test execution method does not
lend itself well to testing GUI modules since the GUI by
definition requires interaction in real/time with the
user.  Having said this, the author is investigating the
idea of integrating a "batch" approach to GUI testing.
Comments or suggestions in this regard may be sent
to the email address listed above.

--------------------------------------------------------

						CONTENTS	

0.0		System Requirements
1.0		TESTMAN driver definition syntax
1.1		TESTING string
1.2		SETTINGS
1.3		CREATE string
1.4		INCLUDE string
1.5		CHECK_DELTA number
1.6		GLOBAL string
1.7		TEST [testidentifier] [LIKE testidentifier] : ... END
1.7.1		LIKE testidentifier
1.8		DECLARE id AS string
1.9		DO string
1.A		CHECK string ...		
1.B		CLEANUP string [string...]	
1.C		Comments in the test driver definition
1.D		TESTFLAGS - driver run/time customziation
2.0		Test Driver Example - ARRYST.TST

--------------------------------------------------------

0.0 System Requirements

TESTMAN was developed and tested under Windows NT,
versions 3.51 and 4.0.  Although Windows95 was not
used, I believe it should also work.

Versions are not available for 16 bit Windows, UNIX, or
DEC VMS.  If demand exists I may port it.

--------------------------------------------------------

1.0			TESTMAN driver definition syntax

TESTING string									
SETTINGS MFC or ANSI                          optional
CREATE string
INCLUDE string [INCLUDE str...]               optional
CHECK_DELTA number                            optional
GLOBAL string [string...]                     optional

TEST [testidentifier] [LIKE testidentifier] :
	DECLARE id AS string [DECLARE...]     optional
	DO string [string...]                 optional
	CHECK string IS string                optional
	CHECK string IS {expression]
	CHECK string NONE
	CHECK ...
	CLEANUP string [string...]            optional
END

[TEST...]                                     optional

--------------------------------------------------------

1.1	TESTING string

The TESTING construct requires the user to provide a
string describing the module being tested.  This string
is included in the output of the test executable.  

Example:

	TESTING "PROJECT: myproject MODULE: mymodule.cpp"

Any string may be used but you should use something that
describes specifically what is being tested.

1.2 SETTINGS

The SETTINGS construct allows the user to change the
target environment for the test driver.  Currently,
the environment is either ANSI or MFC.  Specifying ANSI
generates ANSI compatible code.  Specifying MFC generates
code compatible with Microsoft Visual C++ and the
Microsoft Foundation Classes; this includes the use of
INT, UINT, BOOL, CString as well as tchar.h.

The SETTING construct is optional.  The default SETTING is 
ANSI.

Example:

	SETTING MFC

1.3 CREATE string

The CREATE construct requires the user to specify the
test driver file name that is generated by TESTMAN.
When TESTMAN is invoked, the result is a single file
containing the test driver code matching the test cases
defined in the input file.

Example:

	CREATE "mymodule_tst.cpp"

1.4 INCLUDE string

The INCLUDE construct allows the user the specify
files in the test driver.  TESTMAN allows none, one, or
more INCLUDE constructs in the input file:

Example:

	INCLUDE "<ctype.h>"
	INCLUDE "myincl.hpp"

1.5 CHECK_DELTA number

The CHECK_DELTA construct allows the user to specify
the amount by which floating point numbers may differ
and yet be recognized as equal.  Since two "doubles" may
be similar in value but deviate by a very small amount
due to roundoff, this method allows some leeway when
comparing floating point values.  The value effects
the CHECK construct (see below).

Example:

	CHECK_DELTA 0.00001

Note that the CHECK_DELTA construct is optional with
a default value of 0.001.  The example changes the
value to 1E-005, a value that may be more appropriate
for testing some high precision math routines.

Also note that in cases where a different delta (or no
delta) is required you may simply modify the CHECK
construct from

	CHECK "double_result" IS {other_number}

to

	CHECK "double_result==other_number" IS {TRUE}

1.6 GLOBAL string

The GLOBAL construct allows the user to provide global
definitions to the test driver.  This includes both
data definitions as well as global functions and
procedures.  This construct is optional.

Example:

	GLOBAL
		"int aflagvalue;"
		"SOMECLASS *globalobject;"

Presumably, one or more testcases will somehow use
these values for testing.

1.7 TEST [testidentifier] [LIKE testidentifier] : ... END

The TEST construct allows the user to define a single
testcase.  A testcase may be given a name which is 
included in any failures detected during test execution.

The LIKE construct references a previously defined testcase.
The new testcase (currently being defined) inherits
characteristics from the earlier testcase.  The inherited
characteristics may then be overridden in the body of the
testcase definition.

A test identifier is not required for a testcase.  However,
including one is useful for two reasons.  One, so the
test may be referenced later in a LIKE construct.  Two,
so that if failures occur in the testcase, the test
output will include the name of the test making it
easier to find the testcase definition when investigating
the failure.

1.7.1 LIKE testidentifier

The following sections describe the DECLARE, DO, CHECK,
and CLEANUP constructs that are used to define a testcase.
The LIKE clause causes the new test to inherit the previous
testcase characteristics of these constructs.

In the sections below you can see that the DECLARE and
CHECK constructs may be multiply defined and each has
a unique name.  The DO and CLEANUP constructs however
do not have names and consist of one or more strings.
You may selectively override DECLARE and CHECK 
constructs by referencing the correct name.  Overriding
the DO and CLEANUP constructs is all-or-nothing.
In other words, when specifying a DO construct in a
testcase that inherited a DO construct, all inherited
strings are lost.

1.8	DECLARE id AS string

The DECLARE construct allows the user to declare data
(and make functions calls; TESTMAN doesn't care).  The
key is that each DECLARE construct is named and thus
may be selectively overridden in subsequest testcases
that inherit.

Example:

	DECLARE iandn AS "int i,n=list.Size();"
	DECLARE max AS "double max=1.0;"
	DECLARE str AS "char *str=\"hello world\";"
	DECLARE calls AS "myfunc(a,b); for(i=0;i<n;i++) xfunc(i);"

The id (e.g. iandn) is any sequence of a letter followed
by none or more letters and digits.  Further, the id
may or may not be the actual value defined.

Originally the DECLARE construct was intended as the
place to declare variables to be used in the testcase.
Then the actual test activities would follow in the DO
and CHECK constructs.  However, you may be creative
in your use of these constructs in any way that (hopfully)
makes it easier to test your code.

1.9	DO string

The DO construct allows the user to delcare lines of code
that perform a series of tasks.  The DO construct is optional.
One or more strings are required in the DO construct,
if included.  The combination of the DECLARE and DO
constructs should perform one or more tasks that
may be measured and checked in the CHECK constructs.

Example:

	DO
		"int returncode = Sort(mydata,param1);"
		"double data1 = mydata[0];"

					
1.A	CHECK string ...		

The CHECK construct allows the results (of tasks
performed in the DECLARE and DO constructs) to be
compared to the desired results.  If the CHECK
fails, a line is printed in the test driver
output describing which test and which check
failed.

The CHECK construct can handle double precision values,
integers, and strings (and MFC CStrings if enabled in
the SETTINGS construct).  Strings are defined between
quotes.  Numerical results and expressions are
defined between curly braces.

Example:

	CHECK "returncode" IS {0}
	CHECK "data1" IS {1.444}
	CHECK "abc==def" IS {TRUE}
	CHECK "abc==def" IS {1==1}
	CHECK "strings[1]" IS "a desired string"

CHECK constructs are inherited when using LIKE.  In
some cases, the new testcase does not require a 
CHECK from the previous testcase.  This is accomplished
using the following form:

	CHECK "nthvalue" NONE
	CHECK "strings[1]" NONE
		
1.B	CLEANUP string [string...]	
				
The CLEANUP construct allows lines of code to be
included after the CHECK constructs.  This may be
useful for freeing up memory, closing files, or
some other activity.  CLEANUP is optional.

Example:

	DECLARE str AS "char *str = new char[10];"
	DO
		"strcpy(str,\"XYZ\");"
	CHECK "str" IS "XYZ"
	CLEANUP
		delete [] str;

1.C Comments in the test driver definition

Comments may be included in the definition using a 
slash character.  Any text following the slash character
are ignored up to the end of the current line.  This
does not include slashes inside of strings or
curly braces (see CHECK).

Example:

\
\ Testcases for MYMODULE
\
TEST test1 :
...
END

\ a second test
TEST test2 LIKE test1:
	\ first declare some values
	DECLARE...
	\ then DO some stuff
	DO ...
	\ perform CHECKs
	CHECK...
END

--------------------------------------------------------

1.D TESTFLAGS - driver run/time customziation

TESTMAN automatically includes a variable called
TESTFLAGS in the test driver that converts an
input parameter (i.e. command line argument) to
an integer.  The default value is 0.

In the example that follows,

c:> arryst 255

would pass the value 255 to the test driver and store
in TESTFLAGS.

Use TESTFLAGS to customize the sequence of events in
during the test execution.  For example, in a test
driver I used, I wanted to allow interaction with the
user:

TEST atest:
  "int param=10;"
  "if (TESTFLAGS&1) { printf(\"Param? \"); scanf(\"%d\",&param); }"
  ...
END

While the use of TESTFLAGS should probably be limited,
it may be useful in some applications.

--------------------------------------------------------

2.0 Test Driver Example - ARRYST.TST

The sample test driver (ARRYST.TST) tests the module
ARRYST.HPP and ARRYST.CPP.

The simple module defines a class that investigates
the min, max, and mean of an array of double precision
numbers:

class ArrayStatistics {
	private:
		double minimum;
		double maximum;
		double mean;
		int min_occurred_at;
		int max_occurred_at;
		int status;
	public:
		ArrayStatistics( double *array, int narray );
		int Reset( double *array, int narray );

		double Minimum() { return(minimum); }
		double Maximum() { return(maximum); }
		double Mean() { return(mean); }
		int MinOccurredAt() { return(min_occurred_at); }
		int MaxOccurredAt() { return(max_occurred_at); }
		int Status() { return(status); }
};

There are 7 tests defined in the TESTMAN test driver
definition file, ARRYST.TST.

Build the test driver file by:

C:> testman arryst.tst

This creates the file, ARRY_TST.CPP.

Compile and link this file along with ARRYST.CPP.  The
executable will produce the following output:

ARRYST.CPP - class ArrayStatistics
7 Test 32 Checks      0 failures

There is a build flag "ENABLEBUGS" that you uncomment
in ARRYST.CPP to enable a buggy implementation of
ArrayStatistics.  Recompile, link, and execute the
test executable.  The output should be:

ARRYST.CPP - class ArrayStatistics
Test 1 (positive_array), FAIL: stats.Minimum() == sign*1.0
Test 1 (positive_array), FAIL: stats.MinOccurredAt() == 0
Test 2 (negative_array), FAIL: stats.Maximum() == sign*1.0
Test 2 (negative_array), FAIL: stats.MaxOccurredAt() == 0
Test 5 (count_is_1), FAIL: stats.Minimum() == value
Test 5 (count_is_1), FAIL: stats.MinOccurredAt() == 0
Test 6 (count_is_1b), FAIL: stats.Maximum() == value
Test 6 (count_is_1b), FAIL: stats.MaxOccurredAt() == 0
7 Tests	 32 Checks	 8 Failures