The Beta Tester's Guide

                   by Tom Rawson, JP Software



[Copyright 1988, 1991, 1992, JP Software Inc., All Rights
Reserved.  See end of document or contact J.P. Software, 617-646-
3975, for reprint / distribution restrictions.]


This guide is designed to give you a few ideas about what to
expect and how you should behave as a "beta tester" for a piece
of software.  It is written with beta testing for the small
commercial or shareware developer in mind, but you may also find
it useful if you're beta testing a product from a large software
developer.


What's a Beta Test?

Most programs or program enhancements of any consequence go
through roughly four different phases between the original design
and first release:


(1) Design, coding, and testing by developer.


(2) "Alpha" test by a small group selected by the developer.  In
software development it's pretty easy to be so involved in the
program that significant bugs or design problems are missed. 
It's also almost impossible to foresee and have good judgement
about all the things users might want or need, and many of these
things can't be determined during the design phase.

The alpha testers are extra sets of eyes and ears for the
developer.  They pick up bugs that an end user might look at and
say "gee, they should never have even considered releasing this
thing!"  They try out different approaches to the user interface,
run the software on a variety of hardware configurations, or
debate suggestions on what functions should be included or
excluded.  In a large company the alpha testers may all be
internal to the company, or may include only a very small number
of trusted and technically competent customers.

When the alpha test is complete the product may not be bug free,
but it is known to run reasonably well on a variety of systems
without exhibiting major bugs or destroying other parts of the
system.  It also usually has at least reasonably good agreement
from the alpha test group that it meets their needs and the needs
they perceive users will have.


(3) "Beta" test by a larger and more representative group of
customers, either those who expressed interest in testing a new
product, or those who are already using an older version and are
willing to try out a new release.  

The job these people have is to make sure the product is solid
before it is released, i.e. that it runs properly on the widest
possible variety of appropriate hardware; that both major and
minor bugs are identified and taken care of; that auxiliary items
such as support programs, documentation, and help screens are
well-designed and work correctly; and that any additional design
and interface problems are identified and dealt with.

Most products go through more than one phase of beta testing
before release, and there are usually a number of updates
supplied to beta testers as the product is refined and corrected
in response to their comments.


(4) Release of the product to the public.


Once these four phases are complete there is often ongoing beta
testing of additional enhancements to the program.  This may
begin immediately after release, or be delayed for a while,
depending on the development cycle the developer wants or has the
resources to follow.


Beta Testing Pros and Cons

If you decide to beta test a product you are going to get some
benefits.  You will get the latest release early, usually well
ahead of others; this can be especially important if you need to
use some new features of a product to get your job done or your
application running.  You will have an opportunity to shape the
nature of the final product, which may be important if it's
something you use or plan to use heavily.  And you may have a
chance to learn the internal workings of the product.  You'll
probably get to know the developer and support staff well, and
have an inside connection for any future questions or support
needs.

In most cases, you will receive a free copy of the product or
update (some large companies actually pay beta testers, but it is
not a common practice for PC software vendors, especially
shareware developers and small companies).

You are also going to have some frustrations.  The program is
going to have bugs in it.  Some of them will be in the piece that
you just had to have yesterday.  The user interface will change
on you and you'll have to re-learn the command-line switches, or
the function keys, or some other part of the program's behavior. 
Things that used to work will "break" in a new release, then get
fixed again.  The developer will not be able to fix everything
right away.  Some of your suggestions will be rejected for
reasons you don't agree with.

Most beta testers of course find that the benefits outweigh the
frustrations; if not they don't become (or remain) beta testers.


Beta Testing Guidelines

These guidelines are intended to help you do what's necessary to
make the beta test a productive experience for both you and the
program's developer.  They aren't absolute of course, and some
will vary from situation to situation.  Some of them may seem to
conflict; that's intentional.  Striking a good balance is what
will make your participation in the beta test go well.


(1) Verify what you find.  Does the program behave strangely if
you run it on Sunday morning, and work fine the rest of the time? 
Don't fire off a bug report the first time, try it again.  Don't
overdo it -- there's no point in beating your head against the
wall -- but two or three tries at reproducing the problem and
determining under exactly which conditions it occurs may be
immensely helpful to the developer.  Remember, s/he isn't sitting
there at your computer, and the problem may not show up the same
way on another system.


(2) Don't jump to conclusions.  If you report that "this thing
doesn't work when run with my pop-up calculator", you should try
to verify that that's really what's happening.  Is it really the
calculator, or is it only when the calculator is loaded with the
disk cache that the problem occurs?  Has your favorite
hexadecimal to dodecahedral conversion algorithm really been
eliminated in the new release, or was it just accidentally left
off the help screen?  It is much easier to send the developer off
on a wild goose chase if it appears that you've narrowed the
problem down further than you actually have.  

Another way to say this is, when reporting on something to the
developer, differentiate between what you know and what you're
guessing at:   "With my Paradise VGA board and a Magnavox monitor
the text comes out pink on red, perhaps there's a problem with
the color selections on VGAs" rather than "The color selections
don't work on VGAs any more".  Of course the developer must
"edit" your reports in this same fashion, but the more specific
you can be about what you actually know, the better.


(3) Report everything.  Verify it first, and report only what you
found, but do report it.  The thing you run across once that
appears minor may be a key issue for some other type of customer,
and the developer will want to know about it.  Don't make the
decision for the developer that something is too small to matter.


(4) Express your opinion, positive and negative.  Is there
something you like, especially something new?  Say so!  If you
don't the developer may decide it isn't such a big deal and get
rid of it, when really it made the program much better.  If the
new menu arrangement is really super and worth losing
compatibility with the old one, the developer needs to know.

Is there something you don't like, even though it's not a bug? 
Maybe the human engineering of the keyboard is great with the
function keys at the left, but on a keyboard with top-row
function keys it is incredibly difficult to use.  Say that too --
once (then see the next section!).  Don't restrict yourself to
bug reports; the developer tries to assemble a representative
group of testers, and relies on you to state your opinion
thoughtfully.

Be sure you provide alternatives when you have problems with the
design:  "It would be better if it did Y" rather than just "It
shouldn't do X".  The developer is going to have a much easier
time being responsive if your suggestions are constructive. 
Sometimes this is difficult -- for example, if the program is too
slow, you may not know how to make it faster.  But you can at
least indicate why that is a problem:  explain just which parts
are too slow to meet your needs, and what those needs are.


(5) Expect to be snubbed.  OK, maybe that's a little strong.  But
remember that the developer is dealing with a lot of (usually
conflicting) ideas, suggestions, and reports.  Yours is not going
to be given a high priority simply because it is important to
you, unless it also fits in with other plans, the reasonable way
to do the programming, etc.  

Most developers are quite responsive during a beta test, but they
also have to keep the product operating consistently and do their
work efficiently; as a result, they may choose to make decisions
you don't like.  The best developers will consider the product to
be yours as much as theirs, but they must still be the ultimate
architect of how it looks and works.  

Further, you may have ideas that sound good on the surface, but
turn out to be infeasible because they require rewriting a major
portion of the code, or would need a change to the operating
system, etc.  Often these things cannot be seen from the outside,
so your suggestion may appear like the "obvious" way to do
things, yet still be difficult to impossible to implement.

So go ahead and argue for your point of view.  Present it, see if
you can convince your fellow beta testers (if you have contact
with them) and/or the developer.  But if you lose the battle, let
it go -- if you can't, you probably shouldn't be beta testing the
product.


(6) Keep your files backed up.  There are all sorts of ways that
a bug can cause a disk error that messes up a file, a directory,
or even a whole disk.  These are usually very rare, but they
cannot be entirely eliminated without testing; since you're one
of the testers, you need to be prepared to handle them, and check
for them.  

For example, on an MS-DOS system when testing a program that uses
files (as most do), you should run CHKDSK periodically to
determine that your file allocation table isn't getting mixed up. 
Of course if it is, try to verify that the problem is caused by
the program you're testing -- many things can mess up the disk,
and it is the kind of bug you need to verify as carefully as
possible before sounding an alarm, and report as specifically as
possible.


(7) Don't use a beta test version for critical applications.  In
some cases you may have to, but be prepared for the consequences. 
At least once during the beta test you will probably find that
something you need isn't working, and you'll have to go back to
an earlier version, or the previous release (pre-beta test), to
get your work done.  If you have become dependent on new
features, you may be in trouble.  

Similarly, features or methods of operation added during the beta
test can also be removed if problems crop up, so the thing you
need most may disappear.  This is rare, but sometimes the
developer will find that a feature just doesn't fit with the rest
of the program and has to be changed.  As long as the product
hasn't been released, that's the developer's prerogative.


(8) Use the program as you normally would.  On the one hand, the
developer is expecting you to really use the program, not just
try it a couple of times -- if you're just a casual user, let the
developer know as they may need more regular users to make the
test worthwhile.  But on the other hand, you don't need to overdo
it or try every single feature, unless you want to.  

The developer has recruited beta testers with a variety of
experience levels, needs, and uses of the product, to be sure all
aspects are tested.  If you use the product in your usual way and
pay careful attention to any problems, questions, or suggestions,
you can feel comfortable that you're doing a good job.  

Do try out new features that you're interested in, or test the
program at "boundary conditions" if you can.  ("What happens if I
give it a *really* big file to process?  How about a really small
one?"). But it is probably more valuable to the developer to have
you watching carefully what happens during normal use than to
have you trying all sorts of features you aren't familiar with
but not really paying attention to what happens and how they
work.


(9) Keep things confidential.  In virtually all cases the
developer will require that some or all aspects of the beta test
be kept confidential.  Respect this agreement.  Don't pass copies
of the program around, and don't talk about anything you aren't
supposed to talk about.  The developer has determined what
information s/he wants to make public and when based on what
market the program is in and how the new product or version will
be publicized; it's not up to you to decide this on your own.  Of
course the developer needs to make these restrictions clear at
the beginning (most developers use a formal, signed beta test or
non-disclosure agreement which spells these things out exactly). 
You could even be subject to legal action if you violated such an
agreement.


(10) Keep the old versions.  You will certainly need them.  The
newest version is going to fix some bugs and introduce others,
some of which you may find intolerable.  The best way to handle
this for most people is to save the most recent public release
(if you are working on an enhancement), the most recent
relatively stable beta test version, and the most recent and
next-most-recent beta test versions, whether they are stable for
your use or not.  This allows you to test things on different
versions if you can, and go back to older stable versions if
necessary.


(11) Know when to get on (and off) the beta test "bandwagon". 
Some developers may not know when their product isn't ready for
beta testing yet, and some beta testers may not realize what
they're getting into.  You can find lots of requests for beta
testers on electronic bulletin boards and on-line services. 
Virtually all of them are from reputable developers with a
product that is most likely more than ready to go into beta test. 
But occasionally you may find that the program is just too buggy
to even try, or that it doesn't meet your needs in the way you
thought it would.  

When this occurs just politely inform the developer that you'd
like to withdraw from the test.  Remember that what you were
promised was the right to try the new version and test it; if it
doesn't work the way you wanted it to you can certainly stop.  Be
gracious about it, and the developer usually will also.  
Similarly, if you can't continue using the product reasonably
regularly and reporting the results, or meet other conditions the
developer sets, you should also withdraw, because your
participation won't meet the developer's needs.  

If you do choose to withdraw before you've put in the requisite
amount of effort, or you haven't met all of the developer's
conditions (e.g., filling out an evaluation form) don't expect a
free copy of the program when it's released, as you haven't
continued with your end of the deal.  But do expect the developer
to be polite about it also:  people drop out of beta tests all
the time, and it's the developer's responsibility to assemble a
team that can handle a little attrition.


Conclusion

You'll probably find being a beta tester rewarding and valuable;
for popular products there are often many more people ready to
test than needed.  If you pay attention to what you're doing and
follow the guidelines above, you can make it work well for both
yourself and the developer.




----------------------------------------------------------------

This guide was written by Tom Rawson, JP Software Inc.,
Arlington, MA, and is Copyright 1988, 1991, 1992, JP Software
Inc., All Rights Reserved.  Permission is granted for developers
of public-domain and shareware software to distribute this
material at no charge in conjunction with beta tests of their
products, provided this notice is included.  Any other use
requires specific permission from JP Software.  We can be
contacted via CompuServe ID 75300,210 or by calling 617-646-3975
during normal business hours.  JP Software is the publisher of
4DOS, the award-winning COMMAND.COM replacement for MS-DOS,
PC_DOS, and DR-DOS, and of 4OS2, the superior replacement command
processor for OS/2 1.2, 1.3, and 2.0.