Zipfile Duplicate Checking System (ZDCS) Ver. 2.0 Copyright (C) 1991, 1992 Michael W. Cocke --------------------------------------------------------------------- Technical Reference Manual TABLE OF CONTENTS ----------------- ABOUT THE DOCS.................................................3 PURPOSE OF ZDCS................................................3 GENERAL TECHNICAL INFORMATION..................................4 ZDCS OPTIONS...................................................5 Deletion of Duplicate Files...............................5 Allowed Duplicates........................................6 Privileged Users..........................................6 BBS Ads...................................................6 Pre-Testing...............................................7 INSTALLATION OVERVIEW..........................................7 THE ZDCS DIRECTORY.............................................8 THE ZDCS CONFIGURATION FILE....................................8 Function..................................................8 Line 1....................................................9 Line 2....................................................9 Line 3....................................................9 Line 4...................................................10 Line 5...................................................10 Line 6...................................................11 Line 7...................................................12 Line 8...................................................12 Line 9...................................................13 Line 10..................................................13 Line 11..................................................13 Line 12..................................................14 Line 13..................................................14 THE ZDCS DATABASE BUILD.......................................14 Purpose..................................................14 Creation of the Initial Database.........................15 The Screen Display During the Database Build.............15 Additions to an Existing ZDCS Database...................17 ZDCSDB Without a Separate File Integrity Checker.........17 The Database Build Log File ZDCS-DBB.LOG.................18 THE ZDCS DUPLICATE REPORT GENERATOR...........................18 THE ZDCS DATABASE PURGE.......................................20 BBS ADS.......................................................21 Function.................................................21 Creation of the BBS Ads Database.........................22 Updating the BBS Ads Database............................22 Selecting Deletion or Flagging of BBS Ads................24 ALLOWED DUPLICATES............................................25 - 1 - Zipfile Duplicate Checking System (ZDCS) Ver. 2.0 Copyright (C) 1991, 1992 Michael W. Cocke --------------------------------------------------------------------- Technical Reference Manual TABLE OF CONTENTS ----------------- THE UPLOAD FILE CHECKER.......................................27 Purpose..................................................27 Function.................................................27 Maximum and Actual Percent Dupes in an Upload............28 Updating ZDCS Database(s) After an Upload................29 DOS Error Levels.........................................29 CALLING THE UPLOAD FILE CHECKER...............................30 Calling ZDCSFC.EXE from EXZTEST..........................30 Calling ZDCSFC.EXE from the PCBTEST.BAT File.............30 The %1 %2 %3 Parameters..................................32 PROCESSING LOCAL UPLOADS......................................32 PRE-TESTING...................................................34 ZIPFILES WITHIN ZIPFILES......................................35 CD-ROMS AND DATABASE MERGING..................................36 PKZIP AV STAMP INTEGRITY......................................37 ARJ SECURITY ENVELOPE INTEGRITY...............................38 ACCURACY OF THE CRC32 METHOD..................................39 MEMORY USAGE..................................................41 TROUBLE-SHOOTING GUIDE........................................41 Your system is crashing during the upload test...........41 ZDCSDB.EXE (the database build) is crashing..............42 ZDCS is reporting a device I/O error (in any module).....42 ZDCS reports path/file access errors.....................43 QEMM exception 13 errors occur when running ZDCS.........43 REGISTRATION..................................................43 SUPPORT.......................................................45 Contact Information......................................45 Goodies..................................................46 Database Merge......................................46 Pre-Built CD-ROM Databases..........................46 Local Upload Processing.............................47 Database Browser....................................47 Fixpath.............................................47 BBS Ad Tracker......................................47 Conversion from ZDCS 1.6x to ZDCS 2.0...............47 Documentation............................................48 FUTURE ENHANCEMENTS...........................................48 ACKNOWLEDGEMENTS..............................................48 COPYRIGHTS AND LEGAL STUFF....................................49 - 2 - ABOUT THE DOCS -------------- Welcome to the technical reference manual ZDCS-REF.TXT for ZDCS version 2.0. This manual has been prepared to give you an easy reference guide for the various parts, functions, and options in ZDCS. It is deliberately arranged to be modular in organization so that you can look up whatever interests you without having to read the entire manual. This manual should be the =second= doc file that you read. We recommend that you first read the walk-through ZDCSWALK.TXT if you are new to ZDCS, or the conversion file CONVERT.TXT if you are running ZDCS 1.6x and looking to convert to version 2.0. That will get you up and running painlessly. The walk-through ZDCSWALK.TXT is a friendly guide that is meant to hold your hand and whisper sweet explanations in your ear as you install and first explore ZDCS. It's arranged to take you through all the steps you need to know from beginning to end. The walk-through is the first place that new users of ZDCS should go for guidance when installing and configuring ZDCS. The conversion file CONVERT.TXT is written specially for the user of ZDCS 1.6x upgrading to version 2.0. It provides instructions for quick conversion from the older versions of ZDCS to version 2.0. If you're already familiar with ZDCS 1.6x and want to upgrade to the current version, this is the first place you should look for help in getting ZDCS 2.0 up and running on your system. It is also the place to go for answers to questions specific to the upgrade, like how to convert your old ZDCS 1.6x database to the new database format used in ZDCS 2.0. PURPOSE OF ZDCS --------------- ZDCS is a shareware set of utilities intended to help a PCBoard sysop deal with the problem of duplicate files, whether those files are already on the bbs or are being uploaded by a caller. It provides specific support for looking inside archives made with PKZIP, including PKZIP version 1.93 (ZIP files), archives made with ARJ (ARJ files), and self-extracting files made with either PKZIP or ARJ (SFX files). ZDCS processes all of these archives so that the differences between them are transparent to the sysop and caller. To make this walk-through easier to read, we'll refer to all of these file types as archives. ZDCS also provides support for accepting unzipped GIFs. ZDCS helps to manage the problem of duplicate files in two ways. 1. It provides a method for weeding out duplicate files from an existing collection of files, like a bbs file system or a CD-ROM collection before mastering. - 3 - 2. It provides a duplicate checking method for any prospective addition to the existing file base, such as uploads to a bbs or late additions to the CD-ROM collection. ZDCS was written by a sysop and programmer who knows that sysops never have enough time to do all the things they would like to do on their boards. Making ZDCS friendly to use for both sysop and caller has been important since version 1.0 was first written. Although ZDCS was developed primarily for use on PCBoard bbs's, it can also be used to look for duplicates on other systems as well, such as a shareware CD-ROM or even multiple directories on your hard drive system. However, it's on a bbs that ZDCS really shows its stuff. ZDCS can be told to decline an upload, to automatically remove duplicate files, to delete those pesky little bbs ads from uploads, and to recognize "allowed duplicates" - or any combination of the above. You can tell ZDCS to accept anything that a privileged user uploads, and change your mind on who's "privileged" and who's not. There's even a pre-test capability that lets callers easily find out ahead of time whether or not their intended upload duplicates files already on your board. All of these are discussed in this reference manual. Setup and configuration are also described in the walk- through guide ZDCSWALK.TXT. GENERAL TECHNICAL INFORMATION ----------------------------- ZDCS handles file archives with the .ZIP extension created by PKZIP (including version PKZIP 1.93), with the .ARJ extension created by ARJ, and with the .EXE extension created by PKZIP or by ARJ (self- extracting files, also called SFX). ZDCS is able to see inside each of these archives to deal with the individual files inside them. For simplicity, we will refer to all of these files as zipfiles. ZDCS also handles GIF files. These are individual graphic files that already have their own compression. GIFs may be uploaded to some bbs's as individual files. They are not archives of multiple freestanding files, so ZDCS does not need to look "inside" them. In fact, ZDCS thinks of them rather like poor zipfiles with only a single file in them. Whenever ZDCS encounters a file that is not a ZIP or an ARJ or an SFX, it treats that file like a GIF. Although we will continue to use the term GIF to refer to these files, ZDCS could actually handle any "other" type of file by treating it as a single file, the same as it does with GIFs. ZDCS does not provide support for archiving methods besides ZIP and ARJ. This means that if ZDCS encounters an LZH archive, for example, it cannot see inside the archive to look at the individual files. Instead, the entire archive would be treated as a single GIF-type file. - 4 - ZDCS makes use of the 32-bit CRC, often called the CRC32, used internally by PKZIP for ZIPs and SFXs. When ZDCS encounters any other file, it considers that file to be a GIF and calculates the CRC value for it. ZDCS checks files for duplicates by keeping a database of the CRC values. When new files are added to the system (uploads), ZDCS compares the CRC values of the newcomers to those in the database to determine if there are any matches, indicating duplicate files. This method uses the contents of the files and is independent of the names of those files. The ZDCS database of CRC values uses a B-tree index, so there are no external sort utilities or regular file maintenance requirements of any kind. ZDCS is compatible with any Netbios compatible lan, such as Lantastic or Novell Netware. It has also been tested extensively with many CD- ROM devices and with Alloy slave cards. ZDCS OPTIONS ------------ ZDCS has five sets of options that can be configured independently of each other and changed at any time. You can try out the different options and change your mind about which ones you want to use without re-installing ZDCS. For a good guide to choosing the options to fit your system, take a look at the walk-through ZDCSWALK.TXT. As part of the installation guide it explains what each option can mean to your bbs and what some of the possible consequences are. Each of these options is also covered in greater technical detail in other sections of this reference manual. Running ZDCS without any of the extra options enabled gives you a basic duplicate checking system that will not delete any files automatically. It will still tell you about all duplicate files, but it won't be able to distinguish between files you always want to delete (like bbs ads), files that duplicate exactly ones already on your board, and files that are duplicates but still important to keep (like VALIDATE.COM from Macafee's SCAN program). You can try out the different options and change your mind about which ones you want to use without re-installing ZDCS. All it takes is a change to the appropriate line(s) in the configuration file. Deletion of Duplicate Files --------------------------- ZDCS can be set to either flag or delete duplicate files from uploads to your bbs. Either way, ZDCS will still recognize the files that are duplicates of ones already in the bbs file system and leaves you messages in a log file. This deletion feature does not operate on a GIF file. - 5 - Enabling the deletion of files is the only way that ZDCS could ever have the opportunity to change the AV stamp on a zipfile made with PKZIP, or the security envelope on a zipfile made with ARJ. That happens only if ZDCS deletes a file that is included "inside" the AV stamp or the security envelope. (It is possible to create a either type of archive that has additional files added outside stamp or envelope but still inside the same archive.) ZDCS can be set to delete duplicate files and / or bbs ads, with each option being set independently of the other. Allowed Duplicates ----------------- You can designate certain files as allowed duplicates. Although these files may already be present on your bbs, you can tell ZDCS not to treat them like duplicates. This prevents ZDCS from deleting them if you have selected the option to delete duplicate files. It also prevents the allowed duplicates from being counted in the calculation of the percentage of duplicates in an upload. Let's look at an example to see both of these effects in action. A caller uploads a zipfile containing eleven files. One of those files is in your allowed duplicates list, and three of the files are already present on your board. ZDCS will report that there is one allowed duplicate in the zipfile. Since that one file is not counted in the calculations, ZDCS will ignore it when determining the percentage of dupes in the zipfile: Total Number of Files 10 (not eleven!) Number of Dupes 3 (allowed dupes don't count) Percentage of Dupes 30% (3/10, not 3/11) This feature does work for GIF files. If a GIF is included among the allowed duplicates, then a repeat upload of the same GIF will not be flagged as a duplicate, and the upload will be accepted. Privileged Users ---------------- You can designate one or more user names as privileged users. Any file that is uploaded by a someone on the privileged user list is never declined by ZDCS, no matter how many duplicate files there may be in it. This provides you with the option of excluding one or more usernames from the usual dupe checking requirements you've set for the rest of your callers. BBS Ads ------- You can specify certain files as truly obnoxious. These are collectively referred to as bbs ads after one of our favorites, although they could also include get-rich-quick files or any other nuisance file that you've already seen once and don't ever want to see again. ZDCS can be told to either flag or delete these files from all uploads. - 6 - This option is completely independent of the option to delete duplicate files. A major distinction between the two types of files is that bbs ads are files that are known and despised ahead of time. This option provides a safe way to delete them without risking the removal of authors' unchanged files from newer shareware versions. The bbs ads option does not operate on a GIF file. Enabling the deletion of files is the only time that ZDCS can ever change the AV stamp on a zipfile made with PKZIP, or the security envelope on a file made with ARJ. That happens only if ZDCS deletes a file that is included "inside" the AV stamp or the security envelope. (It is possible to create either archive that has additional files added outside the stamp or envelope but still inside the same archive.) ZDCS can be set to delete bbs ads and / or duplicate files, with each option being set independently of the other. Pre-Testing ----------- ZDCS offers an easy method for callers to your bbs to pre-test an upload before actually sending the full file to the board. The pre- test is quick and simple for them to do, and gives the same information that callers would see after sending the full file. There are no upload credits granted for using just the pre-test feature without sending up the full file. A canned bulletin is included in this package to explain to your callers how to use the pre-test feature. The sysop side of pre- testing is also easy to implement, and is discussed in the PRE-TESTING section of this manual. INSTALLATION OVERVIEW --------------------- There are six basic steps to installing ZDCS to work with the bbs: 1. Copying the ZDCS files to a new ZDCS directory. (required) 2. Setting up the configuration file. (required) 3. Creating the initial database. (required) 4. Creating the bbs ads database). (optional) 5. Creating the list of allowed duplicates. (optional) 6. Setting up the check for uploaded duplicates. (required) There is a walk-through called ZDCSWALK.TXT included in this package to help you install ZDCS. It's a friendly guide to understanding the functions and installing the programs on your bbs. It is especially good for taking you through the steps slowly and in order so that installation goes smoothly on your system. If you're already using ZDCS version 1.6x and you want to convert to version 2.0, then the file you need is CONVERT.TXT. It guides you through the conversion so that you don't have to spend time creating a new database from scratch or reinventing the wheel. - 7 - This technical reference manual has information on all the parts, functions and options of ZDCS, but it's arranged with a different purpose in mind: to allow you to find and read only those specific sections you want to know more about. If you want to install ZDCS, we highly recommend using the walk-through as your primary guide and all- around hand-holder. If you want to convert from ZDCS 1.6x to ZDCS 2.0, then we recommend the conversion file as your best friend for speeding up the process. THE ZDCS DIRECTORY ------------------ All of the working files for ZDCS should be collected together in one directory. The only possible exception to this is the ZDCS database (ZDCS.DAT and ZDCS.NDX) created for your system, which you have the option of locating elsewhere. All of the executables in this package (*.EXE), the configuration file (ZDCS.CFG), and any additional files you create such as the bbs ads database or the list of allowed duplicates, must all be together in this one directory. For simplicity, this directory is referred to as your ZDCS directory. The ZDCS database is by far the biggest set of files for most bbs and CD-ROM applications. The option to locate it somewhere else besides the ZDCS directory is a convenience, especially for sysops who may not have enough disk space where they want to put the ZDCS directory. The ZDCS directory should be part of your PATH statement. Remember that you'll need to reboot after changing the PATH statement in order for the changes to take effect. THE ZDCS CONFIGURATION FILE --------------------------- Function -------- The ZDCS configuration file ZDCS.CFG is central to the entire process. All the modules of ZDCS look for and use at least parts of this file. It's also the place where you set many of the options and all of the parameters of ZDCS. For such a powerful component, the ZDCS.CFG is surprisingly simple: just thirteen short lines of ASCII that you can create with any text editor. A sample configuration file is included with this package. The contents of each line are discussed in excruciating detail further on in this section. Under most circumstances, ZDCS.CFG should be in the same directory as the rest of the ZDCS files. This is always true if you are running DOS version 3.x or higher. - 8 - The one time that the configuration should be located elsewhere is for those poor lost souls still running DOS 2.x. In this unusual case, ZDCS.CFG belongs in whichever directory will be the current directory when the ZDCS program modules are run. Line 1 ------ This line is the complete drive, path and filename of an ASCII text file. This file is one that you create listing all the pathnames, one on each line, for all the directories containing the zipfiles / GIFs to be included in the database. PCBoard refers to this as the download path list and defaults to DLPATH.LST as the file name in PCB setup. You can use that file to point to the complete bbs file directory only if you are =not= using the index file feature in PCBoard 14.5a. You can also use the freeware utility DIRTREE (by Mike Cocke, available on The Hacker Central BBS) to create a text file of all directories on a specified drive. Depending on how your system is arranged, you might be able to use the resulting text file directly, or you might need to delete a set of lines from the file. In any case, you won't have to wonder whether you've remembered to include all the pathnames. There is no upper limit on the number of pathnames that can be processed. It does not matter whether or not you've included the trailing backslash for each pathname. To process a new collection of files into the ZDCS database, like those on a CD-ROM, just change either this line or the contents of the file to which it points. Line 2 ------ This line is the complete drive, path and filename where you want the finished ZDCS database (ZDCS.NDX and ZDCS.DAT) to be located. It makes no difference if you include the trailing backslash here or not. You can put the ZDCS database in the same directory as the rest of the ZDCS files and programs, or you can decide to put it on a different drive or even across the network. This was made possible because the database is the largest pair of files, and some sysops needed the flexibility of locating them on a drive with more spare room. All the rest of the ZDCS files (such as the bbs ads, the allowed duplicates, and whatnot) should be grouped together in the same directory with the executables. Line 3 ------ This line is the complete drive, path and filename giving the location of the privileged user list file. If you don't want to have any privileged users on your system, simply leave this line blank. - 9 - A privileged user is someone who can upload no wrong. Whenever a file is uploaded by a user named on the privileged user list, the file is explicitly passed by ZDCS, no matter how many duplicates there might be in it. Even if you have enabled deletion of duplicate files and / or bbs ads, ZDCS still will not delete them if the original zipfile was uploaded by a privileged user. The privileged user status takes precedence. The format of the file containing the privileged user list is straightforward. Put one username on each line and end each line with a CR/LF. It makes no difference if you use upper case, lower case, or a mixture of the two. There is no maximum number of names you may put in the privileged user list, but remember that if you make this a long list, ZDCS will take longer to check new uploads because it will have to check every name in this list. This feature was added for a couple of sysops who wanted to pass specific files (beta code) to each other as part of unattended mail runs, and wanted those files automatically posted for public access instead of possibly held for later sysop review. Line 4 ------ This line is either the letter Y or the letter N. It controls whether you want ZDCS to add the disposition line to the end of the upload description. The disposition line shows the total number of files in an upload and the number of those that were duplicate files. It is used when ZDCS declines an upload. Here is a sample disposition line: ZDCS: 12 Duplicates / 12 files The actual numbers will change depending on the file. ZDCS has to be running in standalone mode for this, not with any of the current gateways to file integrity checkers, such as with Extest. When ZDCS is running with a gateway, then the actual contents of this line are ignored by the program. Technically, it's the use of the OMP parameter in the gateweay connection and not the existence of the gateway itself that is the controlling factor. If you're really interested in the abstruse technical details, take a look at the third party interface file ZDCS-TPI.ZIP on The Hacker Central BBS. You still need to have this line present in the configuration file so the total line count doesn't fall off, but the line can contain anything as a placeholder. This new feature was added by request. Please note that you must be running PCBoard 14.5a in order to make use of it, and that the third command line parameter must be specified (ZDCSFC %1 %2 %3) in your PCBTEST.BAT file. Otherwise, leave this fourth line of the config set to the letter N. Line 5 ------ This line is either the letter Y or the letter N. It controls whether you want ZDCS to truncate nulls from the end of GIFs (or "other" type - 10 - files) before performing any operations on them. The truncation is actually done on a copy of the file and the original is left intact. Any AV stamp that is present is not disturbed. Nulls (ASCII 0) can be added to the end of a file by some transfer protocols (such as Ymodem) in order to make the entire file come out on a block segment. Other protocols (such as Zmodem) do not add the null characters. That would make the identical file uploaded by the two different protocols slightly different files by the time they arrived on your system. If the nulls were included in the file contents when any calculation or comparison was done, it would look like two different files instead of the same file transferred by two different protocols. Setting this line to the letter Y enables ZDCS to ignore those extra nulls. The advantage of using this feature is increased accuracy when the same file is uploaded by different transfer protocols. The disadvantage is that it adds a bit of time to the upload checking and database build operations. The time is needed for ZDCS to create a temporary copy of the file in order to remove the exrta nulls, and again to delete that temporary file, leaving the original upload intact. Use of the option to truncate nulls also requires enough additional disk space to hold a temporary copy of the file so that the original one is not disturbed. Line 6 ------ This line is an integer - that's a whole number, no decimals - between 0 and 100. It sets the maximum percentage of dupes that your bbs will accept in an upload. ZDCS will calculate the actual percentage of duplicates in the upload and compare it to your maximum percentage. If the actual percentage is lower, the upload is accepted. If the actual percentage is equal to or higher than the maximum you specified, the upload is declined and kept in your private upload directory for your review. Setting the percentage to 100 effectively bypasses this filter, since it permits a duplicated GIF or a zipfile with nothing but duplicates to pass. At the other extreme, setting the percentage to 0 effectively requires that the uploaded GIFs and zipfiles have no duplicates at all. If you make a mistake and enter a decimal on line 6, ZDCS will not crash. It will simply truncate your number (lop off everything after the decimal point) and use the resulting integer as the maximum percentage of dupes. This works out quite well in actual practice. Uploads that are a fraction of a percent under the maximum percentage of duplicates are the only files where this makes any difference, and the use of truncating instead of rounding means that they will always be passed. IF ZDCS rounded the decimal points instead, there would be some uploads that would round up, making their actual percentage of dupes the same as the maximum value. Such files are declined by ZDCS. - 11 - Line 7 ------ This line is the complete drive, path and filename you want ZDCS to use for the log file created by the upload file checker ZDCSFC. This log is an ASCII text file that contains information from the upload file checker ZDCSFC for each upload it has processed. Each message about a specific upload includes the following: name of the uploaded zipfile or GIF list of all component files inside a zipfile which files are flagged as bbs ads which files are marked as duplicates reference to the file already in the database which is duped by the UL which files are marked as allowed duplicates actual percentage of dupes in the upload whether the upload was accepted or declined The fifth line in the list above is new in ZDCS 2.0. It means that when a duplicate file is found in an upload, ZDCS will provide the reference to the zipfile or GIF containing the copy of that file which was already on the bbs. It happened that callers who had uploaded files with lots of dupes were not always able to find where those files already existed on the system, often because files had been renamed. Those callers sometimes pestered the sysop for this information, which wasn't always easy for the sysops to locate, either. Now this reference is written directly to the upload checker log file The information includes the name of the zipfile or GIF and its pathnumber in the system. If PCBOARD.SYS is in the current directory when the upload file checker is run, then the name of the currently logged caller is also included in the log file. Line 8 ------ This line is either the letter Y or the letter N. It controls the switch to tell ZDCS whether to delete bbs ads (Y) in an uploaded zipfile or to just flag them (N). If you've decided not to enable any checking for bbs ads at all, just set this to N. This is not the place where you decide if you want to check for bbs ads. You do that by creating the bbs ads database. If that database is found, then ZDCS will check files against the database of known bbs ads looking for a match. If there is no bbs ads database, ZDCS won't be able to check for them. The choice you are making here is whether to remove those known bbs ads from zipfiles, when they are encountered. This decision is separate from the choosing whether to delete duplicate files in general, which is set in line 9. You may configure these two options independently of each other. - 12 - Line 9 ------ This line is either the letter Y or the letter N. It controls whether you want ZDCS to delete all duplicate files from an upload (Y) or just flag them and leave them intact (N). If you choose to delete all duplicate files, then any duplicate file that ZDCS finds that is not specifically listed as either an allowed duplicate or a bbs ad will be deleted from the zipfile ZDCS is processing. The bbs ads are handled by a separate option, determined by the existence of the bbs ads database ZDCS-BBA.NDX and configured in line 8. Line 10 ------- This line is reserved for a single line of text by the sysop. The contents of this line are appended to the PCBFAIL.TXT file created by ZDCS whenever an upload is declined. The caller who has just uploaded the declined file sees this line of text as a message on the screen. This line is where you can express from 1 to 72 characters' worth of creativity. Some callers have become quite fixated on the idea that "declined" is the same as "thrown out" - which is of course not true. You can use this line to tell the caller what has happened or will happpen with the upload. One possible line to use is: Too many duplicate files - upload must be reviewed by sysop. If you don't want to display any message to the caller, just place something innocuous like a period or even a blank space on this line. Just don't leave the line completely blank! You also shouldn't use any quotation marks in this line. You can make use of PCBoard @variables and &filespec to your heart's content; both are fully supported here. If your file integrity checker overwrites the PCBFAIL.TXT file, your callers won't see this line of text unless you choose to use ZPEND. This is another free utility available on The Hacker Central BBS. By using ZPEND with such an integrity checker, you will make sure that this line of text is displayed to the caller. Also displayed will be a complete ZDCS breakdown of the upload, showing which files are bbs ads, dupes, allowed duplicates, and so on. Line 11 ------- This line contains the filename ZDCSFC.OUT and nothing else. For all practical purposes, this line is planning for the future. Technically, this line points to the name of an ASCII text file that will be created every time an upload is processed. It's a spare copy of the ZDCS summary information that most upload checkers overwrite. ZPEND and some utilities-in-progress use it. Leave this line set to ZDCSFC.OUT until further notice or features. - 13 - Line 12 ------- This line is the complete drive and pathname of a RAM drive that is available to ZDCS for certain types of processing work. ZDCS can make use of a RAM drive to speed up the processing of embedded archives (referred to as zipfiles within zipfiles). It can also use the RAM drive to process GIFs ("other" files) if you set config line 5 to Y in order to truncate nulls. The existence of the RAM drive is verified by ZDCS, but the amount of space available on it is not checked. If you run out of space on the RAM drive while ZDCS is processing an upload, the upload will be declined. If you do not want to use a RAM drive, leave this line blank. Line 13 ------- This line consists of the single letter Y or N. It controls whether ZDCS displays the one line "Registered to..." message after the board receives an upload (Y) or turns off the display of this message (N). Either way, the caller still sees the file by file breakdown of the upload and the status (duplicate, bbs ads, etc.) of each file. This feature is only recognized by the registered version of ZDCS. It has no effect on the three line message displayed by the unregistered version. It is also the only line of the configuration file that you can forget to include without causing major problems. If the line is missing, ZDCS defaults to (Y) and displays the three line message. This would probably be a fine time to wax poetical about the advantages of registration. Instead, we'll just direct you to the section in this manual on registration for more information about that, and to the section on support for some of the reasons why. (Actions speak louder than words.) THE ZDCS DATABASE BUILD ----------------------- Purpose ------- In order for ZDCS to process new uploads, it must first have created a database for the existing files to serve as a standard of comparison. This is referred to as the duplicate file database, the ZDCS database, or the CRC database. For simplicity, we will also refer to it as "the" database. Information stored in this database includes pathname information, the file name of every zipfile and GIF examined by ZDCS, and the names and CRC values of every individual file. - 14 - There is another purpose to building the ZDCS database besides enabling ZDCS to check uploads. It also serves as a tool to help you weed out duplicate files from the existing collection. Creation of the Initial Database -------------------------------- THE ZDCS database of duplicate files is created with the database build program ZDCSDB. The finished database consists of two files: ZDCS.NDX and ZDCS.DAT. A third file is also created during this process: ZDCS-DBB.LOG, which is an ASCII text file logging messages from the database creation process. There is no limit to the number of component files within a zipfile. ZDCS will "see" and process all the member files. The more files in the zipfile, however, the longer the processing of that archive will take. To create the database, you simply run ZDCSDB. This program looks for the ZDCS configuration file to tell it which files you want to process. These can be in multiple directories, on different drives, or across a network. Once you start the build process, there is nothing more to do until it is complete. Everything will run automatically with no need for additional input from you. While the program is running, there is a screen display that gives you the current status of operations and one very important piece of information: use the escape key if you wish to abort the process. Additional explanation of the screen display is covered in the next section of this manual. The Screen Display During the Database Build -------------------------------------------- The ZDCSDB status summary contains both useful and esoteric information. The most important of these is the notice that the escape key is the one to use in order to abort the database build. Aborting the process any other way will almost certainly lead to lost clusters on your hard disk, not to mention a very unhappy sysop. In general, ZDCS treats an SFX file made with either PKZIP or ARJ just like a ZIP or ARJ file, and this documentation has reflected that fact by using the term "zipfile" to refer to all of these archives. For the purposes of this section, however, we will use the individual terms SFX and ZIP, since the screen display does distinguish between them. One line in the middle of the screen will change frequently as ZDCSDB is processing files. Besides letting you know that the system is working, this line tells you something about the processing steps as they take place, just in case you were curious. - 15 - The possibilities are: Initializing. ZDCSDB is pulling its act together and gearing up for work Directory C1.. ZDCS has just begun processing a new directory Directory C2.. ZDCS has just retrieved the next ZIP / ARJ / SFX / GIF file within the same directory ZIP Directry.. ZDCS is retrieving information from within an individual zipfile SFXID......... ZDCS is testing whether the file being procesed is a self-extracting file, and if so which archive method (PKZIP or ARJ) was used to make it SFX(A)........ ZDCS is retrieving information from within an individual SFX file made with ARJ SFX(Z)........ ZDCS is retrieving information from within an individual SFX file made with PKZIP ZIZ........... ZDCS is processing a zipfile (ZIP or ARJ) within a zipfile (ZIP) or ARJ) - often called an embedded archive CRC........... ZDCS is calculating the CRC value for a GIF file Fast Forward.. ZDCS has already encountered and processed this exact same file (It takes good eyes and a slow computer to catch this indicator, which may come up when an aborted database build is resumed.) Indexing...... ZDCS is writing the index and data to the Btree files Underneath this changing line is a line that tells you which ZIP, ARJ, SFX or GIF file is currently being processed, with the full pathname included. Below that is the phrase "Share" or "NoShare" to indicate the type of "file opens" being used. "Share" capability is detected by all ZDCS programs to permit automatic use of the appropriate type of file access. Just to the right of that is the word "Archives" followed by the number of ZIPs, ARJs, SFXs and GIFs that have been processed so far. And to the right of that (no political statements implied) is the word "Members" followed by a number. That number counts all the individual files (those within the ZIP, ARJ, SFX or GIF files) that have been processed so far during this session. (Remember, a GIF shows up as having only one member file.) And all the way to the right of that same line, way out in right field, is the phrase "DB Total" followed by the number of member files that are already in the database. The next line down starts with "$:" (a dollar sign and a colon) followed by a five digit number. This is arcane internal status information that will not be explained here. (Consider yourself lucky.) The only time you are likely to need this is if there is a problem with your system during the database build and we ask you to read the screen display to us. To the right of that esoterica is the start time. The end time will display immediately to the right of that as soon as the database build is complete. If you leave the room for a peanut butter and pickle sandwich while the database is building, this provides a simple way to tell if and when the process has finished. - 16 - All the way to the right, in fact in the lower right hand corner of the display, is the word "Stack:" followed by a four digit number. This is more arcane status information. Additions to an Existing ZDCS Database -------------------------------------- Adding new files to the existing ZDCS database is very simple. If you run the database build ZDCSDB when there already is an existing ZDCS database, the records for the new files will be added to those already in the database. (If you temporarily rename or move the database files mentioned in the database creation section, then ZDCS will not recognize them until you restore them.) This provides a very simple method for adding large or small file collections to an existing database. All you have to do is edit the first line of the configuration file to point to the new files, and you're ready to go. Issue the command ZDCSDB and from here on the process will proceed just the same way as it does for creating the initial database. There is a second way to add files to the ZDCS database. Available free on The Hacker Central BBS is a merge utility that lets you combine two ZDCS databases into one. This allows you to create smaller databases at a time and then merge them together, which can be an advantage if you have a very large collection to process. The main advantage of the database merge option is that it allows sysops to use pre-built ZDCS databases for common file collections, like the popular CD-ROMs of shareware programs. Pre-built databases are also available on The Hacker Central BBS and are mentioned in more detail in the section in this manual on support. We recommend that you copy the necessary files over to a temporary working directory and perform database merging there. This has two advantages: it minimizes the chances that you'll get confused and rearrange part of your "live" system; and it doesn't interfere with the operation of the bbs. Pick your fastest available processor to speed up the operation, and make sure you have enough disk space for the new files, too. The merge utility ZDCSDBMG.EXE, available in Z2-DBMG.ZIP on The Hacker Central BBS, comes with a text file explaining step by step how to perform a database merge using a temporary working directory. The same kind of instructions are also included with every pre-built CD- ROM database available on The Hacker Central BBS. ZDCSDB Without a Separate File Integrity Checker ------------------------------------------------ ZDCS is not a file integrity checker and does not intend to replace the fine checkers that are already available. Normal operation of ZDCS assumes that you have already processed your files with a file integrity checker. - 17 - Human nature and circumstances being what they are, a special switch has been included with the database build module to permit some simple file integrity checking to be done. This method has ZDCS call in PKZIP or ARJ, whichever is appropriate, to test the zipfiles. There is no checking performed on GIFs at all. To use this feature, you issue the database build command as ZDCSDB T instead of ZDCSDB. Because ZDCS now calls PKZIP or ARJ for the file integrity checking in addition to the usual functions of the database build, the processing time is greatly increased. If you are only processing a small collection of files, such as a batch of local uploads, the difference in time is probably unimportant in practice. But if you are processing a large set of files, you'll be better off making your first pass with a file integrity checker and then using the regular ZDCSDB command to build the database. If you run ZDCSDB on a collection of files that includes a corrupt zipfile, the database build may crash. While you can find out after the fact about the damaged zipfile by reading the log, it's still far preferable to avoid the problem entirely by using file integrity checking first. The Database Build Log File ZDCS-DBB.LOG ---------------------------------------- There is a log file called ZDCS-DBB.LOG created by the database build operation. This is an ASCII text file that logs messages from the database creation process. If ZDCSDB encountered damaged zipfiles while trying to build the database, there would be messages alerting you to that fact in the log file. If you have any problems while running ZDCSDB, look in this log file for help in understanding what happened. THE ZDCS DUPLICATE REPORT GENERATOR ----------------------------------- The primary purpose of the duplicate report generator ZDCSDR.EXE is to use the information in your ZDCS database to give you a report of all duplicate files in your file collection. The usual time to do this is after you have created the initial ZDCS database. It is entirely likely that when you first create the initial database you will already have some duplicate files in your collection of zipfiles and GIFs. To find out about them you need to run ZDCSDR.EXE. First, make sure that your CONFIG.SYS file contains the following line: FCBS=32,32 - 18 - This is absolutely essential for the database report generator (and the merge utility). No fooling around. If you have to add this line to your CONFIG.SYS, remember to reboot afterwards so that the change takes effect. Second, switch to a directory (any directory will do) that has enough space available to handle the temporary files that will be created during the duplicate report generation. These files are normally deleted again when the report is completed, but you need to have enough room to store them while the operation is in progress. If you have less space available than ZDCSDR.EXE =estimates= it will need, you'll be given the bad news and the opportunity to cancel the report generation. Third, run ZDCSDR.EXE. This program is part of the ZDCS package and should be in your ZDCS directory with all the rest of the executables. (Your ZDCS directory should be in your path.) This will create an ASCII text file containing a three part report that is ready for viewing or printing. Part one is a list of zipfiles or GIFs that are 100% duplicates of other files in the database. Part two is a list of zipfiles that have some level of duplication, but also contain at least one non- duplicated file. Part three is the complete list of all duplicate files in the database, including the name and CRC value of the duplicated file and the identity with full drive and pathname of the zipfile or GIF containing the dupe. Note that no duplicate files are deleted by ZDCSDR.EXE when you create the initial database. The list of duplicates, ZDCS-DUP.LST, can be used by the sysop to remove any duplicate files in the system. Once you have the list of duplicates ZDCS-DUP.LST in hand, you are ready to clean out your file system. Use the information to remove any duplicate files from the file collection. On some of the larger bbs file systems, this step alone has freed up megabytes of hard drive space. After you are sure you have finished cleaning up the file system, it is possible to purge duplicate entries from the ZDCS database in order to reduce its size. The command to do this is ZDCSDR P and it should =not= be used until you have read the next section in this manual, THE ZDCS DATABASE PURGE. There are two other switches that can be used to change the operation of ZDCSDR.EXE - L and S. Issuing the command ZDCSDR L will cause the usual database report to be generated, but all those temporary work files will not be deleted. This requires a lot of extra disk space to use. The main reason for this switch is debugging purposes. Issuing the command ZDCSDR S will sort the database and regenerate the ZDCS.NDX file from the existing ZDCS.DAT file. This has a couple of uses: 1. If your ZDCS.NDX file has become corrupted or destroyed (we won't ask how, stranger things have happened on computers before), then - 19 - this feature will let you recreate the ZDCS.NDX file without having to recreate the entire ZDCS database from scratch. If you had a ZDCS.NDX file in the directory before using this feature, it will be overwritten by the fresh index file. 2. If you have obtained the *.DAT file for a ZDCS 2.0 database and you want to recreate the *.NDX file, this feature will let you do that. You'll notice that all the pre-built CD-ROM databases available on The Hacker Central BBS only have the .DAT file and no .NDX file. (The *.NDX file isn't needed for the database merge and not including it saves on disk space and transfer time.) Each of these pre-built databases comes with step by step instructions for using this feature to recreate the index file. The instructions include the creation of a temporary working directory so that you won't interfere with the "live" installation of ZDCS on your bbs. 3. If you've added many new files to your ZDCS database (either with lots of uploads, lucky you, or with database merges), then you might want to investigate this feature. Running ZDCSDR S on your system will sort your ZDCS.DAT file and will rebuild the ZDCS.NDX file to correspond to the newly reorganized ZDCS.DAT. This makes for a slightly more efficient operation. The time you regain in faster processing speed will depend on how badly out of sorts the database had become. (no pun intended, of course) We recommend running ZDCSDR S to sort the database and regenerate the index file once a year. Unless you process an unusually high number of files, there's not much need to do it more frequently than that. Certainly you should =not= put this into your daily event on the board, or it will eat up more time than you'll save. While this operation is much faster than the database creation step, it can still take from 20 minutes to 2 hours to complete (depending on the size of your system). If you run ZDCSDR S once an average year, your gain in processing speed for upload checking will probably be somewhere between 1% and 10%. If you forget which switch does what, you can always issue the command ZDCSDR ? and the question mark will tell ZDCS to list the various options for you: = Generate the duplicate report. (default choice, no command line parameters needed) P = Purge duplicate records from your ZDCS database. S = Sort and re-index the ZDCS database without purging. L = Perform the standard duplicate report without purging the database, but leave all temporary work files in place. THE ZDCS DATABASE PURGE ----------------------- Every duplicate file in your initial collection of files on the bbs will be reflected by a duplicate entry in the CRC32 database, which ZDCS uses as its standard of comparison for checking new uploads. - 20 - Only one entry for each unique file is needed in order to recognize future uploads of the same file. You can reduce the size of this ZDCS database by purging duplicate entries from it. This leaves just one single reference to each individual CRC32 in the database. Before you make any changes to the ZDCS database at all, please read the section in this manual on the duplicate report generator ZDCSDR. The reporting function of this module will supply you with an accurate list of all your duplicate files, including the necessary path information - but only if you run it =before= you purge the database. Without this list, you'll be hard pressed to find and remove the duplicate files from your system. We strongly recommend that you not only generate this report but also finish cleaning out the file system before you purge anything. When you feel you are ready to purge the database, start by making a backup copy of it first. This is very important. =Please= make a copy of the database and put it away somewhere safe before you start purging info from the original. Once you purge this information, the only way to restore it is to rebuild the database. While it is always possible to rebuild the database, this takes a fair bit of time, especially for larger systems. And what sysop has enough time? By now you are probably really ready to get on with it. To purge the database of duplicates, you use the duplicate report generator module ZDCSDR.EXE with the P (for Purge) switch. All you have to do is type ZDCSDR P and hit the enter key. The P will redirect the program to purging the duplicates instead of creating a report on them. There is nothing else to do until the purge is complete. This is not a fast process, but it will pay you back with a smaller database. BBS ADS ------- Function -------- There are some files that you really and truly don't want on your bbs at all, ever, period. These could be illegal cracks that make the rounds every now and then, or part of the endless succession of pyramid schemes and scam files, or our own favorites, those rude bbs ads some boards feel compelled to include in every one of their files. This whole group of junk files is collectively and affectionately known as bbs ads. ZDCS offers a way to recognize the reappearance of these files. ZDCS will let you target them for deletion or simply flag them for manual removal, independent of whether or not you have enabled the deletion of duplicate files. The distinction here is that the bbs ads are =known= files that you have given ZDCS specific orders to deal with. In order to recognize a file as a bbs ad, you must first tell ZDCS about it. This is done by setting up the bbs ads database. After that, every time ZDCS sees the same file again it knows that it's a bbs ad. - 21 - Since this option is completely independent of the option to delete duplicate files, you don't have to take a chance on removing authors' unchanged files from newer shareware versions. The bbs ads option does not operate on a GIF file. Creation of the BBS Ads Database -------------------------------- All you need to create the bbs ads database is the program ZDCSBA.EXE and a collection of bbs ads, pyramid schemes, chain letters or other obnoxious files. Once created, the bbs ads database will consist of a single file, ZDCS- BBA.NDX, which will be located in the ZDCS directory along with the rest of the ZDCS files. You can delete this file at any time if you want to start over again with a fresh collection of files. The easiest way to create the bbs ads database involves grouping all those nasty bbs ads together and collecting them into one zipfile. You can use either PKZIP or ARJ and select whatever name you like for the archive. Then you run the program ZDCSBA.EXE from your ZDCS directory. This is easy to see with an example. If your collection of files to be added is called BBS-ADS (.ZIP if created with PKZIP, .ARJ if made with ARJ), then the command you issue to create the bbs ads database with these files is: ZDCSBA BBS-ADS.ZIP or ZDCSBA BBS-ADS.ARJ You don't have to do anything else. The program handles everything for the creation of the bbs ads database. If you want to create a new bbs ads database in the future, just delete the old database file (ZDCS-BBA.NDX) and follow the steps for creating a new bbs ads database. If you don't delete the old database, then the new ads will be added to the old ones in the database, which is an easy way to add new bbs ads. There are many other ways to add bbs ads, which are described in the next section. Updating the BBS Ads Database ----------------------------- There are four easy ways to add new bbs ads as they are foisted off on the bbs community. 1. You can run ZDCSBA.EXE on a new collection of bbs ads when you already have an existing bbs ads database. This is good even if the archived collection only has a single file in it, and it doesn't matter whether you use PKZIP or ARJ to create that archive. ZDCSBA.EXE works just the same way here as it does for creating a new database, with one exception: the files being processed are added to the existing database. That's all there is to it. - 22 - Example. Let's say that you've collected some bbs ads from your board, your own downloads, and even from friends. You zip the entire set together and call it JUNK.ZIP. To include these files in the bbs ads database, just issue the command: ZDCSBA JUNK.ZIP 2. You can run ZDCSBA.EXE on an individual bbs ad without bothering to archive it first. This is a quick way to catch a new bbs ad as soon as you are lucky enough to get it. Example. You've just come across an obnoxious get-rich-quick scheme called MOOLAH.TXT. To put this file in the bbs ads database, just issue the command: ZDCSBA MOOLAH.TXT 3. You can even run ZDCSBA to include a file in the bbs ads database when you don't have the original file, as long as you have the CRC32 value for it. Just run ZDCSBA on the CRC32 value immediately preceded by a dollar sign $, instead of on the file itself. Example. You don't have the original ad anymore, but you have a record that the CRC32 of a new bbs ad is 12345678. To put this file in the bbs ads database, just issue the command: ZDCSBA $12345678 4. You can also run ZDCSBA.EXE on an ASCII list of CRC values. This feature was added because lists of the CRC values for bbs ads have now been assembled by some sysops and have started appearing on boards like Salt Air. Instead of processing each CRC value individually, you can process the whole list at once. In the example immediately above, you would replace the dollar sign by the at symbol @ followed immediately by the complete filename (with path if not located in the current directory) of the list of CRC values. Example. You just picked up a list of CRC32s for a new crop of bbs ads. The filename of the list is MORE-SOM.TXT. To add all the CRC values in this file to the bbs ads database, just issue the command: ZDCSBA @MORE-SOM.TXT You can use any or all of these methods to add new records to the bbs ads database. If you have an assortment of ZIP and ARJ collections, a *.COM program that generates a flashing ANSI bbs ad, a couple of plain text chain letters, a few files known only by CRC32, and a list of pyramid scam file CRCs, the essential data can all be entered into the same database without having to use the same technique for each case. In fact, there's even a way to dump the contents of the bbs ads database to a flat ASCII file. The result is a text file with one CRC value on each line, the same format as the input file for method - 23 - number four above. This means that you can "spit back" the information in your bbs ads database in a format that another ZDCS user can easily incorporate into his or her bbs ads database. The command to regurgitate these CRC values is ZDCSBA #. Example. You'd like to share your accumulated bbs ads information with another sysop, and you want to call the file CRC-DUMP.TXT. Just issue the command: ZDCSBA #CRC-DUMP.TXT The variety of ways you can add new bbs ads to the database makes it easy to share information with other sysops and to catch new annoyances as they are let loose on us. Selecting Deletion or Flagging of BBS Ads ----------------------------------------- If you have built the bbs ads database, ZDCS will automatically detect its presence and will compare uploads against the records in the bbs ads database. What ZDCS does with any matches it finds depends on how you set up the configuration file. In line 8 of the ZDCS configuration file ZDCS.CFG, you typed one of two letters: Y or N. The letter Y tells ZDCS to delete the offending bbs ad. The letter N tells ZDCS not to delete it, but just to flag it as a bbs ad. When a bbs ad in an upload is merely flagged, that information is still displayed to the caller and recorded in the log for the upload file checker. If you wish, you can go back and manually delete such files later on. Whether or not the bbs ad is flagged, as soon as it is identified as a bbs ad it is not counted in the calculation for the percentage of dupes in the upload. Once you know it's a bbs ad, you really don't care if it's a duplicate of one that's managed to get into your database. By definition, a bbs ad is something you don't want and would rather not have. It is possible for this option to disturb an AV stamp on an uploaded ZIP file, but only under the following rather narrow set of circumstances: 1. The upload was created with PKZIP and contains an AV stamp =and= 2. one or more of the files in the ZIP is flagged as a bbs ad =and= 3. the bbs ad file is located =inside= the AV stamp =and= 4. you have enabled deletion of bbs ads. If any of these four steps are missing, then there is no disturbing of an AV stamp as a result of bbs ad processing. You can change your mind about using deletion or flagging to deal with bbs ads at any time. All you have to do is edit line 8 in the - 24 - configuration file. No other changes need to be made to run with the new option. ALLOWED DUPLICATES ------------------ Allowed dupes are a special case. They are not flagged as dupes when they are encountered by ZDCS; they are also not considered "original" or non-dupe files. Whether or not these files are already present on your system, you specifically don't want ZDCS to treat any future occurrences of them as dupes. By identifying these files as allowed duplicates, you are protecting them from deletion, even if you have configured ZDCS to delete duplicate files. Furthermore, when ZDCS evaluates the actual percentage of duplicates in an upload, all files that are allowed duplicates are not counted. They are transparent in any calculations, completely and utterly ignored. They aren't even counted when reporting the number of files in the archive. The reason behind the concept of allowed duplicates is that there are some files that reappear frequently as critical components of more than one major shareware package or upgrade version. These might be order forms, registration guides, general text files, small utilities or other files. Two common examples are the OMBUDSMN.ASP file found in all ASP-ware and the VALIDATE.COM program included in every revision of McAfee's SCAN package. Both files turn up unchanged in many zipfiles. It would be misleading to treat these files as ordinary duplicates. There are two functional advantages to using allowed duplicates. First, by not counting allowed duplicates in the actual percentage of dupes calculation for new uploads, your system does not "penalize" a zipfile for having one or more of these files when it comes to deciding whether to accept or decline the file. This is especially significant if you have set a pretty high standard for the "uniqueness" of uploads that your bbs will accept. This is easier to see with a real example. Example. Suppose a zipfile had a total of 24 files. Four of those were allowed dupes and eight were regular dupes. ZDCS would calculate the following information: 20 files in the archive 8 dupes 40% dupes The number of files in the archive would be 20 and not 24 because the four allowed dupes are excluded from all the calculations. The percentage of dupes would be 40% (100% * 8/20). If those allowed duplicates had been counted in as dupes, then the percentage of dupes would have been reported as 50% instead of 40%. - 25 - (The total number of files would have been 24 instead of 20 and the number of dupes would have risen to 12.) If you had set the maximum percentage of dupes for your system to 45%, then counting in the allowed duplicates as dupes would have caused this upload to be declined. (That maximum percentage is set in line 6 of the configuration file and can be changed without needing to reinstall ZDCS.) The second advantage is for those systems that configure ZDCS to delete duplicate files. The use of allowed duplicates can protect popular shareware packages from being stripped of important files. Not only does the deletion of such files alter the content of the author's shareware package, but it also destroys any AV stamp the author placed on the work. Whether or not the option to recognize allowed dupes is enabled is controlled by the presence or absence of an ASCII text file called ZDCS.ADN in the ZDCS directory on your system. This file is one that you create with any text editor to list all the files, one per line, that are allowed duplicates. You can designate an allowed duplicate by either its file name or its CRC value. You can even mix the two styles in the same file if you want to specify some files by name (like VALIDATE.DOC) and others by CRC value. To specify an allowed duplicate by name, just type the dollar sign $ followed immediately by the name of the file with its extension. Don't leave any blank spaces in between. This preserves a file with a distinctive name (like OMBUDSMN.ASP) even if it undergoes some revisions. To specify an allowed file by its CRC32 value, type the pound sign # followed by the CRC32 for the file. Again, don't leave any blank spaces in between. If you don't know the CRC32 offhand, you can get this information from PKZIP or ARJ. For an archive made by PKZIP, use the command PKZIP -V to give you the CRC values for each individual file inside the zipfile. For an archive made by ARJ, the corresponding command is ARJ L The ZDCS.ADN file may be up to 256 lines long, but must contain no blank lines and no blank spaces. You can edit, delete or recreate this file at any time without having to set any other switches, alter the configuration file, or make any other changes to the setup of ZDCS. While the expected application of the allowed duplicates concept is with zipfiles, the feature does work for GIF files too. If a GIF is included among the allowed duplicates, then a repeat upload of the same GIF will not be flagged as a duplicate, and the upload will be accepted. - 26 - THE UPLOAD FILE CHECKER ----------------------- Purpose ------- Once you have created the initial ZDCS duplicate file database, you can get the bbs to check all uploaded zipfiles and GIFs against it from then on. This is done by processing the uploaded zipfiles and GIFs with the upload file checker ZDCSFC.EXE as the files are received. The primary purpose is to catch uploaded files that duplicate ones already on the bbs. ZDCSFC.EXE can also flag bbs ads, as described in that section of this manual. Depending on whether you have enabled any deletion options in the configuration file, ZDCSFC.EXE can also perform deletions of duplicate files and / or bbs ads. With the use of a list of allowed duplicates, also described in a separate section in this manual, the file checking performed by ZDCSFC.EXE can be made even more sophisticated. Function -------- ZDCSFC.EXE works with the CRC value for each file in an upload. If the uploaded file is a GIF, it first calculates the CRC value for that file. If the uploaded file is a zipfile, ZDCSFC.EXE reads the CRC value for each individual file in the zipfile. Then ZDCSFC.EXE compares the CRC for each file in the upload against the ZDCS database. If you have created the bbs ads database, which you would have left in the ZDCS directory on your system, ZDCSFC.EXE also compares the files in the upload against the bbs ads database. And if you have created the allowed duplicates list, also located in the ZDCS directory, ZDSCFC.EXE will check the upload against that, too. The results are displayed file by file for the caller after the upload has been received and processed, providing real time feedback. The same information is also available to the sysop on screen and as part of the ZDCSFC.EXE log file. The complete path and file name for this log were specified in line 7 of the configuration file. ZDCSFC.EXE also calculates the actual percentage of duplicates in any upload and compares this against the maximum percentage you have set for the board. On the basis of this comparison, an upload may be accepted or declined. Declined uploads are not deleted from the system. PCBoard keeps a declined upload in your private directory for sysop review. The calculation of percent duplicates is covered in more detail in the next section of this manual. - 27 - Maximum and Actual Percent Dupes in an Upload --------------------------------------------- ZDCSFC.EXE calculates the actual percentage of duplicate files in the upload. Since a GIF is a single file, it will either be 0% (not a dupe) or 100% (a dupe). For zipfiles, this actual percentage can vary anywhere between 0 and 100. ZDCSFC.EXE compares this actual percentage against the maximum percentage you set in line 6 of the configuration file. If the actual percentage is lower than the maximum, the upload is accepted. If the actual percentage is equal to or higher than the maximum you specified, the upload is declined. PCBoard keeps these declined files in your private upload directory, where you can review them. Some callers have become frantic at the idea that a "declined" file is thrown out. This does =not= happen with ZDCS. If you want to bypass this filter, set the percentage to 100. This permits a duplicated GIF or a zipfile with nothing but duplicates to pass the filter and never be declined. At the other extreme, you can set the percentage to 0. This effectively requires that the uploaded GIFs and zipfiles have no duplicates at all, or they will be declined. When you are deciding how high to set the maximum percentage of dupes, it helps to understand how the dupe calculation works when a file is uploaded. If any allowed duplicates are found, they are excluded from =all= calculations. Once it identifies an allowed duplicate, ZDCS will not figure it into the math, not even into the total number of files. Allowed duplicates are explained in more detail in an earlier section of this manual. If any bbs ads are found, they are not part of the calculation of original and duplicate files. You already know you don't want any file that you've specified as a bbs ad, so it doesn't matter to you whether this one is "original" on your system or one that's found its way into the ZDCS database already. What does matter is that the file is unwelcome either way, and so it isn't figured into this calculation. To see how this works out in real life, let's look at two examples. Example 1. ICONS-HO!.ZIP is uploaded to your system. There are 61 files in the archive. Of these, four are in your bbs ads database (the file got around) and one is in the allowed duplicates list. Of the remaining 56 files (61-4-1=56), fourteen are already present on your system. ZDCS will figure the results like this: 60 member files (61 minus the one allowed duplicate) 4 bbs ads 14 dupes 25% percent dupes (14 dupes divided by 56 files, not by 60 or 61) - 28 - Example 2. CORNDOG.ARJ is uploaded to your system. There are 14 files in the archive. Of these, one is a bbs ad and five are allowed duplicates. Of the remaining 8 files (14-1-5=8), all eight are dupes. Here's how ZDCS would figure the numbers: 9 member files (14 minus the 5 allowed duplicates) 1 bbs ad 8 dupes 100% percent dupes (8 dupes divided by 8 files, not by 9 or 14) The configuration file section in this manual points out that the maximum percentage of dupes in line 6 must be a whole number, not a decimal. This is because ZDCS deliberately truncates any decimal from this value. ZDCS does not round up or down but performs a straightforward truncation on both the actual and the maximum percentage of dupes. This was done for reasons of memory management. An example may make it clearer how this approach works in actual practice. Suppose that the maximum percentage of duplicates is set to 20. An upload with an actual percentage of 19.9 would be truncated down to 19. ZDCS would see that 19 is less than 20 and would accept this upload. If the actual percentage were rounded instead of truncated, it would be 20 instead of 19. ZDCS would then decline this upload because the actual percentage would not be less than the maximum percentage of dupes. Updating ZDCS Database(s) After an Upload ----------------------------------------- ZDCSFC.EXE automatically updates the duplicate files database with the CRC values of all uploads. You do not have to do anything special to include this information in the database for comparison with future uploads. ZDCSFC.EXE does not modify the bbs ads database at all. It's still not smart enough to recognize a bbs ad until you've pointed it out first - but it does remember them next time it sees them. Please see the section in this manual on the bbs ads database if you want to know more about how the bbs ads database can be updated. DOS Error Levels ---------------- In addition to creating the required PCBPASS.TXT and PCBFAIL.TXT files, ZDCSFC.EXE also sets the DOS error level when it exits. These levels are: 0 No duplicate files were found within the upload. 1 Some duplicates were found, but the upload passed the percentage test. - 29 - 2 Too many duplicates were found, and the upload failed the percentage test. 3 Every file within the upload was a duplicate. 4 There is still no number 4. (Reserved for future expansion.) 5 Upload checking by ZDCSFC was aborted. Please see log file for error message. CALLING THE UPLOAD FILE CHECKER ------------------------------- ZDCS is not a replacement for using a file integrity checker on your uploads. You should always use some kind of integrity testing on files before they are added to the system. ZDCS is called after that integrity checking is performed, so how you call ZDCS may be affected by which integrity checker you are using. Calling ZDCSFC.EXE from EXZTEST ----------------------------- If you have decided to use EXZTEST as your file integrity checker, please stop by The Hacker Central BBS and pick up a copy of EXZTEST version 2.23 beta 7MWC (or higher). This is a freeware version of the program that contains some bug fixes not available in any earlier versions. EXZTEST version 2.23 beta 7MWC provides a seamless integration with ZDCS that takes care of calling the ZDCSFC.EXE program, feeding it the information it needs, and completing any file deletions that you have enabled. This is not necessarily true of earlier versions of EXZTEST. A sample PCBTEST.BAT file that calls the version 2.23 of EXZTEST is included with this package. For more information on the ZDCS integration implemented in EXZTEST 2.23, please see the documentation for EXZTEST. More detailed information can be found in this manual in the section on calling ZDCSFC.EXE from the PCBTEST.BAT file. This section is more general and addresses the use of any file integrity checker. Calling ZDCSFC.EXE from the PCBTEST.BAT File -------------------------------------------- ZDCS should always be used in conjunction with a file integrity checker. When processing new uploads, the file integrity checker should be called from the PCBTEST.BAT file before ZDCSFC.EXE. If you are using the PCBDescribe feature in PCBoard 14.5a, that command should be used first of all. To use ZDCS with the file integrity checker of your choice (even if it's just PKZIP -T), there are four basic pieces you need to include in the PCBTEST.BAT file. - 30 - 1. Include the following three lines to clean out old copies of these files left over from processing other uploads: @IF EXIST PCBFAIL.TXT DEL PCBFAIL.TXT @IF EXIST PCBPASS.TXT DEL PCBPASS.TXT @IF EXIST ZDCS-DEL.LST DEL ZDCS-DEL.LST PCBFAIL.TXT contains information displayed to the caller when an upload is declined. PCBPASS.TXT contains the information displayed when the upload is accepted. ZDCS-DEL.LST is a list of files targeted by ZDCS for deletion from an upload. These might be duplicate files, bbs ads, or both. If you haven't enabled any deletion options in the configuration file, or if there were no files meeting the criteria for deletion, ZDCS would not generate the ZDCS-DEL.LST file for that upload. By deleting the files here, you make sure that no old deletion instructions are hanging around when a new file is processed. 2. Call your file integrity tester to process the upload. The exact command to do this will depend on which integrity checker you are using. The most basic and rudimentary one of all is to make use of this ability in your chosen archiver. In PKZIP you would use the -T switch and in ARJ you would use the T switch. Or you can use a file integrity tester designed to check for file damage, viruses, and much more. 3. Call ZDCSFC.EXE to check the upload. The appropriate command is: ZDCSFC %1 %2 %3 If you are running a version of PCBoard older than the May 22, 1991 version 14.5a, PCBoard will provide only two parameters. Since the third one is absolutely necessary for pre-testing, that means you must disable the pre-testing feature. If you are using a newer version of PCBoard but you don't want to use the pre-testing right now, it is still strongly recommended that you leave the third parameter in place. It does no harm and could save you some grief if you change your mind in the future. At the end of processing by ZDCSFC.EXE, you might have a new file called ZDCS-DEL.LST. This depends on whether you have enabled any deletions and on whether there were any files in the upload that met your criteria for deletion. If no files are to be deleted, this control file won't be created. 4. Perform the deletion of files specified by ZDCSFC.EXE. IF EXIST ZDCS-DEL.LST PKZIP -D %1 @ZDCS-DEL.LST IF EXIST ZDCS-DEL.LST ARJ D %1 !ZDCS-DEL.LST Both the PKZIP-specific and ARJ-specific commands can be included. The system will return an error message when the "wrong" one is encountered, but this error message will not hang the system or cause any problems with the processing. - 31 - Note that the actual deletion is done by PKZIP or ARJ. ZDCSFC.EXE creates the control file ZDCS-DEL.LST to specify the deletions and PCBTEST.BAT passes this file to PKZIP or ARJ at this point. This is also the only time an existing PKZIP AV stamp or an ARJ security envelope on an upload is affected by ZDCS. More information on these is available in two separate sections of this manual: PKZIP AV STAMP INTEGRITY and ARJ SECURITY ENVELOPE INTEGRITY. The %1 %2 %3 Parameters ----------------------- These three variables %1, %2 and %3, are parameters used by PCBoard. They are passed to ZDCS when the upload checking program is being used. That's the only module of ZDCS that actually runs from inside PCBoard, and it's the only one where these parameters are important. The first parameter, %1, can be one of two words in PCBoard: UPLOAD or TEST. For operation of ZDCS, this variable is set to UPLOAD, by the very fact that a caller has just uploaded either a file for the board, or the test file ZDCSTEST.CHK. If ZDCS should ever be passed the word TEST here instead, it will quietly and neatly halt its processing and control will pass on. The second parameter, %2, is the name of the file that needs to be processed by the upload checker. Under normal circumstances this is the name of the file the caller just uploaded. With a few file integrity checkers, the information in this parameter becomes invalid after their processing because they have moved or modified the uploaded file right out from under this parameter. In that case, the FIXPATH.EXE utility will smooth over things. (In the SUPPORT section of this manual there is a list of goodies, including this utility, that are available free on The Hacker Central BBS.) The third parameter, %3, is the name of the file that is holding the description of the upload. When callers upload files to a PCBoard BBS, they are asked for the file name and for a description of the file. The description is held in a separate text file which is then appended to the file directory with the upload file name and other information. If you are running a version of PCBoard older than the May 22, 1991 version 14.5a, PCBoard will provide only two parameters. Since there are two features that require the use of the third parameter, you must not enable those features until you are running a more recent version of the board code that supplies all three parameters. The two features in question are upload pre-testing and the use of the disposition line (set in line 4 of the configuration file). PROCESSING LOCAL UPLOADS ------------------------ There probably isn't a sysop alive who hasn't scavenged new files for the board and uploaded them locally. ZDCS offers two ways of - 32 - processing these local uploads. The first method is to use the database builder ZDCSDB.EXE; the second method is to use the upload file checker ZDCSFC.EXE that processes all the regular uploads from callers. In the first case, ZDCSDB.EXE can be used to add all the new files to the database. Just like the initial creation of the ZDCS database or subsequent database merges, ZDCSDB.EXE adds the CRC values of the new files to the database. The duplicate report generator ZDCSDR.EXE can still be used to tell you about duplicates in the database. A disadvantage to this method is that it does not give the sysop the same kind of immediate feedback for a local upload as the sysop and caller get for a regular upload. It also does not delete bbs ads or duplicate files (if you have those options enabled). To use ZDCSDB.EXE to process local uploads, you only have to change the first line of the ZDCS configuration file. This line points to the paths of the files to be processed when running the database builder. (The line contains the full path / name of the text file that lists all the directories whose files are to be processed.) Change this to point to the uploads directory, a holding directory, or whichever directory will hold the local uploads to be processed. Now just run ZDCSDB.EXE and the files will be added to the existing database. This method does not interfere with the realtime upload checking performed by ZDCSFC on regular uploads, because the path information in line 1 of the configuration file is not used by ZDCSFC.EXE. If you've decided on a permanent holding directory, you won't even have to touch the configuration file again to process any future uploads. Just remember to run ZDCSDB.EXE before posting new files on the board. This is a good time to repeat one note of caution: ZDCS does not do any integrity testing. ZDCS is intended to work with your choice of integrity tester, not to replace it. If you have not used an integrity checker on the local upload files (Why not?), then you need to use the T (for Test) switch. Instead of using the command ZDCSDB to run the database build, use the ZDCSDB T command. This calls either PKZIP or ARJ, whichever is appropriate, to do an integrity check on the zipfiles. Any file that fails this test is not processed by the database builder. There will be a message in the database building log file ZDCS-DBB.LOG for each such damaged file. Don't use the T switch without reason - it adds another step and slows the processing down tremendously. But if you have a small collection of files that haven't passed through an integrity checker, like prospective local uploads, ZDCSDB T is one way to deal with them. If ZDCSDB.EXE has to skip a file because it's flagged as damaged, that information will show up in ZDCS-DBB.LOG. The second method mimics regular uploading to get the local uploads processed by the upload file checker ZDCSFC.EXE. This is handled via a small utility and batch file combination available free on The Hacker Central BBS as LOCALUP.ZIP. You'll have to modify the batch file for your own system, of course. - 33 - The result is that the local uploads will see the same processing that a regular caller's upload does on your board - including your usual file integrity checker, bbs ad deletion (if enabled), duplicate file deletion (if enabled), and all the rest. This has the additional advantage of giving you the ZDCSFC.EXE log file entries for the local uploads, which gives the status of the files inside the zipfiles: duplicates, bbs ads, allowed duplicates, and so on. PRE-TESTING ----------- Callers can pre-test an upload to find out how it compares to files already on the bbs in terms of duplicates and bbs ads. The idea is to see how the prospective upload compares to existing files in terms of duplicates, bbs ads, etc. before actually uploading the full file to the bbs. It can save both caller and sysop a certain amount of frustration. The procedure is wonderfully simple from a caller's point of view and requires nothing that might tax a relatively novice uploader's skills. There are no special files to download, no complicated operations to get right, no arcane rituals to perform. This makes it more likely that callers will take advantage of the pre-test feature. At the time of this release, ARJ is still in a beta cycle. The pre- testing feature has been implemented for PKZIP only, and a maintenance release of ZDCS is planned to incorporate an ARJ pre-testing feature after the new production version of ARJ is released. Included in this the ZDCS 2.0 package is a sample bulletin that can be posted by the sysop on the bbs to explain to callers the ZDCS pre- testing feature for ZIP files. The bulletin holds the caller's hands, figuratively speaking, through the whole process. If your bbs permits the uploading of SFX files, you might want to add those initials where you see ZIP in the bulletin. The pre-test can recognize both ZIP files and self-extractive archives made with PKZIP. A ZIP-only bbs can use the canned bulletin right from the package. To pre-test an upload called GOODSTUF.ZIP, the caller simply issues the DOS command: PKZIP -V GOODSTUF.ZIP > ZDCSTEST.CHK This command is not case sensitive, so it's pretty hard to make a mistake with it. The result of this operation is a new file called ZDCSTEST.CHK. The caller then uploads the file ZDCSTEST.CHK directly to the bbs. ZDCS will read the information in it and will give the caller a breakdown of the zipfile's contents, noting which individual files are duplicates or bbs ads. This output looks just like the response a caller gets after actually uploading the zipfile, with one exception: - 34 - instead of telling the caller whether the file is accepted or declined, ZDCS tells the caller what percentage of files in the upload are duplicates, and also gives the maximum percent duplicates which the sysop has allowed for the board. Now it's up to the caller to decide whether to upload or not. Enabling the pre-test capability is not tough on the sysop, either. The key to whether or not pre-testing is permitted on your bbs lies in the UPSEC file. If you want to prevent callers from using pre- testing, just disallow uploads that have the .CHK extension. On the other hand, if you want callers to have the option to pre-test, make sure that .CHK files may be uploaded to the system. There's one more thing to check before letting your callers use the pre-testing feature. If you are not using EXZTEST version 2.23 or higher, remember to make sure that the PCBTEST.BAT file contains the following line: ZDCSFC %1 %2 %3 This is already discussed elsewhere in this manual as part of the ZDCS installation. That third parameter is necessary to let the pre- testing work for more than just the first caller. If you leave it out, the first ZDCSTEST.CHK file uploaded to your system will work just fine, but all subsequent attempts to pre-test will be told that the ZDCSTEST.CHK file is a duplicate! And that means they won't be recognized as attempts at pre-testing. If you are using EXZTEST version 2.23 and up as your file integrity checker, you won't have a ZDCS line in the PCBTEST.BAT file. That's because EXZTEST has a seamless interface to ZDCS and calls it directly. Since this third parameter is critical to the proper functioning of the pre-test option, you must have a recent enough version of PCBoard that supports the third parameter. The earliest version of the board code that satisfies this requirement is PCBoard 14.5a from May 22, 1991. This version and anything after it can handle ZDCS pre-testing. If you track your callers' uploads and downloads, you might like to know that the ZDCSTEST.CHK will not count as an upload. According to ZDCS and PCBoard, the ZDCSTEST.CHK file always "fails" validation and is not credited to the caller as an upload. The bulletin does mention that pre-testing doesn't count as an upload, just in case some of your callers are too enthusiastic about getting upload credits. ZIPFILES WITHIN ZIPFILES ------------------------ Processing of zipfiles contained within zipfiles is accomplished with some caveats. Zipfiles within a zipfile are only checked one level deep. Remember that we are using the word zipfile to refer to an archive made with PKZIP or ARJ. In fact, you can even have an ARJ within a ZIP or the other way around and ZDCS can still handle it. The simplest explanation is to look at an example. - 35 - Zipfile A contains assorted files and zipfile B. In turn, zipfile B contains more files and another zipfile, C. Zipfile C contains still more files. How does the whole melange get processed? All the individual files in zipfile A and in zipfile B have their CRC32 values entered into the duplicate files database. If you have configured ZDCS to delete any files (dupes or bbs ads), then those deletions are done automatically only for the individual member files inside zipfile A. No files are deleted from inside zipfile B. Of course, all the duplicates in zipfile B are still listed in the log, so you do know about them and you can decide whether to remove them manually from your file system. What about zipfile C? That's easy: any zipfile embedded more than one level deep in the uploaded zipfile (and C is two levels deep) is not processed as a zipfile at all. No file deletions, no CRCs, nothing. In fact, *any* file that has either the .ZIP or .ARJ extension will not be treated as an individual file in the database or the log file. The reason is simple: it's the individual files inside the zipfiles and not the zipfile itself that are important when looking at duplicates. (Besides, it plays havoc with the percentages.) CD-ROMS AND DATABASE MERGING ---------------------------- CD-ROMs have been growing in popularity among bbs's because they enable a large, static collection of files (such as shareware programs) to be available for downloading. ZDCS works just fine with CD-ROM drives on a bbs, but the sheer size of these disks means that it can take quite a while to build the ZDCS database for a CD-ROM. To make life easier, you can use the database merge utility to combine two or more databases into one integrated ZDCS database. By merging databases, it is possible to take advantage of the existence of pre- built ZDCS databases for the more popular CD-ROMS. This lets you merge a pre-built database with your own bbs's ZDCS database in a fraction of the time it would take to add all the CD-ROM information from scratch. The ZDCS database merge utility ZDCSDBMG.EXE is available on The Hacker Central BBS for downloading as Z2-DBMG.ZIP. Included in that package are clear, detailed directions for merging two databases. You will need the utility program ZDCSDBMG.EXE and the *.DAT files from the two ZDCS 2.0 databases you want to merge together. All you really need to do is: 1. Put copies of all three files into a temporary working directory. 2. Make sure one of the *.DAT files is specifically named ZDCS.DAT (doesn't matter which one). The other one can by ANYNAME.DAT. 3. Check your CONFIG.SYS file and make sure that you have FCBS=32,32. - 36 - 4. Run the merge program on the new database by typing ZDCSDBMG That's it. Out the other end of the process you'll get your complete new database, N-ZDCS.DAT and N-ZDCS.NDX. The files are ready to be renamed and used. The improved database merge program ZDCSDBMG.EXE eliminates the need for the *.NDX files when you merge a database. The new N-ZDCS.NDX file is generated right from the newly merged database. This means that the pre-built databases available on The Hacker Central can now be smaller, which cuts down on storage requirements and transfer time, saving us all a little money. Prebuilt ZDCS 2.0 databases for many CD-ROMs are available free on The Hacker Central BBS. If you have a CD-ROM that doesn't already have a prebuilt ZDCS database, please consider creating a ZDCS database for the individual CD-ROM and sharing it with other ZDCS users by uploading it to The Hacker Central. Through the end of 1993, we are offering a free ZDCS registration as a thank you to anyone who uploads such a database. You'll find more about our two free registration offers in the accompanying text file ZDCS-CD.TXT. PKZIP AV STAMP INTEGRITY ------------------------ Registered versions of PKZIP can "brand" a ZIP with the unique AV (authenticity verification) number of the registered owner. The AV stamp is used by some shareware authors for their official distribution packages. It is also abused by some people, including a few bbs sysops, sad to say. Whether ZDCS will always leave that AV stamp intact is determined by the configuration of ZDCS. The first possibility is that ZDCS is configured never to delete any files, whether duplicate files or bbs ads. In this case, the original AV stamp is always retained. The second possibility is that ZDCS is configured to delete files, but no files need to be deleted from a particular zipfile. In that case, any AV on the original zipfile is left intact. The third possibility is that ZDCS deletes one or more bbs ads. Whether the AV stamp is left intact depends on whether the bbs ad was included in the zipfile inside or outside the AV branding. If the offending bbs just added their advertising to the outside of an AV stamped zipfile and left the original author's AV stamp in place, ZDCS will do the same. If that bbs added their overhead file and then rezipped the entire package with their own AV stamp (don't laugh, there's at least one large bbs that's done this), then they have destroyed any original AV stamp. ZDCS does not preserve the AV stamp in this case. This is in keeping with the ideal that the stamp should represent an authenticity verification from the =original= author. - 37 - The fourth possibility is that ZDCS deletes one or more duplicate files. If any deleted file was included in the AV branding, then the AV stamp is not preserved. If all deleted files were outside the original AV stamp, then the AV stamp is intact. (This could happen with some advertising file that you've already been hit with but which doesn't appear in your bbs ads database.) This last possibility deserves a little thought from the shareware author's point of view. It is not uncommon for a new version to contain some files that are unchanged from the previous version. If these files haven't been flagged as allowed duplicates, it is possible to choose to configure ZDCS so that these files are removed. If that happens, the AV stamp is also removed. ZDCS is quite capable of handling and retaining all AV stamps. This is a matter of choice on the part of the sysop and is the direct result of the simple configuration options. In order to disturb an AV stamp, the following conditions must =all= be met: 1. ZDCS must be configured to delete dupes, bbs ads, or both. 2. The uploaded file must be a ZIP file bearing an AV stamp. 3. One or more files in the ZIP must be targeted by ZDCS for deletion. 4. At least one file to be deleted is located "inside" the AV stamping. ARJ SECURITY ENVELOPE INTEGRITY ------------------------------- Registered versions of ARJ can "brand" an ARJ with a security envelope. This envelope is used by some shareware authors for their official distribution packages. It is also abused by some people, including a few bbs sysops, sad to say. Whether ZDCS will always leave that security envelope intact is determined by the configuration of ZDCS. The first possibility is that ZDCS is configured never to delete any files, whether duplicate files or bbs ads. In this case, the original security envelope is always retained. The second possibility is that ZDCS is configured to delete files, but no files need to be deleted from a particular zipfile. In that case, any security envelope around the original ARJ is left intact. The third possibility is that ZDCS deletes one or more bbs ads. Whether the security envelope is left intact depends on whether the bbs ad was included in the ARJ inside or outside the envelope. If the offending bbs just added their advertising to the outside of an enveloped ARJ and left the original author's security envelope in place, ZDCS will do the same. If that bbs added their overhead file and then rearchived the entire package with a new security envelope - 38 - (don't laugh, there's at least one large bbs that's done this, too), then they have destroyed any original envelope. ZDCS does not preserve the security envelope in this case. This is in keeping with the ideal that the envelope should represent a tamper-evident seal from the =original= author. The fourth possibility is that ZDCS deletes one or more duplicate files. If any deleted file was included in the security envelope, then the envelope is not preserved. If all deleted files were outside the original envelope, then the security envelope is intact. (This could happen with some advertising file that you've already been hit with but which doesn't appear in your bbs ads database.) This last possibility deserves a little thought from the shareware author's point of view. It is not uncommon for a new version to contain some files that are unchanged from the previous version. If these files haven't been flagged as allowed duplicates, it is possible to choose to configure ZDCS so that these files are removed. If that happens, the ARJ security envelope is also removed. ZDCS is quite capable of handling and retaining all security envelopes. This is a matter of choice on the part of the sysop and is the direct result of the simple configuration options. In order to disturb a security envelope, the following conditions must =all= be met: 1. ZDCS must be configured to delete dupes, bbs ads, or both. 2. The uploaded file must be an ARJ file having a security envelope. 3. One or more files in the ARJ must be targeted by ZDCS for deletion. 4. At least one file to be deleted is located "inside" the security envelope. ACCURACY OF THE CRC32 METHOD ---------------------------- In order to detect duplicates, some unique signature is needed for each file. While the only guaranteed truly unique signature is by definition as long and cumbersome as the file itself, there are more manageable alternatives that make use of formulas called algorithms to generate a signature. Any algorithm is a trade-off between these two factors: manageability and accuracy. The more unique signatures a method can generate, the smaller the possibility that two different files might generate the same signature. ZDCS makes use of the CRC32 used by PKZIP and ARJ. This signature has a total of eight "places", each filled by =independent= assortment with one of sixteen different values. (It doesn't matter how many places you have if they aren't each filled independently of the others.) That makes for a total of 16*16*16*16*16*16*16*16 (16 to the 8th power) unique signatures. With over four and a quarter billion possible signatures, this method has good accuracy. - 39 - It is possible to use other methods to create an even greater number of unique signatures. These longer signatures and different calculation methods carry a corresponding cost in complexity, disk space and processing speed. But no signature that is shorter than the original file can ever provide an absolute guarantee of uniqueness. When an algorithm produces the same signature for two different files, and eventually every algorithm will, there is a possibility for misidentification =if= you also have both of those files. This is because the heart of misidentification is in the comparison of signatures from different files. If you have only one of the two files generating the same signature, then the comparison step will still show it to be unique on your system. But suppose that you do beat the odds enough to have on your system both of these files that generate the same signature. What effect would that have? The second file you encounter would be flagged as a duplicate of the first one. If a caller uploads a file that contains this second, "misidentified" file, then that upload will show one more dupe. If you have set the percentage of maximum dupes very low and the upload contains few files, then this might be enough to decline the upload. If your percentage is set high and / or there are many files in the upload, this one "extra" dupe will not tip the scale in deciding whether the upload is declined or not. For example, suppose the upload contained only four files. Every one of them that was flagged as a dupe would contribute a value of 25% to the level of duplication in the file. If your maximum percentage of dupes acceptable in an upload were set to 10%, then a single misidentified dupe in this upload is enough to have ZDCS decline the file. Most sysops, however, use a higher maximum percentage. And most files (excluding GIFs) contain a higher number of files on average. So let's change the example to a maximum percentage of 50% and an upload containing 10 files. A single misidentified dupe in this upload would only contribute a value of 10% to the level of duplication in the file. That wouldn't be enough to have ZDCS decline the file. So even though there was a long-shot misidentification, the net effect is that ZDCS still accepts the file. But what about GIFs? A GIF upload is a single file to ZDCS. There are only two possible outcomes for the dupe check: 0% and 100%. To provide extra protection against misidentification in this situation, ZDCS has implemented a =second= algorithm, one completely independent of the CRC32 method, as a cross check. In order for a GIF to be flagged as a dupe, it must be identified as a duplicate first by the CRC32 method, and then again by the second, independent method. ZDCS protects against harmful consequences of misidentification by making use of a well-tested signature algorithm that generates over four and a quarter billion signatures, by implementing a percentage system that lets the sysop adjust the odds even further, by providing independent cross-checking for GIFs, and by never deleting a failed upload from your system. So even if a misidentification does occur and does cause an otherwise original upload to be declined, that upload is still available on your system in your private uploads directory for your review. - 40 - ZDCS provides you with good accuracy while still being manageable. After all, the most accurate method in the world does no good if you can't implement it, right? MEMORY USAGE ------------ The ZDCS upload file checker ZDCSFC.EXE, the bbs ads database builder ZDCSBA.EXE, and the ZDCS 1.6x to 2.0 conversion program Z1TO2.EXE each require a minimum of 384K of free memory (RAM). The database build program ZDCSDB.EXE and the duplicate report generator ZDCSDR.EXE each require a minimum of 512K of free memory (RAM). If your system has at least one megabyte of available EMS (expanded memory), ZDCS can make good use of it when running any of the following programs: ZDCSDB.EXE the database builder ZDCSDR.EXE the duplicate report generator ZDCSFC.EXE the upload file checker ZDCSDBMG.EXE the database merge utility There are no internal limits on the number of member files within a zipfile that can be processed by both the upload file checker ZDCSFC.EXE and the database creator ZDCSDB.EXE. TROUBLE-SHOOTING GUIDE ---------------------- There is no better help for solving problems than experience, especially someone else's experience with the same problems. Your system is crashing during the upload test. ----------------------------------------------- We have tracked this problem down to the following set of circumstances so far: you are using EXZTEST to call ZDCS, =and= you have QEMM installed on a 386 computer with an Adaptec SCSI device. This problem goes back to memory management. Your system is trying to juggle PCBoard, EXZTEST, ZDCS, and PKZIP or ARJ - all in memory. To add to the confusion, QEMM is trying to help out and not always managing that. PCBoard has the /SWAP parameter and ZDCS has the SuperShell. When ZDCS needs to call PKUNZIP or ARJ to deal with an embedded archive (which, by the way, is the ONLY time that ZDCS needs to call an external program), ZDCS swaps itself out of memory to EMS (if available) or to disk. Under certain very specific circumstances on certain hardware configurations, QEMM seems to have a problem with this (to put it very politely). - 41 - Right now your best solution to this is to get a copy of SHROOM (Shell Headroom). This is an excellent shareware product from Davis Augustine (Shroom Shareware) P.O. Box 610 Sausalito, CA. 94966-0610 With SHROOM on your system, you can modify the line in your PCBTEST.BAT file that calls EXZTEST to call SHROOM first. Look for this line: EXZTEST %1 %2 and change it to this: SHROOM EXZTEST %1 %2 That should take care of the memory management problem and prevent the system crashes. ZDCSDB.EXE (the database build) is crashing. --------------------------------------- There is a log file ZDCS-BBA.LOG that collects messages when ZDCSDB is used to create a new ZDCS database or to add to an existing one. Read the messages in the log file to find out why the database build program is crashing. Most of the time, the culprit is a damaged zipfile. ZDCS normally depends on the fact that most sysops use a separate integrity file checker on their boards. If a corrupted zipfile gets into the collection of files being processed into the database, it can bring things to a sudden halt. The recommended solution is to process all files with a file integrity checker first. However, there is a second alternative using just ZDCS. You can use the T (for Test) switch to tell ZDCSDB to call PKZIP or ARJ, whichever is appropriate, and perform a file check first. Only those files passed by this check are processed by the database builder. The syntax is simple: ZDCSDB T instead of ZDCSDB. The penalty in speed for doing it this way is very significant, which is why it is not the usually preferred solution. Unless you have only a small handful of files to process, it's worthwhile to use a real file integrity testing program instead. ZDCS is reporting a device I/O error (in any module). ---------------------------------------------------- So far, we have seen this happen thrice. Each time, ZDCS was running on a LAN. In all three cases, the problem was traced back to bad spots on the hard drives. The best medicine for this error message is to run Norton Disk Doctor, Spinrite, or something similar on your hard drive system. - 42 - ZDCS reports path/file access errors. ------------------------------------ One possibility is that you might have loaded SHARE high under DOS 5.0. There seems to be a problem with loading SHARE high; it's not anything specifically to do with ZDCS. Try loading SHARE low instead. Another possibility is that you haven't specified enough files in your CONFIG.SYS file. Because ZDCS makes such heavy use of disk I/O, you should make sure that at least 30 files are specified in your CONFIG.SYS (FILES=30). QEMM exception 13 errors occur when running ZDCS. ------------------------------------------------ There are four possible reasons for this error message. 1. Your QEMM command line is incorrect. The problem isn't with ZDCS itself, but ZDCS is likely to be the first program to notice this QEMM error because of the amount of data ZDCS needs to juggle. The result is that ZDCS is able to write its data someplace it shouldn't. To solve this problem, check that all areas of memory that should be excluded from QEMM are being properly excluded. Be especially careful if your system has super VGA, SCSI devices or a LAN; all of these devices usually require specific areas of memory to be excluded from remapping by QEMM. 2. You have one or more "mis-behaved" device drivers. You can check out this possibility by removing all drivers from your CONFIG.SYS file, rebooting the system, and trying ZDCS again. One example of this type of problem was with an older version of the Hyperdisk cache. 3. You believed the QEMM optimize program when it told you to get rid of DOS stack space. Put it back! QEMM is egocentric enough to believe it can handle all types of stack requirements for any application properly. Let's just say that putting back the DOS stack space solves this problem nicely. 4. You are trying to use the "stealth technology" feature of QEMM 6.x. There have been persistent reports of problems traced to interactions between stealth and various applications from other vendors. It wouldn't be at all surprising if ZDCS experienced the same trouble with stealth. This is not a bug in ZDCS. Contact Quarterdeck tech support. In the meantime, disable stealth by removing the ST:F or ST:M from the DEVICE=QEMM.SYS line in the CONFIG.SYS file. REGISTRATION ------------ ZDCS is a fully functional shareware package. There are no critical limits, crippled features or "drop dead" dates. The only difference - 43 - in the unregistered version is a brief message to sysop and caller that the version is unregistered. A customized ZDCS.KEY file is available to registered users to change the "unregistered" message to show registration in the name of or to eliminate the display of any message at all. For sysops, registration entitles the user to a license for use of the Zipfile Duplicate Checking System on one bbs, no matter how many nodes it has. For CD-ROM mastering, registration entitles the user to a license for use of one copy of the Zipfile Duplicate Checking System, no matter how many CD-ROM collections are processed. There are no additional fees based on the number of CD-ROMs, the gross revenue, the phase of the moon, or any other factors. This includes =all= future versions of ZDCS. No additional fees will be charged for registration of future versions of this product. Once you register, your keyfile works with all versions of ZDCS from version 1.6x on. Please take a look at the accompanying text file on CD-ROM uses of ZDCS. It describes special registration offers for both bbs and CD- ROM mastering operations. In brief, you can trade a ZDCS 2.0 pre- built CD-ROM database for a full registration, or you can have the registration fee waived if you include the ZDCS 2.0 pre-built database on any CD-ROM that you master. More details are given in ZDCS-CD.TXT. If you try ZDCS out on your system and decide that you want to continue using it, please register your copy by sending a check for $25.00 (US). Include the name of your bbs (up to 25 characters) as you want it to appear to your callers. An order form (ORDER.FRM) is included in this release for your convenience. We will prepare a key file for your copy of ZDCS with this information in it to replace the "unregistered version" line in the display. This keyfile will be available for download from The Hacker Central BBS. If your bbs is located in the continental United States or Canada, you may choose to take advantage of the keyfile delivery service any time during 1992 - 1993. For an additional $5.00 (US), we will prepare your keyfile and upload it to your bbs. Just set up an account in the name of ZDCS SUPPORT with the password KEYFILE and with sufficient security to upload a private file for the sysop. When you send in your registration of $25.00 plus $5.00, remember to include the name and number of the bbs. It's all spelled out on the order form. As of 1991, first time callers to The Hacker Central BBS have been able to register ZDCS online with a valid Visa or Mastercard. Look for the ZDCS script questionnaire in order to register. The keyfile delivery service is also available as part of the online registration. Processing usually takes from one to three days. If you prefer to register by check or money order, please make your registration check payable to Michael W. Cocke and mail it along with the necessary information (like a completed order form) to: Michael W. Cocke 11 Cedar Road Montville NJ 07045-9582 - 44 - Please be assured that ZDCS will continue to receive support through future revisions. For over two years now it has been in use on its home board, The Hacker Central BBS. When the programmer is also a sysop and has to live with the results of that programming work every day, you can be sure that the support will be there! SUPPORT ------- Support is a critical part of ZDCS. It's nice to have a program that's easy to install and use, but we all know that there is nothing to beat direct, involved author support of a product. We are committed to providing that kind of support for ZDCS. Many of the improvements in ZDCS 2.0 grew directly out of providing that kind of support to users of earlier versions of ZDCS. This is also true of the utilities and accessories, which you will find described a little later on in the section called GOODIES. Part of good support is also a willingness to listen to suggestions and requests. If you have any ideas for new features or ways to make existing features better, please let us hear from you. Contact Information ------------------- ZDCS product support is handled on the ZDCS home board and via echoed email conferences. Questions, comments and discussion of ZDCS are welcome in the following conferences: ZDCS-Support (on the ILink BBS network) ZDCS-Support (on the Intelec BBS network) Both conferences are of course available on the ZDCS home board, The Hacker Central BBS. For PCBoard sysops on Salt Air BBS, Mike Cocke is also available there in the ZDCS-Support conference. Additional product support in the form of downloadable files is available on The Hacker Central BBS. These are described in the next section of this manual, appropriately entitled GOODIES. First time access to The Hacker Central BBS must be done via the public node. After completing the new user and visiting sysop scripts, sysops and ZDCS users (whether or not they have registered ZDCS yet) will be upgraded in security level, usually handled in a day or two. This will give you access to the two private nodes, both of which are high speed lines. The Hacker Central phone numbers are: Node 1 201-334-2555 Public 2400 Node 2 201-316-8840 Private HST / Dual Standard Node 3 201-335-9343 Private HST / Dual Standard You can join the ZDCS support conferences on your first call. All the pre-built CD-ROM databases, utilities, and other goodies are available for download =only= from inside one of the ZDCS-Support conferences. - 45 - If you should get confused or need a reminder, there is a bulletin on the board addressed to those interested in ZDCS. You'll find it listed with the rest of the bulletins. Goodies ------- Another aspect of ZDCS product support is the collection of utilities and pre-built CD-ROM databases available for download from The Hacker Central BBS for all ZDCS users. This collection is constantly growing in response to suggestions, requests and cries for customized help. A few of the special goodies are described below. This is not necessarily the complete list of goodies. New CD-ROM databases are constantly being added, and development work on various accessories is an ongoing process. Suggestions for other new utilities are always welcome, of course. All ZDCS utilities to date are free but are only available on The Hacker Central BBS. Please do not upload these modules to other bbs's or distribute them as part of the shareware ZDCS package. Database Merge -------------- The database merge program ZDCSDBMG.EXE lets you combine more than one ZDCS database into a single database. This has the tremendous advantage of permitting pre-built ZDCS databases to be created for the more popular CD-ROMS. Once a single CD-ROM database has been created, can be merged into any bbs's ZDCS database in a fraction of the time it would take to create the CD-ROM database from scratch. The existence of the merge utility also makes it possible for you to build your own ZDCS database a section at a time if that's easier for you, and then merge the different pieces together for your complete system. It gives you a little more flexibility that way. The merge utility is available in a little package called Z2-DBMG.ZIP that includes step by step instructions so you can't get lost. Pre-Built CD-ROM Databases -------------------------- Many pre-built CD-ROM databases are available on The Hacker Central BBS. These are easy to incorporate into your ZDCS database with the database merge utility described above. If you have a CD-ROM that doesn't already have a pre-built database (either on disk or in the collection of databases on The Hacker Central BBS), then we'd like to encourage you to create the ZDCS 2.0 database for that CD-ROM and upload it to share with other sysops. In fact, we are offering a free ZDCS registration to anyone who uploads a new ZDCS 2.0 pre-built CD-ROM database to The Hacker Central BBS from now until the end of 1993. For details, check out the text file ZDCS- CD.TXT in this package. - 46 - Local Upload Processing ----------------------- A small local uploads package has been put together to make processing local uploads easier for the sysop who also scavenges files for the board. This package was revised for ZDCS 2.0 to make it easier for the sysop to use it "out of the box". Changes to the docs and the sample batch file take more of the load off the sysop when processing local uploads. The package is available for download as LOCALUP.ZIP. Database Browser ---------------- There is another small program that permits you to look through the records in a ZDCS database. This database browse utility originally started as a debugging tool and is still intended primarily for debugging purposes. The program name is ZDCSED2.EXE and it is available for download as ZDCSED2.ZIP. Fixpath ------- Some upload file integrity testers move or rename the uploaded file. There is now available a small utility called FIXPATH.EXE for use with these testers. This program acts as a small shell for ZDCS, and adjusts the %1 parameter originally passed from PCBoard to reflect whatever the file integrity tester has done. This utility is available for download as FIXPATH.ZIP. BBS Ad Tracker ------------- This is a new utility to help you track down bbs ads that might have made it into your ZDCS database before they were recognized as bbs ads. The way some bbs's change their ads frequently, it's possible to receive one of these little joys as part of an upload or even to have it in your original collection of files for some time before you discover that it's another bbs ad. The bbs ad tracker ZDCSBAT.EXE can compare your ZDCS database with your bbs ads database and let you know if any of those bbs ads appear in teh database. Armed with that report, you can hunt the ads down and exterminate them. This program is available for download as ZDCSBAT.ZIP. Conversion from ZDCS 1.6x to ZDCS 2.0 ------------------------------------- If you have already installed one of the earlier versions of ZDCS (1.6x), there is a straightforward upgrade path to the smaller and faster version 2.0. But you won't find a separate upgrade kit on The Hacker Central BBS; all you need is already included in this package. And since you are reading this file, you must have the rest of the package, too. Right? However, you =will= need to make sure that any utility programs you are using, such as the database merge, are up to date for ZDCS 2.0 and are not older version intended for ZDCS 1.6x. - 47 - Documentation ------------- A final note about ZDCS support concerns documentation. We feel that clear and helpful documentation is an important part of a sysop utility package like ZDCS. The documentation for ZDCS 2.0 has been revised to answer more questions and give more examples. Of course we've kept the friendly walk-through ZDCSWALK.TXT and the technical reference manual ZDCS-REF.TXT, just improved them a little. In addition to the plain text files included in this package, we also have the documentation available in WinWord 2.0 (also known as MS Word for Windows 2.0) with full use of styles, footers, outline processing and other features. This makes for some really nifty output if you want to print copies of any of the docs. If you are interested in a copy of the documentation files in Winword 2.0 format or in discussing conversion to other formats such as WordPerfect, please contact us on The Hacker Central BBS. In fact, if you have any questions, comments or suggestions concerning the documentation, we would like to hear from you. FUTURE ENHANCEMENTS ------------------- ZDCS is a dynamic product. In response to suggestions, comments and the changing bbs scene, ZDCS has seen a lot of improvements since version 1.5 was first released to the public. And we haven't reached the end of the road yet. Future enhancements under consideration or in progress include support for other archive formats besides PKZIP and ARJ, further improvements to local upload handling, and database path updating for bbs types besides PCBoard. (Support for PCBoard systems was added in version 2.0.) ACKNOWLEDGEMENTS ---------------- Thanks to all the beta testers who have helped to make ZDCS a better and smoother program, from version 1.5 through versions 1.6x and culminating in version 2.0. Their patience and input through all the steps along the way are definitely reflected in the quality of the final product. Thanks to Dave Klein for allowing me to bounce off him everything from data reduction approaches to curses at Quarterdeck. Thanks to Rich Hackney, Greg Wilson, Bob Jacobson, and Michael Leavitt for their help in tracking down the QEMM / Supershell memory incompatability problem. - 48 - Special thanks to Andy Keeves for ExzTest. Come to think of it, special curses to Andy for the same thing. Thanks to Roy Timberman for his help with some CD-ROM related errors. Finally, thanks to everyone who had the patience to wait while version 2.0 went through six intense months of beta bashing and feature testing. COPYRIGHTS AND LEGAL STUFF -------------------------- ZDCS (Zipfile Duplicate Checking System) is copyright (C) 1991, 1992 by Michael W. Cocke. ZDCS 2.0 is fully functional shareware. It may be freely copied and distributed, provided that no files in this package are removed or altered in any way. The individual documentation files ZDCS-REF.TXT, ZDCS.BLT and ZDCSWALK.TXT are copyright (C) 1991, 1992 by Evelyne Stalzer. Permission is granted to distribute these files as part of the complete ZDCS shareware package. Permission is also granted to bbs sysops to modify ZDCS.BLT for use as needed, including as a bulletin, message or downloadable help file. ZDCS.KEY is the individual registration key file and may not be copied, distributed or otherwise shared with individuals beyond the registered user. Neither Michael W. Cocke, Evelyne Stalzer nor MWC Enterprises will accept responsibility for the function, failure to function, or side effects of any function of the Zipfile Duplicate Checking System (ZDCS). ZDCS is provided in good faith, but its use is solely at the risk of the operator. ARJ is copyright Robert Jung. EXZTEST and EXZIP are copyright Andy Keeves. FWKCS is trademark and copyright Fred Kantor. Lantastic is trademark of ArtiSoft Inc. MS-DOS is trademark and copyright Microsoft Corp. Netware is trademark of Novell Inc. PCBoard is trademark and copyright Clark Development Company. PKZIP and PKUNZIP are trademark and copyright PKWARE, Inc. QEMM/386 is trademark and copyright Quarterdeck Office Systems. Word for Windows is copyright Microsoft Corp. ZDCS is copyright (C) 1991, 1992 Michael W. Cocke. - 49 -