A Summary of the Features of SRE-Filter ver 1.2

This outline descibe the features and options of the SRE-Filter (ver 1.2) HTTP (Web) server package for the GoServe Internet server. More detailed information is available in the manual (a 200k file).
SRE-Filter is revised on a regular basis. Upgraders might want to read the list of current changes.

In a nutshell... The SRE-Filter niche is the non-cutting edge, small-to-medium load site being maintained by non-professionals who want a fair degree of functionality in an easily maintained system.

In a slightly larger nutshell (a coconut shell?) ...

Initial Setup

All variables, file names and directory names mentioned here are, unless otherwise specified, set in the INITFILT.80 file. They can be changed either by editing INITFILT.80, or by using the CONFIG2.HTM configurator. For more details on the parameters contained in INITFILT.80, please see INITFILT.DOC.

After unzipping the SRE-Filter package (SREFILTR.ZIP) to a temporary directory, initial setup is accomplished with SRE-Filter's INSTALL.CMD program (a REXX program). At installation, the user provides the names of several directories (or chooses the defaults).

User configurable files

The INSTALL program will install a number of files. In addition to the various program files (such as SREFILTR.80, SREFMON.CMD, and SREFPRC1.RXL), several "user-configurable" parameters/options files are installed.

Although the names of these user-configurable files can be changed by the user, in this summary we refer to them by their default names .

Parameter initialization

INITFILT.80 contains a number of user-configurable parameters; many of which are discussed below. These parameters can be modified either by editing INITFILT.80 (with a text editor). More conveniently, after installation of SRE-Filter you can use CONFIG2.HTM -- which provides a fill in form that allows you change initialization parameters, username/passwords, aliases, virtual directories, SEL-specific access control, and HTACCESS files.

KEYPHRASE Support

REPSTRGS.IN is used to specify customized REPLACEment strings.

ALIAS creation

ALIASES.IN -- A user configurable set of aliases used for document redirection, implementing searchable indices, and a variety of other tasks.

Virtual directories

VIRTUAL.IN -- A user configurable list of "local and remote virtual directories", consisting of a selector substring and an actual drive:\directory (or a "root URL" of the form http://some.site.xxx/directory/*).

Auditing files

Several files are used to keep track of the the number of times your various documents have been requested:
  1. COUNTER.CNT -- keep track of files that have a REPLACE HITS or a REPLACE COUNTS keyphrase.
  2. RECRDALL.CNT -- keep track of all files (when RECORD_OPTION is active)
  3. SENDFILE.CNT -- keep track of all files transfered using the SENDFILE facility of SRE-Filter
  4. COMMON.LOG, BROWSERS.LOG, and REFERER.LOG -- the common-log, browsers and referers log files.
To limit "false hits" caused by repeat requests (from the same client), SRE-Filter can be told to cache recent requests. When a request matches an entry in the cache (based on the request selector and client's IP address) the counter files are not augmented.

Logon and Access control files

USERS.IN is a text file containing a list of usernames,passwords, and privileges.

ALL_FILE.CTL is used for "per selector" access control and permissions.

External REXX procedures

SRE-Filter comes with several files that provide useful features, including:
  1. DOSEARCH.CMD -- A simple but powerful string search utility for text files.
  2. POSTFILT.80 -- A sample of a "post-filter": an event-driven e-mail notifier.
  3. PREFILTR.80 -- A sample of a "pre-filter"; it enables access to the GOREMOTE "server remote control" package.
  4. STATUS.CMD -- A sample "server side" program ("a REXX script")
In addition, several add-ons for SRE-Filter can greatly extend the capabilities of your web site. These include:

More information

For further details on the files that comprise SRE-Filter, please see the SRE-Filter packing list.

Multiple hosts

SRE-Filter supports multi-hosting. Multiple hosting can occur in two different manners:
  1. When multiple numeric IP addresses are handled by a single computer
  2. When multiple aliases exist for a single numeric IP address; and the HOST: request header is provided by the client's browser.
When your server is responsible for multiple hosts, most of SRE-Filter's variables and parameters can be configured to be "host specific"

Basic access (logon) controls

SRE-Filter provides several logon controls. The primary control is provided by the CHECKLOG variable, with possible values:

Defining classes of users

You can pre-specify the "user-class" of clients using using their IP addresses. This provides a means of permitting, or denying, access on a client-by-client basis (without the use of an explicit logon).

Superusers

OWNERS are defined using the OWNERS variable: which should contain a space delimited list of IP addreses. OWNERs are given SUPERUSER privileges; which allows uninhibited access to all files and facilities of SRE-Filter.

In-House Users

The IP addresses, and logon privileges, of IN-HOUSE users are defined using the INHOUSEIPS. "stem" variable. The wildcard character, *, can be used in any of the domain fields. Thus, INHOUSEIPS.1="*.*.*.*" means "everyone is an IN-HOUSE user". IN-HOUSE users have more privileges then the public, though not as many as OWNERs. Note that OWNERs are automatically granted INHOUSE privileges.
Hint: The INHOUSEIPS. variables can be thought of as a simple form of IDENT authorization

Unallowed Users

In some cases, there may be individuals (or groups), with known IP addresses, that you want to keep out. This can be accomplished using the UNALLOWEDIPS.n variable (uses the same syntax as INHOUSEIPS.n).
Note that SRE-Filter firsts check if the client is an OWNER or an IN-HOUSE user; if neither is true, UNALLOWEDIPS.n is checked. Thus, setting UNALLOWEDIPS.1="*.*.*.*" will keep everyone (except OWNERS and IN-HOUSE clients) out.

Domain Name Checking

Optionally, you can check the name of the requester. This is a security measure designed to limit entry to clients with "fake" IP addresses.

The USERS_FILE

When "logon" is required, USERS.IN is checked for username, password, and privilege information. Note that a privilege of INHOUSE or SUPERUSER gives INHOUSE and SUPERUSER privileges (respectively) to a client with this username/password.

Note that logon is never required of OWNERS and IN-HOUSE clients.

PUBLIC_URLS

In cases where you want to allow free access to a limited set of resources, a set of PUBLIC_URLS can be specified. Thse PUBLIC_URLS point to resoures (programs, documents, and directories) that are available to everyone, with or without logon privileges.

Logon and Access failure response files

When access is denied, either at the site (LOGON) or resource (ACCESS) levels, a custom written "response" (the LOGON_FAIL_FILE or the ACCESS_FILE_FILE) file can be returned; or the client can be re-prompted for a new username and password. Furthermore, as a security measure, you can use LOGON_LIMIT to limit the number of attempts per minute per client.

Finding the requested document

In many cases, a request will not specify a document, or specifies a document that can not be found. SRE-Filter has several means of guessing what the client really wants.

Note that in most cases requested files are assumed to be "relative" to the data directory. This is a security feature of SRE-Filter that is typical of most servers.

The DEFAULT document

In the extreme (and typical) case the client asks for nothing (an empty request selector is recieved). When this occurs, the DEFAULT variable is used; it should contain the name of your "home page" document (traditionally, INDEX.HTM is used as your "home page").

Default documents for directories

Upon recieving a request selector that specifies a directory (that ends with a /, or that ends with a name containing no extension), the AUTO_NAME variable is examined to ascertain which document should be used. AUTO_NAME should contain a space delimited list of file names; SRE-Filter will check each name in succession, stopping when it finds a file name that exists (by default, SRE-Filter will look for these files in the "directory" specified in the request selector).

When AUTO_NAME contains the * character (by itself), SRE-Filter looks for a file with the directory.HTM name; where directory is the "last" name in the directory requested (that is, if PROJECTS/HARRY/ is requested, * will mean HARRY.HTM}.

Specifying a HOME directory

The HOME_DIR variable is used whenever a ~ is seen in the request selector -- the contents of HOME_DIR are substituted into the request selector. Thus, if HOME_DIR=USERS/PERSONAL/, then ~GERALD/CV.HTM becomes USERS/PERSONAL/GERALD/CV.HTM.

In addition, you can specify a web-specific subdirectory, for each of a set of users. To do this, just include a $ in the HOME_DIR variable.

For example:

   If  HOME_DIR=USERS/$/WWW
      and the request selector is /~GERALD/RESUME.HTM
  Then SRE-Filter will use:
        /USERS/GERALD/WWW/RESUME.HTM
Summarizing given a $ in the HOME_DIR:
  1. SRE-Filter reads the substring (of the request selecttor) between ~ and the first / following the ~
  2. The substring is deleted from the request selector
  3. The $ in the HOME_DIR is replaced with this substring (from step 1)
  4. The ~ in the modified request selector (from step 2) is replaced with the modified HOME_DIR (from step 3)
Notes:

Specifying a "virtual" directory.

By default, files are assumed to be in the default data directory, and server side programs are assumed to be in the working directory. While a good security feature (files not in these directories are unaccessible), this can be an inconvenience. To remedy this inconvenience, one can define "virtual directories" using VIRTUAL.IN. Basically, SRE-Filter will compare the "starting portion" of a request selector to see if it matches an entry in VIRTUAL.IN. If it does, the "candidate directory" portion of this entry is used (the portion of the request selector after the "starting portion" is used as the file name). Thus, you can make available a wide, but controllable, set of directories on (or WEB, or LAN accessible from) your server

Redirection using ALIASES

The ALIASES.IN file is used to redefine specific request selectors. Among it's many uses is internal and external URL redirection. Internal redirection is used to "map" the request selector (which might be an easily remembered string) to an otherwise unassociated server resource (say,a file deep in the directory tree). External redirection is used when a URL should now point to a different server). Note that "wildcard matching" and "wildcard matching with substitution" can be used in the ALIAS file, which allows requests for any document in a (perhaps no longer existent) directory to be redirected to a new location.

In addition to selector specific redirection, SRE-Filter also supports a simple form of load balancing. When active, if the current number of active requests is too high, further requests are redirected to other servers (i.e; mirror sites).

Document not found?

If, despite the above tricks, the requested document can not be found, a "not found" document is automatically returned. You can specify a short message to include in this "not found" document by changing the NOT_FOUND_URL variable -- it should contain a link to some default document (say, your home page).

To facilitate this, you can use the HOME_NAME variable in the definition of NOT_FOUND_URL. HOME_NAME should contain the "colloquial" (not necessarily IP) name of your site.

Server Side Includes Using KEYPHRASES

A major advantage of SRE-Filter is the ease with which a variety of Server Side Includes can be invoked-- just by adding special keyphrases to your document.

Two types of server side inclusions are supported:

  1. SRE-Filter syntax:

    SRE-Filter syntax for server side includes is:
    DELIM1 KEYWORD ARGUMENT DELIM2
    As shipped, DELIM1 and DELIM2 are

  2. or <!-- and -->
    (the delimiters can be easily changed).
  3. NCSA HTTPD syntax:

    Most of the NCSA HTTPD server side include commands are supported. These are invoked using a syntax of:
    <!-- #command option="value" -->


Headers and Footers

In addition to Keyphrases, you can specify the automatic addition of a Header and a Footer to all documents.
Headers are set in the HEADER. "stem" variables,
Footers are set in the FOOTERS. "stem" variables.

Headers and Footers can contain keyphrases.

String REPLACEments

The REPLACE keyphrase is used to add static and dynamically determined strings to your document.

Dynamic Strings

Dynamic strings are determined at the time of the request. Typically, they are variables that SRE-Filter (or GOSERVE) determines on-the-fly. Examples of such variables are the time of day, the number of "hits" for the document, and the creation date of the document.

SRE-Filter provides a number of these variables, which we list here (see the main documentation for details:

DATE DATEGMT TIME TIMEGMT CREATION CREATION_DATE CREATION_TIME READ_HEAD SERVER USERNAME BROWSER REFERER HITS COUNTS OPTION_COUNTS OPTION_HITS URL.

Static Strings

Static strings are predetermined; one could enter their contents directly into your documents. However, an approriate REPLACE keyphrase may be easier to remember. More importantly, if a phrase occurs in a number of documents, and it changes on a frequent basis (say, a daily message), then one can change the phrase just by changing the appropriate static variable rather then editing a zillion documents.

There are a few static variables built into SRE-Filter, that can be modified in INITFILT.80, including: WEBMASTER HOME_NAME INHOUSE.n SUPERUSERS.n

In addition, you can add your own "static" variable names to the REPSTRGS.IN file. Although the replacement strings specified in REPSTRGS.IN names can be several lines long, for longer "inclusions" it is probably better to use the INCLUDE keyphrase.

FILE includes

The INCLUDE keyphrase is used to include files into the requested document.

OPTION includes (to access information in the request selector)

OPTIONs are used to pass information, from a client, to be incorporated into an HTML document. Specifically, options may appear after a ? character (the ? is after the document name), with each of several options seperated by a & character.
With appropriate specification of the invoking URLs, this provides some ability to customize documents (say, with the name of the document from which the request occurred).

INTERPRET includes (executing REXX code blocks)

The INTERPRET keyphrase is used to execute REXX code blocks, and insert the results into the document. This provides a simple means of doing a limited amount of Server Side Processing; especially when the results may be required by different documents. In a sense, the INTERPRET keyphrase allows one to create new "dynamic" variables, without delving into the SRE-Filter source code!

For example, USETABL2.RXX, which can be used in an INTERPRET keyphrase, provides a simple mechanism of automatically deciding whether to send a "TABLE containing" document, or a "non-TABLE containing" document, to a client (based on the name of the client's browser).

SELECTive includes

The SELECT keyphrase is a little different: it dictates whether the "block of HTML code" that follows the SELECT keyphrase is to be retained. Thus, it is best thought of as a conditional "server side exclude". In a sense, the SELECT keyphrase is similar in function to the combination of INTERPRET and INCLUDE keyphrases discussed above.

NCSA HTTPD style includes

The NCSDA HTTPD server side include options are fully supported. These include: FLASTMOD, FSIZE, ECHO, and CONFIG. EXEC is also supported, with all the "standard output" generated by EXEC invoked scripts inserted into the document.

Automatic response header creation

If desired, SRE-Filter will parse a requested document, extract all LINK and META HTTP-EQUIV elements found in the <HEAD>, and create the appropriate response headers. This is especially useful in response to HEAD requests.
Note that when responding to a HEAD method request, server side includes will not be done on <HEAD> elements.

Imagemaps, Searchable Indices, and Server Side Processing

Imagemaps

SRE-Filter provides support for server side processing of images maps using NCSA and CERN style .MAP files. This done by
  1. specifying a <ISMAP>able image as the target of a URL;
  2. the "location portion" of the URL contains a MAPIMAGE/ (or similar) substring.
  3. the remainder of the "location portion" points to a MAP file
SRE-Filter uses this .MAP file, along with the pixel location (that is found after the ? in the request selector), to perform document redirection to the appropriate URL.

Searchable Indices

Searchable indices are supported in SRE-Filter by using an ALIAS. In combination with the DOSEARCH facility, it is quite easy to set up a searchable index against an ASCII text file (that can be anywhere on the server). For example, by creating the following ALIAS:
SEARCH2.HTM?* DOSEARCH?file=RESOURCE.LST&search=*
a request for SEARCH2.HTM will invoke (and execute) a searchable index against the text file RESOURCE.LST (assuming that SEARCH2.HTM contains an <ISINDEX> tag).

Server Side Processing

The addition of custom-written Server Side Processing routines is a easily accomplished with SRE-Filter. These routines are best implemented with REXX procedures, but it is not difficult to invoke any executable from within SRE-Filter.

CGI-BIN programs

SRE-Filter supports CGI-BIN programs. It uses procedures adapated from the GoHTTP package. REXX procedures and binary executables can be readily invoked -- in the simplest case, just by copying the program to the CGI_BIN_DIR directory.

SRE-Filter also supports the use of alternate (non-REXX) interpreters, such as PERL

Auditing and Access Controls

Assigning Privileges

OWNERs are automatically granted SUPERUSER and the INHOUSE_PRIVS privileges. the IN-HOUSE users are automatically given the INHOUSE_PRIVS privileges, and can be assigned client specific privileges. For all others, a privilege list is read from the USERS.IN file (it follows the username and password). Occasionally, a logon is forced solely to collect privilege information. Lastly, everyone is granted the PUBLIC_PRIVS privileges.

Auditing Requests for File

There are several means of keeping track of the number of times files have been requested (you can also use the GOAUDIT.80 file maintained by GOSERVE).

The REPLACE HITS / REPLACE COUNTS counter file

When REPLACE HITS or REPLACE COUNTS keyphrase is encountered (in an HTML document), the COUNTER.CNT file is examined. If the request selector is found, the "count of hits" is read and augmented. If not found, a new entry is automatically added. In addition, COUNTER.CNT can contain an optional (2-part) message to be used when displaying the "you are the nth hit" message.

The common-log file

SRE-Filter can record all requests to a common-log audit file. It can also save browser and referer information to seperate log files. The WRITE_LOGS parameter can be used to enable/disable this auditing.

RECORD_OPTION and recording all requests

The RECORD_OPTION variable is used to tell SRE-Filter to record all request selectors , or files, in RECRDALL.IN. If you use the YES_ALL option, be aware that the RECRDALL.IN file can become quite large.
Note that wildcard matching can be very useful when requests for temporary files occur -- it reduces the clutter in RECRDALL.IN without losing any essential information.

Hint: if a more sophisticated transaction recorder is available (written in REXX), you can "plug it in" to SRE-Filter using the POST_FILTER option (for an example, see POSTRCRD.80).

Controlling access to your server on a SEL-specific basis.

The ACCESS_CONTROL variable is used to limit access to your server on a SEL-specific basis. If ACCESS_CONTROL = 'YES', then SEL-specific "access information" is checked for all users (except SUPERUSERS). If ACCESS_CONTROL='INHOUSE', then access information is checked for non-INHOUSE (and non-SUPERUSERS) only.

ALL_FILE.CTL contains the SEL-specific access information. Each entry in ALL_FILE.CTL should contain a (space delimited) list containing:
SEL, resource_privileges_list , permissions, SEL-specific-realm
When the requested SEL (or a wildcard match to it) is found , this resource_privilege list is compared to the the client's privileges: if there is a common element, the request is honored.
If there is no match in ALL_FILE.CTL, or none of the client's privileges appears in the resource_privilege list, the request will not be honored.

Note that if the resource_privilege list equals the * character, then the request is always honored. Conversely, if the resource_privilege list equals NO, then it is never honored (assuming, of course, that ACCESS_CONTROL is binding).

ALL_FILE.CTL can also contain a number of "SEL-specific permissions" -- such as whether server side includes and server side processing is allowed, whether the DELETE and PUT methods will be honored, and whether caching is permitted,

ALL_FILE.CTL can contain a realm name associated with this request selector. This realm name is displayed by the client's browser when requesting username/password information. In addition, you can specify a realm-specific list of "additional resource privileges", which will be granted to all SEL-specific entries with the same realm name .

Using the HTACCESS method

To provide compatability with HTTPD style servers (such as Don Meyer's GOHTTP), SRE-Filter now supports the HTACCESS method of access control (see HTACCESS.DOC for details). This support uses code graciously loaned to us by Don Meyer, suitably modified for SRE-Filter.

The SEND_FILE

The SENDFILE facility uses SENDFILE.CNT to keep track of requests, (that is, files requested through the SENDFILE facility). It's function is similar to RECRDALL.CNT. One difference is that files are recorded only after completion of the transfer -- if the client kills the connection, the SENDFILE.CNT file is not augmented.

Useful Facilities that come with SRE-Filter

A few facilites, and some general purpose Server Side Processing routines, are included with SRE-Filter.

File Uploads

SRE-Filter supports three mechanisms for clients to transfer files to the server.
  1. Getting a URL: This requires that the client has the ability to place the files on her own web server. She can then direct SRE-Filter to get this file, and save it in SRE-Filter's upload directory.
  2. Uploading a file from the client's own computer: If the client is using a browser that supports the type=file FORM attribute, then she can easily transfer files directory from her own machine to the server (for example, NetScape 2.01 has this capability).
  3. The PUT HTTP method can be used to write a file to a server's "data directory" (or appropriately specified virtual directory) -- but only if ALL_FILE.CTL allows such access (on a SEL-specific basis). The DOPUT.CMD REXX program demonstrates how this can be used. The DELETE method is also supported.

Special Requests

Several special requests are recognized as "special requests" (some of which require CONTROL or SUPERUSER privileges). These include:
  • !authorize !ping !statistics !ssi?option !host+ip.add.res.s !reset !save !variable?varname
    • In addition, several special directives can be invoked, including:
  • !SENDAS_MIMETYPE_SUBTYPE/sel !DELSEND/tempfile_sel !FORCE/sel !NORECORD/sel

    • The !DIR built-in directory display facility

    !DIR is SRE-Filter built-in directory display (and file transfer) facility (updaters note: it replaces GETAFILE.CMD, which is now available as an add-on). !DIR can be invoked with a URL of
  • http://your.server/!DIR?sel
    • . There are a number of options that you can set -- either by adding additional arguments after the ?sel, or by modifying the DIR_OPTIONS and DIR_EXCLUSION SRE-Filter parameters.

    Note: !DIR is also used by the AUTO_NAME feature; it is called when a request for a directory is recieved, and no INDEX (or equivalent) file can be found in the requested directory.

    For further information on !DIR, please read the DIR.DOC file.

    The SHOWDIR dynamic directory list processor

    As an alternative to SRE-Filter's !DIR facility and GETAFILE add-on, SRE-Filter supports the "GOHTTP-style" SHOWDIR dynamic directory list processor. More details on SHOWDIR can be found in SHOWDIR.DOC.

    The SENDFILE file transfer facility

    SENDFILE is a facility that can be used to record transfer of files. It is called with a file name, and an optional "transfer as text file" parameter. It uses the SENDFILE.CNT file to record information.

    The DOSEARCH text file search facility

    DOSEARCH.CMD is a simple but powerful text-file search routine. DOSEARCH looks for occurences of words and phrases in paragraphs. It supports a variety of boolean conditions, and it provides several display options (such as highlighting matches).

    The DOSEARCH.HTM demonstrates the use of DOSEARCH

    DOSEARCH, when combined with an appropriate ALIAS, provides a simple means of implementing searchable indices.

    The Message-Box Facility

    SRE-Filter supports a text-file based message-box facility, with entries made using MESSAGE?option1&option2&.. requests. These message boxes can be used to support a no-frills bulletin board system, with contents viewed using the VIEWMESS and ASKMESSBOX faciltites. Of greater interet, the FORUM facility uses message boxes to support multi-threaded discussion groups. In addition, the READMAIL and MAILLIST facilities can be used to record e-mail into message boxes.
    The combination of MESSAGE, FORUM, READMAIL, and MAILLIST can be used to implement a combination "discussion group/ list server"
    .

    A Note on User Modifications to SRE-Filter

    SRE-Filter has several mechanisms by which user extensions can be incorporated. These are:
    1. Creating your own "pre-filter" .
    2. Creating your own routines for handling Server Side Processing Requests, or using CGI-BIN programs (even PERL scripts, if you have an OS/2 PERL interpreter installed).
    3. Creating your own "post-filter"
    4. Adding INTERPRET code blocks.
    5. Creative use of the SELECT keyphrase can accomplish quite a bit of customization of HTML documents.
    6. A library of useful routines is available (via macrospace) to ambitious REXX programmers (see SREFPRC1.DOC for details).
    7. SRE-Filter uses several "helper programs" (running in seperate threads) to: Ambitious REXX programmers can modify these; say to use a database manager (perhaps located on a secondary server), rather then the flat ascii files used by SRE-Filter. For further information on programming with SRE-Filter, see SREFPRC1.DOC.