Outline of SRE-Filter, ver 1.2i

Since SRE-Filter has numerous options, many of which have similar effects, at times it can get confusing!

This document outlines the steps followed by SRE-Filter when a new request is recieved. The goal is to provide a general outline, without burdening the reader with the details of how it actually works. That is, some familiarity with the various SRE-Filter parameters is assumed, or a willingness to check the main documentation or the description of the initialization parameters !

Note that both variants of SRE-Filter (SREFILTR.80, or SREFQUIK.80) adhere to this outline. For simplicity sake, we abstract from the intialization of SRE-Filter (which occurs on the first call to SRE-Filter). Also note that VARIABLE_NAMES refer to variables that can be set in INITFILT.80.

The outline

Multiple hosts

1. A HOST: request header is read (if it exists)
2. The IP address of the client is extracted.
3. Using the HOST: information and the IP address, the HOSTS. stem variables are searched for a match.
4. If one is found, the appropriate host nickname is returned (and used throughout SRE-Filter).

Preliminary checks

1. The LOAD_THRESHOLD and BACUPSERVERLIST variables are examined to see if this request should be forwarded to another server.
2. If PRE_FILTER="YES", call the (several) pre-filter(s) (listed in the PREFILTER_NAME variable).
3. Check to see that GoServe's caching has not taken care of this request, and that the pre-filter didn't deal with it. If either occured, goto the "post -filter step" (see below).
4. Check the PUBLIC_URLS list. If the requested selector appears on this list (perhaps by matching a wildcarded entry) then:
   • Check if it's a literal public_url.
     If so, then return a file in the data directory, or use an exact name contained in the public_url entry.
     If it's also a literal_norecord public_url, then exit.
     Otherwise, goto the "post-filter step".
   • If it's a public_url, but not a literal public_url;
     lookup dynamic client privileges,
     skipto the "alias check" step (skip the logon checks).

Logon and access controls

Logon

1. If the client is an OWNER or has an IP address that matches an INHOUSEIPS., skip the logons (grant her entry to the site).
2. If the client is an UNALLOWEDIPS, deny entry.
3. If DNS_CHECK='YES', check for the domain name server for the existence of a matching IP "name".
4. Otherwise, if CHECKLOG is on, look up (or first ask for) the username/password in the USER_FILE.

   ..but, first check the LOGON_LIMIT

5. If LOGON_LIMIT is violated, or an incorrect username/password is entered, SRE-Filter can either,
   Re-request username/password, (authorization responses use THE_REALM as the "realm name").
   Send the LOGON_FAIL_FILE

Determining client-privileges

1. If the client is an OWNER, grant her SUPERUSER "client privileges"
2. If the client is an INHOUSEIPS., grant her any privilege mentioned on her INHOUSEIPS. entry, and the privileges contained in INHOUSE_PRIVS.
3. Based on the client's username/password (if provided), client-privileges are read from the USER_FILE
4. Everyone gets PUBLIC_PRIVS;
5. If CHECK_ADD_PRIVS is on, the "dynamic" privileges cache is checked for client-specific transient privileges.
6. If ADD_USER_NAME is on, add the "username" to the client-privileges list.

Determining Access Rights using Resource-Privileges

1. Lookup SEL-specific resource privileges, permissions, etc. in the ACCESS_FILE. Add the "action" portion of the URL if ADD_RESOURCE_NAME is on.
2. If ALLOW_ACCESS is on, the list of resource privileges is compared against the client privileges.
   • All the "MUST_HAVE" resource privileges must be satisfied,
   • at least on of the "ONE_OF" resource privileges must be satisfied.
3. If the resource privileges are not "satisfied":
   • If ACCESS_FAIL_FILE=0, re-request a username/password (possibly using the "SEL-specific" realm name).
   • Otherwise,
     If a "SEL-specific" access failure file is available, send it to the client
     If there is no "SEL-specific" access failure file, send the ACCESS_FAIL_FILE
4. In addition to the "resource privileges list" and the SEL-specific realm, several permissions are read from the ACCESS_FILE.

   Any of the following permissions might appear: NO_SSI, NO_SSP, CACHE, NO_HTACCESS, NO_ALIAS, NO_VIRTUAL, NO_POSTFILTER, DELETE, and PUT.

5. If CHECKLOG< >"NO" or ALLOW_ACCESS< >"YES", then caching is suppressed. This suppression is withdrawn when the CACHE permission appears.

Request selector modifications and URL redirections

1. But first, if PRE_FILTER equals YES (rather then FIRST), call the pre-filter (and check for an "all done" return status).
2. If it's an empty selector, then set the selector to be the DEFAULT
3. NOTE: Non-literal public_urls jump to here !
4. If CHECK_ALIAS is on, and the NO_ALIAS permission is not present: Compare the request selector to the aliases contained in the ALIAS_FILE.
   Matches may indicate "local redirection", "remote" redirection, or "immediate file transfer".
   If a local redirection, the request selector is modified.
   If a remote URL redirection (or a file transfer); do the"redirection" (or the "transfer"), and goto the post-filter step.
   
5. If the http method is GET, then check "auto naming"
   If NOEXT_TYPE is on, check the (now alias modified) selector for an extension.
   If no extension found, then (depending on the value of NOEXT_TYPE) either add an extension, or search a directory for a file (using the entries in AUTO_NAME), or display a directory of files in the requested directory (using the !DIR facility).
6. Perform the HOME_DIR for ~ replacement (with possible use of the $ to signifiy "www specific subdirectories").

Processing the Request

Note that in all cases,

• If the request selector can not be matched to a file, a response that uses HOME_NAME and NOT_FOUND_URL is returned.
  Note that NOT_FOUND_URL may invoke a custom-written "not found" response file.
• If a NO_VIRTUAL permission appears, then virtual directories will not be checked
• Post-filter processing is done before exiting, but after closing the connection to the client.
• If DO_HTACCESS is on, then examine the directory (and parent directories) of the requested file for HTACCESS files, and use information therein for access control (and possibly for redirection). Note that for CGI-BIN programs this directory will be the location of the CGI-BIN program (similar rules hold for other requests for server side processing).
• If an advanced options file was specified (in the ACCESS_CONTROL_FILE), EXEC directives will be executed before "processing the verb".

• Image map requests

  The request string is compared against the NCSA_ISMAP and CERN_ISMAP variables. If a match is found, the image map routine is called, which invokes a redirection to the chosen URL.
  Note that the MAX_POINTDIST variable is used by the image map routines.
  Note that HTACCESS is never checked for image map requests (but HTML documents that contain clickable images are subject to HTACCESS controls!)

• CGI-BIN requests.

  If a GET or POST request starts with a CGI-BIN/, and the CGI_BIN_DIR variable is set, then the request is assumed to be a CGI-BIN requests.

  1. The alias file is checked again, this time for a match for a matching CGI-BIN script name. If found, the alias entry should contain the fully qualified directory containing the CGI-BIN script. If not found, the CGI-BIN script is assumed to be in the CGI_BIN_DIR directory. Exception: If the CGI-BIN portion of the request selector is preceeded by a directory entry, then this subdirectory (of CGI_BIN_DIR), or this virtual directory, will be usd.
  2. The CGI-BIN processor is called, and the results from the CGI-BIN script are returned (or an error message if no such script exists).

     Note:By default, CGI-BIN/MAPIMAGE/ requests are treated as NCSA imagemap requests, and handled by the standard SRE-Filter mappable image requests (see above description).

  3. Jump to the post-filter step.
• Special Requests If the request starts with a !, then one of the several special request might be processed.
  1. The valid special requests are: ping, statistics, host, reset, save, and variable.

     Save,reset, and variable require SUPERUSER or CONTROL privileges. Otherwise,an error response is returned.

  2. The !DIR special request will create a directory listing. It looks at the DIR_OPTIONS and DIR_EXCLUSION parameters for options. It also will check the "directory cache" for a match to the requested directory, and use it if possible.
     
  3. Several special directives are also recognized:
     • !SENDAS_type_subtype/sel is used to force SRE-Filter to return the sel with a content-type:type/subtype response header (that is, the normal SRE-Filter methods for determining MIME type are suppressed).
     • !DELSEND/filename.ext will transfer a file from the TEMPDATA_DIR, and then delete it.
     • !FORCE/asel will force a logon, and will suppress use of the SSI-Cache.
  4. For special requests: Jump to the post-filter step.
     For special directives: preceed to GET step
     
• HEAD requests
  1. The selector is converted to a file name, using the data directory or a local virtual directory (assuming that NO_VIRTUAL has not been set).
  2. If DO_HTACCESS is on, then check appropriate HTACCESS file(s) for access permissions.
  3. If AUTO_HEADER is on, and the selector is recognized as an HTML file; then the file is parsed, and all LINK, NAME and META-EQUIV elements in the HEAD are used to create extra response headers.
  4. If advanced options are enabled, process any HEADER or REQUEST directives.
  5. The response headers are returned to the client.
  6. Jump to the post-filter step
• GET requests
  There are 4 classes of GET requests
  1. Request for non-html files.

     The request selector is matched to a file, using the data directory or a local or remote virtual directory. If DO_HTACCESS is on, then check appropriate HTACCESS file(s) for access privileges. If the ACCEPT_RANGE variable is on, then check for a RANGE response header. Penultimately, if advanced options are enabled, HEADED and RESPONSE directives are processed. Lastly, the file is returned to the client.

  2. Requests for Html files.

     The request selector is matched to a file, using the data directory or a local or remote virtual directory. If DO_HTACCESS is on, then check the appropriate HTACCESS file(s) for access privileges.

     • HTML files are recognized by extensions of .HTM, .HTML, .SHT, .SHTML, or .HTML-SSI. You may add to the list of HTML file "extensions" by modifying MEDIATYP.RXX (MEDIATYP.RXX can also be used to add new MIME type definitions). You can also change the "server side includes allowed" list by modifying the SSI_EXTENSIONS variable.
     • If AUTO_HEADER="ALWAYS", then LINK, NAME and META-EQUIV elements in the HEAD are used to create extra response headers.
     • If SSI_SHTML_ONLY is on, then check the SSI_EXTENSIONS (the "server side includes allowed" list) to see if this file can have server side includes added. Otherwise, all HTML files are checked for server side includes. In both cases, if NO_INCLUDE is on, or a NO_SSI permission applies to the selector, server side processing will not occur.
     • If SSI_CACHE_ON is on, then check to see if the SSI-Cache contains an entry for this file. If it does, either send it "as is", or use it in the following steps.
     • If the advanced options are enabled, check for SSI_suppression directives.
     • If the request selector includes a ? followed by a text string, the text string is assumed to contain OPTIONS, which can be used in server side includes.
     • Searchable indices, which are often specified as HTML files followed by ?string, should be interpreted through the use of an alias (the alias maps the request for an HTML file to a call to an appropriate external program, such as DOSEARCH).
     • Note that server side includes are processed recursively
     • The DELIM_1. and DELIM_2 variables are used to define keyphrases
     • The following variables may be used for various server side includes: HEADERS., FOOTERS., INHOUSE., SUPERUSER., WEBMASTER, OPTION_HIT_LINE . In addition, the REPSTRGS_FILE may be checked for custom written "static REPLACEment strings".
     • The COUNTER_FILE may be used to record the "number of hits" -- after checking the repetitive hits cache (the repetitive hits cache is configured by the HIT_CACHE_LEN, HIT_CACHE_DURATION, and HIT_OWNER_SUPPRESS variables).
     • If NO_INTERPRET_CODE is on, or a NO_CODE permission applies to this request, then SELECT and INTERPRET CODE keyphrases are ignored.
     • Dynamic privileges can be added by using ADDPRIVS.RXX in an INTERPRET FILE keyphrase. Note that the added client privilege is modified by the ADD_PRIVS_PREFIX variable.
     • If FIX_EXPIRE is greater then 0, the "expire immediately" header is modified.
     • If DO_SEND_PIECE is on (and other conditions hold), send pieces of server-side include containing documents "as they become available" (rather then waiting for the entire file to be completely processed).
     • If advanced options are enabled, HEADER and RESPONSE directives are processed.
     • Lastly, the HTML document (perhaps with server side includes) is returned to the client.
  3. Server side processing requests
     • If NO_PROCESSING is on, or a NO_SSP permission applies to this selector, then no server side processing will be permitted.
     • If the request is to one of the several "built in" routines (SENDFILE, ASKMESSBOX, VIEWMESS, MESSAGE, GET_URL, !DIR, PUT_FILE), then SRE-Filter handles it directly.
     • Requests to SRE-Filter add-ons (such as DOSEARCH, GETAFILE, STATUS) are processed by calling the appropriate external procedure, which is located either in the GoServe working directory (the directory containing GoServe) or in a local virtual directory.

       These programs should return results directly to GoServe (i.e.; by using 'FILE xxx' types of commands). Note that failures in these "external" programs will result in error messages being returned to the client -- SRE-Filter will not crash.

     • If DO_HTACCESS is on, then check any appropriate HTACCESS file(s) (in the directory-tree that contains the program). For "built in" and "included" programs, the GoServe directory is checked for an HTACCESS file.
• POST requests POST requests are handled just like GET requests for server side processing. The only difference is that the argument list is pulled from the "body" of the request, rather then from the "request selector".

  If a "file upload" request occurs, the UPLOAD_MAXSIZE, UPLOAD_MINFREE and UPLOAD_DIR variables are used. (note that this applies to GET method transactions also).

  If DO_HTACCESS is on, then check any appropriate HTACCESS file (in the directory tree that contains the program). For "built in" and "included" programs, the GoServe directory is checked for an HTACCESS file.

• PUT requests PUT request (for transfering files to the server) are honored only if a PUT permission applies to the selector.

  If DO_HTACCESS is on, then check any appropriate HTACCESS file(s) (in the directory-tree that is the target of the PUT).

• DELETE requests DELETE request (for deleting files on the server) are honored only if a DELETE permission applies to the selector.

  If DO_HTACCESS is on, then check any appropriate HTACCESS file(s) (in the directory tree that contains the file to be deleted).

Post-filter processing

1. If the WRITE_LOGS parameter is enabled, then (perhaps) write the common-log audit file, the browser log, and the referer log. The SREFLOGS.INI will dictate exactly where and how to write these log files.
2. If the RECORD_OPTION is on and the RECORD_ALL_FILE is set, then record this selector.
   ... but first check the repetitive hits cache to see if it's been recently recorded.
   Note: recording may be stored in an internal cache for several minutes.
3. If the POST_FILTER variable is on, then call a (set of) post-filter(s) with listed in POSTFILTER_NAME (note that the SMTP_GATEWAY can be used by the POST_FILTER).