Synchronet Message Base Specification Version 1.21 Updated 05/31/95 Copyright 1995 Digital Dynamics PO Box 501 Yorba Linda, CA 92686 Voice: 714-529-6328 BBS: 714-529-9525 V.32/V.32bis FAX: 714-529-9721 714-529-9547 V.FC Fido: 1:103/705 ftp: netcom.com /pub/sb/sbbs Table of Contents ================= &&Contents Introduction....................................................@@INTRO___ Implementation Levels...........................................@@IMPLEVEL Definitions.....................................................@@DEFINES_ Acronyms................................................@@ACRONYMS Data Types..............................................@@DATATYPE File Formats....................................................@@FILEFORM Index.....................(*.SID).......................@@SID_FORM Header....................(*.SHD).......................@@SHD_FORM Header Allocation.........(*.SHA).......................@@SHA_FORM Data......................(*.SDT).......................@@SDT_FORM Data Allocation...........(*.SDA).......................@@SDA_FORM CRC History...............(*.SCH).......................@@SCH_FORM Header Field Types..............................................@@HFIELD_T Data Field Types................................................@@DFIELD_T Messsage Attributes.............................................@@ATTRBITS Translation Types...............................................@@XLATTYPE Agent Types.....................................................@@AGENTTYP Network Types...................................................@@NETWORKS Media Types.....................................................@@MEDIATYP Message Storage Pseudo Code.....................................@@STORPCOD Message Retrieval Pseudo Code...................................@@READPCOD SMBUTIL.........................................................@@SMBUTIL_ CHKSMB..........................................................@@CHKSMB__ FIXSMB..........................................................@@FIXSMB__ SMBLIB (C library)..............................................@@SMBLIB__ Data Types and Constants..(SMBDEFS.H)...................@@SMBDEFS_ Global Variables..........(SMBVARS.C)...................@@SMBVARS_ Function Prototypes.......(SMBLIB.H)....................@@SMBLIB.H Library Functions.........(SMBLIB.C)....................@@SMBLIB.C Miscellaneous.............(CRC*.* and LZH.*)............@@SMB_MISC SMBLIB Storage Example..........................................@@SMB_PUT_ SMBLIB Retrieval Example........................................@@SMB_GET_ SMBLIB Performance Issues.......................................@@PERFORM_ Bibliography....................................................@@BIBLIOGR Implementations.................................................@@IMPLEMEN Introduction ============ &&Introduction $$INTRO___ Q. What is SMB? A. SMB (Synchronet Message Base) is a technical specification for the storage format of electronic mail messages. These e-mail messages may all be contained in one database, or, more commonly, separated into catagorized databases. These message databases (or message bases) are also referred to as "sub-boards", "forums", "conferences", and "SIGs". The messages may be directed to an individual person, sent to a group of individuals, or sent to everyone who can read messages in that message base. Messages may be created and read soley at one physical location, or imported from and exported to a message network that may span continents. Message bases that are connected to a message network are often called "echoes". Q. Why SMB? A. The Synchronet Message Base is designed to store high volumes of messages while maintaining optimum search, retrieval, and creation performance. These messages are not limited to mere text. In addition to text, SMB defines the storage of digitized sound, MIDI, graphics, fonts, animation, as well as other multimedia data and triggers for localized multimedia. SMB thrives on a multi-user environment where messages are being created, read, modified, and deleted by multiple tasks simultaneously. With the large message networks of today being the rule, rather than the exception, and high volumes of messages being imported on a daily, sometimes hourly basis, creation and deletion speed is of the utmost importance. This is where SMB really shines. Being extensible enough to handle message formats from networks of today and tomorrow, and fast enough to import more messages that humanly readable, the SMB format will more than meet your message storage needs. Q. Why a specification? A. Message bases are often accessed and modified by a number of different programs. Often these programs are developed by individuals or companies other than the original designer of the message base format. This specification is an attempt to aid developers in creating programs that access or modify a message base stored in the SMB format. Q. Who can use this specification? A. Anyone that has interest in the Synchronet Message Base format at either an educational or professional level. Specifically, software developers interested or currently involved in the development of message readers, editors, echomail (toss/scan) programs, message transfer agents (MTAs), network gateways, and bulletin board systems. Much of the information in this specification is intended for those with preexisting programming knowledge, so those with little or no programming experience may find it hard to comprehend. Q. What does the SMB specification include? A. The text you are reading is part of the SMB specification: a single text document that defines the storage format of each of the six files of an SMB format message base and how they are related to each other. Included with this specification is C source code to be used as an example to programmers of how to access an SMB format message base and public domain library functions (SMBLIB) that can be compiled and linked into programs that access an SMB format message base developed by third parties. An SMB utility program (SMBUTIL) is also included with C source code as an example of how to use the SMBLIB functions. Q. Where did the SMB specification come from? A. Digital Dynamics (southern California based software development company) released "Synchronet Multinode BBS Software Version 1a" in June of 1992 as one of the first BBS packages to be designed from the ground-up to operate in a multinode environment with incredible speed and reliability, with a large suite of multinode specific features and design innovations. The original message base format was designed with localized messaging and low volume message networks in mind. By January of 1993, it was clear that high volume message networks (FidoNet, RelayNet, Usenet, etc.) were the preference of most BBS users and a new message base format was required to allow for high volume message storage, improved storage, retrieval, and maintenance performance, as well as lower storage space requirements. Rather than introduce another new message format, Digital Dynamics sought to implement an existing public specification for a format that would meet current and future message storage needs. More than a few specifications were seriously considered at one time or another, but after careful examination, design flaws and lack of extensibility eliminated them from the long term plans of Digital Dynamics and Synchronet BBS Software. Thus began the design of the "Synchronet Message Base" (SMB) format. At the request of many message related program developers, Digital Dynamics created and released the SMB specification before the release of "Synchronet Version 2.00" to allow lead-time on developing support programs for the new format. Digital Dynamics strongly encourages developers of message related programs (including software that directly competes with Synchronet or other Digital Dynamics products) to implement support for SMB. Though this is a public specification and Digital Dynamics encourages developer suggestions, it will remain under the sole control of Digital Dynamics unless specifically stated otherwise in a future revision of this specification. Digital Dynamics requests that any organizations that wish to adopt or ratify this specification, in part or whole, notify Digital Dynamics through any of the contact methods listed at the beginning of this document. Q. How does SMB store messages? A. Each message base is stored in a set of binary files. This set consists of between three and six files depending the storage method used. The base filename (maximum of eight characters under DOS) is the same for all six files of the same message base and unique amoung the filenames of other message bases in the same directory. The six files each have a different three character extension. The first character of the extension is always the letter 'S' (for SMB), while the second and third characters define the contents of the file. Two of the six files associated with each message base are not recreatable and therefore are the most important when considering data integrity. These two files are the data file (with a .SDT extension) and header file (.SHD extension). Both of these files use 256 byte blocks and have associated block allocation tables (stored in .SDA and .SHA respectively) so that deleted message blocks may be used by new messages without creating odd sized unused 'holes' in the files. The block allocation table files (.SDA and .SHA) can be recreated with the information stored in the header (.SHD) file. When using Hyper Allocation storage method, the allocation files (.SDA and .SHA) are not used. For fast indexing, there is a small fixed length index file (with a .SID extension). This file allows for the immediate location of message header records based on sender's name or user number, recipient's name or user number, subject, message number, or message attributes. This file can be recreated with the data stored in the header (.SHD) file. The last file is an optional CRC history (.SCH) file. It contains 32-bit CRCs of a configurable number of messages imported or created locally. This is to help eliminate duplicate messages created by user or program error. The CRC history file can be recreated with the combination of information stored in the data (.SDT) and header (.SHD) files. Q. How fast do messages import into an SMB message base? A. This is a very important question for systems for that import large volumes of messages. Of course, the answer depends on the storage format which you are importing from, the average length of messages, the design of the program which is peforming the import process, as well as the hardware and system software being used. What's important is that SMB will allow the fastest import process possible with any given combination of the above factors. Since system storage capacity is rarely infinite, neither is the number of messages which can be stored in a message base. System operators must define the maximum number of messages to be stored in a message base, the maximum age of the messages in that message base, or a combination of both. When using the Self-packing storage method (defined later in this document), the smaller the number of messages stored in a message base, the faster the import process. The SMB format is flexible enough to support multiple levels of import performance based on optimizations for storage space or speed. Most system operators will almost invariably choose speed over space, but which choices are available is determined by the importing program. This specification defines three storage methods, from slowest to fastest: Self-packing, Fast Allocation, and Hyper Allocation. Other options defined in this specification may affect storage performance, including duplicate message checking and message compression/encryption. Q. How much storage is required for an SMB message base? A. The biggest factor in determining storage requirements for a message base is the maximum number of messages to be stored in the base (defined by the system operator) and the average size of each message. The minimum required storage for a message base is 32 bytes plus 532 bytes per message (plus four bytes per message if duplicate message checking is used and three bytes per message if Self-packing or Fast Allocation storage methods are used). The SMB format was originally designed to be "self-packing", meaning purged (deleted) message header and data blocks will be used automatically by new messages. Relying solely on self-packing, an SMB format message base will never "shrink" in size. This is not to say that it will continually "grow" in size, but that without specific packing procedures, deleted message blocks may remain unused for extended periods of time, meanwhile using some amount of storage space that could be recovered using specific packing procedures. The Fast Allocation and Hyper Allocation storage methods do not use deleted message blocks for new messages so specific packing procedures must be used if any messages are deleted and that storage space is to ever be recovered. Limiting the maximum age of messages in an SMB message base is another way to control the storage requirements. While maximum message age definition is optional, the definition of the maximum number of messages is not. Q. How many messages can be stored per SMB message base? A. Without considering storage limitations or message data lengths greater than 256, the theoretical maximum number of messages that can be stored in a single SMB message base is 16.7 million. Considering the variable length nature of message and header data, it is suggested that the system operator allow no more than 1 million messages per base. To determine an estimated maximum number of messages for a message base using the average message data length as a factor, use the following formula: 4.2 billion divided by the average message length rounded up to be evenly divisible by 256. If the average message data length is 1500 bytes, the estimated maximum number of messages would be 2,734,375 (4.2 billion divided by 1536). Implementations of this format may be further limited by available system memory. Implementation Levels ===================== &&Implementation Levels $$IMPLEVEL The SMB format can be implemented to varying degrees between programs without creating compatibilty issues. Rather than have developers specifically state which features they have and have not implemented, we have defined seven levels of implementation (represented by Roman numerals I through VII). For a program or software package to meet an implementation level, it must have all of the features listed for that level and all of those for each level below it. The minimum suggested imlementation is level I. The SMBUTIL program included with this specification is an example of a level I implementation with features from some of the higher implementation levels. Level I ------- The minimum suggested level of implementation. Messages contain merely ASCII text displayable on an ANSI terminal. Messages can be added to the message base and if the maximum number of messages is exceeded, messages are removed or marked for deletion. Level II -------- The addition of file attachments, multiple index/header entries per message (multiple destinations), multiple text bodies for the separation of message text and tag/origin lines (for example), forwarding, threading, and specific FidoNet kludge header field support makes this level of implementation more realistic for bulletin board system and EchoMail software implementation. Synchronet Multinode BBS Software v2.00 has a level II implementation of this specification. Level III --------- This implementation adds support for translation strings defined later in this document for data compression, encryption, escaping, and encoding. This level is still limited to basic ASCII text and ANSI escape sequence entry and retrieval. Synchronet Multinode BBS Software v2.10 has a level III implementation of this specification. Level IV -------- The storage and retrieval of embedded and attached images is added in this level of implementation. Supported images are limited to single binary or text data blocks that can be displayed or transferred to the user (automatically, or by request) if their display and translation protocols define specific support for the image type. Level V ------- This level of implementation adds support for embedded and attached sound data. This includes digitized sound and MIDI data. Supported sounds are limited to single binary or text data blocks that can be played or transferred to the user (automatically or by request) if their presentation and translation protocols define specific support for the sound type. Level VI -------- Localized sound and image data can be triggered by messages stored and retrieved in an implementation of this level. Level VII --------- Complete multimedia support is reached in this implementation level with support for embedded and attached animation, sound, and video data. Definitions =========== &&Definitions $$DEFINES_ Control Characters ------------------ When specifying control characters (ASCII 1 through 31), the caret symbol "^" or the abreviation "ctrl-" followed by a character will be used to indicate the value. ^A is equivalent to ASCII 1, ^B ASCII 2, etc. The case of the control character is not significant (i.e. ^z and ^Z are equivalent). The control character ^@ (ASCII 0) will be specified as NULL or 0. Hexadecimal ----------- Base sixteen numbering system which includes the digits 0-9 and A-F. Hexadecimal numbers are represented in this document with a prefix of "0x" or "\x" or a suffix of "h". Hexadecimal letter digits are not case sensitive (i.e. the number 0xff is the same as 0xFF). File dump --------- When example file dumps are displayed, the format is similar to that of the output from the DOS DEBUG program. With the exception of the ASCII characters, all numbers are in hexadecimal. Offset Byte values ASCII characters 000000 53 4D 42 1A 10 01 20 00 F4 01 00 00 F4 01 00 00 SMB... .ô...ô... 000010 20 00 00 00 D0 07 00 00 D0 07 00 00 00 00 00 00 ...Ð...Ð....... Bit values ---------- Bit (or flag) values are represented in C notation as (1<" Summary : Message-ID field as specified in RFC-822. Name : RFC822REPLYID Value : B2h Data : ASCII Multiple : No Required : No Format : "<" addr-spec ">" Summary : In-Reply-To field as specified in RFC-822. Name : UNKNOWN Value : F0h Data : undef Multiple : Yes Required : No Summary : Undefined header field of undefined type This field is useful for retaining binary header fields (that do not have an equivalent representation here) between message storage formats. Name : UNKNOWNASCII Value : F1h Data : ASCII Multiple : Yes Required : No Summary : Undefined header field of type ASCII This field is useful for retaining ASCII header fields (that do not have an equivalent representation here) between message storage formats. Name : UNUSED Value : FFh Data : undef Multiple : Yes Required : No Summary : Unused (deleted) header field The data contained in this header field is of an unknown type and should not be processed. Note: ---- Specifically, not defined are the values F000h through FFFFh. These values are to be used for user or system defined header fields. Digital Dynamics requests that any developers or organizations that wish to have additional header fields added to this specification notify Digital Dynamics through any of the contact methods listed at the beginning of this document. Data Field Types: ================ &&Data Field Types $$DFIELD_T These are the defined valid values for dfield_t.type: Val Name Data Description --- ---- ---- ----------- 00h TEXT_BODY mtext_t Displayable text (body of message). Included in duplicate message checking. All terminating white space and control characters are to be truncated from data (except when multiple contiguous CRLFs terminate the text, only the last CRLF is removed). 01h TEXT_SOUL mtext_t Non-displayed text. Not normally displayed. Not necessarily displayable. Included in duplicate message checking. 02h TEXT_TAIL mtext_t Displayable text (tag/tear/origin lines, etc). Not included in duplicate message checking. All terminating white space and control characters are to be truncated from data. 03h TEXT_WING mtext_t Non-displayed text. Not normally displayed. Not necessarily displayable. Not included in duplicate message checking. 10h FTEXT_BODY ftext_t Formatted equivalent of TEXT_BODY to be displayed in place of TEXT_BODY if format is supported. See Image Types for valid values of ftext_t.type. 12h FTEXT_TAIL ftext_t Formatted equivalent of TEXT_TAIL to be displayed in place of TEXT_TAIL if format is supported. See Image Types for valid values of ftext_t.type. 20h IMAGEEMBED membed_t Type and data of embedded raster image file for display. See Image Types for valid membed.type values. 21h ANIMEMBED membed_t Type and data of embedded graphical animation file for display. See Animation Types for valid membed.type values. 22h FONTEMBED membed_t Type and data of embedded font definition file. See Font Types for valid membed_t.type values. 23h SOUNDEMBED membed_t Type and data of embedded sound file for playback. See Sound Types for valid membed_t.type values. 24h PRESENTEMBED membed_t Type and data of embedded presentation definition file. See Present Types for valid membed_t.type values. 25h VIDEOEMBED vembed_t Type and data of embedded video/sound file for playback. See Video Types for valid vembed_t.type values. See Video Compression Types for valid vembed_t.comp values. 26h APPDATAEMBED membed_t Type and data of embedded application data file for process/display. See Application Data Types for valid membed_t.type values. FFh UNUSED undef Space allocated for future update/expansion Specifically, not defined are the values F000h through FFFFh. These values are to be used for user or system defined data fields. Digital Dynamics requests that any developers or organizations that wish to have additional data fields added to this specification notify Digital Dynamics through any of the contact methods listed at the beginning of this document. Message Attributes: ------------------ &&Message Attributes $$ATTRBITS These are the bit values for idxrec_t.attr and msghdr_t.attr: MSG_PRIVATE (1<<0) // Private MSG_READ (1<<1) // Read by addressee MSG_PERMANENT (1<<2) // Permanent MSG_LOCKED (1<<3) // Msg is locked, no editing possible MSG_DELETE (1<<4) // Msg is marked for deletion MSG_ANONYMOUS (1<<5) // Anonymous author MSG_KILLREAD (1<<6) // Delete message after it has been read MSG_MODERATED (1<<7) // This message must be validated before export MSG_VALIDATED (1<<8) // This message has been validated by a moderator Auxillary Attributes: -------------------- These are the bit values for msghdr_t.auxattr: MSG_FILEREQUEST (1<<0) // File request MSG_FILEATTACH (1<<1) // File(s) attached to Msg MSG_TRUNCFILE (1<<2) // Truncate file(s) when sent MSG_KILLFILE (1<<3) // Delete file(s) when sent MSG_RECEIPTREQ (1<<4) // Return receipt requested MSG_CONFIRMREQ (1<<5) // Confirmation receipt requested MSG_NODISP (1<<6) // Msg may not be displayed to user Network Attributes: ------------------ These are the bit values for msghdr_t.netattr: MSG_LOCAL (1<<0) // Msg created locally MSG_INTRANSIT (1<<1) // Msg is in-transit MSG_SENT (1<<2) // Sent to remote MSG_KILLSENT (1<<3) // Kill when sent MSG_ARCHIVESENT (1<<4) // Archive when sent MSG_HOLD (1<<5) // Hold for pick-up MSG_CRASH (1<<6) // Crash MSG_IMMEDIATE (1<<7) // Send Msg now, ignore restrictions MSG_DIRECT (1<<8) // Send directly to destination MSG_GATE (1<<9) // Send via gateway MSG_ORPHAN (1<<10) // Unknown destination MSG_FPU (1<<11) // Force pickup MSG_TYPELOCAL (1<<12) // Msg is for local use only MSG_TYPEECHO (1<<13) // Msg is for conference distribution MSG_TYPENET (1<<14) // Msg is direct network mail Translation Types: ----------------- &&Translation Types $$XLATTYPE Definition for values of *.xlat[x]: XLAT_NONE 0 // No translation/End of translation list XLAT_LF2CRLF 1 // Expand sole LF to CRLF XLAT_ESCAPED 2 // 7-bit ASCII escaping for ctrl and 8-bit data XLAT_HUFFMAN 3 // Static and adaptive Huffman coding compression XLAT_LZW 4 // LZW (Lempel-Ziv-Welch) encoding for compression // Terry Welch, IEEE Computer Vol 17, No 6 // June 1984, pp 8-19 XLAT_LZC 5 // LZC (modified LZW) encoding for compression // Unix compress program XLAT_RLE 6 // Run length encoding compression XLAT_IMPLODE 7 // Implode compression (PKZIP v1.x) XLAT_SHRINK 8 // Shrink compression (PKZIP v1.x) XLAT_LZH 9 // LZH dynamic Huffman coding // Haruyasu Yoshizaki, LHarc // November, 1988 Agent Types: ----------- &&Agent Types $$AGENTTYP AGENT_PERSON 0 // To or from person AGENT_PROCESS 1 // Unknown process, identified by agent name Agent types E000h through EFFFh are reserved for Synchronet process types (defined specifically by Digital Dynamics). Note: ---- Specifically not defined are agent types F000h through FFFFh. These values are to be used for user or system defined agent types. Digital Dynamics requests that any developers or organizations that wish to have additional agent types added to this specification notify Digital Dynamics through any of the contact methods listed at the beginning of this document. Network Types: ------------- &&Network Types $$NETWORKS // Net Type Address Format // ----------------------------------- NET_NONE 0 // Locally created none NET_UNKNOWN 1 // Unknown undef NET_FIDO 2 // FTN network fidoaddr_t NET_POSTLINK 3 // PostLink network none NET_QWK 4 // QWK based network ASCII NET_INTERNET 5 // The Internet ASCII NET_WWIV 6 // WWIV based network ulong Media Types: =========== &&Media Types $$MEDIATYP Image Types: ----------- IMAGE_UNKNOWN 0x00 // Use image signature header to determine format IMAGE_ASC 0x01 // ASCII text/IBM extended ASCII graphics IMAGE_ANS 0x02 // ANSI X3.64 terminal escape sequences IMAGE_AVT 0x03 // AVATAR terminal escape sequences IMAGE_LVI 0x04 // LVI terminal escape sequences IMAGE_GIF 0x05 // Compuserve Graphics Interchange Format (GIF) IMAGE_TIF 0x06 // Tagged Image Format (AKA TIFF) IMAGE_JPG 0x07 // Joint Photographers Electronics Group (JPEG) IMAGE_T16 0x08 // TrueVision 16-bit bitmap (TGA) IMAGE_T24 0x09 // TrueVision 24-bit bitmap (TGA) IMAGE_T32 0x0a // TrueVision 32-bit bitmpa (TGA) IMAGE_PCX 0x0b // ZSoft PaintBrush graphics IMAGE_BMP 0x0c // Windows bitmap IMAGE_RLE 0x0d // Windows bitmap (compressed) IMAGE_DIB 0x0e // Display independant bitmap IMAGE_PCD 0x0f // Kodak PhotoCD IMAGE_G3F 0x10 // Group 3 FAX IMAGE_EPS 0x11 // Ecapsulated PostScript IMAGE_RTF 0x12 // Rich text format IMAGE_RIP 0x13 // Remote Imaging Protocol Script (RIPscrip) IMAGE_NAP 0x14 // NAPLPS IMAGE_CDR 0x15 // Corel Draw! IMAGE_CGM 0x16 // Computer graphics metafile IMAGE_WMF 0x17 // Windows metafile IMAGE_DFX 0x18 // Autodesk AutoCAD IMAGE_IFF 0x19 // Amiga Interchange File Format Animation Types: --------------- ANIM_UNKNOWN 0 // Use file signature header to determine format ANIM_FLI 1 // Autodesk animator ANIM_FLC 2 // Autodesk ANIM_GL 3 // Grasprt ANIM_IFF 4 // Amiga Interchange File Format Video Types: ----------- VIDEO_UNKNOWN 0 // Use file signature header to determine format VIDEO_QTIME 1 // Apple Quick-time VIDEO_FQTIME 2 // Apple Flattened Quick-time VIDEO_AVI 3 // Windows Auto/Video Interleave VIDEO_ULT 4 // OS/2 Ultimotion Video Compression Types: ----------------------- VCOMP_UNKNOWN 0 // Use file signature header to determine codec VCOMP_RLE 1 // Apple animation VCOMP_SMC 2 // Apple graphics VCOMP_RPZA 3 // Apple video VCOMP_KLIC 4 // Captain crunch VCOMP_CVID 5 // CinePak VCOMP_RT21 6 // Intel indeo R2 VCOMP_IV31 7 // Intel indeo R3 VCOMP_YVU9 8 // Intel YVU9 VCOMP_JPEG 9 // JPEG VCOMP_MRLE 10 // Microsoft RLE VCOMP_MSVC 11 // Microsoft video 1 Font Types: ---------- FONT_UNKNOWN 0 // Use file signature header to determine format FONT_TTF 1 // Windows TrueType FONT_PFB 2 // PostScript Type 1 Font Binary FONT_PFM 3 // PostScript Type 1 Font Metric FONT_AMIGA 4 // Amiga Bitmapped FONT_AGFA 5 // CompuGraphic Fonts Sound Types: ----------- SOUND_UNKNOWN 0 // Use file signature header to determine format SOUND_MOD 1 // MOD format SOUND_VOC 2 // Sound Blaster VOC format SOUND_WAV 3 // Windows 3.1 WAV RIFF format SOUND_MID 4 // MIDI format SOUND_GMID 5 // General MIDI format (standardized patches) SOUND_SMP 6 // Turtle Beach SampleVision format SOUND_SF 7 // IRCAM format SOUND_AU 8 // Sun Microsystems AU format SOUND_IFF 9 // Amiga Interchange File Format Application Data Types: ---------------------- APPDATA_UNKNOWN 0 // Use file signature header to determine format APPDATA_WORDPERFECT 1 // WordPerfect Document APPDATA_WKS 2 // Lotus 123 Worksheet (?) APPDATA_WK1 3 // Lotus 123 Worksheet rev 1 APPDATA_WK2 4 // Lotus 123 Worksheet rev 2 APPDATA_WK3 5 // Lotus 123 Worksheet rev 3 APPDATA_DBF 6 // dBase III data file APPDATA_PDX 7 // Paradox data file APPDATA_EXCEL 8 // Excel data file APPDATA_QUATRO 9 // Borland Quatro Pro file APPDATA_WORD 10 // Microsoft Word Message Storage Pseudo Code =========================== &&Message Storage Pseudo Code $$STORPCOD The following is a "C like" pseudo code listing example of adding a message to an SMB message base. SMBLIB contains C functions to do most of the following operations. We are supplying this pseudo code as a general definition of the order of required operations in writing to the message base. Many details have been left out to simplify the code and to demonstrate only the basic principles. shd = open ( MSGBASE.SHD , READ/WRITE/DENY_NONE ) sdt = open ( MSGBASE.SDT , READ/WRITE/DENY_NONE ) sid = open ( MSGBASE.SDT , READ/WRITE/DENY_NONE ) lock ( shd , smbhdr ) read ( shd , smbstatus ) if ( smbstatus.attr & SMB_HYPERALLOC ) msg.hdr.offset = filelength ( sdt ) else { number_of_blocks = length_of_message_data / SDT_BLOCK_LEN if ( length_of_message_data % SDT_BLOCK_LEN ) /* unevenly divisible */ number_of_blocks = number_of_blocks + 1 sda = open ( MSGBASE.SDA , READ/WRITE/DENY_ALL ) if ( fast_allocation_mode ) seek ( sda , END_OF_FILE ) else { seek ( sda , BEGINNING_OF_FILE ) while ( not end_of_file ( sda ) ) { read ( sda , allocated , number_of_blocks * 2 ) if ( allocated = 0 ) { seek_backwards ( sda , number_of_blocks * 2 ) break } } } msg.hdr.offset = ( current_position ( sda ) / 2 ) * SDT_BLOCK_LEN allocated = 1 write ( sda , allocated , number_of_blocks * 2 ) close ( sda ) } seek ( sdt , msg.hdr.offset ) write ( sdt , message_data ) if ( smbstatus.attr & SMB_HYPERALLOC ) msg.idx.offset = filelength ( shd ) else { number_of_blocks = length_of_message_header / SHD_BLOCK_LEN if ( length_of_message_header % SHD_BLOCK_LEN ) /* unevenly divisible */ number_of_blocks = number_of_blocks + 1 sha = open ( MSGBASE.SHA , READ/WRITE/DENY_ALL ) if ( fast_allocation_mode ) seek ( sha , END_OF_FILE ) else { seek ( sha , BEGINNING_OF_FILE ) while ( not end_of_file ( sha ) ) { read ( sha , allocated , number_of_blocks ) if ( allocated = 0 ) { seek_backwards ( sha , number_of_blocks ) break } } } msg.idx.offset = ( current_position ( sha ) * SHD_BLOCK_LEN ) msg.idx.offset = msg.idx.offset + smbstatus.header_offset allocated = 1 write ( sha , allocated , number_of_blocks ) close ( sha ) } seek ( shd , msg.idx.offset ) msg.hdr.number = smbstatus.last_msg+1 write ( shd , msg.hdr ) smbstatus.total_msgs = smbstatus.total_msgs + 1 smbstatus.last_msg = msg.hdr.number write ( shd , smbstatus ) write ( sid , msg.idx ) unlock ( shd , smbstatus ) Message Retrieval Pseudo Code ============================= &&Message Retrieval Pseudo Code $$READPCOD shd = open ( MSGBASE.SHD , READ/WRITE/DENY_NONE ) sdt = open ( MSGBASE.SDT , READ/WRITE/DENY_NONE ) sid = open ( MSGBASE.SDT , READ/WRITE/DENY_NONE ) read ( sid , msg.idx ) seek ( shd , msg.idx.offset ) lock ( shd , msg.hdr ) read ( shd , msg.hdr ) seek ( sdt , msg.hdr.offset ) read ( sdt , msg.hdr.data_length ) unlock ( shd , msg.hdr ) SMBUTIL ======= &&SMBUTIL $$SMBUTIL_ SMBUTIL is a utility that can perform various functions on an SMB message base. The primary purpose of SMBUTIL is as an example to C programmers of how to use the SMBLIB functions to access and modify an SMB message base. The complete C source code for SMBUTIL is included and functions from it can be used or modified by developers at their own discretion. The following files make up SMBUTIL: SMBUTIL.EXE Compiled and linked for 16-bit DOS (ready to run) SMBUTIL.C C functions SMBUTIL.H C definitions and variable prototypes SMBUTIL.WAT Makefile for Watcom C/C++ (type wmake -f smbutil.wat) SMBUTIL.BOR Makefile for Borland C/C++ (type make -f smbutil.bor) The usage syntax is as follows: SMBUTIL [/opts] cmd smb_filespec.shd where cmd is one or more of the following: l[n] = list msgs starting at number n r[n] = read msgs starting at number n v[n] = view msg headers starting at number n k[n] = kill (delete) n msgs i = import from text file f s = display msg base status c = change msg base status m = maintain msg base - delete old msgs and msgs over max p[k] = pack msg base (k specifies minimum packable Kbytes) where opts is one or more of the following: a = always (force) packing z = set time zone (n=min +/- from UT or 'EST','EDT','CST',etc) and smb_filespec is the base filename or file specification (wildcards) for the message base. If wildcards are used, the ".SHD" extension must be specified. An example command line: SMBUTIL MP C:\SBBS\DATA\SUBS\*.SHD would maintain and pack all the message bases found in the C:\SBBS\DATA\SUBS directory. CHKSMB ====== &&CHKSMB $$CHKSMB__ CHKSMB is a utility that performs a comprehensive analysis of a message base to find any possible errors and calculate the number of packable bytes. It does not "fix" a message base if any errors are found, it only reports the specific errors (and exits with a non-zero error level). If any errors are reported, packing the message base with SMBUTIL may rebuild the damaged files. If that doesn't work, then use FIXSMB as a last resort. C source code for CHKSMB is also included as an example to programmers of how to use SMBLIB functions. The usage syntax is as follows: CHKSMB [/opts] smb_filespec.shd where opts is one or more of the following: q = quiet mode (no beeps) s = stop after an errored message base (for use with wildcards) p = pause after an errored message base (wait for key press) t = don't check for unsupported translation strings (faster) e = display extended information on corrupted messages An example command line: CHKSMB /QP C:\SBBS\DATA\SUBS\*.SHD would check all the message bases in the C:\SBBS\DATA\SUBS directory, without beeping on errors, and pausing after an errored message base. FIXSMB ====== &&FIXSMB $$FIXSMB__ FIXSMB is a utility that will rebuild the index and allocation files for a message base. Since the message headers are not necessarily stored sequentially, the order of the messages in the index may be changed when the index is rebuilt. Messages are also re-numbered, so only use this program if the index is corrupted and the messages are extremely important. C source code for FIXSMB is also included as an example to programmers of how to use SMBLIB functions. The usage syntax is as follows: FIXSMB [/M] smb_file An example command line: FIXSMB \SBBS\DATA\MAIL Only use the "/M" command line switch if fixing an older Synchronet e-mail message base (created with SBBS v2.1 or earlier). Once the SMB_EMAIL status attr is set ("SMBUTIL S" will report a status attr of 1), the "/M" is not required. SMBLIB ====== &&SMBLIB $$SMBLIB__ SMBLIB is a library of C functions for accessing and storing messages in an SMB format message base. It can eliminate much of the development time for developers that wish to use the library in whole or in part, or use the functions as examples for their own message base function library. The library consists of the following files: SMBDEFS.H Constant definitions, macros, and data types SMBLIB.H Library constants and function prototypes SMBLIB.C Function definitions SMBVARS.C Global variable definitions (doubles as declaration file) For developers to use this library with their program, they must include the "SMBLIB.H" header file at the top of each C file that uses any of the library functions, global variables, data types, macros, and constants. This can be done by simply adding the following line to each .C file: #include "smblib.h" If SMBLIB.H is included, there is no need to include SMBDEFS.H or SMBVARS.C. To link the library functions and variables with a main program, the files SMBVARS.OBJ and SMBLIB.OBJ must be linked with the main program .OBJ files. If the operating system is DOS, be sure that all .OBJ files are compiled for the same memory model. Example MAKEFILEs for compiling and linking SMBUTIL with Borland C/C++ (SMBUTIL.BOR) and Watcom C/C++ (SMBUTIL.WAT) are included. SMBDEFS.H ========= &&SMBDEFS.H $$SMBDEFS_ The SMBDEFS.H file contains important constant definitions and data types (also defined in this document). If ever this document and SMBDEFS.H are inconsistent with each other, then SMBDEFS.H is to be considered correct and this document in error. If such a discrepency is found, please notifiy Digital Dynamics so it can be corrected in a future revision of the specification. Most notable of the data types is a structure called smbmsg_t (not defined in this document). It contains the fixed and variable portions of a message's header record as well as convenience pointers to the sender's name (smbmsg_t.to), recipient's name (smbmsg_t.from), network addresses, and more. If multiple SENDER header fields are included (for example), then smbmsg_t.to will point to the last SENDER header field in the header record. Convenience pointers for other data items work in the same fasion if multiple header fields of the same type exist in the header record. Variables of the smbmsg_t data type (and pointers to variables of smbmsg_t type) are used as arguments to many of the SMBLIB functions. SMBVARS.C ========= &&SMBVARS.C $$SMBVARS_ The SMBVARS.C file contains definitions of the global variables used by the SMBLIB functions. It is a fairly small file since their are a small number of global variables (by design). This file is used for both definitions and declarations, so no "extern" declarations need to be made in developers source code as long as SMBVARS.C or (preferably) SMBLIB.H is included in the source code. SMBLIB.H ======= &&SMBLIB.H $$SMBLIB.H The SMBLIB.H file contains prototypes of all the functions in the SMBLIB.C file. It is necessary to include this file in C source code if any of the SMBLIB functions are used. The following C source line will include this file: #include "smblib.h" and should be placed near the top of all C source files that use SMBLIB functions, variables, constants, or data types. Function prototypes are necessary for compilers to know the correct calling syntax of a function and detect incorrect usage. Prototypes are also useful as a quick reference for programmers as to the correct calling syntax of a specific function. SMBLIB.C ======= &&SMBLIB.C $$SMBLIB.C The SMBLIB.C file contains the actual SMBLIB library functions. This source file is not a stand alone program, but instead must be compiled and linked with a main source file to create the executable program. The functions in this file are organized in a logical order, but their order is actually irrelevant to the compiling, linking, and execution of the resulting program. A comment block preceeds each function, explaining what the function does, how the passed parameters are used, and what the return code (if any) indicates. A more detailed explanation of each function is included here: int smb_open(int retry_time) ---------------------------- The smb_open() function must be called before the message base is accessed (read from or written to). The parameter, retry_time, is the maximum number of seconds to wait while retrying to lock the message base header. If retry_time is 0, then the message base header is not locked or read (this is called "Fast Open" and should only be used when speed is more important than checking for compatibility and validity upon opening). The global variable smb_file must be initialized with the path and base filename of the message base. This function returns 0 on success, 1 if the .SDT file could not be opened, 2 if the .SHD file could not be opened, and 3 if the .SID file could not be opened. If the message base header could not be locked, this function returns -1. If the message base ID is incorrect, it returns -2. And if the message base is of an incompatible version, it returns -3. The errno global variable (standard of most C libraries) will most likely contain the error code for open failure. int smb_open_da(int retry_time) ------------------------------- The smb_open_da() function is used to open the data block allocation file for writing messages to a message base. The parameter, retry_time, is the maximum number of seconds to wait while retrying to open the file. This function returns 0 on success. -1 is returned if an open error other than "Access Denied" is returned from the operating system, and the global variable errno will contain the error code. -2 is returned if the retry_time has been reached, and -3 is returned if the file descriptor could not be converted to a stream by the fdopen() function. fclose(sda_fp) should be called immediately after all necessary file access has been completed. This function is not used with the Hyper Allocation storage method. int smb_open_ha(int retry_time) ------------------------------- The smb_open_ha() function is used to open the header block allocation file for writing messages to a message base. The parameter, retry_time, is the maximum number of seconds to wait while retrying to open the file. This function returns 0 on success. -1 is returned if an open error other than "Access Denied" is returned from the operating system, and the global variable errno will contain the error code. -2 is returned if the retry_time has been reached, and -3 is returned if the file descriptor could not be converted to a stream by the fdopen() function. fclose(sha_fp) should be called immediately after all necessary file access has been completed. This function is not used with the Hyper Allocation storage method. int smb_create(ulong max_crcs, ulong max_msgs, ushort max_age, ushort attr ,int retry_time) -------------------------------------------------------------------------- The smb_create() function is used to create a new message base or reset an existing message base. The parameters max_crcs, max_msgs, max_age, and attr are used to set the initial status of the message base status header. The parameter, retry_time is the maximum number of seconds to wait while retrying to lock the message base header. This functions returns 0 on success or 1 if the message base header could not be locked. int smb_trunchdr(int retry_time) -------------------------------- The smb_trunchdr() function is used to truncate the header file when packing the message base and writing the new header information back to the header file. The parameter, retry_time is the maximum number of seconds to wait while retrying to truncate the header file. Returns 0 on success, -1 if error was other than "Access Denied", or -2 if retry_time reached. int smb_locksmbhdr(int retry_time) ---------------------------------- The smb_locksmbhdr() function is used to lock the first message base (status) header. The parameter, retry_time is the number of seconds to wait while retrying to lock the header. The smb_unlocksmbhdr() function should always be used to unlock the header after accessing the message base header (usually with smb_getstatus() and/or smb_putstatus()). Returns 0 if successful, -1 if unsuccessful. int smb_unlocksmbhdr() ---------------------- The smb_unlocksmbhdr() function is used to unlock a previously locked message base header (using smb_lockmsghdr()). Returns 0 on success, non-zero on failure. int smb_getstatus(smbstatus_t *hdr) ----------------------------------- The smb_getstatus() function is used to read the status message base header into the hdr structure. Returns 0 on success, 1 on failure. int smb_putstatus(smbstatus_t hdr) ---------------------------------- The smb_putstatus() function is used to write the status information to the first message base header. The parameter hdr, contains the status information to be written. Returns 0 on success, 1 on failure. int smb_getmsgidx(smbmsg_t *msg) -------------------------------- The smb_getmsgidx() function is used to get the byte offset for a specific message header in the message header file based on the message base index. If msg->hdr.number is non-zero when this function is called, then the index will be searched for this message number. If the message number is found in the index, the msg->idx.offset is set to the byte offset of the message header record in the header file and msg->offset is set to the record offset of the index record in the index file, and the function returns 0. If the message number is not found in the index, the function returns 1. If msg->hdr.number is zero, msg->idx.offset and msg->idx.number are obtained from the index record at record offset msg->offset. If msg->offset is an invalid record offset when this function is called, the function returns 1. Otherwise, the function returns 0. int smb_getlastidx(idxrec_t *idx) --------------------------------- Reads the last index record of the currently open message base into the idxrec_t structure pointed to by idx. Returns 0 if successful, -1 if the index is empty or unopened, or -2 if the record can't be read. int smb_getmsghdrlen(smbmsg_t msg) ---------------------------------- The smb_getmsghdrlen() function is used to calculate the total length of message header msg including both fixed and variable length portions. This function returns the length of the header record in bytes. long smb_getmsgdatlen(smbmsg_t msg) ----------------------------------- The smb_getmsgdatlen() function is used to calculate the total length of the data for message msg. This function returns the length of all data fields combined. int smb_lockmsghdr(smbmsg_t msg, int retry_time) ------------------------------------------------ The smb_lockmsghdr() function is used to lock the header record for message msg. The parameter retry_time is the maximum number of seconds to wait while retrying to lock the header. Returns 0 on success, -1 on failure. The function smb_unlockmsghdr() should immediately be called after accessing the message header (usually with smb_getmsghdr() or smb_putmsghdr()). int smb_getmsghdr(smbmsg_t *msg) -------------------------------- The function smb_getmsghdr() is used to read the header record for message msg. msg->idx.offset must be initialized to the byte offset of the header record in the header file before this function is called. The function smb_freemsgmem() must be called to free the memory allocated by this function for the header and data felds. This function returns 0 on success, -1 if the fixed portion of the message header record could not be read, -2 if the message header ID was incorrect, -3 if memory could not be allocated, -4 if a data field could not be read, -5 if the fixed length portion of a header field could not be read, -6 if the variable length portion of a header field could not be read, -7 if one or more of the mandatory header fields (SENDER, RECIPIENT, or SUBJECT) are missing, -8 if total_dfields extends beyond the end of the header record, or -9 if incompatible header version. Several convenience pointers in the msg structure are initialized by this function to point to the last occurance of the SENDER (msg->from), RECIPIENT (msg->to), SUBJECT (msg->subj), etc. int smb_unlockmsghdr(smbmsg_t msg) ---------------------------------- The smb_unlockmsghdr() function is used to unlock a previously locked message header (with smb_lockmsghdr()). This function returns 0 on success, non-zero on failure. int smb_addcrc(ulong max_crcs, ulong crc, int retry_time) --------------------------------------------------------- The smb_addcrc() function is used to add a CRC-32 to the CRC history file for a message base, automatically checking for duplicates. The parameter max_crcs should be the max_crcs defined in the status header of the message base. The parameter crc, is the CRC-32 of the TEXT_BODY and TEXT_SOUL data fields for the message. The parameter retry_time is the maximum number of seconds to wait when retrying to open the CRC history file. This function returns -1 if there was an open error, -2 if the retry_time was reached, -3 if there was a memory allocation error, 1 if the CRC already exists in the CRC history file (indicating a duplicate message), or 0 on success (and no duplicate). int smb_hfield(smbmsg_t *msg, ushort type, ushort length, void *data) --------------------------------------------------------------------- The smb_hfield() function is used to add a header field to the structure msg. The parameters type, length, and data, must be specified according to the header field values listed in this specification. This function returns 0 on success, non-zero on memory allocation error. The function smb_freemsgmem() must be called to free the memory allocated by this function. int smb_dfield(smbmsg_t *msg, ushort type, ulong length) -------------------------------------------------------- The smb_dfield() function is used to add a data field to the structure msg. The parameters type and length must be specified according to the data field values listed in this specification. This function returns 0 on success, non-zero on memory allocation error. The function smb_freemsgmem() must be called to free the memory allocated by this function. int smb_addmsghdr(smbmsg_t *msg,smbstatus_t *status,int storage,int retry_time) ------------------------------------------------------------------------------- The smb_addmsghdr() function is used to add a new message header to the message header file and update the index file. The msg and status structures are updated to reflect the new total messages, last message number, etc. The storage parameter is used to indicate the storage method to use (either SMB_SELFPACK, SMB_FASTALLOC, or SMB_HYPERALLOC). If the storage type is SMB_SELFPACK, the header block allocation file will be searched for unused block(s) to store this header. If the storage type is SMB_FASTALLOC or SMB_HYPERALLOC, the header is stored at the end of the header file. Returns 0 on success, non-zero on failure. The parameter retry_time is the maximum number of seconds to wait while retrying to lock and open files. int smb_putmsg(smbmsg_t msg) ---------------------------- The smb_putmsg() function calls both the smb_putmsghdr() and smb_putmsgidx() functions to write the header and index elements of a message to the appropriate files. Returns 0 on success, non-zero on failure. int smb_putmsgidx(smbmsg_t msg) ------------------------------- The smb_putmsgidx() function is used to store a message index in the message index file. The message index can be for a new message or an existing message. Returns 0 on success, non-zero on failure. int smb_putmsghdr(smbmsg_t msg) ------------------------------- The smb_putmsghdr() function is used to store a message header in the message header file. The message header can be for a new message or an existing message. Returns 0 on success, non-zero on failure. void smb_freemsgmem(smbmsg_t msg) --------------------------------- Frees allocated memory for the header and data fields in the msg structure. This function must be called to free the memory allocated by the functions smb_hfield(), smb_dfield(), and smb_getmsghdr(). long smb_hdrblocks(ulong length) -------------------------------- The smb_hdrblocks() function is used to calculate the number of blocks required to store a message header of length size (in bytes). This function returns the number of blocks required. long smb_datblocks(ulong length) -------------------------------- The smb_datblocks() function is used to calculate the number of blocks required to store message data of length size (in byte). This function returns the number of blocks required. long smb_allochdr(ulong length) ------------------------------- The smb_allochdr() function is used to search for free blocks to store a message header of length bytes and mark the free blocks as allocated in the header allocation file. This function returns the byte offset to the header record or a negative number on error. The function smb_open_ha() should be called prior to calling this function and fclose(sha_fp) should be called after. The function is called from smb_addmsghdr(), so you probably have no need to call this function directly. long smb_fallochdr(ulong length) -------------------------------- The smb_fallochdr() function works exactly the same as the smb_allochdr() function except it is much faster because the header allocation file is not searched for free blocks. The function is called from smb_addmsghdr(), so you probably have no need to call this function directly. long smb_hallochdr(ulong header_offset) --------------------------------------- This smb_hallochdr() functions works exactly the same as the smb_fallochdr() function except the status.header_offset is passed as the argument and the header allocation (.SHA) file is not updated so smb_open_ha() need not be called. The function is called from smb_addmsghdr(), so you probably have no need to call this function directly. long smb_allocdat(ulong length, ushort headers) ----------------------------------------------- The smb_allocdat() function is used to search for free blocks to store length amount of data for a message. The parameter headers, indicates the number of message headers that are associated with this data. Normally, the headers parameter will be 1, unless this message is part of a mass mailing. The offset to the allocated data blocks is returned, or a negative value on error. The function smb_open_da() should be called prior to calling this function and fclose(sda_fp) should be called after. long smb_fallocdat(ulong length, ushort headers) ------------------------------------------------ The smb_fallocdat() function works exactly the same as the smb_allocdat() function except it is much faster because the data allocation file is not searched for free blocks. long smb_hallocdat() -------------------- The smb_hallocdat() function works exactly the same as the smb_hallocdat() function except no argument is passed and the data allocation file (.SDA) is not updated so smb_open_da() need not be called. int smb_incdat(ulong offset, ulong length, ushort headers) ---------------------------------------------------------- The smb_incdat() function is used to increment the header counter in the data allocation file for the data starting at the byte offset and length size in bytes. The parameter headers, indicates the number of headers to add to the current allocation value in the data allocation file. Returns 0 on success, non-zero on failure. int smb_freemsg(smbmsg_t msg, smbstatus_t status) ------------------------------------------------- The smb_freemsg() function is used to free the disk space allocated for the header and data fields of the message msg. Returns 0 on success, non-zero on failure. The parameter, status, must be the current status from the message base header for this message base. int smb_freemsgdat(ulong offset, ulong length, ushort headers) -------------------------------------------------------------- The smb_freemsgdat() function is used to decrement the data block allocation records in the data allocation file associated with the data in the data file by the value of the headers parameter (normally 1). The parameter offset indicates the byte offset to the beginning of the message data in the data file and the parameter length is the total length of the message data. Returns 0 on success, non-zero on failure. int smb_freemsghdr(ulong offset, ulong length) ---------------------------------------------- The smb_freemsghdr() function is used to set the header block allocation records in the header allocation file to 0 (indicated non-allocated block). The parameter offset indicates the byte offset to the beginning of the header record being freed and the parameter length indicates the total length of the header record. Returns 0 on success, non-zero on failure. int smb_stack(int op) --------------------- The smb_stack() function is used to save and restore message base information so that multiple message bases can be open simultaneously. The stack can save up to 4 message bases (allowing 5 simultaneously open message bases). The stack is a "last in, first out" storage area for open message bases. If the op parameter is SMB_STACK_PUSH, smb_stack() will save (push) the current message base onto the stack. Calling smb_stack(SMB_STACK_POP) will restore (pop) the most recently pushed message base off the stack. Calling smb_stack(SMB_STACK_XCHNG) will exchange the most recently pushed message base and the current message base (replacing the top of the stack with the current message base). void smb_close() ---------------- Closes the header, data, and index files for the currently open message base. Miscellaneous SMBLIB Files ========================== &&Miscellaneous SMBLIB Files $$SMB_MISC CRC32.H C header file for CRC-32 calculations ----------------------------------------------------- This file contains a static 32-bit CRC table (crc32tbl[]) and a macro (ucrc32) that uses this table to calculate 32-bit CRCs one byte at a time. Example: ulong crc=0xffffffff; for(i=0;i msgbase[0] // 0 opened msg[1] --> msgbase[1] // 0 pushed 1 opened msg[2] --> msgbase[1] msg[3] --> msgbase[2] // 1 closed 0 popped 0 closed 2 opened msg[4] --> msgbase[0] // 2 pushed 0 opened msg[5] --> msgbase[2] // 0 pushed 2 popped (exchanged) msg[6] --> msgbase[3] // 2 closed 0 popped 0 closed 3 opened msg[7] --> msgbase[0] // 3 pushed 0 opened The second example would be of significant performance gain over the first example (4 open operations versus 8) if the messages to import were in the following order: msg[0] --> msgbase[0] // 0 opened msg[1] --> msgbase[1] // 0 pushed 1 opened msg[2] --> msgbase[0] // 1 pushed 0 popped (exchanged) msg[3] --> msgbase[1] // 0 pushed 1 popped (exchanged) msg[4] --> msgbase[0] // 1 pushed 0 popped (exchanged) msg[5] --> msgbase[2] // 0 pushed 1 popped (exchanged) 1 closed 2 opened msg[6] --> msgbase[3] // 2 pushed 0 popped (exchanged) 0 closed 3 opened msg[7] --> msgbase[2] // 3 pushed 2 popped (exchanged) More advanced use of "stack-like" message base file handle storage can easily reduce the number of open operations, therefore increasing import performance under more adverse message base ordering conditions. Compression ----------- If any message data compression features are offered by the application, it is important the the application not unnecessarily compress data that will not save any storage space. While this may seem an obvious statement, please review the following pseudo-code example: if ( message_data_length < SDT_BLOCK_LEN ) // Store uncompressed data else { // Compress data if ( ( compressed_data_length / SDT_BLOCK_LEN ) < ( message_data_length / SDT_BLOCK_LEN ) ) // Saves a block or more // Store compressed data else // Store uncompressed data } Since the SMB format stores message data in fixed length blocks, there is no point in storing a message in compressed format if it requires the same number of blocks as the uncompressed format (i.e. a message that is two blocks in length in uncompressed format and only a block and a half in length when compressed should not be stored in compressed format since it still requires two full blocks of storage). It is important to note that in the above example, the length of the data translation string was not taken into account in determining the number of required blocks. Also, the smb_datblocks() function is normally used in determing the number of required blocks to store a given data length and it is a little more involved than simply dividing the length of the data by SDT_BLOCK_LEN. Bibliography ============ &&Bibliography $$BIBLIOGR Title : The C Programming Language Publisher : Prentice Hall Author : Brian W. Kernighan and Dennis M. Ritchie Document : ARPANET Request for Comments (RFC) #822 Title : Standard for the Format of ARPA Internet text messages Publisher : SRI International Author : David H. Crocker, University of Delaware Document : FTS-0001 Publisher : FSC Author : Randy Bush, Pacific Systems Group Document : FTS-0004 Title : EchoMail Specification Publisher : FSC Author : Bob Hartman Document : FTS-0009 Title : A standard for unique message identifiers and reply chain linkage Publisher : FSC Author : Jim Nutt Document : FSC-00046 Title : A Product Idenfifier for FidoNet Message Handlers Publisher : FSC Author : Joaquim H. Homrighausen Document : FSC-00053 Title : Specifications for the ^aFLAGS field Publisher : FSC Author : Joaquim H. Homrighausen Implementations =============== &&Implementations $$IMPLEMEN Product : Synchronet Multinode BBS Software Developer : Digital Dynamics Level : III Version : 2.10 Product : Synchronet/FidoNet Import/Export Utility (SBBSFIDO) Developer : Digital Dynamics Level : III Version : 2.10 Product : Synchronet UTI (Universal Text Interface) Driver Developer : Digital Dynamics Level : III Version : 2.10 Product : SBBSecho FidoNet Packet Tosser for Synchronet Developer : Digital Dynamics Level : III Version : 1.00 Product : NetXpress Internet UUCP for Synchronet Developer : Merlin Systems Level : II Version : 1.00 Product : InterEcho FidoNet Packet Tosser Developer : InterMail Sales Inc Level : II Version : 1.07