Synchronet Message Base Specification Version 1.00 Updated 01/04/94 Copyright 1994 Digital Dynamics PO Box 501 Yorba Linda, CA 92686 Voice: 714-529-6328 BBS: 714-529-9525 2400bps FAX: 714-529-9721 529-5313 V.32/V.32bis FIDO: 1:103/705 529-9547 HST RIME: 5115 529-9721 ZyXEL Table of Contents ================= &&Contents Introduction....................................................@@INTRO___ Implementation Levels...........................................@@IMPLEVEL Definitions.....................................................@@DEFINES_ Acronyms................................................@@ACRONYMS Data Types..............................................@@DATATYPE File Formats....................................................@@FILEFORM Index.....................(*.SID).......................@@SID_FORM Header....................(*.SHD).......................@@SHD_FORM Header Allocation.........(*.SHA).......................@@SHA_FORM Data......................(*.SDT).......................@@SDT_FORM Data Allocation...........(*.SDA).......................@@SDA_FORM CRC History...............(*.SCH).......................@@SCH_FORM Header Field Types..............................................@@HFIELD_T Data Field Types................................................@@DFIELD_T Messsage Attributes.............................................@@ATTRBITS Translation Types...............................................@@XLATTYPE Agent Types.....................................................@@AGENTTYP Network Types...................................................@@NETWORKS Media Types.....................................................@@MEDIATYP Message Storage Protocol........................................@@STORPROT Message Retrieval Protocol......................................@@READPROT SMBUTIL.........................................................@@SMBUTIL_ SMBLIB (C library)..............................................@@SMBLIB__ Data Types and Constants..(SMBDEFS.H)...................@@SMBDEFS_ Global Variables..........(SMBVARS.C)...................@@SMBVARS_ Function Prototypes.......(SMBLIB.H)....................@@SMBLIB.H Library Functions.........(SMBLIB.C)....................@@SMBLIB.C Bibliography....................................................@@BIBLIOGR Implementations.................................................@@IMPLEMEN Introduction ============ &&Introduction $$INTRO___ Q. What is SMB? A. SMB (Synchronet Message Base) is a technical specification for the storage format of electronic mail messages. These e-mail messages may all be contained in one database, or, more commonly, separated into catagorized databases. These message databases (or message bases) are also referred to as 'sub-boards', 'forums', 'conferences,' and 'SIGs'. The messages may be directed to an individual person, sent to a group of individuals, or sent to everyone who can read messages in that message base. Messages may be created and read soley at one physical location, or imported from and exported to a message network that may span continents. Message bases that are connected to a message network are often called 'echoes'. Q. Why SMB? A. The Synchronet Message Base is designed to store high volumes of messages while maintaining optimum search, retrieval, and creation performance. These messages are not defined as merely text. In addition to text, SMB defines the storage of digitized sound, midi, graphics, fonts, animation, as well as other multimedia data and triggers for localized multimedia. SMB thrives on a multi-user environment where messages are being created, read, modified, and deleted by multiple tasks simultaneously. With the large message networks of today being the rule, rather than the exception, and high volumes of messages being imported on a daily, sometimes hourly basis, creation and deletion speed is of the utmost importance. This is where SMB really shines. Being extensible enough to handle message formats from networks of today and tomorrow, and fast enough to import more messages that humanly readable, the SMB format will more than meet your message storage needs. Q. Why a specification? A. Message bases are often accessed and modified by a number of different programs. Often these programs are developed by individuals or companies other than the original designer of the message base format. This specification is an attempt to aide developers in creating programs that access or modify a message base stored in the SMB format. Q. Who can use this specification? A. Anyone that has interest in the Synchronet Message Base format at either an educational or professional level. Specifically, software developers interested or currently involved in the development of message readers, editors, echomail (toss/scan) programs, message transfer agents (MTAs), network gateways, and bulletin board systems. Much of the imformation in this specification is intended for those with preexisting programming knowledge, so those with little or no programming experience may find it hard to comprehend. Q. What does the SMB specification include? A. The text you are reading is part of the SMB specification: a single text document that defines the storage format of each of the six files of an SMB format message base and how they are related to each other. Included with this specification is C source code to be used as an example to programmers of how to access an SMB format message base and public domain library functions (SMBLIB) that can be compiled and linked into programs that access an SMB format message base developed by third parties. An SMB utility program (SMBUTIL) is also included with C source code as an example of how to use the SMBLIB functions. Q. Where did the SMB specification come from? A. Digital Dynamics (a software development company in southern California) released "Synchronet Multinode BBS Software Version 1a" in June of 1992 as one of the fist BBS packages to be designed from the ground-up to operate in a multinode environment with incredible speed and reliability, with a large suite of multinode specific features and design innovations. The original message base format was designed with localized messaging and low volume message networks in mind. By January of 1993, it was clear that high volume message networks (FidoNet, RelayNet, Usenet, etc.) were the preference of most BBS users and a new message base format was required to allow for high volume message storage, improved storage, retrieval, and maintenance performance, as well as lower storage space requirements. Rather than introduce another new message format, Digital Dynamics sought to implement an existing public specificatoin for a format that would meet current and future message storage needs. More than a few specifications were seriously considered at one time or another, but after careful examination, design flaws and lack of extensibility eliminated them from the long term plans of Digital Dynamics and Synchronet BBS Software. Thus began the design of the "Synchronet Message Base" (SMB) format. At the request of many message related program developers, Digital Dynamics created and released the SMB specification before the release of "Synchronet Version 2.00" to allow lead-time on developing support programs for the new format. Digital Dynamics strongly encourages developers of message related programs (including software that directly competes with Synchronet or other Digital Dynamics products) to implement support for SMB. Though this is a public specification and Digital Dynamics encourages developer suggestions, it will remain under the sole control of Digital Dynamics unless specifically stated otherwise in a future revision of this specification. Digital Dynamics requests that any organizations that wish to adopt or ratify this specification, in part or whole, notify Digital Dynamics through any of the contact methods listed at the beginning of this document. Q. How does SMB store messages? A. Each message base is stored in a set of six binary files. The base filename (maximum of eight characters) is the same for all six files of the same message base and unique amoung the filenames of other message bases. The six files each have a different three character extension. The first character of the extension is always the letter 'S' (for SMB), while the second and third characters define the contents of the file. Two of the six files associated with each message base are not recreatable and therefore are the most important when considering data integrity. These two files are the data file (with a .SDT extension) and variable length header file (.SHD extension). Both of these files use 256 byte blocks and have associated block allocation tables (stored in .SDA and .SHA respectively) so that deleted message blocks may be used by new messages without creating odd sized unused 'holes' in the files. The block allocation table files (.SDA and .SHA) can be recreated with the information stored in the header (.SHD) file. For fast indexing, there is a small fixed length index file (with a .SID extension). This file allows for the immediate location of message indices based on sender's name, recipient's name, subject, message number, or message attributes. This file can be recreated with the data stored in the header (.SHD) file. The last file is an optional CRC history (.SCH) file. It contains 32-bit CRCs of a configurable number of messages imported or created locally. This is to help eliminate duplicate messages created by user or program error. The CRC history file can be recreated with the combination of information stored in the data (.SDT) and header (.SHD) files. Q. How fast do messages import into an SMB message base? A. This is a very important question for systems for that import large volumes of messages. Of course, the answer depends on the storage format which you are importing from, the average length of messages, the design of the program which is peforming the import process, as well as the hardware and system software being used. What's important is that SMB will allow the fastest import process possible with any given combination of the above factors. Since system storage capacity is rarely infinite, neither is the number of messages which can be stored. System operators must define the maximum number of messages to be stored in a message base, the maximum age of the messages in that message base, or a combination of both. Generally, the smaller the number of messages stored in a message base, the faster the import process. The SMB format is flexible enough to support multiple levels of import performance based on optimizations for storage space or speed. Most system operators will almost invariably choose speed over space, but which choices are available is determined by the importing program. Q. How much storage is required for an SMB message base? A. The biggest factor in determining storage requirements for a message base is the maximum number of messages to be stored in the base (defined by the system operator) and the average size of each message. The minimum required storage for a message base is 32 bytes plus 531 bytes per message (535 bytes per message if duplicate message checking is used). The SMB format is designed to be "self-packing", meaning purged (deleted) message header and data blocks will be used automatically by new messages. Relying solely on self-packing, an SMB format message base will never "shrink" in size. This is not to say that it will continually "grow" in size, but that without specific packing procedures, deleted message blocks may remain unused for extended periods of time, meanwhile using some amount of storage space that could be freed using specific packing procedures. Limiting the maximum age of messages in an SMB message base is another way to control the storage requirements. While maximum message age definition is optional, the definition of the maximum number of messages is not. Q. How many messages can be stored per SMB message base? A. Without considering storage limitations or message data lengths greater than 256, the theoretical maximum number of messages that can be stored in a single SMB message base is 16.7 million. Considering the variable length nature of message and header data, it is suggested that the system operator allow no more than 1 million messages per base. To determine an estimated maximum number of messages for a message base using the average message data length as a factor, use the following formula: 4.2 billion divided by the average message length rounded up to be evenly divisble by 256. If the average message data length is 1500 bytes, the estimated maximum number of messages would be 2,050,781 (4.2 billion divided by 2048). Implementation Levels ===================== &&Implementation Levels $$IMPLEVEL The SMB format can be implemented to varying degrees between programs without creating compatibilty issues. Rather than have developers specifically state which features they have and have not implemented, we have defined five levels of implementation (represented by Roman numerals I, II, III, IV, and V). For a program or software package to meet an implementation level, it must have all of the features listed for that level and all of those for each level below it. The minimum suggested imlementation is level I. The SMBUTIL program included with this specification is an example of a level I implementation. Level I ------- The minimum suggested level of implementation. Messages contain merely ASCII text displayable on an ANSI terminal. Messages can be added to the message base and if the maximum number of messages is exceeded, messages are removed or marked for deletion. The SMBUTIL program included with this specification, is the perfect example of level I implementation. Level II -------- The addition of file attachments, multiple index/header entries per message (multiple destinations), multiple text bodies for the separation of message text and tag/origin lines (for example), forwarding, threading, and specific FidoNet kludge header field support makes this level of implementation more realistic for bulletin board system and EchoMail software implementation. Synchronet Multinode BBS Software v2.00 has a level II implementation of this specification. Level III --------- This implementation adds support for translation strings defined later in this document for data compression, encryption, escaping, and encoding. This level is still limited to basic ASCII text and ANSI escape sequence entry and retrieval. Level IV -------- The storage and retrieval of embedded and attached images is added in this level of implementation. Supported images are limited to single binary or text data blocks that can be displayed or transferred to the user (automatically, or by request) if their display and translation protocols define specific support for the image type. LEVEL V ------- This level of implementation adds support for embedded and attached sound data. This includes digitized sound and midi data. Supported sounds are limited to single binary or text data blocks that can be played or transferred to the user (automatically or by request) if their presentation and translation protocols define specific support for the sound type. LEVEL VI -------- Localized sound and image data can be triggered by messages stored and retrieved in an implementation of this level. LEVEL VII --------- Complete multimedia support is reached in this implementation level with support for embedded and attached animation, sound, and video data. Definitions =========== &&Definitions $$DEFINES_ Control Characters ------------------ When specifying control characters (ASCII 1 through 31), the caret symbol "^" or the abreviation "ctrl-" followed by a character will be used to indicate the value. ^A is equivalent to ASCII 1, ^B ASCII 2, etc. The case of the control character is not significant (i.e. ^z and ^Z are equivalent). The control character ^@ (ASCII 0) will be specified as NULL or 0. Hexidecimal ----------- Base sixteen numbering system which includes the digits 0-9 and A-F. Hexidecimal numbers are represented in this document with a prefix of "0x" or "\x" or a suffix of "h". Hexidecimal letter digits are not case sensitive (i.e. the number 0xff is the same as 0xFF). File dump --------- When example file dumps are displayed, the format is similar to that of the output from the DOS DEBUG program. With the exception of the ASCII characters, all numbers are in hexidecimal. Offset Byte values ASCII characters 000000 53 4D 42 1A 00 01 20 00 F4 01 00 00 F4 01 00 00 SMB... .ô...ô... 000010 20 00 00 00 D0 07 00 00 D0 07 00 00 00 00 00 00 ...Ð...Ð....... Bit values ---------- Bit (or flag) values are represented in C notation as (1<.SID Record ³ ³ ÉÍÑÍËÍÑÍËÍÑÍËÍÑÍËÍÑÍÑÍÑÍËÍÑÍÑÍÑÍ» ³ ³ ºT³TºF³FºS³SºA³AºO³O³O³OºN³N³N³Nº ³ ³ ÈÍÏÍÊÍÏÍÊÍÏÍÊÍÏÍÊÍÏÍÏÍÏÍÊÍÏÍÏÍÏͼ ³ ³ T = CRC-16 of to user ³ ³ F = CRC-16 of from user ³ ³ S = CRC-16 of subject ³ ³ A = Attributes (flag bits) ³ ³ O = Offset to header record ³ ³ N = Message number (1 based) ³ ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ Example file dump (16 messages starting with message number 1): -------------------------------------------------------------- 000000 36 4F 2B 7F 5E F5 00 00 20 00 00 00 01 00 00 00 6O+^õ.. ....... 000010 4C C7 5E 56 22 7F 00 00 20 01 00 00 02 00 00 00 LÇ^V".. ....... 000020 04 BE 96 82 CA 9B 00 00 20 02 00 00 03 00 00 00 .¾–‚Ê›.. ....... 000030 D8 2E B5 6C 0C 00 00 00 20 03 00 00 04 00 00 00 Ø.µl.... ....... 000040 7B 82 EA C4 FF CD 00 00 20 04 00 00 05 00 00 00 {‚êÄÿÍ.. ....... 000050 32 2A 53 CD 9F 82 00 00 20 05 00 00 06 00 00 00 2*SÍŸ‚.. ....... 000060 D2 E2 7E 7E A0 4C 00 00 20 06 00 00 07 00 00 00 Òâ~~ L.. ....... 000070 08 97 7E 7E AE F9 00 00 20 07 00 00 08 00 00 00 .—~~®ù.. ....... 000080 74 93 7E 7E 49 E6 00 00 20 08 00 00 09 00 00 00 t“~~Iæ.. ....... 000090 36 4F 57 16 4E 67 00 00 20 09 00 00 0A 00 00 00 6OW.Ng.. ....... 0000A0 D1 3C 56 F3 13 0B 00 00 20 0A 00 00 0B 00 00 00 Ñ" Summary : Message-ID field as specified in RFC-822. Name : RFC822REPLYID Value : B2h Data : ASCII Multiple : No Required : No Format : "<" addr-spec ">" Summary : In-Reply-To field as specified in RFC-822. Name : UNKNOWN Value : F0h Data : undef Multiple : Yes Required : No Summary : Undefined header field of undefined type This field is useful for retaining binary header fields (that do not have an equivalent representation here) between message storage formats. Name : UNKNOWNASCII Value : F1h Data : ASCII Multiple : Yes Required : No Summary : Undefined header field of type ASCII This field is useful for retaining ASCII header fields (that do not have an equivalent representation here) between message storage formats. Name : UNUSED Value : FFh Data : undef Multiple : Yes Required : No Summary : Unused (deleted) header field The data contained in this header field is of an unknown type and should not be processed. Note: ---- Specifically, not defined are the values F000h through FFFFh. These values are to be used for user or system defined header fields. Digital Dynamics requests that any developers or organizations that wish to have additional header fields added to this specification notify Digital Dynamics through any of the contact methods listed at the beginning of this document. Data Field Types: ================ &&Data Field Types $$DFIELD_T These are the defined valid values for dfield_t.type: Val Name Data Description --- ---- ---- ----------- 00h TEXT_BODY mtext_t Displayable text (body of message). Included in duplicate message checking. All terminating white space and control characters are to be truncated from data (except when multiple contiguous CRLFs terminate the text, only the last CRLF is removed). 01h TEXT_SOUL mtext_t Non-displayed text. Not normally displayed. Not necessarily displayable. Included in duplicate message checking. 02h TEXT_TAIL mtext_t Displayable text (tag/tear/origin lines, etc). Not included in duplicate message checking. All terminating white space and control characters are to be truncated from data. 03h TEXT_WING mtext_t Non-displayed text. Not normally displayed. Not necessarily displayable. Not included in duplicate message checking. 10h FTEXT_BODY ftext_t Formatted equivalent of TEXT_BODY to be displayed in place of TEXT_BODY if format is supported. See Image Types for valid values of ftext_t.type. 12h FTEXT_TAIL ftext_t Formatted equivalent of TEXT_TAIL to be displayed in place of TEXT_TAIL if format is supported. See Image Types for valie values of ftext_t.type. 20h IMAGEEMBED membed_t Type and data of embedded raster image file for display. See Image Types for valid embed.type values. 21h ANIMEMBED membed_t Type and data of embedded graphical animation file for display. See Animation Types for valid embed.type values. 22h FONTEMBED membed_t Type and data of embedded font definition file. See Font Types for valid membed_t.type values. 23h SOUNDEMBED membed_t Type and data of embedded sound file for playback. See Sound Types for valid membed_t.type values. 24h PRESENTEMBED membed_t Type and data of embedded presentation definition file. See Present Types for valid membed_t.type values. 25h VIDEOEMBED vembed_t Type and data of embedded video/sound file for playback. See Video Types for valid vembed_t.type values. See Video Compression Types for valid vembed_t.comp values. 26h APPDATAEMBED membed_t Type and data of embedded application data file for process/display. See Application Data Types for valid membed_t.type values. FFh UNUSED undef Space allocated for future update/expansion Specifically, not defined are the values F000h through FFFFh. These values are to be used for user or system defined data fields. Digital Dynamics requests that any developers or organizations that wish to have additional data fields added to this specification notify Digital Dynamics through any of the contact methods listed at the beginning of this document. Message Attributes: ------------------ &&Message Attributes $$ATTRBITS These are the bit values for idxrec_t.attr and msghdr_t.attr: MSG_PRIVATE (1<<0) // Private MSG_READ (1<<1) // Read by addressee MSG_PERMANENT (1<<2) // Permanent MSG_LOCKED (1<<3) // Msg is locked, no editing possible MSG_DELETE (1<<4) // Msg is marked for deletion MSG_ANONYMOUS (1<<5) // Anonymous author MSG_KILLREAD (1<<6) // Delete message after it has been read MSG_MODERATED (1<<7) // This message must be validated MSG_VALIDATED (1<<8) // This message has been validated by a moderator Auxillary Attributes: -------------------- These are the bit values for msghdr_t.auxattr: MSG_FILEREQUEST (1<<0) // File request MSG_FILEATTACH (1<<1) // File(s) attached to Msg MSG_TRUNCFILE (1<<2) // Truncate file(s) when sent MSG_KILLFILE (1<<3) // Delete file(s) when sent MSG_RECEIPTREQ (1<<4) // Return receipt requested MSG_CONFIRMREQ (1<<5) // Confirmation receipt requested MSG_NODISP (1<<6) // Msg may not be displayed to user Network Attributes: ------------------ These are the bit values for msghdr_t.netattr: MSG_LOCAL (1<<0) // Msg created locally MSG_INTRANSIT (1<<1) // Msg is in-transit MSG_SENT (1<<2) // Sent to remote MSG_KILLSENT (1<<3) // Kill when sent MSG_ARCHIVESENT (1<<4) // Archive when sent MSG_HOLD (1<<5) // Hold for pick-up MSG_CRASH (1<<6) // Crash MSG_IMMEDIATE (1<<7) // Send Msg now, ignore restrictions MSG_DIRECT (1<<8) // Send directly to destination MSG_GATE (1<<9) // Send via gateway MSG_ORPHAN (1<<10) // Unknown destination MSG_FPU (1<<11) // Force pickup MSG_TYPELOCAL (1<<12) // Msg is for local use only MSG_TYPEECHO (1<<13) // Msg is for conference distribution MSG_TYPENET (1<<14) // Msg is direct network mail Translation Types: ----------------- &&Translation Types $$XLATTYPE Definition for values of *.xlat[x]: XLAT_NONE 0 // No translation/End of translation list XLAT_LF2CRLF 1 // Expand sole LF to CRLF XLAT_ESCAPED 2 // 7-bit ASCII escaping for ctrl and 8-bit data XLAT_HUFFMAN 3 // Static and adaptive Huffman coding compression XLAT_LZW 4 // Limpel/Ziv/Welch compression XLAT_MLZ78 5 // Modified LZ78 compression XLAT_RLE 6 // Run length encoding compression XLAT_IMPLODE 7 // Implode compression (PKZIP) XLAT_SHRINK 8 // Shrink compression (PKZIP) Agent Types: ----------- &&Agent Types $$AGENTTYP AGENT_PERSON 0 // To or from person AGENT_PROCESS 1 // Unknown process, identified by agent name Agent types E000h through EFFFh are reserved for Synchronet process types (defined specifically by Digital Dynamics). Note: ---- Specifically not defined are agent types F000h through FFFFh. These values are to be used for user or system defined agent types. Digital Dynamics requests that any developers or organizations that wish to have additional agent types added to this specification notify Digital Dynamics through any of the contact methods listed at the beginning of this document. Network Types: ------------- &&Network Types $$NETWORKS NET_NONE 0 // Locally created NET_UNKNOWN 1 // Unknown network type NET_FIDO 2 // FTN network NET_POSTLINK 3 // PostLink network NET_QWK 4 // QWK based network NET_INTERNET 5 // The Internet Media Types: =========== &&Media Types $$MEDIATYP Image Types: ----------- IMAGE_UNKNOWN 0x00 // Use image signature header to determine format IMAGE_ASC 0x01 // ASCII text/IBM extended ASCII graphics IMAGE_ANS 0x02 // ANSI X3.64 terminal escape sequences IMAGE_AVT 0x03 // AVATAR terminal escape sequences IMAGE_LVI 0x04 // LVI terminal escape sequences IMAGE_GIF 0x05 // Compuserve Graphics Interchange Format (GIF) IMAGE_TIF 0x06 // Tagged Image Format (AKA TIFF) IMAGE_JPG 0x07 // Joint Photographers Electronics Group (JPEG) IMAGE_T16 0x08 // TrueVision 16-bit bitmap (TGA) IMAGE_T24 0x09 // TrueVision 24-bit bitmap (TGA) IMAGE_T32 0x0a // TrueVision 32-bit bitmpa (TGA) IMAGE_PCX 0x0b // ZSoft PaintBrush graphics IMAGE_BMP 0x0c // Windows bitmap IMAGE_RLE 0x0d // Windows bitmap (compressed) IMAGE_DIB 0x0e // Display independant bitmap IMAGE_PCD 0x0f // Kodak PhotoCD IMAGE_G3F 0x10 // Group 3 FAX IMAGE_EPS 0x11 // Ecapsulated PostScript IMAGE_RTF 0x12 // Rich text format IMAGE_RIP 0x13 // Remote Imaging Protocol Script (RIPscrip) IMAGE_NAP 0x14 // NAPLPS IMAGE_CDR 0x15 // Corel Draw! IMAGE_CGM 0x16 // Computer graphics metafile IMAGE_WMF 0x17 // Windows metafile IMAGE_DFX 0x18 // Autodesk AutoCAD Animation Types: --------------- ANIM_UNKNOWN 0 // Use file signature header to determine format ANIM_FLI 1 // Autodesk animator ANIM_FLC 2 // Autodesk ANIM_GL 3 // Grasprt Video Types: ----------- VIDEO_UNKNOWN 0 // Use file signature header to determine format VIDEO_QTIME 1 // Apple Quick-time VIDEO_FQTIME 2 // Apple Flattened Quick-time VIDEO_AVI 3 // Windows Auto/Video Interleave VIDEO_ULT 4 // OS/2 Ultimotion Video Compression Types: ----------------------- VCOMP_UNKNOWN 0 // Use file signature header to determine codec VCOMP_RLE 1 // Apple animation VCOMP_SMC 2 // Apple graphics VCOMP_RPZA 3 // Apple video VCOMP_KLIC 4 // Captain crunch VCOMP_CVID 5 // CinePak VCOMP_RT21 6 // Intel indeo R2 VCOMP_IV31 7 // Intel indeo R3 VCOMP_YVU9 8 // Intel YVU9 VCOMP_JPEG 9 // JPEG VCOMP_MRLE 10 // Microsoft RLE VCOMP_MSVC 11 // Microsoft video 1 Font Types: ---------- FONT_UNKNOWN 0 // Use file signature header to determine format FONT_TTF 1 // Windows TrueType FONT_PFB 2 // PostScript Type 1 Font Binary FONT_PFM 3 // PostScript Type 1 Font Metric Sound Types: ----------- SOUND_UNKNOWN 0 // Use file signature header to determine format SOUND_MOD 1 // MOD format SOUND_VOC 2 // Sound Blaster VOC format SOUND_WAV 3 // Windows 3.1 WAV RIFF format SOUND_MID 4 // MIDI format SOUND_GMID 5 // General MIDI format (standardized patches) SOUND_SMP 6 // Turtle Beach SampleVision format SOUND_SF 7 // IRCAM format SOUND_AU 8 // Sun Microsystems AU format Application Data Types: ---------------------- APPDATA_UNKNOWN 0 // Use file signature header to determine format APPDATA_WORDPERFECT 1 // WordPerfect Document APPDATA_WKS 2 // Lotus 123 Worksheet (?) APPDATA_WK1 3 // Lotus 123 Worksheet rev 1 APPDATA_WK2 4 // Lotus 123 Worksheet rev 2 APPDATA_WK3 5 // Lotus 123 Worksheet rev 3 APPDATA_DBF 6 // dBase III data file APPDATA_PDX 7 // Paradox data file APPDATA_EXCEL 8 // Excel data file APPDATA_QUATRO 9 // Borland Quatro Pro file APPDATA_WORD 10 // Microsoft Word Message Storage Protocol ======================== &&Message Storage Protocol $$STORPROT 1. Open SDT, SHD, and SID files read/write deny-none (shareable). 2. Determine length of all message data and number of 256 byte blocks required to store the data. 3. Open SDA file read/write deny-all. 4. If fast allocation mode, seek to end of SDA file and go to step 6. 5. Search SDA file for enough consecutive unused blocks to store all of the message data. If found, seek back to beginning of unused blocks. Otherwise stay at end of file. 6. Write to the SDA file the number of index entries that are going to point to this data (normally 1) for the number of blocks that will be used. The data block(s) have now been allocated. 7. Close the SDA file. 8. Determine length of header record and number of 256 byte blocks required to store the record. 9. Open SHA file read/write deny-all. 10. If fast allocation mode, seek to end of SHA file and go to step 12. 11. Search SHA file for enough consequetive unused blocks to store all of the header record. If found, seek back to beginning of unused blocks. Otherwise stay at end of file. 12. Write to the SHA file a 1 (single byte) for each block that will be used. The header block(s) have now been allocated. 13. Close the SHA file. 14. Lock message base header in SHD file. 15. Read SHD base header #1 (config info). 16. Increment the total number of messages and last message number in header. 17. Write SHD base header #1 (config info). 18. Write header record to SHD file. 19. Write index record to SID file. 20. Unlock SHD base header. 21. Write message data to SDT file. Message Retrieval Protocol ========================== &&Message Retrieval Protocol $$READPROT 1. Open SDT, SHD, and SID files read/write deny-none (shareable). 2. Read index record from SID file. 3. Seek to the byte offset in the SHD file specified in the index record. 4. Lock the message header record. 5. Read the message header record. 6. Unlock the message header record. 7. Compare the message number to the one specified in the index record. If they don't match, re-read the index record and goto step 3. If they continue to mismatch, the index has been corrupted and must be recreated. 8. For each data field specified in the header, seek to the byte offset in the SDT file plus the offset specified in the data field, read from the SDT file the length (in bytes) speicifed in the data field. SMBUTIL ======= &&SMBUTIL $$SMBUTIL_ SMBUTIL is a utility that can perform various functions on an SMB format message base. The primary purpose of SMBUTIL is as an example to C programmers of how to use the SMBLIB functions to access and modify an SMB message base. The complete C source code for SMBUTIL is included and functions from it can be used or modified by developers are their own discretion. The following files make up SMBUTIL: SMBUTIL.EXE Compiled and linked (ready to run) SMBUTIL.C C functions SMBUTIL.H C definitions and variable prototypes SMBUTIL.MAK Makefile (for Borland C++) The usage syntax is as follows: smbutil [/opts] cmd smb_file where cmd is one of the following: l[n] = list msgs starting at number n r[n] = read msgs starting at number n v[n] = view msg headers starting at number n k[n] = kill (delete) n msgs i = import from text file f s = display msg base status c = change msg base status m = maintain msg base - delete old msgs and msgs over max p = pack msg base where opts is one or more of the following: f = fast msg creation mode d = disable duplicate message checking z = set time zone (n=min +/- from UT or 'EST','EDT','CST',etc) and smb_file is the base filename for the message base. An example command line: smbutil r forsale would read all the messages in the forsale message base. If the forsale message base files are not stored in the current directory, the complete path must be specified. (i.e. smbutil r c:\msgs\forsale) SMBLIB ====== &&SMBLIB $$SMBLIB__ SMBLIB is a library of C functions for accessing and storing messages in an SMB format message base. It can eliminate much of the development time for developers that wish to use the library in whole or in part, or use the functions as examples for their own message base function library. The library consists of the following files: SMBDEFS.H Constant definitions, macros, and data types SMBLIB.H Function prototypes SMBLIB.C Function definitions SMBVARS.C Global variable definitions (doubles as declaration file) For developers to use this library with their program, they must include the "SMBLIB.H" header file at the top of each C file that uses any of the library functions, global variables, data types, macros, constants. This can be done by simply add the following line to each .C file: #include "smblib.h" If SMBLIB.H is included, there is no need to include SMBDEFS.H or SMBVARS.C. To link the library functions and variables with a main program, the files SMBVARS.OBJ and SMBLIB.OBJ must be linked with the main program .OBJ files. If the operating system is DOS, be sure that all .OBJ files are compiled for the same memory model. An example MAKEFILE for compiling and linking SMBUTIL with Borland C++ is included. SMBDEFS.H ========= &&SMBDEFS.H $$SMBDEFS_ The SMBDEFS.H file contains important constant definitions and data types (also defined in this document). If ever this document and SMBDEFS.H are inconsistent with each other, then SMBDEFS.H is to be considered correct and this document in error. If such a discrepency is found, please notifiy Digital Dynamics so it can be corrected in a future revision of the specification. Most notable of the data types is a structure called smbmsg_t (not defined in this document). It contains the fixed and variable portions of a message's header record as well as convenience pointers to the sender's name (smbmsg_t.to), recipient's name (smbmsg_t.from), network addresses, and more. If multiple SENDER header fields are included (for example), then smbmsg_t.to will point to the last SENDER header field in the header record. Convenience pointers for other data items work in the same fasion if multiple header fields of the same type exist in the header record. Variables of the smbmsg_t data type (and pointers to variables of smbmsg_t type) are used as arguments to many of the SMBLIB functions. SMBVARS.C ========= &&SMBVARS.C $$SMBVARS_ The SMBVARS.C file contains definitions of the global variables used by the SMBLIB functions. It is a fairly small file since their are a small number of global variables (by design). This file is used for both definitions and declarations, so no "extern" declarations need to be made in developers source code as long as SMBVARS.C or (preferably) SMBLIB.H is included in the source code. SMBLIB.H ======= &&SMBLIB.H $$SMBLIB.H The SMBLIB.H file contains prototypes of all the functions in the SMBLIB.C file. It is necessary to include this file source code if any of the SMBLIB functions are used. The following C source line will include this file: #include "smblib.h" and should be placed near the top of all C source files that use SMBLIB functions, variables, constants, or data types. Function prototypes are necessary for compilers to know the correct calling syntax of a function and detect incorrect usage. Prototypes are also useful as a quick reference for programmers as to the correct calling syntax of a specific function. SMBLIB.C ======= &&SMBLIB.C $$SMBLIB.C The SMBLIB.C file contains the actual SMBLIB library functions. This source file is not a stand alone program, but instead must be compiled and linked with a main source file to create the executable program. The functions in this file are organized in a logical order, but their order is actually irrelevant to the compiling, linking, and execution of the resulting program. A comment block preceeds each function, explaining what the function does, how the passed parameters are used, and what the return code (if any) indicates. A more detailed explanation of each function is included here: int smb_open() -------------- The smb_open() function must be called before the message base is accessed (read from or written to). The global variable smb_file must be initialized with the path and base filename of the message base. This function returns 0 on success, 1 if the .SDT file could not be opened, 2 if the .SHD file could not be opened, and 3 if the .SID file could not be opened. The errno global variable (standard of most C libraries) will most likely contain the error code for open failure. int smb_open_da(int retry_time) ------------------------------- The smb_open_da() function is used to open the data block allocation file for writing messages to a message base. The parameter, retry_time, is the maximum number of seconds to wait while retrying to open the file. This function returns 0 on success. -1 is returned if an open error other than "Access Denied" is returned from the operating system, and the global variable errno will contain the error code. -2 is returned if the retry_time has been reached, and -3 is returned if the file descriptor could not be converted to a stream by the fdopen() function. fclose(smb_sda) should be called immediately after all necessary file access has been completed. int smb_open_ha(int retry_time) ------------------------------- The smb_open_ha() function is used to open the header block allocation file for writing messages to a message base. The parameter, retry_time, is the maximum number of seconds to wait while retrying to open the file. This function returns 0 on success. -1 is returned if an open error other than "Access Denied" is returned from the operating system, and the global variable errno will contain the error code. -2 is returned if the retry_time has been reached, and -3 is returned if the file descriptor could not be converted to a stream by the fdopen() function. fclose(smb_sha) should be called immediately after all necessary file access has been completed. int smb_create(ulong max_crcs, ulong max_msgs, ushort max_age, int retry_time) ------------------------------------------------------------------------------ The smb_create() function is used to create a new message base or reset an existing message base. The parameters max_crcs, max_msgs, and max_age are used to set the initial status of the message base status header. The parameter, retry_time is the maximum number of seconds to wait while retrying to lock the message base header. This functions returns 0 on success or 1 if the message base header could not be locked. int smb_trunchdr(int retry_time) -------------------------------- The smb_trunchdr() function is used to truncate the header file when packing the message base and writing the new header information back to the header file. The parameter, retry_time is the maximum number of seconds to wait while retrying to truncate the header file. Returns 0 on success, -1 if error was other than "Access Denied", or -2 if retry_time reached. int smb_locksmbhdr(int retry_time) ---------------------------------- The smb_locksmbhdr() function is used to lock the first message base (status) header. The parameter, retry_time is the number of seconds to wait while retrying to lock the header. The smb_unlocksmbhdr() function should always be used to unlock the header after accessing the message base header (usually with smb_getstatus() and/or smb_putstatus(). Returns 0 if successful, -1 if unsuccessful. int smb_unlocksmbhdr() ---------------------- The smb_unlocksmbhdr() function is used to unlock a previously lock message base header (using smb_locksmghdr()). Returns 0 on success, non-zero on failure. int smb_getstatus(smbstatus_t *hdr) ----------------------------------- The smb_getstatus() function is used to read the status message base header into the hdr structure. Returns 0 on success, 1 on failure. int smb_putstatus(smbstatus_t hdr) ---------------------------------- The smb_putstatus() function is used to write the status information to the first message base header. The parameter hdr, contains the status information to be written. Returns 0 on success, 1 on failure. int smb_getmsghdroff(smbmsg_t *msg) ----------------------------------- The smb_getmsghdroff() function is used to get the byte offset for a specific message header in the message header file based on the message base index. If msg->hdr.number is non-zero when this function is called, then the index will be searched for this message number. If the message number is found in the index, the msg->idx.offset is set to the byte offset of the message header record in the header file and msg->offset is set to the record offset of the index record in the index file, and the function returns 0. If the message number is not found in the index, the function returns 1. If msg->hdr.number is zero, msg->idx.offset and msg->idx.number are obtained from the index record at record offset msg->offset. If msg->offset is an invalid record offset when this function is called, the function returns 1. Otherwise, the function returns 0. int smb_getmsghdrlen(smbmsg_t msg) ---------------------------------- The smb_getmsghdrlen() function is used to calculate the total length of message header msg including both fixed and variable length portions. This function returns the length of the header. long smb_getmsgdatlen(smbmsg_t msg) ----------------------------------- The smb_getmsgdatlen() function is used to calculate the total length of the data for message msg. This function returns the length of all data fields combined. int smb_lockmsghdr(smbmsg_t msg, int retry_time) ------------------------------------------------ The smb_lockmsghdr() function is used to lock the header record for message msg. The parameter retry_time is the maximum number of seconds to wait while retrying to lock the header. Returns 0 on success, -1 on failure. The function smb_unlockmsghdr() should immediately be called after accessing the message header (usually with smb_getmsghdr() or smb_putmsghdr()). int smb_getmsghdr(smbmsg_t *msg) -------------------------------- The function smb_getmsghdr() is used to read the header record for message msg. msg->idx.offset must be initialized to the byte offset of the header record in the header file before this function is called. The function smb_freemsgmem() must be called to free the memory allocated by this function for the header and data felds. This function returns 0 on success, -1 if the fixed portion of the message header record could not be read, -2 if the message header ID was incorrect, -3 if memory could not be allocated, -4 if a data field could not be read, -5 if the fixed length portion of a header field could not be read, -6 if the variable length portion of a header field could not be read. Several convenience pointers in the msg structure are initialized by this function to point to the last occurance of the SENDER (msg->from), RECIPIENT (msg->to), SUBJECT (smg->subj), etc. int smb_unlockmsghdr(smbmsg_t msg) ---------------------------------- The smb_unlockmsghdr() function is used to unlock a previously locked message header (with smb_lockmsghdr()). This function returns 0 on success, non-zero on failure. int smb_addcrc(ulong max_crcs, ulong crc, int retry_time) --------------------------------------------------------- The smb_addcrc() function is used to add a CRC-32 to the CRC history file for a message base, automatically checking for duplicates. The parameter max_crcs should be the max_crcs defined in the status header of the message base. The parameter crc, is the CRC-32 of the TEXT_BODY and TEXT_SOUL data fields for the message. The parameter retry_time is the maximum number of seconds to wait when retrying to open the CRC history file. This function returns -1 if there was an open error, -2 if the retry_time was reached, -3 if there was a memory allocation error, 1 if the CRC already exists in the CRC history file (indicating a duplicate message), or 0 on success (and no duplicate). int smb_hfield(smbmsg_t *msg, ushort type, ushort length, void *data) --------------------------------------------------------------------- The smb_hfield() function is used to add a header field to the structure msg. The parameters type, length, and data, must be specified according to the header field values listed in this specification. This function returns 0 on success, non-zero on memory allocation error. The function smb_freemsgmem() must be called to free the memory allocated by this function. int smb_dfield(smbmsg_t *msg, ushort type, ulong length) -------------------------------------------------------- The smb_dfield() function is used to add a data field to the structure msg. The parameters type and length must be specified according to the data field values listed in this specification. This function returns 0 on success, non-zero on memory allocation error. The function smb_freemsgmem() must be called to free the memory allocated by this function. int smb_addmsghdr(smbmsg_t *msg, smbstatus_t *status, int fast, int retry_time) ------------------------------------------------------------------------------- The smb_addmsghdr() function is used to add a new message header to the message header file. The msg and status structures are updated to reflect the new total messages, last message number, etc. The fast parameter is used to indicate if the fast allocation mode should be used. If the fast parameter is 0 (off), the header block allocation file will be searched for unused block(s) to store this header. If the fast parameter is 1 (on), the header is stored at the end of the header file. Returns 0 on success, non-zero on failure. The parameter retry_time is the maximum number of seconds to wait while retrying to lock and open files. int smb_putmsghdr(smbmsg_t msg) ------------------------------- The smb_putmsghdr() function is used to store a message header in the message header file. The message header can be for a new message or an existing message. Returns 0 on success, -1 on failure. void smb_freemsgmem(smbmsg_t msg) --------------------------------- Frees allocated memory for the header and data fields in the msg structure. This function must be called to free the memory allocated the functions smb_hfield(), smb_dfield(), and smb_getmsghdr(). long smb_hdrblocks(ulong length) -------------------------------- The smb_hdrblocks() function is used to calculate the number of blocks required to store a message header of length size (in bytes). This function returns the number of blocks required. long smb_datblocks(ulong length) -------------------------------- The smb_datblocks() function is used to calculate the number of blocks required to store message data of length size (in byte). This function returns the number of blocks required. long smb_allochdr(ulong length) ------------------------------- The smb_allochdr() function is used to search for free blocks to store a message header of length bytes and mark the free blocks as allocated in the header allocation file. This function returns the byte offset to the header record or a negative number on error. The function smb_open_ha() should be called prior to calling this function and fclose(sha_fp) should be called after. long smb_fallochdr(ulong length) -------------------------------- The smb_fallochdr() function works exactly the same as the smb_allochdr() function except it is must faster because the header allocation file is not searched for a free block(s). long smb_allocdat(ulong length, ushort headers) ----------------------------------------------- The smb_allocdat() function is used to search for a free blocks to store length amount of data for a message. The parameter headers, indicates the number of message headers that are associated with this data. Normally, the headers parameter will be 1, unless this message is part of a mass mailing. The offset to the allocated data block(s) is returned, or a negative value on error. The function smb_open_da() should be called prior to calling this function and fclose(sda_fp) should be called after. long smb_fallocdat(ulong length, ushort headers) ------------------------------------------------ The smb_fallocdat() function works exactly the same as the smb_allocdat() function except it is much faster because the data allocation file is not searched for a free block(s). int smb_incdat(ulong offset, ulong length, ushort headers) ---------------------------------------------------------- The smb_incdat() function is used to increment the header counter in the data allocation file for the data starting at the byte offset and length size in bytes. The parameter headers, indicates the number of headers to add to the current allocation value in the data allocation file. Returns 0 on success, non-zero on failure. int smb_freemsg(smbmsg_t msg) ----------------------------- The smb_freemsg() function is used to free the memory allocated for the header and data fields in the msg structure. Returns 0 on success, non-zero on failure. int smb_freemsgdat(ulong offset, ulong length, ushort headers) -------------------------------------------------------------- The smb_freemsgdat() function is used to decrement the data block allocation records in the data allocation file associated with the data in the data file by the value of the headers parameter (normally 1). The parameter offset indicates the byte offset to the beginning of the message data in the data file and the parameter length is the total length of the message data. Returns 0 on success, non-zero on failure. int smb_freemsghdr(ulong offset, ulong length) ---------------------------------------------- The smb_freemsghdr() function is used to set the header block allocation records in the header allocation file to 0 (indicated non-allocated block). The parameter offset indicates the byte offset to the beginning of the header record being freed and the parameter length indicates the total length of the header record. Returns 0 on success, non-zero on failure. Bibliography ============ &&Bibliography $$BIBLIOGR Title : The C Programming Language Publisher : Prentice Hall Author : Brian W. Kernighan and Dennis M. Ritchie Document : ARPANET Request for Comments (RFC) #822 Title : Standard for the Format of ARPA Internet text messages Publisher : SRI International Author : David H. Crocker, University of Delaware Document : FTS-0001 Publisher : FSC Author : Randy Bush, Pacific Systems Group Document : FTS-0004 Title : EchoMail Specification Publisher : FSC Author : Bob Hartman Document : FTS-0009 Title : A standard for unique message identifiers and reply chain linkage Publisher : FSC Author : Jim Nutt Document : FSC-00046 Title : A Product Idenfifier for FidoNet Message Handlers Publisher : FSC Author : Joaquim H. Homrighausen Document : FSC-00053 Title : Specifications for the ^aFLAGS field Publisher : FSC Author : Joaquim H. Homrighausen Implementations =============== &&Implementations $$IMPLEMEN Product : Synchronet Multinode BBS Software Developer : Digital Dynamics Level : II Version : 2.00 Product : Synchornet/FidoNet Import/Export Utility (SBBSFIDO) Developer : Digital Dynamics Level : II Version : 2.00 Product : Synchronet UTI (Universal Text Interface) Driver Developer : Digital Dynamics Level : II Version : 2.00