S Y M B O L T A B L E M A N A G E M E N T by Riley Paulk The Author of ClipCASE Out Of Memory Continue [Y/N] If you are a Clipper Developer and you have written anything approaching a medium to large application in S'87 then you have probably had occasion to see the message above. One is espe- cially intrigued by Nantucket's recovery option "Continue [Y/N]" -=- what they should have said is "Continue [Y/N/Well Maybe, But I don't Think So]". Memory management problems have been a thorn in the side of Clipper Developers for as long as they have been writing Clipper Applications. Roger Donay of dClip fame says in an article he wrote on Clipper Linkers "If you ask the Clipper programmer about his biggest frustrations, he/she will virtually always answer the same way - memory usage and development time." This writer considers memory management problems to be something like an albatross hanging around his neck. The following infor- mation is provided in the hope that it will help others through the process of solving these problems. This information should be of use to you if: 1. You are still writing applications in Clipper S'87 Code. 2. You are maintaining applications that were written in Clipper S'87 Code. 3. You have S'87 applications that you are considering investing the time necessary to port them over to Clipper 5.0(1). 4. You have applications in which you had to use Nan- tucket's SWITCH or any number of third-party SWAP rou- tines for clearing up enough RAM to allow your applica- tion to run in that little 640K box we all have to live in. 5. You use compilers such as WarpLink, Blinker, dClip with a myriad of pre-processors, dynamic overlays, and other magic aimed at dealing with memory management problems. 6. And, maybe, even if you have already made the quantum leap to Clipper 5.0(1), are writing exclusively in 5.0(1) and have nothing to do with S'87. The number one perpetrator of memory management problems in any Clipper application is the Symbol Table. With well managed overlays whether static, as with executables produced using Plink86 Plus, or dynamic overlays produced with dClip, WarpLink, Blinker, or RTLink 5.0 a tremendous amount of code can success- fully be incorporated into a Clipper application. The problem is that no matter how well the overlays are managed the Symbol Table keeps growing. A topic so dear to the heart of Clipper programmers as this has already been addressed extensively by numerous authorities in the field. The documentation that accompanies Steve Steiner's excellent memory management tools library called SmartMem and a text file available on many Clipper oriented BBS's called CLI- PLINK.ZIP written by Roger Donay of dClip are both especially informative on the subject of Symbol Tables and the management of problems related to Symbol Tables. Rather than covering the same material again in this article, it is recommended that, if you are not already familiar with Clipper's use of Symbol Tables, you read those documents or other sources of similar information. In brief, memory variable references as well as references to procedures and functions, including those in Clipper itself and its extend library, are held in the root overlay area [that portion of the executable that stays in RAM throughout execution] in an area referred to as the Symbol Table. Essentially each uniquely named "symbol" in a logical overlay area requires 22 bytes of RAM. Therefore, as your application grows, so does your Symbol Table. The code for your application can be split into manageable size parcels either through static overlays or the use of dynamic overlay techniques but the Symbol Table must remain intact and must, in most cases, remain in the root area. Unless corrective action is taken, this Symbol Table will eventually become the limiting factor on the size -=- read that power -=- of your applications. There are many techniques available for coping with the size of the Symbol Table. The first, and obviously most effective, technique is don't create the symbol to begin with. Use con- stants instead of memory variables. A simple example of this technique is: t = 10 l = 20 b = 16 r = 59 @ t, l, b, r BOX "ÉÍ»º¼ÍȺ " Which is a perfectly ridiculous way to accomplish: @ 10, 20, 16, 59 BOX "ÉÍ»º¼ÍȺ " If you use the first method to draw a box in each of fifty code segments compiled into twenty-five object files with two code segments in each object file you will have wasted 2.2K of RAM. Remember that it is uniquely named symbols within an object file that counts. Change the variable names to top, left, bot- tom, and right in alternate code segments and you can waste an- other 2.2K. Do this with several hundred unnecessary memory variables and you will most certainly get to see "Out of Memory Continue [Y/N/Well Maybe, But I Don't Think So]". The second technique involves re-using symbol names as often as possible within a given object file. You could use var_1, var_2, ..... var_n in every code segment to minimize the number of unique symbol names in each of your object files. A second aspect of this technique is to compile as many of your code segments as feasible into individual object files. Most popular Linkers available for Clipper applications have made inroads into solving the Symbol Table problem through tech- niques generically referred to as Symbol Table Compaction -=- much too diverse a subject to cover at this time (read the docu- mentation on WarpLink among others for information on this tech- nique). Some including Clipper 5.0(1)/RTLink address the problem by pre-processing the Symbol Table during compile time -=- con- verting memory variables into array elements (one array contain- ing as many as 4092 memory variables only requires one 22 byte space in the Symbol Table). There is a great deal to be said for each of these techniques. Most, if not all, appear to be quite effective. ClipCASE is unique among third party Clipper applications development platforms in that it addresses the Symbol Table prob- lem at its source rather than dealing with its symptoms. Clip- CASE was designed to provide Clipper Developers with a "ready- made" front-end or System Shell as well as assisting throughout the development and maintenance life-cycle of an application. Products designed to help solve one of the major problems facing developers, "development time", should definitely not be guilty of contributing to the second most often mentioned prob- lem, "memory usage". For this reason ClipCASE incorporates a Symbol Table management technique called Source Level Symbol Ta- ble Compression. This technique is virtually the same as that used by pre-processors in that memory variables/symbols are con- verted to array elements. The major difference is that ClipCASE does it at the source code level prior to compiling rather than between compile and link times. In ClipCASE this involved reduc- ing more than 1,700 unique symbols to 56 arrays. The following tables delineate the results of this approach. Note: The graphics were trimmed to fit in sixty-five columns rather than the eighty columns ClipCASE originally drew them in; therefore the scale is not to correct proportion. Before Source Level Symbol Table Compression ClipCASE's analysis of its own Memory Map provided the following results: Clip CASE OVERLAY ARCHITECTURE/MEMORY MAP ANALYSIS ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²ÛÛÛ±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛ ±±±±±±±±±±±±± Root ±±±±±±±±±±±±±1AÛ1B1C²²1D²²ÛÛÛ Symbol Table ÛÛÛ ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²ÛÛÛ±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛÛ Overlay Area Size B/L Overlay Area Size B/L Root 178.6 Kb 178.6 Kb Overlay # 1C 9.2 Kb 9.2 Kb Overlay # 1A 7.3 Kb 7.3 Kb Overlay # 1D 23.4 Kb 23.4 Kb Overlay # 1B 12.4 Kb 12.4 Kb Symbol Table 91.3 Kb 91.3 Kb Total Load Module Size: 322.1 Kb -=|=- Baseline: 322.1 Kb After Source Level Symbol Table Compression ClipCASE's report on its Memory Map is: Clip CASE OVERLAY ARCHITECTURE/MEMORY MAP ANALYSIS ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²²ÛÛÛÛ±±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛ ±±±±±±±±±±±±±± Root ±±±±±±±±±±±±±±±±²1AÛ1BÛ±1C²²1D²²Û Symbl Tbl Û ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±²²²ÛÛÛÛ±±±²²²²²²ÛÛÛÛÛÛÛÛÛÛÛÛÛ Overlay Area Size B/L Overlay Area Size B/L Root 180.0 Kb 178.6 Kb Overlay # 1C 11.6 Kb 8.6 Kb Overlay # 1A 9.2 Kb 7.3 Kb Overlay # 1D 21.0 Kb 18.4 Kb Overlay # 1B 14.0 Kb 12.4 Kb Symbol Table 53.4 Kb 91.3 Kb Total Load Module Size: 289.2 Kb -=|=- Baseline: 322.1 Kb By converting most (exceptions explained below) of its own memory variables to arrays, ClipCASE reduced its own Symbol Table from 91.3K to 53.4K or a reduction of 37.9K -=- 42%. This data is based on 26,808 lines of code before compression and 26,432 lines of code after compression. The reduction in the number of lines of code was due primarily to shorter array element names making less need for wrapped code lines (wrapping done for read- ability) and [blush] some lines of trash that were found during debugging after compression. Both represent the same product with absolutely no substantive change other than Symbol Table Compression. Symbols that cannot be compressed at the source level in- clude parameter passing variables (receiving end), counter vari- ables in FOR ... NEXT loops, and variables used to receive the results of COUNT TO commands. Of course, function and procedure names, including Clipper's functions such as MEMOEDIT(), STUFF(), etc., continue to require a place in the Symbol Table. Developers using ClipCASE as their applications development platform may use ClipCASE to compress their own App's Symbol Ta- ble if they desire. This technique is not recommended for the faint hearted. ClipCASE accomplishes 80-85% of the compression automatically but there is still a significant amount of editing and debugging necessary after ClipCASE finishes the drudge work. The real advantage to developers is that when they use ClipCASE's System Shell for their application's front end, they do so with essentially no inherited Symbol Table load. After source level symbol table compression, editing and modifying the code becomes something of an adventure. One does, however, adjust. To accommodate the problem of being faced with a myriad of array element numbers, rather than slightly more descriptive mem- ory variable names, ClipCASE produces a script file for compiling with a Norton's Guide Engine. The developer is then able to run the Norton's Guide for the specific application being developed or maintained behind his or her editor. This technique provides the developer with context sensitive, hot-key access to the ap- plication's complete Symbol Table (or the substituted array ele- ments); Data Encyclopedia with system and descriptive names and narrative explanations for all data bases and fields; data ele- ment parameters, validation, and picture requirements; relational integrity requirements; a complete system architecture or flow diagram depicting internal relationships between all functions and procedures; indexes including primary, secondary, and foreign key fields; predefined filters or view criteria sets; and the specifications and requirements that they are writing the code to accommodate. The ability to access all of this information from within their editor greatly simplifies the development process, reduces development time, and helps to further insure that what the developer is building is precisely what the client wants them to build. A somewhat, though not totally, unexpected fringe benefit of ClipCASE doing Source Level Symbol Table Compression on itself was a significant decrease in processing time. The tables below were taken from screen captures of two of ClipCASE's source code analysis routines and of packing two of ClipCASE's larger Data Bases. The net results are, after compression ClipCASE operates from 10 to 15% faster on average than it did before compression. All processing from which this data was derived was done on the same machine (AST Premium 386C/25MHz) -=- the hard disk was opti- mized before each set of data was collected. The Estimated Time to complete these processes was zeroed-out in order to clearly show the time over estimates. Before Compression: ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Clip CASE Source Code Analysis ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍ» º Cataloging Clip CASE Source Code Files Tot No Files: 55 º º Processing: Screen Saver Record # 16 º ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ º Program/Procedure/Function Name Ln Cnt Ln No Action º ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ º System User's Manual Report 684 684 Analysis Compº º Next Action Recommended/Required 903 903 Analysis Compº º Index Criteria Definition 612 612 Analysis Compº º Look-Up Tables & Pick-Lists 428 428 Analysis Compº º System User Preferences 509 509 Analysis Compº º System Profile 215 215 Analysis Compº º Data Encyclopedia - System Data Bases 422 422 Analysis Compº º Reports Management & Generation 991 991 Analysis Compº º Screen Saver 145 145 Analysis Compº ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ º Totals: Files = 55 Line Cnt = 26,808 Code Cnt = 23,024 º ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Escape To Abort ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ Processing Source Code File For: Screen Saver Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 46 Min 27 Sec After Compression: ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Clip CASE Source Code Analysis ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍ» º Cataloging Clip CASE Source Code Files Tot No Files: 55 º º Processing: Screen Saver Record # 16 º ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ º Program/Procedure/Function Name Ln Cnt Ln No Action º ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ º System User's Manual Report 684 684 Analysis Compº º Next Action Recommended/Required 903 903 Analysis Compº º Index Criteria Definition 612 612 Analysis Compº º Look-Up Tables & Pick-Lists 428 428 Analysis Compº º System User Preferences 509 509 Analysis Compº º System Profile 215 215 Analysis Compº º Data Encyclopedia - System Data Bases 422 422 Analysis Compº º Reports Management & Generation 991 991 Analysis Compº º Screen Saver 145 145 Analysis Compº ÇÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄĶ º Totals: Files = 55 Line Cnt = 26,432 Code Cnt = 21,205 º ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ͵ Escape To Abort ÆÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ Processing Source Code File For: Screen Saver Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 42 Min 29 Sec Net Decrease in Processing Time: 3 Min 53 Sec -=- 8.54% Before Compression: ÉÍÍÍÍÍ͵ Clip CASE System Cross References & Flow Analysis ÆÍÍÍÍ» Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 1 Min 20 Sec After Compression: ÉÍÍÍÍÍ͵ Clip CASE System Cross References & Flow Analysis ÆÍÍÍÍ» Est Time To Comp: 0 Min 0 Sec -=- Time Over Est: 1 Min 12 Sec Net Decrease in Processing Time: 8 Seconds -=- 10% Before Compression: ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ» º Packing Data Base! -- No Esc At This Time º º Data Base File Name: SYS_PRGS.DBF º º Packing Will Require Approximately: 00:01:14 º ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ After Compression: ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ» º Packing Data Base! -- No Esc At This Time º º Data Base File Name: SYS_PRGS.DBF º º Packing Will Require Approximately: 00:01:00 º ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ Net Decrease In Processing Time: 14 Seconds -=- 19% Before Compression: ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ» º Packing Data Base! -- No Esc At This Time º º Data Base File Name: SYS_DDIR.DBF º º Packing Will Require Approximately: 00:00:54 º ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ After Compression: ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ» º Packing Data Base! -- No Esc At This Time º º Data Base File Name: SYS_DDIR.DBF º º Packing Will Require Approximately: 00:00:46 º ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ Net Decrease In Processing Time: 8 Seconds -=- 15% In the examples above comparing processing times for packing two different Data Bases it was, at first, assumed that the ef- fective decrease in processing time was due to more Free Pool Memory being available. However, when Free Pool was reduced to the pre-compression value using Tom Rettig's MEMHOG.COM function there was no increase in processing time. In summary, ClipCASE provides a uniquely different approach to dealing with memory usage problems. Whether the developer elects to compress the symbol table for his or her own applica- tion is secondary to the fact that ClipCASE provides an extensive library of data management functions and features with virtually no load added to the application's symbol table and improves exe- cution speed by as much as 10-15% for all of those library func- tions. The ability to automatically create Norton's Guides "on- the-fly" for applications while they are still being developed should prove invaluable to developers under any circumstance.