How the Index is maintained
(or "Adama's Ego Page")

Glossary and Program List

Daily operations

Each day I check the Recent Episodes lists to see what's new. When I spot certain things, I add them to the index:

• New storyline codes - I look for acronyms that I haven't seen before (sometimes I have to double-check with the existing list). Then I do a quick read of the episode and its parent to figure out what the code stands for. Usually it's just been the title of the parent episode, but not always. I appreciate the help some authors have given me on this!
• German or other foreign-language episodes - These are usually pretty easy to spot, because the author nearly always fills in a title for the episode. But I also automatically open any untitled episode just to check.
• New "ten-thousand" or "one-thousand" episodes - This check is intermittent; sometimes I"ll wait a week or two before checking. I generally just edit the URL in the browser by hand to find the "unfilled" thousand episodes.

End-of-month operations

Why wait till the end of the month? At the time, it just seemed like a convenient block of time to operate with. Another justification might be the fact that the Recent Episode Lists go back 30 days, so there's less need for an Index more recent than that.

The primary utility I use to generate the end-of month files is the 'stat_proc' program (written by me), whose main screen looks like this:

There is a similar program called 'back_proc' which contains functions for manipulating the Backlink file.

1. Downloading and repairing the the Stat file

• Saving off the Recent Episode list for the past month - Because the episode titles on the Recent lists come from the episodes rather than the Stat file, they are free of the bugs that the Stat file has. I download the 30-day Recent Episode list so that I can have a check for the fixes to be done later. In months having 31 days, I have to download the file for the 1st through the 30th and later append to it the episodes from the 31st.
• Downloading the Stat file - I save off the current Stat file to my hard drive. I chop off any entries from after the end of the month, then I split off a temporary file containing just the records for this month (this will be replaced in a later step).
• Generating the comparison file - The "Comparison file" option on stat_proc generates a version of the 30-day Recent Episode list based on the contents of the Stat file.
• Diffing the comparison file with the Recent Episode list - I run WinDiff on the comparison file and the 30-day Recent episode list. This highlights all the cases of the double-quote bug, as well as cases where the Stat file title has been corrupted or contains slightly different characters from the title in the episode.
  • Why don't you just do a string-search on the Recent list for double-quotes? Because that won't find cases where the quotes are within HTML tags, such as <a> links.
  • Then why don't you edit the Recent list source and search that? Because that would find double-quotes on every line, because they're part of the episode <a> links.
• Reviewing the comparison file for bad HTML - I do a quick scan through the comparison HTML file, looking for HTML code that needs to be corrected -- usually unclosed tags (bolding, italics, etc.)
• Detecting missing Stat file records by generating an Orphan list - The "Identify orphan episodes" option on stat_proc generates a list of all orphan episodes. I then check each new orphan (the ones at the end of the list) to make sure that it really is an Orphan. If it's a legitimate child of another episode, then the record of its parent must have been deleted from the Stat file, so it needs to be added.
• Modifying the stat_fix.h header file to include new fixes needed - I modify the stat_fix file to fix the errors found in the above three steps, and recompile stat_proc.
• Running stat_proc to generate a repaired Stat file - The "Fix the stats file" option of stat_proc applies the repairs to the downloaded Stat file.
• Saving off the Stat records for the current month - I split off the current month's records into a separate file (e.g., stat0802.txt). This makes certain functions (such as generating Author information) easier.

2. Downloading and repairing the Backlink file

• Downloading the Backlink file - I save off the Backlink file to my hard drive.
• Running back_proc to repair the known bugs in the Backlink file - I run the "Fix the backlink file" option of back_proc to fix the existing problems in the Backlink file.
• Checking the Backlink file for truncated episode records - I grep the repaired file to look for truncated records, and make note of the line numbers.
• Modifying the back_fix header file to include new fixes needed - I look up the full title of the truncated records, and add the info to the back_fix file. To find the missing titles, I look at the episodes before and after the truncated one, and look at the Recent Episode lists to find out what episode that occurred between them had double-quotes (usually only one episode qualifies). After adding the new fixes, I recompile back_proc.
• Running back_proc to generate a repaired Backlink file - I run the "Fix the backlink file" option of back_proc again to make a fully repaired Backlink file.

3. Generating the simple indices

The stat_proc program is used to generate the simple indices directly. These are:

• Episodes Listed by Date ("Episodes by date lists" option)
• Episodes Listed by Number ("Episodes by number lists" option)
• Episode Master List ("Master list" option)
• Episode Count statistics ("Count of episodes file" option)
• Individual Episode statistics ("Ep stat files" option)

The stat_proc program does not currently generate the "index.html" files for each of these groups; I have to modify that by hand. Maybe someday I'll get around to changing the program...

4. Generating the Orphan Lists

• New Taken Orphans - This has gotten simpler; I just do a string search on the Master List index files for the words "Taken Orphan" on any episode created in the current month and year. I use the results to create new entries on the "Taken Orphans" page.
• Newly created Orphans -
  • The "Identify orphan episodes" option of stat_proc generates an HTML file containing links for all known orphans.
  • I then do a diff of the new file with the last one generated (which had been saved to a separate name earlier).
  • For each newly-created Orphan (highlighted by the diff) I go to that episode, then follow the "Cancel" link to its parent. This tells me what the parent episode number was. It may also reveal that the Orphan isn't really an Orphan at all, but only looked like one due to some quirk in the Stat file.
  • Once I have determined the episode's parent, I create an entry on the Orphan Episode pages for the episode.

5. Generating the Author indices

The stat_proc program has a separate screen for working with the Author lists:

This screen allows me to combine author entries under the same name or alternate names using the left-hand box; the right-hand box allows me to assign the author entries to a given HTML file name.

• After going through the entire list (which takes a couple of hours), the screen generates this month's list of episode counts by author, and it generates the HTML text for the additions to each author's individual HTML file. The program inserts the HTML text into each existing author's file, but for new authors it writes the text to single file.
• I create new author pages for each new author and append the list of their episodes for this month.
• I apply the newly generated author-episode-count list by hand to the master author count list (which covers the entire Addventure to date).
  • I sort the master list by name
  • I go through both lists, adding new authors and updating the counts of existing authors on the master list
  • For each author, if their new count puts them in a higher group than before, I update the group listings and add the info to the Author Home page.
  • I re-sort the master list by episode count, and restore the headings and spacings.
• I then sort the current month's author-episode-count list and add headings and spacing. A new entry is added to the monthly episode-count index page, including the name of the top author for the month.
• Author graphs - The program "auth_graph" reads an author's episode listing page and generates a graph page that corresponds to it. I run the program for all authors with 20 or more episodes. After running, because three authors have multiple episode-list pages I have to combine their multiple graphs into a single one.

Starting in 2011 I've got a new author-handling program, with similar look and functionality but easier to maintain:

6. Generating the Linkable Episode Index

• The "all linkable" option of back_proc generates a list of all linkable episodes, formatted as a flat list.
• The "taken linkable" option of back_proc generates the same list, formatted as "defined terms" (<dt> tags).
• The "backlink list" option of stat_proc generates a list of all episodes that link back to earlier episodes, formatted as "definitions" (<dd> tags).
• I then sort the "all linkable" list and run the "postprocess all" option of back_proc, which generates the formatted pages of linkable episodes.
• I combine the "taken linkable" and "backlink list" lists, then sort the results. I run the "postprocess linked" option of back_proc, which generates the formatted pages of linkable episodes that have been linked to.
• There are two episodes in the latter lists whose signature lines are so long that they cause errors in this process, so I repair them by hand.

7. Other

• Downloading the new month's episodes - See "Addventure Mirror" below.
• Updating the Episodes with Images list - After downloading this month's episodes for the Mirror (see below), I run a string search on all the HTML files looking for image references ("jpg", ".gif" or "swf"). I open each of those episodes on the server, and find out whether the the images are actually viewable. I then add those episodes to the Episodes with Images Index page.
• Updating Descendant counts - I run the "set_descend" program to update the descendant counts on the Storyline Tree pages.

When-I-have-time operations

• Character Index updates -
  • The "Episodes by date, timestamped" option of stat_proc generates a list of episodes for a specified month, with the timestamp next to each episode for easier cut-and-paste.
  • I then step through the list, reading each episode and often checking a few parent episodes back up the chain, looking for new character information (see Character Indexing Policies) and add or modifiy the entries in the Index.
  • New entries are also added to the alphabetical character list, though I prefer to let that run a little behind the point where I'm at with the monthly files.
  • When I've built up enough updates, I add the special table and color formatting to the files, and update the main Character Index page.
• Storyline Tree updates - Originally, I built the Storyline Trees as I worked on the Character Index; as I encountered episodes that seemed like branching-off points, I added them to the trees. However, that proved to be too distracting, so I stopped doing the Storyline Trees for a while. Eventually, I started up again using the new method:
  • I pick a starting point in the existing trees, and click on every option in every episode to explore the entire tree. I add episodes to the tree that I consider to be plot branches. Which episodes I consider plot branches is highly idiosyncratic, and depends partly on the type of storyline.
  • If a tree is getting excessively long (or excessively wide) I will try to find logical sub-trees which can be split off into separate files, to save on download time and making viewing easier.
  • Descendant counts - After a Storyline page is updated or newly created, I run the "set_descend" program to add descendant counts to the new links.
• Backstage Index updates - I keep an eye out for episodes which give new geographical information about the Backstage Ring. I save off episodes with a lot of detail, especially ones which describe new areas. Maps are created and modified using MS Paint -- not a sophisticated tool, but good enough for this. I also correspond with some of the Backstage authors (especially Dabbler, but also others) to clarify details about the maps.
  • Backstage history - I update this by going through the Episodes by Date lists, reading all the Backstage episodes and looking for significant events. Most of these are the starts of new storylines or the establishment of new general facts or rules about the Backstage. I used to do this in tandem with the Character Index (which is why they're both at about the same point), but now I'll have to find some other way to make myself do it...
• Old monthly author-rankings lists - When I work on a new year-to-date list, I open both the corresponding month-only list and the previous year-to-date list. I sort both, strip out the "Now" numbers, add the new months' data to the new year-to-date list, then re-sort the results and format the new list appropriately.

Indices maintained by others

• German (and other) translations - These are created by anyone who's interested; most of the German translations have been done by Vampire Miyu, Dabbler, and me. The people email me their translations, and I reformat them into a uniform layout and add them to the Index.
• Celebrity Index - These pages were created in their entirety by RaoulDuke. He would email me a zip file containing the HTML files, and I would post them to the Index without editing them. He eventually got tired of working on them.
• Backstage Author Suite Diagrams - These diagrams are created by Dabbler. He posts diagrams to his MSN Communities page, and emails the commentary to me separately. I add the diagrams and commentary to Dabbler's page, and turn the episode references into links.
• Storyline trees - Sometimes other authors will offer to construct a part of the Storyline Tree that they're familiar with, and I gladly accept. A few authors have done this so far -- Scriptfox, Zorlond, JH and FlySpek1947. They have mailed me the trees in raw text form, and I have added HTML formatting and links to the episodes. Others are welcome to contribute.

The Addventure Mirror

I maintain a copy of all Addventure episodes on my hard drive. I do this so that I don't have to be online to work on the Index, and so that I don't have to deal with the lengthy delays that occur in pulling each episode down from the server just to view it. With the episodes on my hard drive, I can flip through them pretty rapidly.

Reassurance: I will not use these files to claim their contents as my own work, or to get the writers in trouble, or for any other nefarious purpose.

• Downloading the new month's episodes - I run the "get_ep" program against the Stat file for the current month, which downloads all the HTML files for this month's episodes. I scan the episodes for images (see "Other" above). I then run "AddReformat" to distribute the files into the proper sub-directories.
• Validating the episode list - Every so often, I check that the Mirror contains exactly the same episodes as the Addventure.
  • I run the "Master dir/id list" option of stat_proc, which generates a list of all episodes in the Stat file.
  • I run the "Orig dir/id list" option of stat_proc, which generates a list of all episodes in the Mirror.
  • I sort both files.
  • I run a diff on the sorted files, then either delete episodes from the Mirror or download new episodes into it.

The Future

One of the things I've done some work on is creating an SQL database to hold the Stat and Backlink file information, which would make it easier and faster to generate the Indices. I've gotten tables set up, and gotten data loaded into them; now I just have to revise the existing utilities to use the tables rather than the original flat files.

It would be nice if this could also lead to the Addventure using an SQL database, but I'm not holding my breath...