Subject: Inn 1.4 makehistory -- performance patch
From: [email-masked] (Otto J. Makela)
Date: 1997/04/23
Newsgroups: news.software.misc,sfnet.tietoliikenne.tekniikka

There is a rather serious bug in the standard distribution inn 1.4 (and all
unoff versions, but has been fixed in inn 1.5.1) makehistory, which affects
innd performance very much. Makehistory can be used to create the history
dbz database files from the raw text file and for example dexpire does this
with every histtrim operation.

The problem is that makehistory is given just one parameter specifying the
size of the expected dbz database in lines, but it uses this parameter also
for the dbzfresh() tagmask parameter where the maximum key value is expected
to be given.  This means the dbz database is built with a very small hash
table causing most of the history database to be accessed without hashing,
a very serious performance hit.

As noted in the inn FAQ section 7.45, you can check the size of your dbz
hash table with the following commands:
        head -1 history.dir | perl -ane 'print 2 ** $F[7], "\n";'
if the number returned is smaller than your history text file, you are
being affected by this problem.

The problem becomes worse as the history file grows, so an indication of this
problem is that your news server worked fine at first (small history file)
but started really beating the living daylights out of the hard disk where
history is after it grew a bit larger.

This problem can be patched with the following extremely simple change to
makehistory, which just multiplies the number of lines in the dbzfresh call
with 70 which is (supposedly) an average number of characters per history
text file line.  This same fix has been implemented in inn 1.5.1 makehistory.
-- 
diff -u expire/makehistory.c{.orig,}
--- expire/makehistory.c.orig   Mon Jul 31 22:18:46 1995
+++ expire/makehistory.c        Wed Apr 23 11:26:50 1997
@@ -125,7 +125,8 @@
     /* Open the new database, using the old file if desired and possible. */
     (void)dbzincore(1);
     if (IgnoreOld) {
-       if (dbzfresh(p, dbzsize(size), HIS_FIELDSEP, 'C', dbztagmask(size)) < 0) {
+      /* Assume average history line length of 70 characters */
+       if (dbzfresh(p, dbzsize(size), HIS_FIELDSEP, 'C', dbztagmask(size*70)) < 0) {
            (void)fprintf(stderr, "Can't do dbzfresh, %s\n",
                    strerror(errno));
            if (temp[0])
-- 
   /* * * Otto J. Makela  <[email-masked]> * * * * * * * * * * * * * * * * */
  /* Phone: +358 14 613 847, BBS: +358 14 211 562 (V.32bis/USR-HST,24h/d) */
 /* Mail: Cygn.k.7 E 46/FIN-40100 Jyvaskyla/Finland, ICBM: 62.14N 25.44E */
/* * * Computers Rule 01001111 01001011 * * * * * * * * * * * * * * * * */