Tuesday, September 19, 2006 - Tue, 19 Sep 2006

The “Speed is more important” quote from my last entry got me thinking. Right now, Behold is really limited to a file of about 5,000 names. It takes about 40 seconds for it to load one that big. Practically, it is fast enough for 2,000 or fewer names, but is only adequate for 2,000 to 5,000.

It no longer is a memory limitation. Computers now with 512MB or even 1 or 2 GB of RAM are the ones you can buy now at reasonable prices. Files of up to 50,000 names will load into these machines … if … given … enough … time.

So what needs to be done is to somehow speed Behold up. Could you guess what is taking over 80% of the time to load these files? The answer is the formatting of the Everything Report. This is done deep in the bowels of the TRichview package which I use. Of that formatting time, over half of it is in the FormatLine routine, and most of that is from hundreds of thousands of calls to the MaxFitA routine. What that routine does is figure out how many characters of the current line can fit before the right side of the window, and this is simply to implement word wrap. A call is made to the Windows System routine called GetTextExtentPoint32 which is very fast, but millions of calls to it do tend to add up.

I looked around to see what other tools there are. There are a few other word processing plug ins for Delphi. I took another look at them last night. The ones I tried are slower than TRichview for very large files. Some are 10 to 20 times slower.

An example GEDCOM I tried is 1.5 MB in size and has 5800 names and takes about a minute for Behold to load. Clicking on a hyperlink works instantly as do all operations that don’t require reformatting. This is a huge file by anyone’s standards. Behold’s print preview indicates that it is a 950 page document. Exporting it to HTML produces a 9 MB HTML file. Internet Explorer takes about 15 seconds to load it. Clicking on a hyperlink takes a noticeable couple of seconds for it to transfer location.

But then export it from Behold to an RTF (Rich Text) document and the file created is 32 MB. Now use Microsoft Word to load it and it takes a whole 5 minutes to read it. In Word, it is a 1500 page document and every time you do any operation, there are noticeable delays.

The key is that this document is “rich text”. This the way you will ultimately want it. It allows for different fonts, bold, italics, text size, superscripts, colors, highlighting and eventually different languages in the same text via Unicode (in Version 1.5). If I went to straight text, it could be a lot faster. UltraEdit is a fantastic text editor. It can load huge files, even hundreds of megabytes, in barely a few seconds. But it is a text editor, not a word processor. I could switch Behold to using text only. It would be much faster. But it would lose a lot. Would you go back to DOS because it’s faster than Windows?

I think the answer may be to bite the bullet and to admit that Behold will only work well, as I said above, for files of 2000 names or less and adequately for files of up to 5000 names right now. I’d estimate that 90% of people have less than 5000 names in their family file. My personal family file has 1500 names and I’ve got about another 1500 to add.

What will happen over time is that computers will get more powerful and Behold’s capacity will go up. Also, I’m not giving up on my optimizing work either. This version is going to have major speed improvements, and I see many potential things to possibly speed up TRichview and other parts of Behold some more. That will further increase Behold’s capacity. Maybe in 10 years, Behold will handle 50,000 names adequately.

… and to those who believe it when programs tell you they can handle 1,000,000 names: Maybe they can load those names into some sort of database. But I’d like to see them present those names and all their associated data to you in a way you can really make use of.

Enough ranting. I better get back to work.