Saturday, February 3, 2007 - Sat, 3 Feb 2007

I’ve been thinking about the speed of data loading. Reading the GEDCOM should be fast, and as I mentioned in my Jan 30th post, I’ve done pretty well in comparison to other programs.

So I added some interesting statistics about reading time that will show up in the Log file Behold produces. For my very large test file, it looks like this:

Reading Files Time: 11.405 secs, Memory: 47.085 MB
Read 841,769 lines at 73,805 lines per second
Read 33,790 people at 2,963 people per second
Read 21,243,265 bytes at 1.78 MB per second

As you see, I’ve even improved on the 13 seconds and 58 MB from a few days ago. All the original data plus data linkage structures for the 21 MB file only take 47 MB of memory.

Reading and building the data structure at 1.78 MB per second is pretty good! To put this speed in context, there are an average of 30 lines per page in each Harry Potter book. The seven books (once the 7th comes out on July 21st) will be about 3,500 pages in total. At the speed of 73,805 lines per second, it would take just over a second for Behold to read all the Harry Potter books.

Still, it’s not the speed of reading the files that takes the time in Behold - it’s the generation of the Everything Report. And I’m working on that.