Louis Kessler’s Behold Blog

A few days ago, Family Tree DNA released a new version of its chromosome browser. There are other people that have already described the improvements, including Kitty Cooper and Roberta Estes, but I’d like to focus on one item that’s changed that affects Double Match Triangulator users.

New Link to Download Segment Matches

The segment matches that you download for use in Double Match Triangulator are downloaded into a file that I’ll call the Chromosome Browser Results (CBR) file, since it is named: 

nnnnnn_Chromosome_Browser_Results_yyyymmdd.csv

where:

• nnnnnn is your Family Tree DNA kit number,
• yyyymmdd is the date of your download, and
• .csv indicates this is a comma delimited file which can be read by Excel and other programs.

It contains a header line and one line for each segment match that you have with every person who you match to.

The way you download the file has changed. Previously, you used to go to the Chromosome Browser page and click on the “Download All Matches to Excel (CSV Format)” link that I’ve shown below highlighted in orange:

Now, you get it a different way. You must first go to your home screen and click on the Chromosome Browser box:

That will take you to the new Chromosome Browser tool page. Before you select anybody, scroll down to the bottom of your list of DNA matches, and you’ll see a “DOWNLOAD ALL MATCHES” link.

Click on that download link, and it will start to download your segment matches.

As before, you still have to be patient after you click the link. There is no immediate indication that anything is happening. I have a lot of matches (17,462 people with whom I have 347,193 segment matches) and I find it takes about 40 seconds for anything at all to happen and then a window pops up to ask what I want to do with the file, that it says is 19.9 MB in size.

File Format Changes

There are a number of changes to the file itself as well.

1. The file now has a Byte Order Mark (BOM).

The BOM is a few characters at the beginning of a file to tell programs that read the file what type of character set it has. The BOM that FTDNA added says that this is a UTF8 file meaning it can contain any Unicode character. The BOM is somewhat of a technical detail you don’t need to worry about, but what this BOM indicates is that the file may contain names and/or words written in almost any language. They sort alphabetically after English letters, so you’ll find names starting with non-English letters at the end of the CBR file.

If you have downloads of the CBR file prior to this, they did include Unicode text, but without a BOM or a program knowing this, the foreign letters would only appear to be gobbledygook:

 

Now that the BOM has been added, text programs, Excel, and DMT all read and display the names correctly. I’m currently working to release version 3.0 of DMT, and I’ll make sure it will also read the names correctly as Unicode from older files you may have downloaded before FTDNA included the BOM.

2. They changed the file format.

The lines used to end with a Carriage Return and Line Feed (CRLF) which is the Windows file format standard. Now they end with just a Line Feed (LF) which is the Unix file format standard. I can’t imagine why they might have wanted to change this.

3. They removed the double quotes from the text fields.

They used to have the name of the person and the name of the match always in double quotes:

They’ve removed them and it now looks like this:

Either way is fine for csv (comma delimited files), but a program will need to be able to handle it both ways and give the same results. 

One case that causes problems is a name with double quotes in it, e.g. “Buddy” John Williams.  They had previously included this as:  “”Buddy” John Williams”.

If you load the former into Excel, it will display as Buddy John Williams without the quotes around the Buddy and it will no longer match the “Buddy” John Williams that the previous form gave you.

This change caused a very strange bug in DMT that took me two mornings of debugging and over 100 compiles before I solved it. I have to trace the problem step by step to discover that it was caused by the quoting.

4. They changed the header line.

It used to be:  

Now it’s:

Well all they really changed was from using Upper case to using Mixed case for the field names. That might not seem like much, but DMT used the first line to checking to see if you have a FTDNA segment file. It would be easy enough to simply uppercase all the letters and compare those … but then, they did add a space in “Match Name” in the new file as well.

The Effect of These Changes

The difficulty with writing programs that read in files produced by anyone else is that  the file format can be changed at any time and break a developer’s utility program. For a desktop program like DMT, you then have to wait for its next release and hope that the programmer noticed the changes and the program has been updated to handle them. (p.s. Thank you to all of you who report problems to me. If I don’t know about them, I can’t fix them.)

With regards to handling file format changes, I envy those who write online programs, because they can squeeze in a fix online at any time. Changing a packaged program like DMT is a bit more involved.

All utility programs are subject to the whims of the programmers and web developers at FTDNA, GEDmatch, 23andMe, MyHeritage DNA and AncestryDNA. Any time they make a change, they affect the utility programs that use their data.

I’m still working on version 3.0 of Double Match Triangulator. Corrections for these FTDNA file format changes will be included, which should allow files in the new format to be compatible and work with files of the old format.

---

Update: Nov 5, 2018:  Since I wrote this article 3 weeks ago, Family Tree DNA made one other tiny change to its Chromosome Browser Results file. They added a space after the comma but before the chromosome number on each line. This breaks Double Match Triangulator.

---

Update: Nov 28, 2018:  Here is a temporary fix to get the new format CBR files to work with DMT version 2.1.1. There are two possible methods:

1. If you have a text processor, open the CBR file with the text processor and change all comma spaces to just a comma, i.e. change all “, “ to “,”.  Then save the file.

2. If you don’t have a text processor, you open your file with Excel.  Select Column C (Chromosome) and open the “Find and Replace” box and change space-X to X, i.e. change “ X” to “X”. Then save the files as CSV UTF-8 (Comma delimited) (*.csv). This will remove the extra spaces that FTDNA has added and will convert their file back to standard .csv format.

   

Fixes to handle FTDNA’s new format have been developed and will be in version 3.0 of DMT when it is released.

---

Update: Dec 10, 2018:  Less than 2 months after Family Tree DNA changed their Chromosome Browser Results file and the method to download it, they’ve managed to change the method to download it again.

Now, on their Chromosome Browser page, you click on the “Download All Segments” link that’s at the top right of the DNA Matches listing. You still have to do this before you select matches to compare.

Then you wait a while. It took about 30 seconds for me before anything happened and my browser said the file to download was available.

This is actually a good change. Where they first had it at the bottom of the page was not easy to find. Now it is readily accessible.

Since it takes a while to respond, they really should pop up a box right away saying: “Please wait while the download is assembled”. But unfortunately they don’t, so most people will press it 100 times to get it to work, when really only one time will do.

The instructions shown in the Update Nov 28, 2018 still must be followed to put the new file into a format that DMT can read it until Version 3.0 is released.

---

Update: Jan 8, 2019:  Version 2.9.5 of Double Match Triangulator has been released to handle the new Family Tree DNA format.

Today at the #kdgs2018 in Kelowna B.C. was a full slate of talks in 4 time slots, 2 in the morning and 2 in the afternoon, with 7 speakers to choose from in each time slot.

The venue today was another beautiful space, this time at the Okanagan College.

 

Claire Smith-Burns opened the day from the grand staircase.

The first talk I attended was in this amazing circular room on the second floor:

In that room, Mary Read gave me many new ideas for “Research in the Prairie Provinces.” This was the perfect talk with respect to my own research.

Then I went to Tara Shymanski who talked about “Sloooooow Genealogy” and methods to methodically and thoroughly organize and carry out research.

And continuing my theme of going to “organizing” talks, I couldn’t miss Cyndi Ingle’s “Maintaining an Organized Computer” talk.

The fourth talk I went to for the day was my own on “Using Double Match Triangulation to Find DNA Ancestors”. I thought it was well attended with 16 in the room, especially considering I was talking opposite Blaine Bettinger, Cyndi Ingle, Helen Smith, Dave Obee, Lesley Anderson and Mary Read.

At 4:50 was the big raffle, so the entire crowd assembled together for one final farewell while all the vendors gathered up their wares.

The 12 speakers were then invited to join the KDGS executive and all the volunteers to a very nice buffet dinner at the Harvest Golf Course. 

The view out the window was spectacular.

It was a wonderful way to conclude the Conference and I got to talk to other speakers who talks I couldn’t attend, including Blaine Bettinger, Dave Obee, Andrea Lister, Lesley Anderson and Geoff Doherty.

This was a very well organized and attended Conference. My congrats to Claire, her committee, and all the volunteers. It was a pleasure to be a part of it.

It was a good day at #kdgs2018 here in Kelowna, B.C. The venue during the day was the beautiful Kelowna district library downtown.

The second floor of the library contains two full rows of shelves that are the reference materials belonging to the Kelowna District Genealogical Society. I’m sure there are others, but this is the first library that I’ve heard of that is allowing its local genealogical society to have its home in the library. Conference organizer Claire Smith-Burns explained to me how they worked hard to get this cooperation and the results seem amazing, with the resources of the library and the genealogical society working together for the benefit of both.

 

The library is very proud of its Innovation Library who have been working hard to make valuable genealogical materials available online to genealogists. Here, several of us are being given a tour of the services available.

I waved through the glass to Helen Smith, who along with Blaine Bettinger was giving individual help to people with DNA questions.

Then it was down to the main floor of this beautiful library…

… where I enjoyed three talks. First was Kelowna local genealogist and Eastern European expert Xenia Stanford with “Advance Your Eastern European Research with a Baker’s Dozen of Unusual Sources.”

Next I heard Cyndi Ingle: “Be Your Own Digital Archivist – Preserve Your Research”

Helen Smith ended the afternoon with “Newspapers,: Bringing Your Family History to Life”.

The room could seat about 60 and it was full for each talk. Andrea Lister was giving computer labs in another room on searching, Microsoft Word, and PowerPoint.

In the evening, we all headed over to the LDS church. We first had a meet and greet. Then we headed into the chapel. Our group filled it up. There must have been 200 of us.

 

Claire Smith-Burns introduced all the speakers in alphabetical order, and each of us listed off our top four surnames and places we are researching. Here’s Geoff Doherty providing his interesting ancestors.

And Dave Obee amusing us with his ancestor’s adventures.

Blaine Bettinger closed if off with his keynote address entitled “The Stories behind the segments.”

Looking forward to day 2 tomorrow.