I have done some deep thinking about what would be best for Behold for its database. I had debated with myself whether to use SQLite or go NoSQL.
Since then, I have become one of the 96 million people who have subscribed to and started using MyHeritage's online website. They have a free program Family Tree Builder that you can download and use to edit your data offline. The program isn't bad actually, and can sync your data with your online data. Family Tree Builder stores its data on your computer as a SQLite database. It is an open and unlocked database whose data and structures can be browsed with tools that read SQLite databases.
I have now decided to see if I can mimic the SQLite database that Family Tree Builder uses and use the same database structure for Behold. Doing so would allow you to:
Read the database that Family Tree Builder creates with Behold
Save Behold to a database that Family Tree Builder can read
In the future, maybe MyHeritage will allow me to use their API and Behold might be able to become a program that can download your online MyHeritage tree directly, and maybe even upload back to MyHeritage and Sync like Family Tree Builder does.
Having Family Tree Builder as a backup program for Behold (and vice versa) would make the use of either a less risky proposition for your valuable genealogical data, and allow you to see and work with your data in two different ways..
So my question to all of you in this topic in my forum is: "What do you think of this proposal, good or bad?"
I have used Family Tree Builder for many years and like it a lot. My only concern with your plan, if I understand it correctly, is that MyHeritage has a limit of people you can add under the free Basic Plan and it is very small. That would preclude the possibility of uploading my Family Tree Data back to MyHeritage.
I definitely like the idea of being able to read the Family Tree Builder database with Behold, and vice versa.
That is very true. MyHeritage only allows online trees of 250 people with their free plan. They do allow any size on your own computer with their free Family Tree Builder software, but you won't be able to sync with your free online tree if you have more than 250 on your computer.
SQLite is a good choice, My Heritage is not. I have a subscription to their data plan, and found that for Europe, most of their records come from FamilySearch, where they can be accessed for free, and for the English speaking countries, Ancestry and FindMyPast have better data, which can both be accessed with subscription plans that can be canceled on a monthly basis, unlike My Heritage. Tying Behold to My Heritage would be way too restrictive for me, also, because most users' on-line trees are a mess.
If you choose SQLite, connecting to the RootsMagic database is a much better choice, assuming that you can solve the legalities. I say that, because RootsMagic can connect to Ancestry, FamilySearch, FindMyPast, and My Heritage, so that users have a far bigger choice, including the unified tree on FamilySearch. Moreover, for those who want their personal tree on-line, Ancestry has no size limit in their free plan, and a much nicer app for Android and iOS too. And although there is no official documentation for RootsMagic's database, there is an unofficial one on
I looked at their data model, and found that it's very good, and has full support for EE style citations, which is far better than the convoluted and proprietary URLs of My Heritage. And assuming that you don't plan to support all 40 languages supported by My Heritage, sticking to the English language of RootsMagic is not a real restriction for your audience.
Louis, I agree with Enno on all points. SQLite and RootsMagic would both be good choices, and a lot of the groundwork has already been done with SQLite Tools for RootsMagic. Like Family Tree Builder, there is also a free version of RootsMagic, which has very few limitations. I just don't know about the legal implications, since Behold is a commercial product, while SQLite Tools for RootsMagic is not, but the RootsMagic owner is pretty approachable.
That's exactly why I asked for opinions on this. I had neglected to think of RootsMagic's database structure as an alternative possibility. Thank you for mentioning this.
There are several considerations that I'll have to take into account:
1. The database itself: Both use SQLite and I know RootsMagic's is well designed because Tamura Jones reviewed it and found it to be one of the fastest and most capable. Family Tree Builder's was just designed for Version 8 a few years ago, so it will be up-to-date. I have seen and understand what the SQLite Tools For RootsMagic group is doing.
2. Handling of Sources: RootsMagic can handle Evidence Explained structures which is good. (I thought you didn't like EE because it's US-Centric). I haven't looked at MyHeritage's closely, but I expect they are mostly freeform fields that follow GEDCOM's sourcing model (which I don't think is that bad).
3. What is it I'm Tying Behold To? This would not tie Behold to any online database. What it is tying Behold to is another desktop program that it will share its database structure with which both will be able to open and edit. It will either be RootsMagic's, or Family Tree Builder's but it can't be both. The data available on the online services is really not relevant. Nor is the app (but do note that I think MyHeritage has an excellent Android App)
4. Syncing: I looked at that in my recent blog post: Finally Interesting Possibilities to Sync Your Data. RootsMagic can only sync with Ancestry and FamilySearch, and only one record at a time. From FindMyPast and MyHeritage, it can only get hints. Family Tree Builder has a full automatic sync with MyHeritage that seems to work quite well to me.
5. Legalities: A very interesting point. I personally have met and talked with Bruce Buzbee a few times. If I choose RootsMagic, I'll be sure to contact him personally and let him know beforehand what I plan to do. I see this as a win-win, but if he insists I don't then I won't. For MyHeritage, they're a bit more corporate. I likely would want their agreement before I use their structure. They have their own API and I'll let them know my plans that way.
6. Intangibles: The MyHeritage API is a bonus that in the future could let me download/update their data and hints from Behold. With RootsMagic, I wouldn't get that since Ancestry is not letting anyone other than RootsMagic and Family Tree Maker use their restrictive API. I could get access to FamilySearch using GEDCOM X and their API, but that's a whole other problem that doesn't depend on whether I pick the RM or FTB database because its structure is completely different from both of the. Geni might be another site I could interface with, and it's owned by MyHeritage. The key worry here that they would have is that my program corrupts a user's database and makes it unreadable by their program, so I'll have to be very diligent to make that a top priority and ensure that doesn't happen.
So thank you, Enno, for bringing up RootsMagic. This is now not cut and dried as to which one to use. Once I finish DMT 3.0, I'll have to look at both database structures in detail and make a decision.
Louis,
Of course, if you choose to use someone else’s db design, then you’ll be beholden (sorry, I just had to say that) to them for db design changes. You will have to adapt your db to their structure when they make changes.
That said, I know how hard it is to design a solid, fast db and it would be a shame to let someone else’s hard work go to waste, providing you can get the OK. Using a developed db structure would save you hundreds, if not thousands, of hours of design and testing time.
I’ve been a RM user for quite a while and like what Bruce has done with the software.
Keith: I don't think the correct implementation of GEDCOM input and output has as much to do with the database structure, as it does with whether the programmers have been diligent enough to check that all of the input and output was following the standard.
Rich: It really shouldn't matter to you what I'm doing under the hood. Behold itself will work exactly the same. But if I end up picking the RootsMagic database, then you'll be able to read it with RM and Behold will be able to read your RM database directly. I guess whether I have more RootsMagic or MyHeritage users might be a consideration in my decision. Also, databases are not changed very often. It is a major effort. That's exactly why this decision is an important one.
I also like what Bruce has done with RootsMagic. But he's added many extras that I believe have added complications to his database. I'll see when I look in more detail, but I think simpler will be better. I won't want to have to support features that Behold won't have.
It's true that I don't like EE, because it's too much work to translate to all languages that we support with Gramps, but I assume that your main audience is in English speaking countries, for which it can work very well. And if EE support benefits your audience, it's good for Behold too, isn't it? I also know that in RootsMagic you can define your own citation templates, which is something that you can't do in FTB.
With RootsMagic's TreeShare, you can download a whole tree from Ancestry, and upload a new one too. That would be good enough for me, as long as I have the discipline to make all updates in one place, because then, I can always refresh one by replacing it with the other.
Also, most independent programs that I know, not FTB, can download any reasonable number of generations from FamilySearch. I often use RootsMagic for that myself, and then move the new data to Gramps via GEDCOM. Bulk uploads to FamilySearch are not allowed, I think, for a good reason. It's a shared tree, like Geni's, and FamilySearch things that blocking bulk uploads is a good way to protect its integrity. It's true though that RootsMagic can't merge data downloaded from FamilySearch, but I bet that you can do that once you decide to support the FamilySearch API yourself, which is like you say a decision that may be independent from the database choice. I do think however that, once that citation elements are included in GEDCOM X, they will match better with the RM data model than with FTB's.
I just looked at the My Heritage API for sources and citations, and found that although they have more fields than the GEDCOM standard has, it looks quite limited to me. And for me, one thing in particular is nasty, and that is that the link field in both source and citation is defined as a link to the source on My Heritage! And to me, that looks like vendor lock-in. It does of course not mean that if you use that link in the Behold data model, that it must point to My Heritage too, but you may be forced to provide a My Heritage link if you upload data nevertheless.
I've thought of recommending Gramps' SQLite too, but there's a problem with that, because we currently store Python objects in that, so you can't read that. We have a normalized SQLite export, but that's not a model that Gramps can work with. It's only supported for import and export.
Model wise, I think that RootsMagic's database can support all fields used by My Heritage, so if you connect to My Heritage, you can still download and store anything you want.
When you tie Behold to FTB, simple access to on-line services will be limited to My Heritage, and for me, that would't be a good thing, because of the things that I mentioned earlier, like source links always going to My Heritage. It means that, if you download data with such links, these links will only work as long as you pay the My Heritage fees, and that does look like a tie to me.
Please note that I will stick with Gramps no matter what, unless you find a volunteer, not me, to translate the new Behold to Dutch. And for me, Gramps is better than FTB, because it's independent.
You're probably right that FTB's database is simpler than RM's, and maybe citation elements are not a good idea for Behold anyway, because they don't look natural in an editable everything report, and only make things more complicated for your intended audience. Formatted citations actually work pretty well if you keep them simple, and there is no need for translations.
I am no programmer, but for what it is worth, I will state my simple opinion. The best thing about Behold is it is a unique and different program that is like no other. Don't lose that. I have been using Family Historian lately and it does a good job of using a GEDCOM file as it's main "database". It also works with My Heritage and FindMyPast. I like the smart match idea but I would ultimately prefer a program with no online features at all to hold my family data. Just too much grabbing info from here and there and merging it in with the data one has with little effort to cite the original documents. I have found lots of sources on Family Search that only reference an Ancestry source that I can't see because I don't have a subscription. A lot of good a source is if one cannot look at it themselves. That said, I did have a subscription to My Heritage for a year, and I did find it well worth it even though it was a bit pricey. I does, as another user said, use Family Search databases a lot. I was disappointed most of their own transcriptions do not have any images to the original source. Finally, I see SQLite used a great deal and if you use that, I can't see too much going wrong in terms of other programs.
Thank you Enno and Fletch. I also got a few responses and thoughts via email.
Everyone has a valid and personal viewpoint, because it is what works best for them. I'll take all these ideas into consideration. But in the end, I'm going to have to decide what is best for Behold and where I want to take it. Some of you will be pleased and some of you won't. But that will be the best I can do because we all work differently.
Some of the concerns I got was that Behold was better in the earlier versions because it provided just the facts. I will be making just those facts accessible again when I release version 1.3.
Joined: Sun, 9 Mar 2003
288 blog comments, 245 forum posts
Posted: Sat, 15 Jun 2019
I have done some deep thinking about what would be best for Behold for its database. I had debated with myself whether to use SQLite or go NoSQL.
Since then, I have become one of the 96 million people who have subscribed to and started using MyHeritage's online website. They have a free program Family Tree Builder that you can download and use to edit your data offline. The program isn't bad actually, and can sync your data with your online data. Family Tree Builder stores its data on your computer as a SQLite database. It is an open and unlocked database whose data and structures can be browsed with tools that read SQLite databases.
I have now decided to see if I can mimic the SQLite database that Family Tree Builder uses and use the same database structure for Behold. Doing so would allow you to:
In the future, maybe MyHeritage will allow me to use their API and Behold might be able to become a program that can download your online MyHeritage tree directly, and maybe even upload back to MyHeritage and Sync like Family Tree Builder does.
Having Family Tree Builder as a backup program for Behold (and vice versa) would make the use of either a less risky proposition for your valuable genealogical data, and allow you to see and work with your data in two different ways..
So my question to all of you in this topic in my forum is: "What do you think of this proposal, good or bad?"
Joined: Tue, 16 Aug 2011
0 blog comments, 1 forum post
Posted: Sat, 15 Jun 2019
I have used Family Tree Builder for many years and like it a lot. My only concern with your plan, if I understand it correctly, is that MyHeritage has a limit of people you can add under the free Basic Plan and it is very small. That would preclude the possibility of uploading my Family Tree Data back to MyHeritage.
I definitely like the idea of being able to read the Family Tree Builder database with Behold, and vice versa.
Keep up the good work!
Joined: Sun, 9 Mar 2003
288 blog comments, 245 forum posts
Posted: Sat, 15 Jun 2019
That is very true. MyHeritage only allows online trees of 250 people with their free plan. They do allow any size on your own computer with their free Family Tree Builder software, but you won't be able to sync with your free online tree if you have more than 250 on your computer.
Joined: Wed, 9 May 2012
15 blog comments, 3 forum posts
Posted: Sun, 16 Jun 2019
Hi Louis,
SQLite is a good choice, My Heritage is not. I have a subscription to their data plan, and found that for Europe, most of their records come from FamilySearch, where they can be accessed for free, and for the English speaking countries, Ancestry and FindMyPast have better data, which can both be accessed with subscription plans that can be canceled on a monthly basis, unlike My Heritage. Tying Behold to My Heritage would be way too restrictive for me, also, because most users' on-line trees are a mess.
If you choose SQLite, connecting to the RootsMagic database is a much better choice, assuming that you can solve the legalities. I say that, because RootsMagic can connect to Ancestry, FamilySearch, FindMyPast, and My Heritage, so that users have a far bigger choice, including the unified tree on FamilySearch. Moreover, for those who want their personal tree on-line, Ancestry has no size limit in their free plan, and a much nicer app for Android and iOS too. And although there is no official documentation for RootsMagic's database, there is an unofficial one on
https://sqlitetoolsforrootsmagic.com/
I looked at their data model, and found that it's very good, and has full support for EE style citations, which is far better than the convoluted and proprietary URLs of My Heritage. And assuming that you don't plan to support all 40 languages supported by My Heritage, sticking to the English language of RootsMagic is not a real restriction for your audience.
Regards,
Enno
Joined: Sun, 7 Apr 2013
17 blog comments, 1 forum post
Posted: Sun, 16 Jun 2019
Louis, I agree with Enno on all points. SQLite and RootsMagic would both be good choices, and a lot of the groundwork has already been done with SQLite Tools for RootsMagic. Like Family Tree Builder, there is also a free version of RootsMagic, which has very few limitations. I just don't know about the legal implications, since Behold is a commercial product, while SQLite Tools for RootsMagic is not, but the RootsMagic owner is pretty approachable.
Whichever direction you go, please keep in mind that while both Family Tree Builder and RootsMagic have problems implementing GEDCOM 5.5.1, Family Tree Builder's are more serious: https://www.dropbox.com/s/qj9ta3dcs03w9h7/FTM-GEDCOM-Crosswalk.xlsx?dl=0
Keith
Joined: Sun, 9 Mar 2003
288 blog comments, 245 forum posts
Posted: Sun, 16 Jun 2019
Enno,
That's exactly why I asked for opinions on this. I had neglected to think of RootsMagic's database structure as an alternative possibility. Thank you for mentioning this.
There are several considerations that I'll have to take into account:
1. The database itself: Both use SQLite and I know RootsMagic's is well designed because Tamura Jones reviewed it and found it to be one of the fastest and most capable. Family Tree Builder's was just designed for Version 8 a few years ago, so it will be up-to-date. I have seen and understand what the SQLite Tools For RootsMagic group is doing.
2. Handling of Sources: RootsMagic can handle Evidence Explained structures which is good. (I thought you didn't like EE because it's US-Centric). I haven't looked at MyHeritage's closely, but I expect they are mostly freeform fields that follow GEDCOM's sourcing model (which I don't think is that bad).
3. What is it I'm Tying Behold To? This would not tie Behold to any online database. What it is tying Behold to is another desktop program that it will share its database structure with which both will be able to open and edit. It will either be RootsMagic's, or Family Tree Builder's but it can't be both. The data available on the online services is really not relevant. Nor is the app (but do note that I think MyHeritage has an excellent Android App)
4. Syncing: I looked at that in my recent blog post: Finally Interesting Possibilities to Sync Your Data. RootsMagic can only sync with Ancestry and FamilySearch, and only one record at a time. From FindMyPast and MyHeritage, it can only get hints. Family Tree Builder has a full automatic sync with MyHeritage that seems to work quite well to me.
5. Legalities: A very interesting point. I personally have met and talked with Bruce Buzbee a few times. If I choose RootsMagic, I'll be sure to contact him personally and let him know beforehand what I plan to do. I see this as a win-win, but if he insists I don't then I won't. For MyHeritage, they're a bit more corporate. I likely would want their agreement before I use their structure. They have their own API and I'll let them know my plans that way.
6. Intangibles: The MyHeritage API is a bonus that in the future could let me download/update their data and hints from Behold. With RootsMagic, I wouldn't get that since Ancestry is not letting anyone other than RootsMagic and Family Tree Maker use their restrictive API. I could get access to FamilySearch using GEDCOM X and their API, but that's a whole other problem that doesn't depend on whether I pick the RM or FTB database because its structure is completely different from both of the. Geni might be another site I could interface with, and it's owned by MyHeritage. The key worry here that they would have is that my program corrupts a user's database and makes it unreadable by their program, so I'll have to be very diligent to make that a top priority and ensure that doesn't happen.
So thank you, Enno, for bringing up RootsMagic. This is now not cut and dried as to which one to use. Once I finish DMT 3.0, I'll have to look at both database structures in detail and make a decision.
Louis
Joined: Tue, 1 Feb 2005
1 blog comment, 30 forum posts
Posted: Sun, 16 Jun 2019
Louis,
Of course, if you choose to use someone else’s db design, then you’ll be beholden (sorry, I just had to say that) to them for db design changes. You will have to adapt your db to their structure when they make changes.
That said, I know how hard it is to design a solid, fast db and it would be a shame to let someone else’s hard work go to waste, providing you can get the OK. Using a developed db structure would save you hundreds, if not thousands, of hours of design and testing time.
I’ve been a RM user for quite a while and like what Bruce has done with the software.
Rich
Joined: Sun, 9 Mar 2003
288 blog comments, 245 forum posts
Posted: Sun, 16 Jun 2019
Thanks Keith and Rich,
Keith: I don't think the correct implementation of GEDCOM input and output has as much to do with the database structure, as it does with whether the programmers have been diligent enough to check that all of the input and output was following the standard.
Rich: It really shouldn't matter to you what I'm doing under the hood. Behold itself will work exactly the same. But if I end up picking the RootsMagic database, then you'll be able to read it with RM and Behold will be able to read your RM database directly. I guess whether I have more RootsMagic or MyHeritage users might be a consideration in my decision. Also, databases are not changed very often. It is a major effort. That's exactly why this decision is an important one.
I also like what Bruce has done with RootsMagic. But he's added many extras that I believe have added complications to his database. I'll see when I look in more detail, but I think simpler will be better. I won't want to have to support features that Behold won't have.
Joined: Wed, 9 May 2012
15 blog comments, 3 forum posts
Posted: Thu, 20 Jun 2019
Hi Louis,
It's true that I don't like EE, because it's too much work to translate to all languages that we support with Gramps, but I assume that your main audience is in English speaking countries, for which it can work very well. And if EE support benefits your audience, it's good for Behold too, isn't it? I also know that in RootsMagic you can define your own citation templates, which is something that you can't do in FTB.
With RootsMagic's TreeShare, you can download a whole tree from Ancestry, and upload a new one too. That would be good enough for me, as long as I have the discipline to make all updates in one place, because then, I can always refresh one by replacing it with the other.
Also, most independent programs that I know, not FTB, can download any reasonable number of generations from FamilySearch. I often use RootsMagic for that myself, and then move the new data to Gramps via GEDCOM. Bulk uploads to FamilySearch are not allowed, I think, for a good reason. It's a shared tree, like Geni's, and FamilySearch things that blocking bulk uploads is a good way to protect its integrity. It's true though that RootsMagic can't merge data downloaded from FamilySearch, but I bet that you can do that once you decide to support the FamilySearch API yourself, which is like you say a decision that may be independent from the database choice. I do think however that, once that citation elements are included in GEDCOM X, they will match better with the RM data model than with FTB's.
I just looked at the My Heritage API for sources and citations, and found that although they have more fields than the GEDCOM standard has, it looks quite limited to me. And for me, one thing in particular is nasty, and that is that the link field in both source and citation is defined as a link to the source on My Heritage! And to me, that looks like vendor lock-in. It does of course not mean that if you use that link in the Behold data model, that it must point to My Heritage too, but you may be forced to provide a My Heritage link if you upload data nevertheless.
I've thought of recommending Gramps' SQLite too, but there's a problem with that, because we currently store Python objects in that, so you can't read that. We have a normalized SQLite export, but that's not a model that Gramps can work with. It's only supported for import and export.
Model wise, I think that RootsMagic's database can support all fields used by My Heritage, so if you connect to My Heritage, you can still download and store anything you want.
When you tie Behold to FTB, simple access to on-line services will be limited to My Heritage, and for me, that would't be a good thing, because of the things that I mentioned earlier, like source links always going to My Heritage. It means that, if you download data with such links, these links will only work as long as you pay the My Heritage fees, and that does look like a tie to me.
Please note that I will stick with Gramps no matter what, unless you find a volunteer, not me, to translate the new Behold to Dutch. And for me, Gramps is better than FTB, because it's independent.
Enno
Joined: Wed, 9 May 2012
15 blog comments, 3 forum posts
Posted: Thu, 20 Jun 2019
You're probably right that FTB's database is simpler than RM's, and maybe citation elements are not a good idea for Behold anyway, because they don't look natural in an editable everything report, and only make things more complicated for your intended audience. Formatted citations actually work pretty well if you keep them simple, and there is no need for translations.
Joined: Sat, 22 Jun 2019
0 blog comments, 1 forum post
Posted: Sat, 22 Jun 2019
I am no programmer, but for what it is worth, I will state my simple opinion. The best thing about Behold is it is a unique and different program that is like no other. Don't lose that. I have been using Family Historian lately and it does a good job of using a GEDCOM file as it's main "database". It also works with My Heritage and FindMyPast. I like the smart match idea but I would ultimately prefer a program with no online features at all to hold my family data. Just too much grabbing info from here and there and merging it in with the data one has with little effort to cite the original documents. I have found lots of sources on Family Search that only reference an Ancestry source that I can't see because I don't have a subscription. A lot of good a source is if one cannot look at it themselves. That said, I did have a subscription to My Heritage for a year, and I did find it well worth it even though it was a bit pricey. I does, as another user said, use Family Search databases a lot. I was disappointed most of their own transcriptions do not have any images to the original source. Finally, I see SQLite used a great deal and if you use that, I can't see too much going wrong in terms of other programs.
Joined: Sun, 9 Mar 2003
288 blog comments, 245 forum posts
Posted: Mon, 24 Jun 2019
Thank you Enno and Fletch. I also got a few responses and thoughts via email.
Everyone has a valid and personal viewpoint, because it is what works best for them. I'll take all these ideas into consideration. But in the end, I'm going to have to decide what is best for Behold and where I want to take it. Some of you will be pleased and some of you won't. But that will be the best I can do because we all work differently.
Some of the concerns I got was that Behold was better in the earlier versions because it provided just the facts. I will be making just those facts accessible again when I release version 1.3.