Janusgraph: ID_STORE_NAME table renamed from Titan
In migrating from Titan, I found (by way of Cassandra table metrics reporting) that the titan_ids table has changed to janusgraph_ids. I think if this table were used it would be a breaking change.
However, it appears to me that this table isn't actually used much if at all? It's defined under the constant ID_STORE_NAME, which seems unused here:
https://github.com/JanusGraph/janusgraph/search?utf8=%E2%9C%93&q=ID_STORE_NAME&type=
Either there should be a migration path for this table from Titan to JanusGraph, or it should not exist at all. So far migrating to JanusGraph from Titan seems to work but things like this make me worry.
sharpau
All 18 comments
jerryjch
on 25 Apr 2017
For HBase backend, it is probably not a big deal because the shortname alias 'i' is normally used as the real cf name in production.
jerryjch
on 25 Apr 2017
@jerryjch I saw that. So it really surprises me that JanusGraph works on an existing Titan 1.0 DB. Maybe the specifying the Cassandra keyspace overrides that name parameter?
sharpau
on 25 Apr 2017
I think we need to make ID_STORE_NAME configurable in light of migration from Titan.
amcp
on 25 Apr 2017
Am I missing something or is this a complete blocker for users that want to migrate from Titan?
FlorianHockmann
on 26 Apr 2017
@FlorianHockmann , I might be missing it too. But I have upgraded from Titan 1.0 to JanusGraph 0.1 without any major issues that would seem to be related. And anecdotally people on the JanusGraph-users mailing list, and tests on PRs such as the one that added the backwards compatibility to version checking, haven't seen problems. So it's either a fairly subtle problem or ID_STORE_NAME isn't used for some backends, maybe?
Still hoping someone more knowledgeable can investigate further so we can be sure.
sharpau
on 26 Apr 2017
@FlorianHockmann @sharpau @jerryjch TITAN_IDS is used to store id reservations assigned to Gremlin Server hosts. If you are heavy on writes and your id block size is small, you will experience heavy writes on TITAN_IDS table. It is not a catastrophe to lose the TITAN_IDS table for reads, but you might experience increased latency on writes especially if you use locking as TITAN_IDS will dish out id ranges that already are in use on the storage layer. If you do not use locking and start with a fresh TITAN_IDS table, it is possible to experience data loss (as we did not change the ID assignment strategy). I think a mechanism to change the name of the ids table is necessary in configuration to support migration.
One consideration for DynamoDB is that we cannot change table names after they are created. In any storage backend, it is possible to copy the TITAN_IDS KCVStore using storage-native tools to a new JANUSGRAPH_IDS table, but you would have to reject all writes to your graph during the migration activity, making migration an offline operation with a service outage, and not just a flip.
amcp
on 27 Apr 2017
I volunteer to pull the KCVStore name into configuration.
amcp
on 27 Apr 2017
@amcp Thanks for the explanation. Inability to change table names isn't just DynamoDB - Cassandra is the same way, so a config option seems like the way to go.
Does this make sense to put in an 0.1.1 release, as the 0.1 branch is intended for a smooth upgrade from Titan?
sharpau
on 27 Apr 2017
@sharpau We will need a new version number. I don't know what version that will be though.
@mbrukman thoughts?
amcp
on 28 Apr 2017
Thanks for the explanation @amcp
I think a hotfix would be a good idea for this change to avoid problems with the migration from Titan which is probably where most users are coming from at the moment.
FlorianHockmann
on 28 Apr 2017
Fix in progress in #240. Items TODO:
- [x] Add documentation for the new configuration option
- [x] Update the point version of 0.1 release
- [ ] Add upgrade-downgrade tests
- [ ] Propose a vote on the point release
- [ ] Release the fix to maven central
amcp
on 28 Apr 2017
Regarding documenting the configuration option, apparently the build generates the configuration tables from the java files, so I do not need to do anything special.
amcp
on 28 Apr 2017
The version on 0.1 branch already appears to be 0.1.1 so it is complete. Also, upgrade testing is difficult to code in Java because of competing versions of many dependencies of Titan and JanusGraph. Also, it is not clear if downgrading from JanusGraph is a possibility for any backend other than DynamoDB (as it has a HTTP API and has never made breaking changes to the v2 API).
The current version of BerkeleyJE is not backwards compatible with the version used in Titan 1.0.0. I know that HBase versions have been increased, and support for a really old version was dropped. Only backend remaining is Cassandra.
@sharpau Do you know if the Cassandra backend (drivers/versions) in JanusGraph 0.1.0 are backward compatible with the Cassandra backend in Titan 1.0.0?
For upgrade-downgrade, I will code a script that downloads and builds the appropriate versions of DynamoDB Storage Backend for Titan and JanusGraph and then execute consecutive Groovy scripts that call on both.
amcp
on 28 Apr 2017
@amcp as far as I am aware the Cassandra backend has not changed from Titan 1.0 to JanusGraph 0.1. We're using Cassandra 2.0.11 and the Cassandra driver 3.1.1. Haven't had to change anything aside from our janusgraph-core and janusgraph-cassandra dependencies, the various imports, and the storage.cassandra.keyspace configuration.
In general for the upgrade, the only issues I've observed are potentially from losing the titan_ids table (multiple vertices getting the same id, I think).
sharpau
on 28 Apr 2017
@amcp – sorry, late to answer your question, but v0.1.1 SGTM (which is what you've already decided on).
mbrukman
on 2 May 2017
@amcp are we good to close this one out?
twilmes
on 11 May 2017
Good to close.
On May 11, 2017, at 10:09 PM, Ted Wilmes <[email protected]notifications@github.com> wrote:
@amcphttps://github.com/amcp are we good to close this one out?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/JanusGraph/janusgraph/issues/228#issuecomment-300784061, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABBHE3jG8JO3UkRAS3wpGfR7MnU7_9BYks5r4wihgaJpZM4NGyhq.
amcp
on 11 May 2017
Related issues
chupman
·
3Comments
robertdale
·
4Comments
bionicles
·
4Comments
amcp
·
5Comments
zifnab87
·
3Comments
Most helpful comment
@FlorianHockmann @sharpau @jerryjch TITAN_IDS is used to store id reservations assigned to Gremlin Server hosts. If you are heavy on writes and your id block size is small, you will experience heavy writes on TITAN_IDS table. It is not a catastrophe to lose the TITAN_IDS table for reads, but you might experience increased latency on writes especially if you use locking as TITAN_IDS will dish out id ranges that already are in use on the storage layer. If you do not use locking and start with a fresh TITAN_IDS table, it is possible to experience data loss (as we did not change the ID assignment strategy). I think a mechanism to change the name of the ids table is necessary in configuration to support migration.
One consideration for DynamoDB is that we cannot change table names after they are created. In any storage backend, it is possible to copy the TITAN_IDS KCVStore using storage-native tools to a new JANUSGRAPH_IDS table, but you would have to reject all writes to your graph during the migration activity, making migration an offline operation with a service outage, and not just a flip.