Go-ipfs: Extracting IPNS + Namesys from IPFS

[aschmahmann]

There have been some requests (e.g. from @hsanjuan at https://github.com/ipfs/go-ipfs/issues/4435#issuecomment-463387209) to be able to utilize IPNS without IPFS.

Given that IPFS solves the problem of resolving Content from Hash(Content) and Namesys solves a different problem of resolving a Path from an Identifier it seems reasonable to be able to ask for these things separately.

Any thoughts on how much of Namesys to extract (e.g. DNSLink and other non-IPNS resolvers) and where to extract it to would be appreciated. Do @lidel @Stebalien @lanzafame have any thoughts on this.

My current thought is to take basically all of Namesys (including DNSLink) and put it into a new package. We can then add other naming related functionality to that library that isn't necessarily appropriate for the IPFS CLI.

[bonedaddy]

I figured I would post my comment from irc/matrix here with regards to the republisher:

The only thing I think that needs to be improved is the republisher, which currently works by pulling every single key from your keystore and publishing that. Granted it's pretty safe to assume most, if not all of the keys in the keystore will be used for IPNS, because that's about all that you can really do with extra IPFS keys.

What I currently do in my own IPNS publishing service, is have an in-memory cache that gets set whenever a record is published like so:

// Publish enables publishing of an IPNS record with a default lifetime of 24 hours
func (r *RTNS) Publish(ctx context.Context, pk ci.PrivKey, cache bool, keyID, content string) error {
    if cache {
        r.cache.Set(keyID)
    }
    return r.ns.Publish(ctx, pk, path.FromString(content))
}

Then when the republisher does its periodic run, it attempts to first list the key IDs stored in cache to ensure that time is spent only on republishing actual records, instead of a catch-all "attempt to republish all keys in keystore"

func (r *RTNS) republishEntries() error {
    keys := r.cache.list()
    if len(keys) == 0 {
        return errNoRecordsPublisher
    }
    for _, key := range keys {
        priv, err := r.keys.Get(key)
        if err != nil {
            return errNoEntry
        }
        if err := r.republishEntry(r.ctx, priv); err != nil {
            return err
        }
    }
    return nil
}

However, an in-memory cache isn't a super great solution, as you pointed our on irc during node restarts we would lose all data in the cache. I wonder if it might be worth taking a look at go-ipfs-provider and modelling a similar system such that instead of republishing blocks, we're republishing IPNS records.

edit:

Perhaps it might be worth putting this into go-ipns? Currently I believe go-ipns is only used for creating/validating records, but it might make sense to store the extracted ipns/namesys in there as well since the aspect of creating, validating, publishing, and republishing records are closely intertwined.

[aschmahmann]

@postables 👍 to trying to emulate what we do for IPFS pinning. It looks like go-ipfs-provider deals with providing CIDs which isn't quite what we're doing with IPNS (although we could potentially rewrite the code to be a bit more generic). As we discussed the lack of local persistence of pins (metadata like "I care about IPNS Key X, please keep it up to date") is a big deal here, but we should be able to just grab a namespace in the datastore.

Using go-ipns was my initial thought as well. However, I don't want to put anything related to DNSLink (or other non-IPNS resolvers) into go-ipns. The fact that DNSLink is accessible via /ipns/domain.com is confusing enough. I'm thinking of using something like go-namesys, or some other name that doesn't imply mutability = IPNS.

[bonedaddy]

Personally speaking, I think having a modified go-ipfs-provider that is more generic is invaluable. Perhaps something like a go-dht-provider for proving and reproviding DHT records makes sense? This would accommodate the "IPNS Pinning" functionality you have talked about before, but then could also be extended to DHT records in general, which would end up allowing the "DHT Seeding" stuff that I've seen mentioned in a few different places. So its effectively killing 2 birds (or dags?) with 1 stone.

Using go-ipns was my initial thought as well. However, I don't want to put anything related to DNSLink (or other non-IPNS resolvers) into go-ipns. The fact that DNSLink is accessible via /ipns/domain.com is confusing enough. I'm thinking of using something like go-namesys, or some other name that doesn't imply mutability = IPNS.

That is a very good point, go-namesys sounds like a very appropriate name.

[Stebalien]

Let's go with namesys.

[aschmahmann]

@hsanjuan any thoughts/requests for what you'd want exposed in a separate namesys library?

[hsanjuan]

publish, republish, unpublish, follow ?

[aschmahmann]

@hsanjuan what is unpublish? Does publish need to only have an IPNS implementation or also DNSLink (I suspect DNSLink publishing is a huge pain compared to resolution).

Otherwise those should all be doable, although it may take some time because namesys has TODOs all over the interfaces that are probably worth tackling now given that we can do a bit of a reset.

[hsanjuan]

unpublish deletes a record (or let's it expire quickly, i don't know). dnslink publish will depend on the dns provider so, out of scope.

[aschmahmann]

unpublish deletes a record

I'm not feeling great about unpublish unless we can come up with a compelling use case and better name for it. I don't want people thinking that we have the ability to "delete" data from the DWeb. I'm even thinking that we should have IPNS records either have EOL be infinity by default, or alternatively allow for republishing/propagating EOL'd records.

dnslink publish will depend on the dns provider so, out of scope.

👍

[hsanjuan]

sgtm, you asked :)