Go-ipfs: Using IPFS as a library (not a as deamon)

[JustinDrake]

Over at OpenBazaar (see here) we're using IPFS as a library (not as a deamon). The main author of the current code is @cpacia. As an example, this is his AddFile function (taken from here)

func AddFile(ctx commands.Context, fpath string) (string, error) {
    args := []string{"add", fpath}
    req, cmd, err := NewRequest(ctx, args)
    if err != nil {
        return "", err
    }
    res := commands.NewResponse(req)
    cmd.PreRun(req)
    cmd.Run(req, res)
    var fileHash string
    for r := range res.Output().(<-chan interface{}) {
        fileHash = r.(*coreunix.AddedObject).Hash
    }
    cmd.PostRun(req, res)
    if res.Error() != nil {
        return "", res.Error()
    }
    if fileHash == "" {
        return "", addErr
    }
    return fileHash, nil
}

I was curious about what the functions PreRun, Run, and PostRun do, and this is what Chris had to say

The functions are just defined in the ipfs code and are different for each command.

To be honest we could probably handle issuing ipfs commands better. When I first wrote that section of code it took my two weeks just to figure out how to do that because I couldn't find any way in the ipfs code to pragmatically issue commands without mocking like it's coming from the command line.

So what you see there is the code that behaves like a command line command. I would prefer to just call the relevant portions of the code directly but I couldn't figure out how to do it.

Could the go-ipfs team help the openbazaar-go team use go-ipfs as a library? Specifically,

• Are we using the internal functions PreRun, Run, etc. properly in the above code?
• Where can we find documentation about the relevant interface functions to use go-ipfs as a library?
• What's the easiest way to contact you? Do you have a Slack channel?
• Do you have any other tips or tricks for using go-ipfs as a library?

Thanks! 👍

[whyrusleeping]

You can avoid using the commands lib entirely, For example, AddFile could be written as:

import (
    "os"

    "github.com/ipfs/go-ipfs/core"
    "github.com/ipfs/go-ipfs/core/coreunix"
)

func AddFile(ipfs *core.IpfsNode, file string) (string, error) {
    fi, err := os.Open(file)
    if err != nil {
        return "", err
    }

    return coreunix.Add(ipfs, fi)
}

You can also get a little more integrated and use the key.Key instead of the string (which is base58 encoded). If you want, i can provide an example of that.

Also, let me know of any other commands you want to use, the more popular of the 'commands' will have parallel bits of code under core.

[jbenet]

Yeah, sorry, that part of the code is a bit of a quagmire. it's very complicated.

The PreRun, Run, PostRun parts happen at different moments of "executing the command" -- these command functions are what run through the API or cli.

As a library, you can do without much of the commands, and call the ipfs node library directly. But problems are:

• Lots of implementation details are still in commands and not in the core library. these need to move for it to be easier for all the things.
• Documentation is lacking
• The core ipfs node code should really match the new interface spec: https://github.com/ipfs/interface-ipfs-core but that's WIP too, and we've been wanting to finish _that_ before we can level up the Go core node interface. (cc @RichardLitt)

I think for now we can do two things:

• help you get by with just the core ipfs node, without the commands (if it makes sense for your use cases)
• re-write commands' logic into the core interface as you need it

But long term, let's finish the core-ipfs-interface and levelup go-ipfs and go-ipfs-api with it.

[whyrusleeping]

Note: for each command line command, the actual code it executes will be in the relevant run function. So for ipfs cat, the code being run is in core/commands/cat.go in the Run function on that command. The majority of it is 'user input' handling stuff and purely running that functionality in go is generally just a few lines of code.

[Kubuxu]

You are using it correctly as far as I can see but we are working on something that will make your life much easier.

It is interface-ipfs-core (https://github.com/ipfs/go-ipfs/pull/2876) which defines simple and small API for interfacing go-ipfs directly, it would work perfectly for using go-ipfs as library, and we also plan to create bindings for it so it can be used when go-ipfs in different process.

We are currently finishing work on 0.4.4 release (this is the reason it hasn't been merged for a long time), but it is important to point out that the interface-ipfs-core isn't finished in go. Bad news is that, in next few months we will be focusing on implementing IPLD into IPFS and we won't have much time to finish it. You could either try working on it or just interface go-ipfs directly.

Best way to contact us directly is #ipfs@freenode.

[jbenet]

It's still great to ask these questions here-- for others to find.

[duosearch]

Thanks all, that's very helpful.

@whyrusleeping Stupid question, what's the easiest way to properly initialise a working core.IpfsNode? One thing we'd like to do is create a testbench for stress testing with ~100 IPFS nodes in a private network. Any tips for doing that? Is there existing code we can borrow?

[jbenet]

@duosearch

• do you want to use full processes? or all in one process? the transports (extra-proc) will matter a lot. for that, there's iptb: https://github.com/whyrusleeping/iptb
• please be wary that _tonnnnns_ of optimizations still need to happen. we would _love_ any traces that you may find of things being slow or wonky.

[duosearch]

We probably want a mixture of full processes and a bunch in one single process. For example, three physical computers on three continents, each with 10 processes, and each process with 10 nodes.

[whyrusleeping]

For example, three physical computers on three continents, each with 10 processes, and each process with 10 nodes.

@duosearch thats something I want too! My goal was to enable iptb to be able to orchestrate this sort of behaviour. development on it has gone slowly so far, but currently it can spin up either local processes, or docker nodes on the host. I'm hoping to find time soon to add an 'ssh' type node where iptb connects to a remote machine and manages one or more daemons there.

[whyrusleeping]

Can we try and transform some of this knowledge into a doc somewhere? either in the docs folder or in the faq?

cc @RichardLitt @flyingzumwalt

[RichardLitt]

Yes. I'll work on this.

[ghost]

Pointing to the IPFS Core API milestone because it's relevant here, and I'm filling it in right now.

[faddat]

I'd also love to use ipfs as a library in my app.

Is there a standard way of doing this?

[JustinDrake]

I think the Core API is ultimately intended to become the "standard way". At the moment you can interact with IPFS in a number of ways:

• Invoking the command line tool
• Using HTTP requests to a gateway
• Programmatically calling internal go-ipfs functions
• Directly instantiating standalone libraries such as go-libp2p

[whyrusleeping]

This is a simple way to use embed ipfs within your go app: http://ipfs.git.sexy/sketches/minimal_ipfs_node.html

[faddat]

@whyrusleeping thanks!

[arunthampi]

The dependency that go-ipfs has on gx makes it hard to use with existing toolchains (such as godep, dep or govendor) -- one user has taken the entire repo an un-gx'ed it so that it can be used as part of a conventional go toolchain: https://discuss.ipfs.io/t/using-go-ipfs-as-a-library-problems-with-dep/2697

Are there plans to make it simple to integrate go-ipfs with go projects in a "standard" way (i.e. use vendoring techniques that other projects use)?

[djdv]

@arunthampi
There's talk about it here https://github.com/ipfs/go-ipfs/issues/4831
This may also be interesting to you: https://github.com/ipfs/go-ipfs/pull/4920

[whyrusleeping]

@arunthampi you also might find @karalabe's fork interesting: https://github.com/ipsn/go-ipfs

[arunthampi]

yup @karalabe pointed me to his fork but I got it working with govendor and the make go-ipfs-source.tar.gz task! thanks @djdv for the pointer :pray:

blog post incoming

[AminArria]

For anyone having issues using @karalabe fork and Go 1.11 modules set your go-ipfs dependency to use the latest commit in master because the latest tag v0.4.17 is missing the version.go file in the root directory.

The only bad side is that using go-libp2p-crypto will gause duplicate enum registered error and there is no nice way (that I found) to fix it except importing from from gxlibs in the fork.

[probonopd]

Trying to run https://github.com/ipfs/go-ipfs/tree/master/docs/examples/go-ipfs-as-a-library, I am getting

me@host:~/GOPATH/src/github.com/ipfs/go-ipfs/docs/examples/go-ipfs-as-a-library$ go get ...
go get: warning: modules disabled by GO111MODULE=auto in GOPATH/src;
    ignoring ../../../go.mod;
    see 'go help modules'
package crypto/ed25519: unrecognized import path "crypto/ed25519" (import path does not begin with hostname)

me@host:~/GOPATH/src/github.com/ipfs/go-ipfs/docs/examples/go-ipfs-as-a-library$ 
me@host:~/GOPATH/src/github.com/ipfs/go-ipfs/docs/examples/go-ipfs-as-a-library$ go run main.go
../../../../../marten-seemann/qtls/auth.go:11:2: cannot find package "crypto/ed25519" in any of:
    /home/me/go/src/crypto/ed25519 (from $GOROOT)
    /home/me/GOPATH/src/crypto/ed25519 (from $GOPATH)

What am I missing?

[aschmahmann]

@probonopd it looks like you're running an old version of Go. Try using Go 1.13.

[probonopd]

Thanks @aschmahmann. "Old", was using 1.12.
Now with 1.13.3 getting:

me@host:~/GOPATH/src/github.com/ipfs/go-ipfs/docs/examples/go-ipfs-as-a-library$ go version
go version go1.13.3 linux/amd64

me@host:~/GOPATH/src/github.com/ipfs/go-ipfs/docs/examples/go-ipfs-as-a-library$ go run main.go
(...)
go: finding github.com/syndtr/goleveldb v1.0.0
go: finding github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
# runtime
/home/me/go/src/runtime/stubs_x86.go:10:6: stackcheck redeclared in this block
    previous declaration at /home/me/go/src/runtime/stubs_amd64x.go:10:6
/home/me/go/src/runtime/unaligned1.go:11:6: readUnaligned32 redeclared in this block
    previous declaration at /home/me/go/src/runtime/alg.go:321:40
/home/me/go/src/runtime/unaligned1.go:15:6: readUnaligned64 redeclared in this block
    previous declaration at /home/me/go/src/runtime/alg.go:329:40

[Stebalien]

Go 1.12 would have worked but you need to export GO111MODULE=on.

Now you just have a problem with your go install (possibly because you have go 1.12 and 1.13). You may want to look at https://golang.org/doc/install#extra_versions.

[probonopd]

This whole GO111MODULE stuff is already one of my least favorite "features" in Go. But anyway thanks for your help.

[probonopd]

Thanks @aschmahmann and @Stebalien, it is working for me now :+1:

How can I achieve the equivalent of --nocopy when using go-ipfs as a library add a file?

https://github.com/ipfs/go-ipfs/tree/master/docs/examples/go-ipfs-as-a-library

[aschmahmann]

@probonopd there is a pattern used throughout the ipfs (and libp2p) codebases which looks like https://github.com/tmrts/go-patterns/blob/master/idiom/functional-options.md.

You can see that https://github.com/ipfs/go-ipfs/blob/3a793a4cc6caee145c0d5a9aec4c6aa6444cbeb8/docs/examples/go-ipfs-as-a-library/main.go#L240

adds a file. However, you can also pass an arbitrary number of options.UnixfsAddOptions into the Add function (where options is the package https://github.com/ipfs/interface-go-ipfs-core/options).

Looking around in that package we see that options.Unixfs.Nocopy(true) is the function that we can pass into Add to get --nocopy. The function then looks like ipfs.Unixfs().Add(ctx, someFile, options.Unixfs.Nocopy(true)).

P.S. You're probably better off asking questions like this on discuss.ipfs.io or the IPFS public chat (available via IRC, Matrix and Discord)