Notes: IPFS Implementation - Haskell

[jbenet]

@cleichner and @ali01 want to start a Haskell implementation. Please coordinate here.

[bitemyapp]

Hello @cleichner @ali01!

@raptros, @puffnfresh, and myself are interested in exploring this. Please let us know if you've got anything anywhere.

[puffnfresh]

Think this was the repo:

https://github.com/cleichner/haskell-ipfs

Doesn't look like much was done but it's really good to know we're not the only ones wanting this.

[puffnfresh]

Think a Merkle DAG in Haskell would be a good place to start:

https://github.com/jbenet/random-ideas/issues/20

[bitemyapp]

@puffnfresh after having read the paper, I was thinking the same thing. Has the possibility to be extracted into a separate library as well. Do you know how to represent this in a persistent data structure? Being acyclic seems like it would help.

[jbenet]

@bitemyapp @puffnfresh great to hear guys! come by #ipfs on freenode to discuss. We're shooting to ship the go-ipfs alpha by the end of the year, so _a lot_ has stabilized (DRAFT 3 of the paper needs to be updated to reflect a few changes, too). some things still to change, but the formats + protocols are reaching a good place. (also, we'll be ripping out all the p2p networking related things into its own package, so it will be easier to implement things as two pieces).

on the Merkle DAG, important to note go-ipfs uses go-multihash and soon will use multicodec (though not there yet), so that people who dislike protobuf dont have to use it.

[tomberek]

Any progress on a Haskell implementation? This would be a good to have in many settings. I didn't seem much going on at https://github.com/cleichner/haskell-ipfs, perhaps the project has moved?

[davidar]

I'd also be interested in this. Haskell's first-class support for laziness and immutable data seems like a perfect fit for IPFS. For example, implementing an IPFS-backed library for transparent persistent memory (cf. VCache where VRef ~ IPFS block and PVar ~ IPNS name) would allow distributed IPFS objects to be treated like any other native (in-memory) datastructure, lazily pulling blocks from the network as necessitated by the computation.

Haskell can also provide strong safety guarantees allowing untrusted code received over the network to be safely executed. This could make it a practical option for the "compression as program" idea mentioned in #47 . Going further than that, I'm also picturing it allowing the ability to run distributed computation across the IPFS network (participating nodes run a given program over their locally-stored blocks, publishing the results back to IPFS).

It's also worth noting that there's a number of Haskell-to-JS compilers, which could be useful.

[davidar]

I've made a start on this: https://github.com/davidar/hs-ipfs-api

[bitemyapp]

@davidar I'm interested in this but held off before because there wasn't a proper spec. Has that changed? Can you link to one in the repo you started?

[davidar]

@bitemyapp My immediate goal is just to make a Haskell equivalent of node-ipfs-api, wrapping the RPC API provided by go-ipfs. Once this is done, it should be possible to start removing this dependency by implementing the IPFS protocol in Haskell, but I agree that should probably wait until the specs are a little more final (@jbenet could comment better about that than me). I'll put this in a README soon.

It's great that there's multiple people with an active interest in Haskell support for IPFS. It looks like you know Haskell much better than I do, so any thoughts you have on this will be greatly appreciated :)

[bitemyapp]

@davidar my suggestion, write a bit of code and ping me periodically for review.

I'd avoid wrapping something via FFI if you're not comfortable in Haskell, it'll force learning too much all at once. Just start writing it in pure Haskell, it'll be easier.

[davidar]

@bitemyapp Thanks. I'm not using FFI, just http-conduit+protocol-buffers. I'll probably move the (HTTP) RPC API bits into a separate module, which should (hopefully) make getting up to parity with node-ipfs-api relatively simple. I'll ping you once I get a chance to do this.

[bitemyapp]

@davidar looking forward to it :)

[jbenet]

@davidar this is a great way to start. And API bindings on more langs would be _great_ to have. (This motivated https://github.com/ipfs/ipfs/issues/83)

[davidar]

@bitemyapp I've separated out the API request logic, and implemented a couple of commands with it. Implementing the remaining read-only commands should be reasonably straightforward. Still need to extend the API to handle writes. Anyway, if you felt like doing a quick CR at this point, that'd be great.

At the moment I'm prioritising coordinating the archival efforts (and my thesis) so might be a little while until I can get these bindings up to scratch. Happy to accept pull requests though :)

[Ericson2314]

@davidar Thanks, I had never heard of vcache before your comment. That + ipfs seems like a most amazing thing.

[joelburget]

Somewhat related, I'm working on an IPLD implementation in Haskell. Still a work in progress but I'm able to, for example, property test by generating 100 random IPLD values, insert in IPFS, and verify that its generated CID agrees with mine. That entails correctly translating to JSON (for insertion to IPFS), correctly translating to CBOR, hashing the CBOR representation, and producing a CID from that.

We can also do other things like traverse Merkle paths and graft trees in to replace Merkle links.

I'm hoping to upload to Hackage pretty soon. The biggest things I need to do now are clean up a bit and write some documentation.

If you're curious, here's what the core data type looks like:

data Value
  = LinkValue !MerkleLink
  | DagObject !(HashMap Text Value)
  | DagArray  !(Vector Value)
  | TextValue !Text
  | DagNumber !Scientific
  | DagBool   !Bool
  | Null

You might notice it bears a striking resemblance to the core data type from Aeson.

[Kubuxu]

@joelburget that is just awesome. Haskell will be perfect for some data processing needs.

You can also add ipld-cbor straight to ipfs using: ipfs dag put --input-enc=raw as underlying format in this case is cbor.

[joelburget]

@Kubuxu You're right, I should switch to cbor for the dag put. What would be really awesome is if I could also get a value from IPFS as cbor. Is there any reason dag get doesn't support a format option (like ipfs object cat --fmt=yaml)?

[Kubuxu]

You can use ipfs block get CID to get binary CBOR out. We should make it clearer.

[whyrusleeping]

@joelburget

Is there any reason dag get doesn't support a format option

We just havent done it yet. Should be fairly simply to implement if youre interested in helping out :)

[daviddias]

yes please! Also, the remaining options so that I can built the rest of js-ipfs-api to match js-ipfs dag API -> https://github.com/ipfs/go-ipfs/issues/3771

[joelburget]

I am interested. Will try to find time in the next couple days unless someone else beats me to it.

[nek0]

Is there some other attempt at implementing ipfs than https://github.com/cleichner/haskell-ipfs (which seems rather dead)? If there are and need help, I am willing to help wherever I can. If not I am considering giving it a shot myself.

[whyrusleeping]

There are some people working on a Haskell ipfs in this issue:
https://github.com/MatrixAI/Forge-Package-Archiving/issues/1#issuecomment-302873144

Might be worth asking there

On Sat, Sep 23, 2017, 3:39 PM rys ostrovid [e-mail] wrote:

Is there some other attempt at implementing ipfs than
https://github.com/cleichner/haskell-ipfs (which seems rather dead)? If
there are and need help, I am willing to help wherever I can. If not I am
considering giving it a shot myself.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ipfs/ipfs/issues/4#issuecomment-331636560, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABL4HGauFYHoP0hCZevYVmVPJgnQNDh3ks5slQnzgaJpZM4CGesd
.

[CMCDragonkai]

@nek0 Hi, we've been working on a haskell implementation of ipfs's libp2p. All of our repos are here: https://github.com/MatrixAI (see the libp2p middle word) while we record our roadmap here https://github.com/MatrixAI/Forge-Package-Archiving/issues/1

[joelburget]

Hey @CMCDragonkai, I've done some work on my haskell-ipld project which overlaps a bit with what you're doing. I'm not focused on the p2p part, but I've built multihashes, cids, and ipld in Haskell. A few things which are nice about my package:

• For building / inspecting ipld it uses a familiar api similar to Aeson's.
• The tests are very nice. It does property-based testing and verifies against the official ipfs binary.
• It uses the new cborg package to handle all the low-level cbor munging.
• It uses Generics to automatically derive serializations to / from ipld.

It might make sense to collaborate.

[CMCDragonkai]

Hi @joelburget, I had a look at your work. Looks pretty good. It might be pretty useful to us, once we have libp2p running. What do you intend to do with it? Our first goal is to integrate NixOS's build system with ipfs (upstream dependencies).

[daviddias]

Bumping up this thread to ping @lukehoersten and @Unlambder who have been working on implementing multiformats in Haskell

[ghost]

Hey everyone I would've liked to look at an ipld implement after finnishing multicodecs but it looks like there's already someone on it, is there something else that can be done to move towards the ipfs implement?

[ghost]

@CMCDragonkai and I have a libp2p crypto library similar to go-libp2p-crypto, that needs some review:
https://github.com/MatrixAI/haskell-libp2p-crypto

We also have some repos that are a WIP:
Peer Identity info (similar to https://github.com/libp2p/go-libp2p-peer):
https://github.com/MatrixAI/haskell-libp2p-peer

Networking/Swarm/Muxer (just some prototype code atm, nothing working):
https://github.com/MatrixAI/haskell-libp2p-swarm

And a general issues/info page where we post some discussion about implementation:
https://github.com/MatrixAI/Forge-Package-Archiving/issues/1

I think the main thing to do is the networking interfaces (Swarm, Connection, Stream) from LibP2P as well as the Muxer interface and Muxer implementations for the spdy, http2 protocols. I'm currently working on that, but don't have much experience with writing network code, so worried a bit about performance.

For that reason, I'm looking at how the warp server (https://github.com/yesodweb/wai/tree/master/warp) does their multiplexing for HTTP2, as it seems they've put in some significant effort to writing a performant muxer (see: http://www.mew.org/~kazu/doc/paper/http2-haskell-2016.pdf). In this area, I think there's some discussion to be had about whether it's better to re-implement muxers for various multiplexing protocols specifically for libp2p; or whether, if it is possible, to write some glue code to fit the Muxer interface from already existing implementations, such as can be found in the Warp server.

Other related stuff from the libp2p-spec that needs to be looked at is peer discovery and peer routing, but I think that can come after we have some of the basic networking stuff done.

Anyway, any comments, feedback, review would be much appreciated. It would also be nice to have some other people working on or collaborating with us in this area, as I wouldn't consider myself very experienced compared to a lot of the Haskell community.

[whyrusleeping]

@plintx If you guys want to reuse the http2 multiplexing code from another package, I think that we are also interested in having an http2 muxer in go and js. The go standard library technically has all the pieces we need to implement it for go-libp2p, so if it would save you some time, we can work on patching that together.

[TravisWhitaker]

Any progress here? It'd be awesome to have this available.