Go-ipfs: Road to 0.5

[Stebalien]

[Living]

We're working towards a pretty large 0.5.0 go-ipfs release but still have some work remaining.

ETA: April 7th because apparently April 1st +/- a few days is reserved for jokes.

Release blockers

• Working (small-data) content routing. That is, the DHT should be useful again for finding small data sets.

Landed features

• Streaming pin ls results.
• "internal" daemon plugin for perma-unstable plugins with direct access to go-ipfs internals.
• Updated QUIC transport (backwards incompatible)
  
  
  
  • Updates the underlying QUIC protocol to draft 27. This should be the last breaking change to QUIC before it stabilizes.
  
  
  • Updates the handshake protocol to support ed25519 keys.
  
  
• Significantly improved bitswap performance when serving data from high-latency disks.
• Systemd support
  
  
  
  • Socket activation
  
  
  • Example unit files
  
  
  • Systemd startup/shutdown notifications
  
  
• Support for .eth links in /ipns/.
• A new and orders of magnitude faster (than rabin) content based chunker for ipfs add (buzhash).
• Unix domain socket support for the HTTP API.
• Identify Push - Libp2p now pushes announcements of address/protocol support changes made at runtime.
• Significant extensions to the bitswap protocol https://github.com/ipfs/go-bitswap/pull/189
• Async datastore writes in badger (2x performance improvements). NEEDS CONFIGURATION.
• Write coalescing: We previously added support for batching small writes but ran into a significant bug (https://github.com/libp2p/go-libp2p/issues/644) and had to pull out at the last minute. We have re-enabled this feature.
• We will now prefer the TLS security transport over SECIO.

Next Up

• QUIC by default. We've been working towards switching over to QUIC (a UDP-based transport) by default for quite some time. However, we've recently made some significant breaking changes to both the underlying protocol (bringing our implementation in-line with the draft spec) and the way in which we structure our QUIC TLS certificates.
• Badger based datastore by default. We've been testing out badger for well over a year now and plan on switching over in the near future. However:
  
  
  
  • Badger just released a 2.0 with some significant changes and an incompatible database format.
  
  
  • Badger 1.0, in our current configuration, uses a _lot_ of memory.
  
  
  • We still haven't thoroughly stress-tested badger.
  
  

Todo

• [x] Finish the remaining TODOs in testground: https://github.com/ipfs/testground/issues/196
• [x] Fix p2p circuit addresses https://github.com/ipfs/go-ipfs/issues/6868
  
  
  
  • Can't reproduce. May have failed to upgrade something properly.
  
  
• [x] Bitswap refactor (see linked issue for the checklist): https://github.com/ipfs/go-ipfs/issues/6782
• [x] Switch to asynchronous datastore writes by default in badger. - https://github.com/ipfs/go-ipfs/issues/6775
• [x] Switch to /p2p multiaddrs: https://github.com/multiformats/go-multiaddr/pull/115/files
• [x] Support base32 PIDs for IPNS.
• [x] Base32 encode keys in the keystore
  
  
  
  • [x] Feature: https://github.com/ipfs/go-ipfs/pull/6955
  
  
• [x] Fix logging to files on windows: https://github.com/ipfs/go-log/issues/73
• [x] Handle bootstrapper fallout: https://github.com/ipfs/go-ipfs/issues/6797
  
  
  
  • [x] Fix outstanding issues.
  
  
  • [x] Implement migration.
  
  
  • [x] Cut a migrations release and update go-ipfs to use it.
  
  
• [x] Add support for a libp2p event plugin (https://github.com/ipfs/go-ipfs/issues/6827). Our current network stats (i.e., number of nodes, we don't collect IP addresses) rely on crawling the DHT. However, nodes behind NATs will no longer join the DHT. We'll have to passively count peer IDs on the bootstrappers using a custom plugin.
• [x] Downgrade to go 1.13 to avoid scheduling/timing performance issues on go 1.14.
  
  
  
  • https://github.com/ipfs/go-ipfs/pull/7054
  
  
• [x] TLS by default: https://github.com/ipfs/go-ipfs/pull/7055
• [x] Fix a bitswap wantlist leak https://github.com/ipfs/go-bitswap/issues/310
• [ ] Fix libp2p address filtering https://github.com/ipfs/go-ipfs/issues/6995
• [ ] Upgrade the QUIC transport (downgraded as it was incompatible with go 1.13).
• [ ] Fix the DHT: https://github.com/libp2p/go-libp2p-kad-dht/milestone/3
• [ ] Maybe, maybe badger by default. It's back on the menu because we got it down to ~20MiB of memory.

[godcong]

~https://github.com/dgraph-io/badger 2.0.0 was released a period of time.
I tried to compiled it into 0.5.0.
it works well.
perhaps we can integrate it when 0.5.0 released.~

I have seen this plan.

[Stebalien]

Status update from testground on unblocking the release.

[momack2]

Very glad this list exists - thank you! To help ease the tracking of the remaining burn-down items as we get closer to launch - can you add owners to each of the remaining todos and create or link tracking issues (with scope of required tests) for each of the "test with testground" items?

[bonedaddy]

Status update from testground on unblocking the release.

Is there any more information about timeline? While this is great and all, it mentions nothing about timeline for v0.5.0. Is it one month? Two months? Or 6 months.

[Stebalien]

The ETA is end of march +/- a week.

[lordcirth]

The build process hasn't changed, right? Hoping to submit 0.5 to nixpkgs when it releases.

[Stebalien]

The build process is still make build. There shouldn't be any significant changes since 0.4.23.

Thanks!

[RubenKelevra]

I resurrected the aur-package from the death (last update was 2017), have fun:

https://aur.archlinux.org/packages/go-ipfs-git/

[Stebalien]

Switching over to a proper release issue: https://github.com/ipfs/go-ipfs/issues/7109