Js-ipfs: Add support for Rabin fingerprinting to js-ipfs

[daviddias]

For feature parity with go-ipfs

https://github.com/ipfs/notes/issues/1

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__This issue now has a funding of 0.4 ETH (247.87 USD @ $619.68/ETH) attached to it.__

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $14,616.85 more funded OSS Work available on the Gitcoin Issue Explorer

[vs77bb]

Hi @diasdavid, Vivek from Gitcoin here. We posted a small bounty on this on Gitcoin to see if it brings some attention. If you'd like any other bounties, let us know - we'd love to support!

[gitcoinbot]

@rafaelboyero Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] warning (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[timxor]

Still need help on this?

[haraldrudell]

The ECMAScript client is in need of a rewrite: it uses some pre-2015 non-standard streams implementation, closures and dictionaries of functions when today's answer would be async iterators. Not a class in sight

with the go client, one can do
ipfs add --chunker=rabin-48-96-192 LICENSE
added QmaJfZamtgZAPgYfPe5HbjEzbgf3YHgCUCu7yfDadwNXSd LICENSE
1.06 KiB / 1.06 KiB [================================================]
ipfs object links QmaJfZamtgZAPgYfPe5HbjEzbgf3YHgCUCu7yfDadwNXSd
QmYcQE8seMKSqWq2GZfcpnfK3WgPXLqhtryNEfRX6TXqsy 185
QmdH3L1zu7hWu7nrpoCe33zfn3hDm6baYpTSENbRitpcFu 140
Qmaw7roFAKSj3V3QxmFUfpqU3dqAEMPMqHynqXxJmhhGhB 64
Qmewm77gB4V7cgnZLsuikXgrG8Q38M1TX8qPRBcKrw4yJ4 90
QmRd6EoLAyp6TXZmSRTByaAgEn5632J8u9NwFSiHfXurLx 95
QmeLhozoTP1z3RiLRBRUuLJ7xpmDh5oi1ShN91B4UPjRwA 172
QmdxhoXpK74gGbm2G8bAoCiuqDfbq44rN6RZKefCAFEhoZ 99
QmZNEhqU6uVjMc7p7xfqfwQK5YGxe1bdUNdG6Hq81149iw 139
Qmd1yFj2ZuNd28r5MaQaWkr3vcUwPF61i8oBdfRjaUC4V5 64
QmeJhcwipWbjXA5DdXVGe6Sh8kwBzADNnw3Zd3HKcPtPkm 82
Qme1G8Mk1PLThzCr11NzChQVy1FJoHCQkjK7rB2iTWKkaJ 58

[ceresstation]

Hey @tcsiwula - yes! If you'd still like to take it on let me know.

[mikeal]

https://www.npmjs.com/package/rabin

On Fri, Jul 13, 2018, 9:41 AM Scott Moore [e-mail] wrote:

Hey @tcsiwula https://github.com/tcsiwula - yes! If you'd still like to
take it on let me know.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ipfs/js-ipfs/issues/1283#issuecomment-404767964, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAACQxuqhUciU8BCkcaplXmDWsFi06_Rks5uGF1RgaJpZM4S9D0X
.

[dordille]

Looking for help on this? Took a look at the rabin module linked by @mikeal and was wondering if there is any concern about compatibility with the browserify build of js-ipfs, not too familiar with that process and whether modules that have c dependencies are included elsewhere.

[mikeal]

While there isn't a browser version we can just set the browser field in package.json to{'rabin': null}. That way, in the browser, the require() statement will just return null and we can check for it and opt out of these feature later in the code.

[dordille]

Submitted datproject/rabin/pull/24 to allow for configuring the window size and polynomial used by the chunker. This will allow chunking used by js-ipfs to match go-ipfs-chunker.

Will have PRs up shortly for ipfs/js-ipfs-unixfs-engine and ipfs/js-ipfs

[mikeal]

Should we consider surfacing these chunker settings in unixfs so that, when files are changed, they can be re-chunked to more closely match the prior chunking?

[dordille]

I'd have to defer to someone that knows more about how often settings like window and the polynomial will be changed. Looking at the equivalent go source, it doesn't seem like there is a practical way to alter the window size or the polynomial used when chunking.

Polynomial: https://github.com/ipfs/go-ipfs-chunker/blob/master/rabin.go#L11
Window: https://github.com/whyrusleeping/chunker/blob/master/chunker.go#L16

It would be pretty straightforward though to update the chunkers in unixfs to export default values used in addition to the stream handler.

[gitcoinbot]

⚡️ A tip worth 0.40000 ETH (106.31 USD @ $265.77/ETH) has been granted to @dordille for this issue from @vs77bb. ⚡️

Nice work @dordille! To redeem your tip, login to Gitcoin at https://gitcoin.co/explorer and select 'Claim Tip' from dropdown menu in the top right, or check your email for a link to the tip redemption page.

• $17237.84 in Funded OSS Work Available at: https://gitcoin.co/explorer
• Incentivize contributions to your repo: Send a Tip or Fund a PR
• No Email? Get help on the Gitcoin Slack

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

__The funding of 0.4 ETH (106.31 USD @ $265.77/ETH) attached to this issue has been approved & issued.__

• Learn more on the Gitcoin Issue Details page
  
  
  
  • Questions? Checkout Gitcoin Help or the Gitcoin Slack
  
  
  • $17,131.53 more funded OSS Work available on the Gitcoin Issue Explorer
  
  

[vs77bb]

Hi @dordille great work here. We tipped you for your work via the Gitcoin bounty which was posted on the issue 🙂