Gun: Try Switching to @PeculiarVentures/webcrypto instead of @PeculiarVentures/node-webcrypto-ossl for Electron?

[tongtwist]

Context

I published an application based on [email protected] and build+publish+update using last version of electron-builder.
I want to add a new feature in which I want to use Gun.

Problem

I cannot add Gun as a new dependency since its own native dependency node-webcrypto-ossl failed to build on [email protected]^.

Symptoms

node-gyp fail on node-webcrypto-ossl build tries since it assume to find an external openssl library dep for the nodejs crypto module used in webcrypto, and so in node-webcrypto-ossl.

But the SSL part of crypto seems to not be furthermore an external dep of recents nodejs versions (and at least on [email protected]^ that is embed in [email protected]^)

==> After a small search, It seems to only be an Electron concern since Electron's NodeJS' headers don't include openssl files. The nodejs embeded in these electron releases seems to be customized on this point.

I'm not sure, but on the node-webcrypto-ossl's author side, they seems to not try to support these recent versions of electron, and they often suggest to switch to @peculiar/webcryptoin this case pattern.

Quick patch attempt

Add a postinstall entry in gun's package.json scripts that replace the optional dependency node-webcrypto-ossl with an alias on one of the adapted module (node-webcrypto-ossl or @peculiar/webcrypto) and modify the module requirement in AXE (in a weird and optimistic way to hack the problem without having to deep dive in the concerned code) ... fail ...

How to reproduce

Create an empty repo with the following package.json content and try a npm install

{
  "name": "gun-electron6",
  "version": "0.0.1",
  "main": "index.js",
  "scripts": {
    "start": "electron .",
    "postinstall": "electron-builder install-app-deps"
  },
  "license": "MIT",
  "dependencies": {
    "electron": "~6.0.3",
    "electron-builder": "^21.2.0",
    "gun": "^0.2019.726"
  }
}

Christmas list

Be able to add all (including Axe) GunDB in my [email protected]^ app (without having to rollback to any old Electron release)
If any idea, I will take it gratefully.
...
Nice job with Gun anyway

[amark]

@tongtwist WELCOME TO THE COMMUNITY!!!! :) :) :) :)

Thanks for your kind comments.

Yeah, me too, I'm considering trying out their other crypto library because ossl and node-gyp seem to fail a lot, and that is causing devs a bad GUN experience :( :( :(

Another option, if it lets me potentially mark this issue as "solved" for the meanwhile... is, do you even need SEA / cryptography in your Electron app? If you're mostly dealing with local-only data, then all your data is trusted.

Or does your electron app talk to other users? In which case, cryptography is still needed to keep the convos secure or private.

[tongtwist]

Thanks for your response.

Unfortunately, I need to protect access to some of my users private data in any cryptographic way because all of my users' data will be in the same p2p space. That's an important point of the feature I want to add in my app.
Then, SEA benefits were strong reasons for me to use GUN to handle this p2p data space.

I can wait for some weeks.

I hope you will be able to produce a gun release that do not fall in the "openssl/boringssl" Electron's native dependancy problem (maybe by Switching on @peculiar/webcrypto or any other lib (why not a wasm compiled one if you have performance issues? ))

If I find the time I may try to do some tests on a wasm solution.

[amark]

@tongtwist yeah, my current priority is to fix these RAD (storage system) errors "no ack, lack", but I agree this is important.

Any chance you could try swapping the ossl package for the webcrypto one and re-run the tests? If fewer than 3 fail, this should be easily doable.

[tongtwist]

@amark Hi. I prefer to not do any promise I cannot "resolve" in a short time. Too much work here.
But, I put it in my todo. I'll be back.

[amark]

I tried swapping ossl for the same team's https://github.com/PeculiarVentures/webcrypto

but I started getting btoa not defined errors in NodeJS since btoa doesn't exist there, but this is odd, because I know @mhelander already coded it to work in both environments already, so I find it odd that NOT having ossl suddenly results in btoa being used, when it should be using NodeJS's native buffers (or a wrapper over it) regardless of what Crypto library is used.

@mhelander any idea?

[tongtwist]

Hi everyone.

@amark As I tried to swap ossl, I encountered the same pb, and to have a big picture of the swapping exercice, I wrote this kind of fast workaround on each btoa() invocation; for example in SeaArray.prototype.toString():

if (enc === 'base64') {
    const result = Buffer.from(this.slice(start, end)).toString(enc)
    ...

After, you will have some same problems with atob() use; I solved it with the same kind of quick fix.
After that, I encountered some unit test fails on mismatch signatures.
Then my ideas were

• Even if the @Peculiar/webcrypto lib should implement the same interface that the current webcrypto lib, I may fall in some hidden behavior differences between these two libs
• I changed something in the SEA behavior with my quick workarounds that disturb any step in the signature verification process (maybe in the custom SafeBuffer or SeaArray implementations)

But I did not have the time to investigate furthermore.

I hope this can help you

[amark]

ahhhh @tongtwist thanks, I must have missed that.

Yeah, @mhelander it seems like @PeculiarVentures 's 2 projects maybe are not interchange-able (?).

Any chance @rmhrisk or @microshine willing to reply?

node-webcrypto-ossl is the ONLY library we've found that actually works as promised, so we've proudly been using it and it is fantastic!

Though, needing to node-gyp OpenSSL & Python complicates the install process. Which is fine (I'd rather have real working cryptography than not!), but that is why I got really excited when I saw your other project @PeculiarVentures/webcrypto is NodeJS only and looks like it supports all the curves we need (I think), that maybe we could ditch ossl for this other repo of yours.

Should that be possible?
Or are the different libraries incompatible?

[rmhrisk]

First, we do support updated versions of Electron with node-webcrypto-ossl but as an OSS project, we provide support in between other work so it may not happen instantly.

Second, both node-webcrypto-ossl and @peculiar/webcrypto are based on the webcrypto standard and with the exception of additional algorithms and underlying crypto implementation they should be interchangeable. In fact. both of these libraries are based on a common library called webcrypto-core that implements the majority API conformity aspects of the libraries.

We do encourage projects not familiar with C++ and native bindings to use @peculiar/webcrypto, 99% of the time when people encounter an issue with node-webcrypto-ossl it has to do with environmental differences, usually missing headers, native dependencies not being in standard locations, etc.

Long term we do want to see people move to the @peculiar/webcrypto package mainly because it takes less for us to support. Many projects have already made this transition successfully.

As for it not working in node (the btoa error) it does, we run our regression tests in node.

There is no use of btoa in @peculiar/webcrypto - or in webcrypto-code.

It is was in node-webcrypto-ossl but it was removed (see here for why). So I am not sure why you're getting this error or how our two libraries are the source.

Since no bug has been filed in either of the respective projects this tagging us here is the first we have heard of this issue. We are not experts in gun, so the best way to get our help on these libraries is to file a bug in one of those repositories with a small repro of the issue and we can try to use that to fix it.

In general though since your a purejs project we thing moving to @peculiar/webcrypto is the right thing to do and are happy to help answer questions.

[amark]

@rmhrisk you're fantastic! Thank you so much, this answered exactly what I was looking for:

That yes, @peculiar/webcrypto is compatible and worthwhile to switch to.

Thank you!!!!

The other issues, including btoa is our fault, not yours. I was just wanting to make sure switching was a smart choice. I'll be putting more effort into this transition.

Thank you and your team once again for such incredible and important libraries. Go Open Source!!!

[mboperator]

First, Thank You

Thanks for making Gun.
I recently switched my project off of Firebase and it was fairly painless.

What's going on

I'm getting this issue with Electron 4.X.X as well.

The node-webcrypto-ossl build error is the last thing that's preventing one of my projects from being fully decentralized.

What I did

I gave migrating the library a quick stab and got the test suite passing.
PR is linked above.
Let me know if you want me to make any changes.

What's next

I'm going to attempt linking gun locally to see if this solves my issue.

[mboperator]

Update

My project now builds properly.
[email protected]
electron-builder@^21.2.0

@tongtwist, if you'd like to attempt rebuilding your project with the webcrypto changes I rebuilt gun with my changes and put them in this branch:
https://github.com/mboperator/gun/tree/sea-switch-to-webcrypto-built

[amark]

@mboperator woah dang!!! That was fast! I tried something similar... you must have the magic touch!!!

I'm very excited for this, will check it out Monday and try getting it pulled & published. THANK YOU!

[amark]

FIXED BECAUSE OF @mboperator WITH https://github.com/amark/gun/pull/814 !!!!

Closing :D :D :D :D :) :) PARTY PARTY PARTY 🎉 🎉 🎉 NO MORE PYTHON/OSSL NEEDED because of THE AMAZING WORK of @PeculiarVentures @rmhrisk @microshine @LiraNuna on a PURE NodeJS WebCrypto polyfill!!!

This should be great with Electron @tongtwist !