Stacks.js: The storage API is difficult to use in Node.js

[jcnelson]

The storage API (getFile, putFile) depends on the following browser-isms to work correctly:

• location.search must exist as a global
• window must exist as a global
• window.location.origin must be set
• window.location.search must be set to ?authResponse=...
  
  
  
  • The developer must create a fake but well-formed authResponse JWT for the above
  
  
• window.localStorage must exist, and must implement the semantics of the LocalStorage API

This makes using Gaia in Node.js programs somewhat painful and unintuitive. I have had to add the following to my top-level index.js get it to work at all (and even then, I'm not 100% sure this is sufficient):

function makeFakeGaiaSessionToken() {
  const appPrivateKey = '583d0cf5572735a95bf2d6de10952131fedbe9c72b272d5081e11e7f77e576c6'
  const ownerPrivateKey = '24004db06ef6d26cdd2b0fa30b332a1b10fa0ba2b07e63505ffc2a9ed7df22b4'
  const transitPrivateKey = 'f33fb466154023aba2003c17158985aa6603db68db0f1afc0fcf1d641ea6c2cb'
  const transitPublicKey = '0496345da77fb5e06757b9c4fd656bf830a3b293f245a6cc2f11f8334ebb690f1' + '9582124f4b07172eb61187afba4514828f866a8a223e0d5c539b2e38a59ab8bb3'

  window.localStorage.setItem('blockstack-transit-private-key', transitPrivateKey)

  const authResponse = blockstack.makeAuthResponse(
    ownerPrivateKey,
    {type: '@Person', accounts: []},
    null,
    {},
    null,
    appPrivateKey,
    undefined,
    transitPublicKey,
    null);

  return authResponse;
}

const localStorageRAM = {};

global['window'] = {
  location: {
    origin: 'localhost',
    search: `?authResponse=${makeFakeGaiaSessionToken()}`
  },
  localStorage: {
    getItem: function(itemName) {
      return localStorageRAM[itemName];
    },
    setItem: function(itemName, itemValue) {
      localStorageRAM[itemName] = itemValue;
    },
    removeItem: function(itemName) {
      delete localStorageRAM[itemName];
    }
  }
};
global['localStorage'] = global['window'].localStorage

Moreover, this only works with the upcoming blockstack.js 18.x.x (I haven't gotten it to work with 17.x.x).

Any thoughts or recommendations on what we can do here? @larrysalibra @kantai @yknl

[jcnelson]

Also, to use getFile(), I first have to call handlePendingSignIn() and call getFile() in the then() branch. The reason for all of this is that handlePendingSignIn() both depends on the availability of window.localStorage and location.search, which it then uses to set the local Gaia hub connection (via getOrSetLocalGaiaHubConnection()).

[kantai]

Multi-player reads are fairly straight-forward. There is one offending line at:

https://github.com/blockstack/blockstack.js/blob/develop/src/storage/index.js#L315

The following code works:

window = { location: { origin: false } }
bsk.getFile('statuses.json', { 
    decrypt: false, verify: false, username: 'collectivism.id',
     app: 'https://app.travelstack.club' }).then(console.log)

If we fix that line, then multi-player reads will work (buffer support still needs the cross-fetch update from #492) without setting the fake window global.

putFile and single-player getFile --- I'm not sure if they make sense outside of the context of the browser, which is where blockstack.js mostly assumes client sessions should live.

This seems like a great argument in favor of moving a lot of the gaia functionality to a gaia.js library, which blockstack.js would just include and call --- blockstack.js is mostly a webapp SDK, whereas gaia.js would be callable from node.

[jcnelson]

The following code works:

Ah, I didn't realize I needed to explicitly set verify: false and decrypt: false. Makes sense given that there's no app key set in localstorage.

putFile and single-player getFile --- I'm not sure if they make sense outside of the context of the browser, which is where blockstack.js mostly assumes client sessions should live.

Node.js apps that need to either upload data or download and decrypt/verify data still seem to need my above hacks to work (the CLI needs them, for example).

This seems like a great argument in favor of moving a lot of the gaia functionality to a gaia.js library, which blockstack.js would just include and call --- blockstack.js is mostly a webapp SDK, whereas gaia.js would be callable from node.

Definitely!

[bodymindarts]

Since making blockstack.js work in node.js requires decoupling from browser primitives, it would be great if the web-worker context would also be taken into account. I've used blockstack.js in web-workers, but with similar hacks on the top level context as are required for node.js.

[x5engine]

@jcnelson @bodymindarts why not add a test in blockstackjs that checks if window and other web browser globals are defined otherwise it won't throw an error and assume it is a server-side thing?

Example

 const defaults = {
    decrypt: true,
    verify: false,
    username: null,
    app: window ? window.location.origin : "",//or whatever it should be
    zoneFileLookupURL: null
  }

Instead of hacking around it ?

[larrysalibra]

I'm also seeing a similar problem with our encrypt/decryptContent methods. Will put this on the list to address for our next release. Specifically, if you don't specify a private key it gives you an error about window not existing instead of something useful.

[sirpy]

FYI loadUserData is also used in storage and uses window

[AC-FTW]

@larrysalibra, Stealthy @prabhaav is working on a project that could really benefit from this functionality. @jcnelson mentioned you were looking into this.

We're specifically interested in doing the following all within node:

• Blockstack Auth
• GAIA read and write
• GAIA multiplayer read (kinda easy as @kantai mentioned--we're already spoofing this in iOS)

We're shooting for the end of this month and are happy to guinea pig your changes if possible.

[larrysalibra]

We're shooting for the end of this month and are happy to guinea pig your changes if possible.

@AC-FTW That would be great! I'll hopefully have updates for you soon!

[jcnelson]

Just adding to this list of known difficulties (happy to move to another issue if you think it's too different): there is a growing use-case for allowing users to generate and pass in authentication tokens and addresses to the putFile() and getFile() calls. For example, if user A is storing data to user B's Gaia hub, then user A needs either an authorization token signed by user B's private key, or an authorization token with an embedded association token signed by user B. User A would need to obtain this information out-of-band (i.e. via the app), and then supply it in these calls. Was thinking this could be done by expanding the options argument to include a Gaia hub config, and was thinking we could add/expose the methods for generating authorization tokens (and later, association tokens).

[florin0x01]

Hi @jcnelson , used the snippet above with blockstack 19.x (calling makeAuthResponse) and got

Authenticating with protocol > 1.1.0 requires transit key, and none found.

Full code :

function makeFakeGaiaSessionToken() {
    const appPrivateKey = '<my app private key>'
    const ownerPrivateKey = '<owner private key>'
    const transitPrivateKey = 'f33fb466154023aba2003c17158985aa6603db68db0f1afc0fcf1d641ea6c2cb'
    const transitPublicKey = '0496345da77fb5e06757b9c4fd656bf830a3b293f245a6cc2f11f8334ebb690f1' + '9582124f4b07172eb61187afba4514828f866a8a223e0d5c539b2e38a59ab8bb3'

    window.localStorage.setItem('blockstack-transit-private-key', transitPrivateKey)

    const authResponse = blockstack.makeAuthResponse(
      ownerPrivateKey,
      {type: '@Person', accounts: []},
      null,
      {},
      null,
      appPrivateKey,
      undefined,
      transitPublicKey,
      null);

    return authResponse;
  }

  const fakeToken = makeFakeGaiaSessionToken()
  window['location']['search'] = `?authResponse=${fakeToken}`


function blockstackListFiles() {
    const appConfig = new blockstack.AppConfig()
    const userSession = new blockstack.UserSession({ appConfig: appConfig })
    userSession.handlePendingSignIn(fakeToken).then(userData => {
        userSession.listFiles((fileName) => { console.log(fileName); return true; }).then(numFiles => {
            console.log(numFiles)
        })  
    })
}

blockstackListFiles()

[florin0x01]

Ok, so I called userSession.generateAndStoreTransitKey() before handlePendingSignIn and got the following :

Failed decrypting appPrivateKey. Usually means that the transit key has changed during login.

[florin0x01]

Ok , so the transit keypairs are wrong, but how to get the transit public key ?

[florin0x01]

Finally got it working, so you can call blockstack.getPublicKeyFromPrivate(transitPrivateKey) .

[yknl]

This should be possible now in the collections branch by specifying a custom Gaia hub config in the storage functions. https://github.com/blockstack/blockstack.js/pull/684

[ruslankonev]

Hi, I was look this issue moves to Mongo/Collections way, but tell me please, at now possible to read public files, with defined put/get file options and without have an account in blockstack?
I want to add an ability to read files from blockstack without create account.

[zone117x]

@ruslankonev Sure, that has been possible for a while:

const userSession = new blockstack.UserSession({ 
  appConfig: new blockstack.AppConfig()
})
const content = await userSession.getFile('testfile.txt', { 
  decrypt: false, 
  zoneFileLookupURL: 'https://core.blockstack.org/v1/names/',
  username: 'test_matt_24482723.id.blockstack',
  app: 'http://localhost:9000', // Set to your app domain
})
console.log(`Result: ${content}`)

If you plan on doing this in a production node.js app, you should run your own core node rather than using the public core node 'https://core.blockstack.org/v1/names/' -- otherwise your app will likely be rate limited / throttled.

[ruslankonev]

@zone117x In some cases I need a my own node of Blockstack core? What the rate limit on public node now?