Metamask-extension: Add Web3 1.0 support

[danfinlay]

Appears to include:

• [ ] Adding support for websockets to provider-engine https://github.com/MetaMask/provider-engine/issues/189

[kumavis]

or just polyfill subscriptions
also: we're already no longer using provider-engine! : )

[GrandSchtroumpf]

Is it normal if I cannot find MetaMask provider with Web3 1.0 ?
When I check on Web3.givenProvider || Web3.currentProvider, I always got "null", even after several seconds...

[danfinlay]

Not capitalized. Just web3.currentProvider

[GrandSchtroumpf]

Ok, I tried, it works, thanks. It tooks 500ms to find.
Just for you to know, here is what we can read on Web3 1.0 documentation :
"After that you need to create a web3 instance and set a provider. Ethereum supported Browsers like Mist or MetaMask will have a ethereumProvideror web3.currentProvider available, web3.js is setting this one to Web3.givenProvider. If this property is null you need to connect to a remote/local node."

Which is wrong because I can find web3.currentProvider, while Web3.givenProvider is null.

[danfinlay]

Since the Web3 class is the one you're importing from web3 1.0, I think they're instructing you how to assign the provider to their object. They're not telling you what to check. If this was confusing to you, you should open an issue on their documentation.

[GrandSchtroumpf]

Hmmm, indeed it makes sense. Maybe my English is guilty here, but if I misunderstood, maybe other will. Thanks anyway, it makes my life much more simple.

[GrandSchtroumpf]

Any idea when "web3.eth.subscribe" will be available with MetaMask ?

[gitcoinbot]

__This issue now has a funding of 0.3 ETH (241.43 USD) attached to it.__

• If you would like to work on this issue you can claim it here.
• If you've completed this issue and want to claim the bounty you can do so here
• Questions? Get help on the Gitcoin Slack
• $11752.94 more Funded OSS Work Available at: https://gitcoin.co/explorer

[edkek]

If no one is currently working on this issue, I'd like to tackle it.

What's the best way to start this? Is there a gitter or something similar to discuss details?

[owocki]

@edkek you can use the github issue page :) or https://gitter.im/MetaMask/Lobby

[kumavis]

here is my roadmap for this feature:

• [ ] identify new subscription rpc methods and their spec used by web3 1.0
• [ ] identify how web3 1.0 gets its server-push notifications on its provider
• [ ] add a matching api to https://github.com/MetaMask/eth-json-rpc-middleware/blob/master/providerFromEngine.js
• [ ] add a server-push interface to json-rpc-engine
• [ ] create a eth-json-rpc-middleware that something that implements the subscriptions + tests

i'd like to separate the notions of websocket transport and subscription support. I think it should be easy enough to provide subscription support on top of http rpc (which will Infura's life easier) with the help of eth-block-tracker.

[awgneo]

Will Infura be content with the many polling calls introduced by eth-block-tracker?

[kumavis]

@awgneo we already use the block-tracker, it effectively reduces the number of requests made by a dapp via per-block client side caching

[gitcoinbot]

__The funding of 0.3 ETH (221.33 USD) attached has been claimed by @muhammaddava1321.__

@muhammaddava1321, please leave a comment to let the funder (@owocki) and the other parties involved your implementation plan. If you don't leave a comment, the funder may expire your claim at their discretion.

• Learn more on the gitcoin issue page
• Questions? Get help on the Gitcoin Slack
• $11701.18 more Funded OSS Work Available at: https://gitcoin.co/explorer

[owocki]

@muhammaddava1321 please introduce yourself

[owocki]

@muhammaddava1321 if i dont hear from you soon, im going to reject your claim and let someone else take this. nothing personal, just want to make sure someone who's making coordinating with everyone a priority has the claim. let us know!

[owocki]

i am rejecting this claim due to radio silence from the claimee

[choochootrain]

@owocki i'd like to claim this! i'll start poking around and get a preliminary changeset out

[owocki]

@choochootrain can you please submit a claim on https://gitcoin.co/funding/details?url=https://github.com/MetaMask/metamask-extension/issues/2350 ? thats the source of truth for who has it claimed :)

[gitcoinbot]

__The funding of 0.3 ETH (221.28 USD) attached has been claimed by @choochootrain.__

@choochootrain, please leave a comment to let the funder (@owocki) and the other parties involved your implementation plan. If you don't leave a comment, the funder may expire your claim at their discretion.

• Learn more on the gitcoin issue page
• Questions? Get help on the Gitcoin Slack
• $10765.28 more Funded OSS Work Available at: https://gitcoin.co/explorer

[choochootrain]

here's my mental model of this so far: i have to read up on the new web3 surface area and work backwards to see what changes need to be made to the metamask provider (this is json-rpc-engine with the eth-json middleware?), which uses eth-block-tracker under the hood. @kumavis is this accurate?

i'll dig into this further on monday - happy new year!

[owocki]

@choochootrain ill let the metamask folks comment on the scope... (but they might be busy catching up from the holidays)

looking forward to seeing this done :)

[kumavis]

@choochootrain yep. see my roadmap for this feature above:
https://github.com/MetaMask/metamask-extension/issues/2350#issuecomment-353437246

[benjamincburns]

I've been working on websocket, pub/sub, and web3 1.0 support for ganache-core over on trufflesuite/ganache-core#14 (though don't look too closely yet, as the PR does not yet include my WIP changes which are fairly subsantial).

We use the provider engine in ganache-core, so my work has kind of bumped up against this issue, as well as MetaMask/provider-engine#189. Here's what I've done so far in the context of @kumavis' roadmap above.

• [x] identify new subscription rpc methods and their spec used by web3 1.0
  new methods are eth_subscribe and eth_unsubscribe, with the first param of eth_subscribe denoting the subscription type. Thankfully, the subscription types map quite cleanly back to the existing RPC filter calls

• [x] identify how web3 1.0 gets its server-push notifications on its provider
  From what I could find in the source, notifications are sent via a data event emitted by the provider. That is, web3 1.0 pukes if you use contract Events but your provider doesn't have an on method (or maybe it was once, I forget).

• [ ] add a matching api to https://github.com/MetaMask/eth-json-rpc-middleware/blob/master/providerFromEngine.js
  I have done this already for ganache's provider type -- will link to it in a future comment here when I update the PR (might be a couple of days)

• [ ] add a server-push interface to json-rpc-engine
  I'm not familiar w/ this component, sorry

• [ ] create a eth-json-rpc-middleware or something that implements the subscriptions + tests
  So far this has been the biggest stumbling block for me. I tried to do this by creating a SubscriptionSubprovider which extends the FilterSubprovider and makes use of its existing filter types for subscription handling. I wanted to do this without modifying FilterSubprovider, but the existing structure of it made it impossible to achieve w/o a race condition. Since there aren't good extension points, I'm figuring out how best to modify it without spilling its guts on the floor. Will submit a PR for that and its child, SubscriptionSubprovider. Relevant issue: MetaMask/provider-engine#189

Apart from the above, I've also modified heavily classes called WebSocketServer and ConnectionManager from the PR which I linked above. The latter type is meant to handle routing of subscription notifications to the correct WebSocket connection, and clean up subscriptions when websocket connections close.

HTH.

[kumavis]

add a matching api to https://github.com/MetaMask/eth-json-rpc-middleware/blob/master/providerFromEngine.js

this part is easy (here's a PR), but requires

add a server-push interface to json-rpc-engine

to wire it up

[kumavis]

I think create a eth-json-rpc-middleware that implements the subscriptions should be pretty easy, but will check in with @benjamincburns tomorrow on that specific part

[benjamincburns]

Managed to get the SubscriptionSubprovider working today with only minor changes to the FilterSubprovider. Will clean it up in the morning and submit a PR.

[benjamincburns]

I've raised MetaMask/provider-engine#207 for the SubscriptionSubprovider work mentioned above.

[choochootrain]

@benjamincburns it looks like you have a good handle on what needs to be done :)

it looks like tackling the json-rpc-enginee and related work is where i'll start tomorrow.

[choochootrain]

didnt get a chance to get to this today unfortunately - if anyone wants to take this on feel free, otherwise ill hopefully get time this weekend

[benjamincburns]

@choochootrain & others on this thread - I work for ConsenSys, so to the best of my knowledge, I'm not eligible for this bounty. My main reason for doing the work that I've done is to get trufflesuite/ganache-cli#257 finished, which is blocked on some, but not all, of the items in @kumavis' list above. We use the web3-provider-engine, not json-rpc-engine, so any work I'd do there wouldn't be in service of my goal.

I'm posting here because the work I'm doing is very analogous to the work that'll need to be done for MetaMask. I figured people trying for the bounty here might benefit from seeing how I'm accomplishing this task in a similarly structured project.

Speaking of which, have a look at websockets-pr-redux over on trufflesuite/ganache-core and compare it to the develop branch if you'd like to see my work thus far.

[benjamincburns]

My WIP for ganache and friends is over at trufflesuite/ganache-core#49, should anyone want to use it as an example. Explanation as to how it all works is in the description.

[vs77bb]

@choochootrain Howdy, I just joined Gitcoin alongside @owocki. Checking in to see your progress here? Happy new year everyone!

[choochootrain]

hey @vs77bb :wave:

i'm way more swamped that i expected right now so the earliest i can get to this is next week..if anyone wants to take this on in the meantime feel free!

[owocki]

@choochootrain if it's okay with you i will reject your claim and let someone else take this

[choochootrain]

:+1: go for it @owocki

[owocki]

ok i've rejected the claim so that someone else can pick it up! @vs77bb

[vs77bb]

@owocki If I recall correctly, @danfinlay said this one is near completion with work from @benjamincburns!

[owocki]

@benjamincburns

we talked internally about you taking on this bounty... basically the question boiled down to: 'is it ethical for consensys team members to compete w. the general public for the bounty? does it fit within the mission to push open source forward'.

i think the answer we decided on internally was that probably no for the first question, but yes for the second question.

in the spirit of being fair to the community and accomplishing our goals as a development community, we then talked internally and we like the idea of taking the ETH you would have earned on this metamask bounty and staking it on some truffle related issue after that .. thereby paying it forward to allow the gitcoin community to work on some truffle issues

id be curious if @kumavis and @danfinlay would be into that.. does that sound like a good middle path to the group?

[danfinlay]

I was just thinking about this yesterday!

• We want to put bounties on more and more things
• We don't want to rely on external contributors.

Therefore, we need to be able to work on issues with bounties, and yeah, this creates potential for unfairness, so we have to seek that middle path, as you've said.

• Consensys members are already salaried to work on this stuff, so it would be double dipping to pay us out. Salaried OSS jobs are like politicians: Security of income in exchange for consistency of dedication. Each time we decide a priority it's a potentially political move, so our salary is there to diffuse any incentive to act in a corrupt way.
• If we regularly closed our own bounties out from under external contributors, we would create an environment that's frustrating to contribute to, so we need to be incredibly transparent about when we're working on a bountied issue. This could be both with formal assignee/bounty claiming, or also with less formal "general protocols", like "we will always work on our highest priority issues first."

Anyways, I think we've done a good job here so far. Ben was very transparent about working on this, and then you've formally dropped the claim to open it for someone else, so looking good!

[benjamincburns]

Just to be clear to anyone reading along, at time of writing this bounty is still very much open and eligible for the community to scoop up.

I just want to clarify that I was under no expectation of receiving a bounty for this. I pinged @owocki yesterday just to ask what the official rules are.

While I agree that me being eligible to receive the bounty in this case would cross some ethical lines, I don't necessarily agree that ConsenSys members being paid out bounties is always double dipping. For example, it's shaping up to be an abysmal and rainy weekend, and I could see myself hacking on some bountied bugs for (edit: profitable) fun. I'd hope that so long as I steer clear of ConsenSys projects I'd wouldn't be ineligible just because GitCoin is facilitating.

That said, I'm saying all of this publicly because I want to make sure I'm doing the right thing, so I encourage others to disagree with my position on this, so long as they help me out by teaching me where I've gone wrong.

If we regularly closed our own bounties out from under external contributors, we would create an environment that's frustrating to contribute to, so we need to be incredibly transparent about when we're working on a bountied issue. This could be both with formal assignee/bounty claiming, or also with less formal "general protocols", like "we will always work on our highest priority issues first."

I share this concern as well, and I'd think that in cases where we intend to complete our own bountied issues, we should be participating in the claiming process, or delisting the bounty. It's tricky, because we're likely to align bounties with the highest priority issues, meaning we're probably likely to work on them in relatively short order if the community doesn't pick them up.

we like the idea of taking the ETH you would have earned on this metamask bounty and staking it on some truffle related issue after that

@danfinlay - I could be mistaken, but I didn't see you weigh in on this... what was your opinion on the above?

Just to be clear to anyone reading along, at time of writing this bounty is still very much open and eligible for the community to scoop up.

[danfinlay]

we like the idea of taking the ETH you would have earned on this metamask bounty and staking it on some truffle related issue after that

To me this is the same as returning the bounty value to the pool. I think GitCoin has done a great job of actively putting bounties on all sorts of stuff, I don't have opinions on sub-pools or anything right now.

[owocki]

thanks for the thoughtful comments here folks. im going to ask some folks on the gitcoin slack channel what they think of the situation... truffle/metamask bounties are quite hot commodities for the community these days, so i'm hopeful they'll want to chime in

[ethikz]

I'm just getting into the blockchain, cryptocurrency, gitcoin community so I apologize for any ignorance that may come out as I am still learning. From what I've read here is my view/take on it, also please correct any mistakes I have as well about relationships between Companies and Projects.

I don't necessarily feel it is unethical, so to speak, for salaried OSS devs to claim bounties but could seem unfair. The perceived mission, from my perspective and maybe others, is that Consensys is paying people across multiple projects/initiatives, but still in the same space, to work full time on their initiatives. With that said, the employees have full-time work to be done to improve the technologies/efforts, and the bounties put on issues for the same reason, it is Open Source. At some point the paths will cross where salaried employees will work on bountied issues, I'm not sure there are ways of getting around it, but maybe there is.

To me, it's the same concept as being a Developer/Engineer working for a company, but in your free time you freelance and do side work or even contribute to Open Source. So why not still allow that to work the same way here? You're a salaried OSS employee working on issues the company has put forth and during your "day job," you work on those issues. When you are off work, and it's personal time then I don't see an issue claiming a bounty if you can do it, unless of course, you put a bounty on it to claim it on your off time, then yes that would be unethical. If caught, that will do the opposite of what is trying to accomplish, that, however, is a whole different discussion.

Of course what was stated:

we like the idea of taking the ETH you would have earned on this metamask bounty and staking it on some truffle related issue after that

is also a viable option if you aren't comfortable taking a bounty. It is nowhere near a perfect solution, but there will have to be trust and communication through the process.

I hope this makes sense to others if not please let me know, and I'll try to explain lol.

[e-n-l]

I'd say 'yes,' everyone should be able to claim / complete / collect a bounty, because a major value proposition for GitCoin is that developers getting paid to work on FOSS. That approach would also eliminate dependency on identity / employer identity, policy enforcement overhead, and generally has the advantage of simplicity.

Digging deeper, there are valid concerns that you'd be defeating that use case and inviting corruption by allowing 'insiders' who are already have contracts in place to double dip, and furthermore, may have influence on what gets a bounty to exacerbate that problem. At that point, it really is more of an internal organizational issue though... perhaps outside technical scope.

The more I think about it, the more I love the idea of facilitating bounty recycling / forwarding. Perhaps allow for organizational agreements to be part of the claim process, such that an organization could 0) pay people to spend their time on OSS, 1) have bounties claimed for that work pooled within the organization to 3) be used for posting new bounties on their own issues.

This may be simplified by simply incentivizing recycling, rather than cash out.

In any case, that would incentivize cross-pollination among teams / businesses, dramatically increase workforce flexibility and scalability: Let's say company A is in an intense sprint to meet a hard deadline, while company B completed a seasonal peak. A could place bounties to get more dev time on issues to make their deadline, which B could collect and save for the next peak season.

[danfinlay]

In any case, that would incentivize cross-pollination among teams / businesses, dramatically increase workforce flexibility and scalability

You just blew my mind a bit. This makes sense, if a person salaried on one team completes an issue, that bounty could be recycled under their team, making the total allocation per project more important than the division of human alignment.

Re: the double dipping: It’s not bad to do stuff on your free time, but it can be unclear which time is free and not when salaried, making it hazardous territory.

[GrandSchtroumpf]

Hey guys, sorry for disturbing the debate. I just would like to know if someone is actually working on this issue right now ? My company has a heavy need of events, so we're stocked with web3 0.20 until this issue is fixed...

[owocki]

it sounds like @benjamincburns has made some progress but is not going to claim the bounty to avoid (even the appearance of) impropriety. sounds like the bounty is back up for grabs for that reason.

[e-n-l]

Turns out, there's a word for inappropriately leveraging your position within a company to get the company's money into your own pocket: embezzlement

It's only embezzlement if an employee inappropriately siphons company money into their pocket. E.g., some scumbag in finance cooks the books to move money from the pension fund into some bogus investment that directly benefits him. It's a form of theft.

Bounties are already explicitly being earmarked for a purpose, and can only paid when that purpose is fulfilled. The moral quandary comes from the question of whether the employee is "stealing time" from their employer, which is outside the GitCoin scope completely; it's up to the relationship the company has with the worker.

Salary / hourly / contract are not the only compensation paradigms, obviously. In fact, studies have shown^[1] that a salary + performance-based compensation structure is the superior paradigm, preferred by many professionals in roles that require significant autonomy, e.g., customer-facing / business development and management roles.

Performance-based compensation, like GitCoin, is most effective where individual performance is a) easily and accurately measured and attributable, and b) the time delta between worker action, desired outcome, and payout is minimized. Simply put, it is extremely motivating for humans to have clear, explicit incentives for tasks. Furthermore, every iteration provides an opportunity for objective measurement, feedback and optimization.

Compare that to the salary / hourly compensation model: role expectations are outlined, and a manager (usually receiving some kind of performance-based compensation himself) is required to police / babysit workers to ensure expectations are met. Absent effective systems, management and worker incentives are vague, and subjective; they often conflict with each other, resulting in exploitation and unethical behavior.

The additional layers of organizational complexity (middle management) bear a heavy cost; effective systems which increase organizational efficiency, transparency and ethicality are of critical importance to all organizations at scale. This is a class of challenges to which GitCoin, and blockchain in general, offer novel, disruptive solutions.

The big downside to pure performance-based compensation approach is that often it lacks the illusion^[2] of continuity and security. The absence of assurance leads to fear, stress, and desperation, which trash productivity, makes humans stupid (literally), and creates a self-fulfilling prophecy of failure.

The salary is a clear, simple, explicit way to provide assurance. It increases the odds that the worker will successfully operate in a mindset of abundance, where they can be most effective.

Therefore, whenever possible, every company can increase efficiency and lower organizational overhead by reallocating some percentage of HR funds away from (coughmanagementcough) salary / hourly / contract compensation, into bounties for their salaried people (along with everyone else who has access) to pursue.

_{[1]:Full disclosure: by saying "in fact" and "studies," I mean that I have a vague recollection that I read something (possibly confabulated / entirely imagined) authoritative about this at some point in the distant past, agreed with it, and decided that it should be presented here with authority.
[2]: I say that it is an illusion because there is never certainty that a company will continue employment, that it will even continue to exist. Successful contractors and free-lancers that live in 100%-performance-based-comp-land construct their own illusions of assurance, mostly from subjective confidence, with a degree of objective historical and statistical performance data. It's essentially a mental construct / model of the "manager" role, and far from invulnerable. (I speak from deep personal experience here)}

[danfinlay]

Related to #1645.

[vs77bb]

Hi @choochootrain - many moons ago (3 weeks, to be exact) you let this go because you were swamped. Any interest in picking it back up at this time?

[gitcoinbot]

__Work has been started on the 0.3 ETH (230.34 USD) funding by__:

1. @pablasso

__Please work together__ and coordinate delivery of the issue scope. Gitcoin doesn't know enough about everyones skillsets / free time to say who should work on what, but we trust that the community is smart and well-intentioned enough to work together. As a general rule; if you start work first, youll be at the top of the above list ^^, and should have 'dibs' as long as you follow through.

On the above list? Please leave a comment to let the funder (@owocki) and the other parties involved what you're working, with respect to this issue and your plans to resolve it. If you don't leave a comment, the funder may expire your submission at their discretion.

• Learn more on the gitcoin issue page
• Questions? Get help on the Gitcoin Slack
• $8903.58 more Funded OSS Work Available at: https://gitcoin.co/explorer

[pablasso]

@owocki there's a lot of guidance already here by @benjamincburns, I think I can follow up and complete this issue. I'll ping you back when I have some progress.

[owocki]

@pablasso sounds good

[ghost]

@danfinlay and others: could one say that web3 is one of the most important libraries that MetaMask uses? Possibly the top-most important one?

@kumavis , could you please place this info: https://github.com/MetaMask/metamask-extension/issues/2350#issuecomment-353437246 into the 1st comment? In essence, everything important that comes up shoudld be moved into the 1st message (and deleted from comments, to shrink down discussion size).

Thus whoever hits on the issue gets kind of immediately the most important info & status. For example me: I would have been able to assess: can I pick this up if @pablasso is drinking margaritas in Hawaii?

(ethics level discussion in this issue... will read it when i need to relax! )

[pablasso]

I don't think we should delete comments as they give you a good sense of where the conversation was heading and why decisions where taken. Update the top comment with the latest info is fine, although I'm not sure if this is a good use of time for the maintainers while we can just invest more time reading the threads — big threads like these are the exception not the rule.

I'm not drinking margaritas in Hawaii (yet!), this was harder than expected and I don't yet have progress for a PR. I'm only dedicated some time on the weekends, hopefully I have something in the next one.

However if anyone else has progress here, I don't have a problem with you taking on this.

[ghost]

@pablasso, all fine then. We all continue to go fishing for relevant info (sometimes missing this way even a bug-fix which was posted within a critical issue).

I do not want to take this over (for now). I just wanted that @danfinlay & @kumavis realize that this issue should possibly be labled "highest priority", and not "good first issue".

But most importantly I just wanted an answer to this question:

could one say that web3 is one of the most important libraries that MetaMask uses? Possibly the top-most important one?

[danfinlay]

could one say that web3 is one of the most important libraries that MetaMask uses? Possibly the top-most important one?

I don't think this is the case anymore. Web3 is now basically a convenience layer on top of the provider API. Eventually, the Provider API will be injected probably without web3 at all (Mist has already deprecated the web3 global).

For example, the Ethereum provider can be used to initialize other convenience libraries, like my favorite, ethjs.

Web3 1.0 requires a new type of provider (which relies on websockets, or server-sent events), which is why it requires special rework to make compatible.

Since other alternatives work fine, I don't consider this a top priority at all. We have issues where there is actual security issues involved (privacy, hardware support, storage resilience), and those are much higher priority to me than being compatible with the latest breaking version of web3.

I'll remove the good-first-issue tag.

[ghost]

@danfinlay , ok then, taking in account what you wrote, and this here:

As Mist 0.9.0 release is approaching, we'd like to inform that in the near future, from 0.9.0 onwards it won't inject web3 object by default on Dapps. This is a measure to keep Dapps stable on the long run as ethereum ecosystem evolves. We'll provide a developer preview version, so Dapp developers will have time to update their Dapps accordingly.

The web3.currentProvider object will still remain for a period of time for backwards compatibility and a new provider object will be introduced.

I looks like the task should be:

• remove internal dependency to web3 (as in: get rid of web3)

(what we discuss here should be visible/documented immediately, thus interested contributors can contribute faster. This relates to #3302)

[cmditch]

Hey @pablasso and @owocki thanks for working/funding this issue! This will finally make elm-web3 compatible with metamask.

[ghost]

I understand that it is one thing what MetaMask uses internally, and another thing what the developers of applications use/need.

So it (web3) could have low priority for the MetaMask team (e.g. could be removed, to alight efforts with Ethereum main projects), but it could have high priority for users (although I feel that they should adjust to what the Ethereum mother projects use, too).

I would really like to understand this (@cmditch / @danfinlay ) ?

[cmditch]

@lazaridiscom

MetaMask is understandably very busy. web3-1.0.js is also still in beta. Luckily the repo is open source, and anyone can make a PR to help add the needed features. Better still, there's a GitCoin bounty attached to helped incentivize such work. Ultimately nothing but ourselves is stopping this from getting done.

Also, forgive me, having a bit of trouble understanding what you're trying to understand.

[ghost]

@cmditch , nothing to forgive, I have to apologize for not expressing this clearly. I try to rephrase this:

MetaMask Requirements

MetaMask AddOn:

• must stay compatible to major libraries (Ethereum main projects)
• should reduce internal reliance on web3

and now the question:

• should ( ? ) or must ( ? ) provide/expose web3 functionality to Application Developers?

If this still does not make sense, then I simply have not understood all this.

(@cmditch , looked at your repo, looks nice)

[GrandSchtroumpf]

Hey guys, I'm following this thread since the beginning and I don't see the end of it ...
The aim was to get compatible with web3 1.0 than relies on websocket for events and some subsciptions.

The facts are :

• Many project relies on, and work with, web3: this is a library with many features, well documented, it's quoted in the ethereum wiki, there are videos, blog posts, trainings about it...
• Web3 1.0 is wayyyy easier and better to use than web3 0.x.
• MetaMask is the main tool used by people to interact with ethereum nodes.
• Smart Contracts are, by nature, difficult to change.

-> Many projects are technologically bounded to Web3 AND their clients use MetaMask. SO if they use many events on their smart contracts, they cannot update to web3 1.0.

So I don't know for a general point of view if MetaMask "should ( ? ) or must ( ? ) provide/expose web3 functionality to Application Developers?", but I know that many projects are waiting for websocket support for some times now.
I don't have enough knowledge of how the provider-engine works internally to resolve this issue, but I think that this thread must be only used to solve this problem and should not be used for general thinking (ethical questions about bounties or what metamask should or must be).

My point here is to say that they are real projects relying on the implementation.

[ghost]

@GrandSchtroumpf , all fine, the high-priority label is there, yet one question remains for me:

If web3 is so super as you describe, then why has Mist dropped it?

[GrandSchtroumpf]

Thanks for the label.
@lazaridiscom I cannot speak for Mist. But I guess some people like ethjs better so there is no reason to use web3 then...

[alexvandesande]

@lazaridiscom what are you talking about? Mist did not drop web3 or has any plans of doing so.

[danfinlay]

Sorry @alexvandesande, I think he was going off my earlier citation where the Mist changelog announced web3's deprecation.

We have continued injecting it ourselves, but weren't sure if you had deprecated yet or not.

[ghost]

@alexvandesande

from: https://github.com/ethereum/mist/releases/tag/v0.9.0

NOTE FOR DAPP DEVELOPERS!

From this version on Mist will not ship its own web3.js instance anymore. We only provide for now web3.currentProvider so you can connect to ethereum. In the future, we will provide a special ethereum object with a default provider.

Also this web3.currentProvider will not allow sync calls anymore, as it is already the case in MetaMask (and it's bad practice in general). So make sure to use the async ones e.g. web3.eth.accounts -> web3.eth.getAccounts(function(){...})

from: https://github.com/ethereum/mist/releases/tag/v0.8.10

Deprecation notice

As Mist 0.9.0 release is approaching, we'd like to inform that in the near future, from 0.9.0 onwards it won't inject web3 object by default on Dapps. This is a measure to keep Dapps stable on the long run as ethereum ecosystem evolves. We'll provide a developer preview version, so Dapp developers will have time to update their Dapps accordingly.

The web3.currentProvider object will still remain for a period of time for backwards compatibility and a new provider object will be introduced.

[amitaymolko]

Hey guys,
Any updates/progress?
@pablasso are you working on this?

@benjamincburns Drizzle needs this quick :(

[bgits]

Is there a WIP branch or repo for this issue? Seems like multiple people started work on this, but it's unclear where that code lives.

[owocki]

hi all. i just killed the bounty for this issue for some gitcoin-internal migration reasons but wanted to let you know that, regardless of the issue description on gitcoin.co.. im good to pay out this bounty when the time is right. just @ me back then!

[guix77]

Really looking forward for this

[alexvandesande]

@lazaridiscom @danfinlay it just means that we won't be injecting it by default, but using a web provider to allow the apps themselves choose their own library, given that more than one exists. It's still our library of choice and we use it ourselves, just giving users more options.

[chapati23]

@danfinlay @lazaridiscom Would raising the bounty for this issue help move it forward quicker? If yes, we at Brickblock would be happy to throw some ETH into the hat.

[danfinlay]

@chapati23

If the core team completes this feature, we’ll reject all bounties for it, since we’re already salaried, but a bounty could encourage an external community member to try to implement this sooner.

[ghost]

@alexvandesande , thank yo for the info. You should possibly clarify this in your documentation, because the announcement reads like "Deprecated". Possibly the by default in won't inject web3 object by default implies that devs can still opt to inject/use it. But this has quite a reduced visibility (at least it had for me).

@chapati23 raising the bounty should help, but i've the feeling that providing a concise task-description will help even more!

@danfinlay , @kumavis - I really think its time to create a task description, with the minimum estimated steps needed to fulfill this.

Would possibly be enough to

• move this https://github.com/MetaMask/metamask-extension/issues/2350#issuecomment-353437246 to the issue description of MetaMask/provider-engine/issues/189
• see what is still actual of this: https://github.com/MetaMask/metamask-extension/issues/2350#issuecomment-354912008 (@benjamincburns )
• possibly update the issue description here (or relay everything to MetaMask/provider-engine/issues/189, if its really the only step needed)

Could(!) look like this (to invite people to take a look at this issue, to possibly do one subtask)

Task

MetaMask should be compatible with web3 1.0. This quick overview provides the estimated necessary steps:

Solution Path A (by kumvaris)

source: https://github.com/MetaMask/metamask-extension/issues/2350#issuecomment-353437246

Issue: Add websocket subprovider - MetaMask/provider-engine/issues/189

• [ ] identify new subscription rpc methods and their spec used by web3 1.0
• [ ] identify how web3 1.0 gets its server-push notifications on its provider
• [ ] add a matching api to https://github.com/MetaMask/eth-json-rpc-middleware/blob/master/providerFromEngine.js. "is easy (here's a PR: https://github.com/MetaMask/eth-json-rpc-middleware/pull/4), but requires"...
• [ ] add a server-push interface to kumavis/json-rpc-engine "to wire it up"
• [ ] create a https://github.com/MetaMask/eth-json-rpc-middleware that something that implements the subscriptions + tests

Comments by benjamincburns: https://github.com/MetaMask/metamask-extension/issues/2350#issuecomment-354912008

Solution Path B (by benjamincburns)

info from end of: https://github.com/MetaMask/metamask-extension/issues/2350#issuecomment-374717425
(easier to implement, but without the full benefits of websockets. Would still enable use of web3 1.0)

Base work:

• https://github.com/MetaMask/provider-engine/blob/master/subproviders/subscriptions.js
• polish and QA
• Expose subscription data event in some way
• Detail/Research Work: possibly a new module could be used to wrap provider-enigne to expand its API to include the full PUB SUB API. Figuring out those details would be part of the work.

Solution Path C

This would be my process/path:

• Ignore everything within this thread
• Provide standard test-cases #3618
• Start development
• Once your stuck, review the solution-paths / contents of this issue to get hints

Solution Path D (by yourself)

• Take your own path.

[owocki]

two notes:

1. it might make sense to move the issue scope over to a new, clean, issue.
2. though i broadly defer to the metamask team on their specific policy, i disagree that it's on the metamask team to think through the solution path for the bounty hunters. if they are going to think through the implementation for you, then they might as well just do it themselves. you could generate a pretty plausible implementation path by just looking at whats changed in web3.js 1.0 and grepping the metamask codebase for what needs to be updated & reworked.

[GrandSchtroumpf]

@owocki I think we are beyond this kind of consideration. This is a core feature for many Dapps that rely on web3 1.0.0 and the bounty is so small (0.3Eth) that nobody will take the time needed to solve this issue.
I think that the real problem is that few people know in depth web3-provider-engine and among them fewer have the time to work on it. A not-so-good dapp developer can earn 0.3Eth per hour of work on freelance mission, why would anyone spend some time to solve this problem except if he is facing it in his job ? Therefore if someone is going to solve this issue, it is absolutely not for the bounty !
I, for example, would be glad to work on that, but my knowledge of this project is too low for that. So I think that if the metamask team hasn't the time to work on that issue, we should at least know what to do to solve it ourselves.

[owocki]

the bounty is so small (0.3Eth) that nobody will take the time needed to solve this issue.

this issue was funded back when ETH was 2x the price it is now.

nonetheless, your point about the amount of funding is still valid. i am happy to increase the funding for the issue if @danfinlay and @kumavis want to give it another go (perhaps on another ticket)

[ghost]

we should at least know what to do to solve it ourselves.

@GrandSchtroumpf , the solution-paths are already given, mostly in comments within this issue. I've update my comment above and collected quickly the relevant info of kumvaris path (and added a personal one) : https://github.com/MetaMask/metamask-extension/issues/2350#issuecomment-374070412

[ghost]

For the case anyone wants to make a tiny step: #3618

[guix77]

I totally agree that the bounty is very small. We're talking about a major door into the Ethereum world, here, not a super fancy awesome useless feature.

Furthermore, maybe no one cares because it's clear that the injection of Metamask is deprecated, because changing its default version would break most dApps and because we can always use currentProvider and replace the whole injected Web3.JS

window.web3 = new Web3(window.web3.currentProvider);

[GrandSchtroumpf]

@lazaridiscom web3 uses this version of websocket when websocket is not available on the window or if we are on a server:
https://github.com/frozeman/WebSocket-Node
Is it ok to use this module in the providerFromEngine to handle server-push and send ?

[ryan-rowland]

The wheels have been spinning on this one for months... It seems like the common pattern is:

• Ooh money! I'll give it a shot.
• Wow that's a lot of work.
• I give up / I'm too busy / etc...
• Progress (and time) is wasted due to all-or-nothing bounty
• Repeat

We have a general idea of the steps that need to be taken to get this task knocked out. Can we get a rough consensus on the steps required, and then split those sub-tasks out into their own tasks so this can be tackled in a more gradual and granular fashion?

[ghost]

@GrandSchtroumpf - In this issue here, I just apply the standard-procedure in an abstract manner. With the quite low amount of domain-knowledge I have, I would answer you: yes. But @kumavis and others should know better.

[danfinlay]

Sorry for the low bounty, and the low response rate. The core team has been very busy with other issues at the moment, and we've always planned to do this eventually, and we might just do it ourselves soon. The community doesn't need to feel responsible for completing this.

That said, since this issue clearly needs a re-focusing, I will now write a brief summary of what this issue involves.

The Current State of MetaMask's Provider

MetaMask's most popular open source component to date has by far been web3-provider-engine. This express inspired middleware architecture allowed @kumavis to compose MetaMask's custom provider logic into a stack of "subproviders" that handle different requests, or portions of requests, and then either pass the mutated request down the stack, or reply directly.

This engine was designed for the original Ethereum JSON RPC spec. Since then, there has also emerged a websocket API that includes subscriptions called the RPC Pub Sub API. This new API allowed push subscriptions, and Web3 1.0 to exist.

Currently, MetaMask still uses polling & HTTP to get all of its data, and so the most obvious way to add Web3 1.0 support is to allow us to move onto a websocket based infrastructure.

A Problem With That

Over time, it has become increasingly clear that some of the ethereum-opinionated aspects of provider-engine were actually getting in the way of creating a good provider, and things were being hacked on top of it.

For example, our cache-subprovider needs block awareness to know when to clear its cache, and so block tracking was built into provider-engine, and bit by bit, these pieces of opinionated architecture has made provider-engine a bit harder to work with.

Furthermore, since provider-engine was never designed for handling subscriptions, since it's a middleware for responding to requests, it has no natural way of representing a websocket connection.

The Websocket Solution

Since much of the difficulty of adding a new connection type is that provider-engine encompasses all the middleware, and makes them hard to interact with, we're moving MetaMask over to json-rpc-engine. You'll see us use it a few places in MetaMask instead of provider-engine. This is basically the same general product as provider-engine, except with all the ethereum-opinionated aspects torn out, so they can more easily be accessed by external consumers, and so some types of features (like subscriptions) can more easily be added.

By removing the specialized logic from the engine itself, that kind of wiring can be left to the subproviders themselves. For example, the new arrangement might look a little more manual, but it allows much more diversity in subproviders (pseudocode below):

var engine = new JsonRpcEngine()

var blockTracker = new BlockTrackerSubprovider()
var cache = new CacheSubprovider({ blockTracker })

engine.add(blockTracker)
engine.add(cache)

engine.start()

Moving to json-rpc-engine means every subprovider in the zero subprovider needs to be re-written or modified for json-rpc-engine, so its functionality can be swapped in place. Many of these subproviders have already been written here:
https://github.com/MetaMask/eth-json-rpc-middleware

In particular, the final subprovider that will need to be written for this issue is the websocket subprovider. Once we have this all ready, we'll probably need to compose the zero provider which is currently only imported, because only by composing the subproviders will we have access to the block-tracker's events. Or we come up with a way of accessing specific subproviders from the parent controller.

Anyways, this proposal is hardly trivial, as provider-engine is basically the heart of MetaMask, and so this is something like MetaMask heart surgery. We've already seen some regressions related to moving provider-engine over to json-rpc-engine, and so this needs to be done slowly and carefully, and json-rpc-engine needs to get really solid really fast.

An MVP Solution

While everything above has been accurate and true, there is a way we can provide web3 1.0 support a bit faster, but without the full performance benefits of websockets.

This is to polyfill the subscription API, the way Ben Burns did here already.

If that subprovider got a little polish and QA, it's possible it would be ready to go very soon, and so someone looking for the shortest path to letting their dapp use Web3 1.0 might just take that on, and I think it would still win the bounty for this issue, since this issue is really about web3 1.0 support, not websockets themselves.

The subscription data event would still need to be exposed in some way, and maybe a new module could be used to wrap provider-enigne to expand its API to include the full PUB SUB API. Figuring out those details would still be part of this "MVP".

[owocki]

@danfinlay thanks for the thoughtful re-focusing comment. if its okay to you ill move your latest comment over to a new github issue and issue a new bounty for the MVP scope.

[ghost]

Updated Summary: https://github.com/MetaMask/metamask-extension/issues/2350#issuecomment-374070412

[owocki]

@danfinlay here is a new issue: https://github.com/MetaMask/metamask-extension/issues/3639

i move we

1. close down this issue
2. put a bounty of 1 ETH on that issue.

does that sound good to the group?

[danfinlay]

Either that one or this one, @owocki.

I will do more work to clean up this story tomorrow. I might close this thread in favor of these new, cleaner issues with links to important comments.

[danfinlay]

To help clarify what would be involved in adding websocket support, I've enhanced the issue description here:

https://github.com/MetaMask/provider-engine/issues/189

[owocki]

just bountied https://github.com/MetaMask/metamask-extension/issues/3642 with 1 ETH

[ghost]

I might close this thread

@danfinlay , yeah, before it reaches 100 comments!

[kyriediculous]

I'm sad that the core team doesn't see this as a major issue :(

[ghost]

Well, @kyriediculous, don't be sad. This has quite a priority and there is ongoing work, see e.g. #3642

[ghost]

@danfinlay , as this issue has follow-up's set, you could lock it.

[ghost]

@ all

It looks that there is a first working draft of web3 1.0 support for testing available, see:

https://github.com/MetaMask/metamask-extension/issues/3642#issuecomment-379921721