Notes: Tor onion integration

[david415]

chatting with Juan Bennet at c-base, Berlin about Tor onion service integration

here's what we've identified as necessary for proper Tor integration:

1. adding /onion to go-multiaddr - /onion/<onion-key>/ipfs/<ipfs-key>
2. adding /onion dialing to go-multiaddr-net
3. make a build of go-ipfs that:
4. only uses /onion
5. only bootstrap to onion nodes
6. disable mdns service
7. disable NAT service

I know plenty of people who'd be willing to run some Tor onion bootstrap nodes.

(edited by @jbenet for links)

[whyrusleeping]

:+1: this would be pretty cool!

[david415]

OK I've written the go-multiaddr Tor onion address decoding:
https://github.com/jbenet/go-multiaddr/pull/13

[leif]

This sounds like a good plan for location-anonymous tor users to have a persistent identity and announce themselves to peers, but it doesn't cover the use case of anonymously (unlinkably?) accessing data from ipfs.

For that we'll need a mode that doesn't transmit any linkable identifiers (which includes not announcing pieces to the DHT).

[david415]

@leif, Good point. The only "pieces" we'll be announcing to the DHT is .onion addresses. I suspect it will be very easy for us to create a readonly client mode for non-onion using Tor users to access any IPFS storage server without announcing any identity at all.

[david415]

Today I wrote a TorDialer and OnionListener; This repo also includes examples programs which can be used as an integration test with your system Tor. However it should soon have more ways to use Tor other than just the control port protocol with the system tor:

• new ephemeral onion services via control port
• "just launch me my own tor instance for this Dialer/Listener"

https://github.com/david415/oniondialer

Although oniondialer could be improved, we can do this later now that we have the basic interface implementations working. Therefore the next step should be to see how this Dialer and Listener can fit into go-multiaddr ... Probably oniondialer will have to change slightly to get it to fit into go-multiaddr.

[david415]

I've been looking closely at the source for go-multiaddr and go-multiaddr-net...
However I'm not sure if Tor onion service multiaddr string should look like this:
/onion/timaq4ygg2iegci7.onion:80
or this
/onion/fufu.onion/tcp/80

Whereas the "readonly Tor-ified" multiaddr looks like this:
/tor/example.com/tcp/80

or should the /tor/ multiaddr not encapsulate and instead look like this?
/tor/example.com:80

What do IPFS developers think of all this?

[ghost]

from chat:

• /onion/fufu.onion/ip4/tcp/80
• /tor/example.com/ip6/tcp/80

Similar to /dns in https://github.com/jbenet/go-multiaddr/issues/7#issuecomment-125745700

A more explicit, but very bloated option for /tor: /ip4/127.0.0.1/tcp/9050/socks5/dns/example.net/ip4/tcp/80

[whyrusleeping]

maybe:
/onion/fufu.onion/tcp4/80
and
/tor/example.com/tcp6/80

[david415]

hi. I chatted with Juan last night on #ipfs and we decided that onion multiaddrs will look like this:

• /onion/timaq4ygg2iegci7/80

Tor doesn't transport IP or TCP... but is a realiable stream transport. We don't need DNS. Tor clients use a local SOCKS proxy; therefore the remote end does the DNS resolution. I hope I've cleared up the confusion regarding this. If not then I'm afraid this ticket will grow too big with my explainations. Perhaps find me on irc instead of commenting here with queries.

For the Tor exit node case I think it should be represented like so:

• /tor/example.com/80

So I think this sort of breaks the multiaddr parser model... and so I made my multiaddr/net dev branches do this:

• /onion/timaq4ygg2iegci7:80

That is the colon separates the embedded onion port number.

[whyrusleeping]

@david415 doing /onion/timaq4ygg2iegci7:80 makes sense to me, all the parameters needed by the onion protocol are in the path element following it. Using the / is strange, as the 80 then doesnt have a label, and its not super clear what its trying to be. @jbenet thoughts?

[jbenet]

we'll already have things like:

/ip4/1.2.3.4/tcp/80/http
/dns/foo.com/A/ip4/tcp/80/http

if we made the restriction that only one component be used, why not then:

/ip4:1.2.3.4/tcp:80/http
/dns:foo.com:A:ip4/tcp:80/http

o/ this is actually harder to impl and does not layer well on a FS approach like 9P or linux proc files, which the former one does

[whyrusleeping]

thats not quite what I was getting at, i'm not saying that we require only one component. I'm saying that we combine the onion address and port into one as they both are given meaning by the /onion path element.

[david415]

Any suggestions for changes to go-multiaddr-net?
Do you not want Tor/Onion related stuff in DialArgs since it is not a thin-waist protocol?
See here:
https://github.com/jbenet/go-multiaddr-net/blob/master/convert.go#L142-L144

[jbenet]

@david415 right, DialArgs is for net.Dial mostly.

What we should do is:

• make an onion Dialer + Listener
• make the main Dialer + Listener extensible (so users can "add" protocols they want to allow. key is user adds it because users may not want to allow _any_ dialing, only certain protocols. in this case, regular net dialing should also be done this way as some users would not want to allow arbitrary TCP/UDP dialing...)

(i unfort dont have time to _work directly_ on this until after Sep 15. but can CR)

[david415]

• yes... we have a basic tor dialer + onion listener that we can improve more later
  https://github.com/david415/oniondialer
• go-multiaddr-net needs a "pluggable transport system"...
  and i think we can make iterative progress here wherein the first phase for Tor support you can:
• remove all transport plugins
• load the tor plugin
• do stuff with tor + ipfs!!!

However... later it would be nice to have more granular policy for operating heterogeneous ipfs nodes/gateways/proxies.

[david415]

I'm looking at go-multiaddr-net and go-ipfs to see how things work... and it makes sense for thin-waist protocols... but this type of thing isn't going to work for the OnionListener use case:
https://github.com/ipfs/go-ipfs/blob/master/core/corehttp/corehttp.go#L50-L55

The onion multiaddr isn't enough information to create an onion listener. We need to know the tor process control port in order to create the onion service... then after we auto-create the onion service key material the control port tells us the onion address as well.

Perhaps in this case we need an onion + multiaddr helper function to create a new onion service and it's respective multiaddr? But how to plug it into the normal usage of go-multiaddr-net where the API user expects a multiaddr Listener to be created by only feeding it the multiaddr?

[jbenet]

@david415 maybe the listen onion multiaddr could be different, could include more information, like the proxy:

/ip4/127.0.0.1/tcp/<control-port>/onion/<onion-address>

but it still should be advertised to others as:

/onion/<onion-address>

Perhaps in this case we need an onion + multiaddr helper function to create a new onion service and it's respective multiaddr?

yeah that's definitely doable. harder to make things plug in nicely. the other option is to create a "listener-maker" and mount _that_ onto multiaddr, like:

c := maonion.NewController(NewMultiaddr("/ip4/127.0.0.1/tcp/<control-port>"))
c.NewListener(NewMultiaddr("/onion/<onion-addr>"))

// mount onto a multiaddr-net 
manet.Listen("/onion/<onion-addr>") // doesn't work
manet.AddProtocol("onion", c)
manet.Listen("/onion/<onion-addr>") // now works
// 

but this is kind of "magical" in a way i dont like (i.e. static), probably makes much more sense to rework how multiaddr-net works into something like this:

n := manet.NewNetwork()
n.Add(manet.IP4)
n.Add(manet.IP6)
n.Add(manet.TCP)
n.Add(manet.UDP)
n.Add(onion.Onion)

l, err := n.Listener("/ip4/1.2.3.4/tcp/1234")
c, err := n.Dial("/onion/<onionaddr>") 

// with shortcuts like:
n := manet.NewThinWaistNetwork()
n.Add(onion.Onion)

"network" is a bit odd name, but does give the idea that these networks are stacked and makes more sense why mulriaddrs work the way they do. "transport" is another possible name, but may not capture the full meaning of "network".

[jbenet]

Sorry, let's move multiaddr pluggable-ness discussion to https://github.com/jbenet/go-multiaddr-net/issues/7

[jbenet]

hey @david415 -- @whyrusleeping has made transport changes to go-ipfs libp2p and to https://github.com/jbenet/go-multiaddr-net/ -- chances are we can resume the tor work!

[david415]

cool! i'll take a look.

[jbenet]

I can put some effort toward this in feb too.
On Sat, Jan 2, 2016 at 08:27 David Stainton [email protected]
wrote:

cool! i'll take a look.

—
Reply to this email directly or view it on GitHub
https://github.com/ipfs/notes/issues/37#issuecomment-168390975.

[kumavis]

does running an ipfs server as a tor hidden service work by default?
I guess there are quite a few dimensions to the intersection of tor and ipfs, and its difficult to tell what works out of the box and what needs development

[david415]

No. IPFS does not "work by default" as a Tor onion service.

It is not difficult to tell the state of the IPFS + Tor integration because it is utterly non-existent; That's why we have this ticket!

[cryptix]

@david415 :+1:

To reiterate what @leif posted: It might work for dialing an IPFS node from a censored location but you better have a custom ipfs build that doesn't scream your identity to everybody in it's vicinity. I hope we can see different flavors of ipfs nodes before 1.0. A privacy preserving client is much needed otherwise every content you share and spread becomes a tracking cookie. I personally already have that barbed-wire in my head stopping me from using it to share music with my family and that's only a 1st world problem.

[kumavis]

@david415 @cryptix thanks -- I look forward to seeing this ticket progress. If I get some time I will try to pick up some tasks

[seanlynch]

I just posted a $100 bounty expiring in six months. Also paid to get it tweeted from @bountysource to hopefully get it some more attention. If there are blocker issues, let me know and maybe I can post bounties for those as well. https://www.bountysource.com/issues/26255361-tor-onion-integration

[TheDeafMute]

Hi everybody sorry for spammy wadup, but --- Whats the overview of progress atm?

(as a payment for this spammy question, If you all need a(n) alpha/beta tester I am perfectly willing :D)

[david415]

@seanlynch you wanna pay us $100 for tor integration; ah hahaha. lol.

[harlantwood]

Great work trying to help get this ball rolling @seanlynch.

[cpacia]

One request with this feature we'd like people to be able to run dual stack nodes, reachable via onion and TCP. This way people who run openbazaar stores can opt to service people who are running onion only nodes in addition to regular users.

[Kubuxu]

With dual stack node you can't provide any gaurantees of anonymity, even running ipfs itself in tor only mode will make your tor node identifiable without some major changes.

[whyrusleeping]

@Kubuxu In a lot of cases, thats not necessary. @cpacia just wants to be able to expose his service to peers in the clearnet, and to peers in the tor-net. His node would be identifiable, sure, but it doesnt necessarily compromise the privacy of anyone who would connect to him

[cpacia]

Yeah that's correct. Someone running a dual stack wouldn't necessarily by trying to hide their own IP.

Thinking about it though that might mess up routing if a dual stack node publishes providers on both networks. Those providers wouldn't make it to K nodes on both networks.

[seanlynch]

@david415 I suppose you'd rather I not post a bounty at all? You were already working on it for free. My hope was that others would also contribute. So far an anonymous contributor has added another $15.

At any rate, given that I have been unable to get IPFS to even work reliably on clearnet, I'm not currently comfortable with increasing the bounty. And from reading others' comments, I am not alone in having problems. Once those problems are resolved, assuming I haven't found a better alternative, we can discuss what amount you might consider more palatable.

[whyrusleeping]

I'm working on a websocket ipfs transport and I will be doing a writeup of "how to make ipfs work over X" using websockets as an example. Hopefully me finishing that will make this task a bit easier.

[david415]

@whyrusleeping @cpacia The other reason to allow clearnet + onion access is so that you can host a node at your house and be able to access it via your LAN when you are at home... and otherwise utilize it's onion service for connectivity. I don't see any conflict here with hiding the location of the service in this case.

[spinda]

Any progress on this?

[david415]

@whyrusleeping did you write your howto-write-ipfs-pluggable-transports document yet?

@spinda clearly the next step is to write an onion/tor transport plugin for ipfs/go-multiaddr-net.

After that is done the focus should be on how to integrate that tor/onion plugin with the ipfs source...

[jbenet]

We will have a discussion session at the IPFS Q3 workshop on this. Will post the link here later.

[jbenet]

@seanlynch would love to hear more. Mind trying the 0.4.3-dev branch? It's solved a lot of problems people had with 0.4.2

[Kubuxu]

We have just released 0.4.3-rc1 give it a try.

[david415]

attempting to write a tor onion transport plugin here:
https://github.com/david415/ipfs-onion-transport

[david415]

but is this the wrong way?
there's this other api which seems to make more sense ->
https://github.com/ipfs/go-libp2p-transport

[seanlynch]

@jbenet @Kubuxu I will try out 0.4.3-rc1 as soon as I can (two small kids). Thanks!

[david415]

ok i'm going to fix my previous onion transport that i wrote a couple months ago. the problem i was running into previously is that we need the private key material in order to start listening on a given onion service. i will proceed as if the transport has a map from onions to keys... we can store the keys in files on disk or in a database. the constructor for the transport will specify the directory to store them in. fail if key material not found; although this implies some previous admin command to create the onion keys and onion addresses.

[cpacia]

@david415 I'm a little confused about the need for keys. Isn't it just a matter of listening on a given port on localhost and pointing the Tor hidden service configuration to that port to receive incoming traffic?

[david415]

@cpacia Tor onion services use a 1024 bit RSA key. To create the service you need this private key. The onion address is in fact the truncated hash of the corresponding public key.

[david415]

I fixed the transport. Here, it needs review: https://github.com/david415/ipfs-onion-transport/blob/master/onion_transport.go

[cpacia]

Oh, ok. For my implementation I was going to use the control port to set up the hidden service which, i believe, just returns the .onion address not the key. I can look and check.

[david415]

https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n1433

[david415]

the API returns the key unless you ask it not to... and in this case why would you not want to? of course we want to reuse onions. we are creating a new IPFS network with many onions.

[cpacia]

Yeah. I just looked at my code, I was having Tor put the hostname and private_key files in the .openbazaar directory and just reading in the hostname afterwards.

[david415]

ah ok so you aren't using the new ADD_ONION control port command... it's really the recommended way to create onion services these days.

[seanlynch]

Added a couple of comments to the code, but I don't know the code at all so they're just stylistic/documentation-related. And just because I have a bounty doesn't mean you have to pay attention to my comments ;-)

What more is necessary for support? If this is it and someone can verify the work, should the bounty just go to david415? This is the first bounty I've seen come even close to getting claimed so I have no idea how the claim process works. I am pretty sure the person who wants to claim it should indicate on bountysource they intend to work on it, though.

[cpacia]

I was using this code to create a hidden service. It uses the tor control port. https://github.com/postfix/goControlTor

Edit: but I like the bulb you used much better.

[cpacia]

@david415 I added a comment as well.

[david415]

haha yeah i forked yawning's bulb a long time ago when i was playing around... but instead everyone should use yawning's bulb. he's maintained and improved it.

[david415]

@seanlynch the onion transport may be nearly done but i suspect there's still a lot more work to make ipfs work with tor... hopefully @whyrusleeping and @jbenet will have some time to chat and consider these things soon.

[jbenet]

yeah, have time soon. im in shanghai this comingweek, but if we can find a time that works, im game. maybe around the ipfs calls on monday. this or next.

[cpacia]

@david415 there's a number of compilation errors I'm working on fixing.

@jbenet I have some issues trying to integrate it into ipfs. Specifically the go-libp2p-transport library calls some methods that don't exist in it's dependencies (at least the current versions) and uses some non-exported structs. The main offenders are are reuseport and the utp library.

And it also is using the multiaddr from jbenet.

[cpacia]

Took a lot of hacking around but...

[jbenet]

@cpacia

• wow that's excellent. 👏 :tada: :heart:
• want to make a list of all the problems you ran into and what can make it easier/better for you? you have beggining of list up there, but maybe go beyond?
• cc @whyrusleeping take a look! o/ -- also see https://github.com/ipfs/notes/issues/37#issuecomment-247684039
• do you think https://github.com/ipfs/xtp would make life easier in the short term, or no? (we could dump some quick effort into this)

@david415

• in light of this working, let's accelerate the "get ipfs ready for Tor" part
• generate a list of all the things secrecy sensitive IPFS nodes MUST NOT do (i.e. the "privacy stripped out libp2p") and create milestones for that. (There may be an issue we started already-- i cant search quickly enough to find it on this plane...)
• plan out how we will deploy this-- config to start? different distro to avoid fails? (go-ipfs-tor and go-ipfs-i2p or go-ipfs-anon)

[cpacia]

Yes I will make a list of things I ran into.

I also submitted a PR to @david415 for that transport it probably needs some clean up as it's kind of more of a proof of concept at this point.

I would also be interested in the list of things a node must not do because I only have a partial knowledge of what the codebase is doing.

[david415]

hi @cpacia, thanks for the PR... actually we need to NOT decode the key like you are doing in your patch because the tor control port API ADD_ONION must receive this key in encoding form... it is a line protocol after all. anyway i will including your code changes and also make some additional corrections as well.

[david415]

ok i made some more corrections here:
https://github.com/david415/ipfs-onion-transport

how does this gx vendoring thing work? we need to give it a version of go-multiaddr with the recent tor onion changes. i'm not sure how to properly build this with all the correct versions of each dependency... but if it was to be built correctly then maybe the unit test would pass. anyway you can see yourself what output go test brings.

[cryptix]

Nice work @david415!

I know this sounds cheeky after you went through all this but how would you feel about decoupling this from ipfs so that it can be used by non-multiaddr tooling? sorry.

[david415]

@cryptix what work? can you articulate exactly which code you want decoupled from ipfs? actually I'm going to go ahead and say that makes no sense and it's off topic. please look at https://github.com/yawning/bulb i think that is what you want. it's "already decoupled" so please enjoy using bulb onions in your non-ipfs golang projects.

[david415]

i'm quite frustrated that i was unable to build this software correctly. if the wrong imports are used and a gx hash for go-multiaddr is used that does not contain the onion to string decoder then the unit test fails horribly.

[jbenet]

@david415 sorry you had a hard time with gx-- the UX on it is definitely early and it needs much better docs/errors/help. (i've often been frustrated by it too-- but much less lately, it's been improving at a good pace, we're getting there! :] ).

@whyrusleeping can you help @david415 with gx Qs?

[cpacia]

@david415 re the key. I'm not sure I follow what you're saying. The bulb library takes in an RSA key.

I save the private key on disk in pem format just as Tor does when you create a hidden service using the config file. The following function turns that encoded key into a RSA key which is used by the bulb library.

block, _ := pem.Decode(key)
privKey, _, err := pkcs1.DecodePrivateKeyDER(block.Bytes)

I ran this code and it works. Doing it the other way didn't.

[david415]

@cpacia yeah you wrote this correctly because the bulb API takes the key in the struct form via the crypto library... i was thinking in terms of the tor controller spec. it's a line protocol which takes the key in encoded form... not raw bytes... however this doesn't matter since bulb is the api we are using.

i'm glad you got it working for you! i ran my unit test and it did not work because the multiaddr it was built with didn't have the multiaddr to onion string conversion code... whereas it was just committed to the master branch a few days ago.

[xloem]

hi, just some feedback, as an extensive user of tor I really only want to create services with the config file. opening the control port up to network-connected apps does not make me feel safe. note that I may be running tor on an actual different machine than the service.
that said, it is incredible that ipfs is so very close to supporting an onion transport ! and having that exist is certainly way more important than any specifics of how the service is created.

[david415]

Application developers making tor friendly apps use the control port because it exposes the new ephemeral onion api (ADD_ONION and DEL_ONION control port commands). This is the recommended way to create onion services.

If you are commenting that the control port provides excess authority to the tor controller application, then you are correct. However this is a separable concern; There are Tor control port filter programs which white-list control port commands that are allowed to be used by a tor controller program. For example Subgraph OS has excellent Tor integration and we wrote Roflcoptor to deal with this problem of filter control port access: https://github.com/subgraph/roflcoptor

[leycec]

As expected from an open issue spanning several years, the thread of discussion has become fairly impenetrable to the lay outsider – namely, me.

Would one or more IPFS developers be willing to supply a high-level status update on both this and I2P integration? My gut intuition is that development on both has stalled out, for arcane reasons I'll probably never understand. (I'd love to be demonstrated wrong on that.)

I note that david415's ipfs-onion-transport purports to implement Tor integration support as an external gx package. Naturally, development on that appears to have stalled out as well. It also fails to provide even the thinnest veneer of usage or installation documentation. Naturally.

I'll be blunt. IPFS without anonymity is functionally useless for me.

I doubt I'm the only one. Thanks to the overtly DMCA-friendly jurisdiction in which I live, merely running a deanonymized IPFS node would unacceptably increase my risk profile. Default denylists would do little to nothing (_...probably nothing_) to ameliorate this risk and obviate the entire motivation for using IPFS in the first place. After all, pseudo-decentralized, censorship-resistant, fully anonymized file sharing protocols already exist (e.g., I2P-enabled BitTorrent).

IPFS was supposed to be something more – not less.

[cpacia]

I've had it working for a while now https://github.com/OpenBazaar/go-onion-transport

[leycec]

But... there's no documentation. On the other hand, go-onion-transport is at least well-maintained. On the gripping hand, lgierth of ipfs/infrastructure fame admits that:

Not gonna run an /onion bootstrapper until go-onion-transport has been audited.

So, we're back to square one.

My running assumption is that, like ZeroNet's official Tor integration, go-onion-transport requires (A) a system-wide installation of Tor (as opposed to the Tor instance bundled with Tor Browser), (B) the Tor control port to be enabled, and (C) the cookie authentication file for access to that port to be enabled. I suppose I should probably forward this query to go-onion-transport's issue tracker. </sigh>

Regardless, the larger issue remains: anonymity is a first-class issue. The difference between a cryptographically secure anonymity stack and an unofficial, undocumented, ad-hoc third-party plugin is citizens locked in cages at gunpoint. This is not hyperbole. Anonymity is not a trivial consumer preference to be idly left to unofficial, unverified, unaudited vendors with possibly dubious motives.

ZeroNet provides out-of-the-box support for Tor-based encapsulation with additional support for I2P on the way. So should IPFS.

In the long twilight of the 21st century, anonymity is no longer an option. It's a necessity.

[flyingzumwalt]

Thank you to @david415 @cpacia and all of the other contributors who have been directing their time, creativity and determination towards this feature. Making IPFS and the whole decentralized web support Reader and Writer privacy certainly is one of the many first-class issues we have to tackle. There will be terrible consequences for some users if we don't achieve Reader and Writer privacy soon.

Building IPFS is a huge open source community effort involving hundreds of contributors who are building technology that will impact millions of people. The core IPFS maintainers are constantly forced to choose between many equally relevant features. It often falls to individual contributors (or their employers) to prioritize the features that they need most urgently.

I'm confident that these features will get the attention they need in the coming months because IPFS continues to gain traction worldwide and we keep making it easier for people anywhere to run IPFS nodes. This forces the issue of Reader & Writer privacy.

[cpacia]

My running assumption is that, like ZeroNet's official Tor integration, go-onion-transport requires (A) a system-wide installation of Tor (as opposed to the Tor instance bundled with Tor Browser),

Either or. Browser bundle works fine.

(B) the Tor control port to be enabled, and

Yes

(C) the cookie authentication file for access to that port to be enabled.

I believe at this time only pw based authentication is supported.

[ig0rsky]

Any updates on this?

[flyingzumwalt]

The onion transport for IPFS still has not been audited yet. Until it has been audited you should consider it experimental and should not assume that it guarantees anonymity. When we audit it, we will announce the results. In the meantime, people are welcome to conduct their own audits, or arrange for third parties to conduct audits. That would not be wasted effort; It's good to have multiple entities audit systems like this.

[david415]

ugh.

[seanlynch]

Anyone have suggestions for professional code auditors and/or know the general price range for such a thing?

[ian-gtv]

Personally I would use Cure53 and I would guess of the order of 10,000 pounds depending on the scope, could easily go up to 40,000.

[seanlynch]

Hmm, OK. I don't see personally contributing more than 1500 USD toward an
audit, and even the bottom end of that range seems like it would be a
challenge to raise given that the original bounty I posted hasn't even been
doubled so far. Maybe if someone knows some researchers who might be
interested in doing a paper on the topic?

On Mon, Aug 14, 2017 at 10:38 AM ian-gtv [e-mail] wrote:

Personally I would use Cure53 and I would guess of the order of 10,000
pounds depending on the scope, could easily go up to 40,000.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ipfs/notes/issues/37#issuecomment-322256603, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAKO-vFW_VPGFA_qtjnnOKjkAIqC_fv6ks5sYIX_gaJpZM4FzoVc
.

[Kelketek]

@seanlynch It might be possible to use crowdfunding. I'd chip in a bit. Could notify a few mailing lists and subreddits or what have you.

[seanlynch]

I'll happily chip in if someone finds an auditor and sets up a gofundme or
whatever that's refundable if the goal isn't met, assuming that it has
support of the core devs.

On Mon, Aug 14, 2017 at 11:38 AM Kelketek [e-mail] wrote:

@seanlynch https://github.com/seanlynch It might be possible to use
crowdfunding. I'd chip in a bit. Could notify a few mailing lists and
subreddits or what have you.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ipfs/notes/issues/37#issuecomment-322272648, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAKO-urgiSSRSUe5mnP0Ucm2z7P2bS-Aks5sYJQRgaJpZM4FzoVc
.

[ianopolous]

The first thing to do would be to contact cure53 and ask for a quote:
https://cure53.de/#services-analysis
You can also look into funding sources like the Open Technology Fund.

[jbenet]

Protocol Labs is now capable of funding independent audits of the whole IPFS stack. We have started some work and RFPs on this and we can accelerate it to cover this part first. This shouldn't hold back any other groups wanting to audit. we can take care of at least one of the bills. I think our ETA was in the next few months (IPFS is big), but we can see what's needed.

cc @diasdavid.

[braderhart]

@jbenet @flyingzumwalt Is this the code that you need audited?

https://github.com/OpenBazaar/go-onion-transport

[notzoom]

So what's up with this ? How do we run it over TOR ?

[Stebalien]

We want to get all of IPFS and libp2p audited before we start encouraging people to use it with TOR. Unfortunately, this will take a while.

[llgoer]

Has there been any recent progress?

[Stebalien]

@llgoer we've been pushing through a large refactor of how transports work that'll make this simpler. We've also made some progress internally towards auditing the IPFS stack. However, you shouldn't expect a blessed and fully functional/recommended tor transport for a while.

[cretz]

An issue on https://github.com/OpenBazaar/go-onion-transport just reminded me to update this issue...I wrote https://github.com/cretz/bine which may make it easier to use go-ipfs over Tor. Granted a simple SOCKS5 proxy is enough to reach the public web out of an exit node, but if you want to start ephemeral onion services or statically link Tor, give that project a look.

[andyli]

For the record, Tor 0.3.3.6 was released earlier this week:

There is now a documented stable API for programs that need to embed Tor. See tor_api.h for full documentation and known bugs.

[lkjh654]

Has any progress been made on the issue recently?

[david415]

From my perspective this Tor integration experience was suboptimal. At least Open Bazar is using the code we wrote.

[lkjh654]

If this integration was suboptimal is there any other integration in progress that will be "optimal"?

[Stebalien]

We have no plans on integrating TOR into go-ipfs officially until there has been full security audit.

[david415]

It's Tor not TOR. That policy decision is a bad decision in my humble opinion for many reasons that I shall not elaborate on here. I explicitly told Juan that I was concerned about the prospect of writing a bunch of code for free that will then not get used. Best wishes to you all in your present and future software projects. Cheers.

[lidel]

@Stebalien just for the record, would go-ipfs accept a PR to add Tor transport as _opt-in_ experiment (behind a flag) to go-ipfs? Or would it be parked until audit happens?

[Stebalien]

I believe the best solution here is to:

1. Add support for loading transports via plugins.
2. Create a plugin for the onion transport, out of tree.

[Trumeet]

Any updates on this issue?

[xloem]

@david415 if you were willing to enumerate the reasons for your opinion, maybe people who agree with you but are closer to development decisions than we are could include them in future plans some day

(Edit: it is obviously dangerous to use ipfs for content that anyone with a lot of power or money wants to remove or track, without using tor, as you can be identified by your hosting ip address.)

[david415]

It's too late for this discussion. IPFS has failed to embrace the concept of free Tor integration from volunteer developers.

That having been said, anonymity is a synonym for traffic analysis resistance; that is to say, even encrypted traffic can be analyzed for the metadata it leaks. Tor is the very weakest of the existing designs for anonymous communication networks however it is the most widely used whereas the other designs from academia have not had much field testing; such as: mix networks, dcnets, verified shuffles and other things can be used to form anonymous communication networks such as private information retrieval, oblivious ram, multi party computation etc.

Tor is trivially broken by any sufficient global adversary by means of timing correlation whereas mixnets are not. There are many other ways to break Tor.

Anonymity aka traffic analysis resistance is not yet a popular security feature because these designs are in some respects ahead of their time... just like not every software project embraces deterministic builds. Just because your white middle class platitude doesn't allow you to understand why people in high risk situations might need these things doesn't mean they are not needed. In fact, in dealing with such folks I find the easiest way to impart the importance to them is to describe military scenarios, e.g. if you were in the military, overseas, you might actually be interested in traffic analysis resistance.

Think about a future brighter than Tor! Think about mixnets, hybrid networks, dcnets and so on. Monoculture is death. Why is Tor the only successful anonymity network? And to a lesser degree I2p? Although the I2p observation is less valid because it's design is so similar to Tor in that it can easily be broken by timing correlation from a sufficiently global adversary.

[seanlynch]

So what you're saying is that if something can be broken by a "sufficient global adversary" it's not worth using? This seems like a silly reason not to provide protection against casual snoopers, criminals, and copyright trolls.

You are welcome to pur your own efforts into a mixnet or whatever. But please don't go telling others they can't have any protection at all just because their requirements don't match yours.

On Fri, Sep 4, 2020, at 14:49, David Stainton wrote:

It's too late for this discussion. IPFS has failed to embrace the
concept of free Tor integration from volunteer developers.

That having been said, anonymity is a synonym for traffic analysis
resistance; that is to say, even encrypted traffic can be analyzed for
the metadata it leaks. Tor is the very weakest of the existing designs
for anonymous communication networks however it is the most widely used
whereas the other designs from academia have not had much field
testing; such as: mix networks, dcnets, verified shuffles and other
things can be used to form anonymous communication networks such as
private information retrieval, oblivious ram, multi party computation
etc.

Tor is trivially broken by any sufficient global adversary by means of
timing correlation whereas mixnets are not. There are many other ways
to break Tor.

Anonymity aka traffic analysis resistance is not yet a popular security
feature because these designs are in some respects ahead of their
time... just like not every software project embraces deterministic
builds. Just because your white middle class platitude doesn't allow
you to understand why people in high risk situations might need these
things doesn't mean they are not needed. In fact, in dealing with such
folks I find the easiest way to impart the importance to them is to
describe military scenarios, e.g. if you were in the military,
overseas, you might actually be interested in traffic analysis
resistance.

Think about a future brighter than Tor! Think about mixnets, hybrid
networks, dcnets and so on. Monoculture is death. Why is Tor the only
successful anonymity network? And to a lesser degree I2p? Although the
I2p observation is less valid because it's design is so similar to Tor
in that it can easily be broken by timing correlation from a
sufficiently global adversary.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ipfs/notes/issues/37#issuecomment-687407153, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AABI56RPO7RM2SGL6PHVYS3SEFOGDANCNFSM4BOOQVOA.

--
https://www.literati.org/~seanl/
This is a temporary email address I use for outbound emails. I will be changing it as it starts to receive spam. If you'd like a permanent address to use, please ask for one.

[xloem]

It sounds like there are newer networks that would be even better to integrate into services than Tor? Are any to or near the point of usability?

[xloem]

Just to add, I suspect the reason that the state of public anonymity tools is not stronger is that the existing international powerholders, whose power could be reduced by widespread accessible anonymity, take diverse action to slow the release and hinder the effective use of the research.

The way to make things change would be for people like us to agree to work together on forging one right thing in a development community, and use tools of both interpersonal mediation and software development to bring the result to happen by force of collective determination. It might help if everyone kept themselves more anonymous, collaborated in private as well as in public, and supported people who ran into personal issues so as to resist disruption and keep the work moving forward.

[seanlynch]

I don't think there's any need to posit a conspiracy to explain this. Strong anonymity is extremely difficult to implement, and not enough people both understand and care about it to make it actually happen.

The best proposal I've seen so far is Chaum's. But there is no stronger system than Tor or I2P that's in a usable state. Personally I trust Tor more than I2P because despite its government funding it has received far more attention from security researchers, and because the compromises have tended to be either against high-value, large targets or large numbers of lower value targets. Large targets are an obvious weakness in Tor against an adversary with wide visibility into the network, and the mass compromises have relied on more traditional means, in particular attacks against the browser.

Burning 0days is costly, so unless it's going to result in the neutralization of a high-value target or a lot of lower-value targets, it seems relatively safe to rely on Tor unless you're engaging in some behavior that's going to get the attention of some well-funded adversary. And if you are, you probably shouldn't be using IPFS either, since it could have its own vulnerabilities that would make compromising Tor unnecessary.

On Sat, Sep 5, 2020, at 10:00, xloem wrote:

Just to add, I suspect the reason that the state of public anonymity
tools is not stronger is that the existing powerhouses, whose power
could be reduced by accessible anonymity, take diverse action to slow
the release and hinder the effective use of the research. The way to
make things change would be for people like us to agree to work
together on forging the right thing in a development community, and use
tools of both interpersonal mediation and software development to bring
the result to happen by force of collective determination. It might
help if everyone kept themselves more anonymous, collaborated in
private as well as in public, and supported people who ran into
personal issues so as to keep the work moving forward, so as to resist
disruption.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ipfs/notes/issues/37#issuecomment-687636039, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AABI56RGGGWSL4NVBOFIB43SEJVETANCNFSM4BOOQVOA.

--
https://www.literati.org/~seanl/
This is a temporary email address I use for outbound emails. I will be changing it as it starts to receive spam. If you'd like a permanent address to use, please ask for one.

[xloem]

I don't think there's any need to posit a conspiracy to explain this.

Let's try to speak concisely and directly, and find ways to support each other.

There are a lot of people taking action for a lot of different reasons, but I have no need to call any of it a conspiracy. My experience is that working on democracy software can become dangerous to the project if it has global publicity, and my knowledge is that acting on this can help sustain other projects. Giving warning prominently, early, and relevently, is an easy way to turn danger into just a conspiracy theory.

I'm observing that David worked very hard for Tor here and it didn't pan out.

For example, discussion of newer protocols, or discussion encouraging a return to Tor, both could take focus, energy, time, and that developer excitement that can make hours of work seem like nothing, away from a potential developer, replying to someone but possibly disagreeing. If someone were working and took the safety of collaborating among people already in agreement, they could have had more man-hours in their project.

I'm not sure anybody here is working on this at the moment, but that's not always the case. If David did most of the work on this earlier, it could give reason to support him.

But there is no stronger system than Tor or I2P that's in a usable state.

I'm always looking to find those obscure implementations nobody talks much about. One of them will be the future Tor.

It does seem to me that Tor already has a codebase and community, and forking can be a quick way to keep working through disagreements; I haven't looked into the code.

[david415]

Dear @seanlynch you are belligerent. I am in fact not saying that Tor or I2p aren't worth using. Come off your platitude for a few minutes and think about people who may be in a more high risk situation than yourself. For those people Tor isn't good enough because their adversary may well be the NSA and the FBI and GCHQ and so on. I am welcome to put forth my efforts into mix networks but not because you say so speaking from a place of belligerence. I am a fan of Tor and I do not go around telling people they can't have any protection. Your delusional caricature of myself is offensive and alarming. Check yourself before you wreck yourself.

[aschmahmann]

It's great to see people passionate about adding some level of anonymizing support to IPFS implementations. Just a reminder that discourse in the IPFS repositories is guided by the code of conduct https://github.com/ipfs/community/blob/master/code-of-conduct.md.

As for the state of things here I agree with the suggestion above https://github.com/ipfs/notes/issues/37#issuecomment-447198558 that the best way to move this forward is for some work to be done improving the plugin system (e.g. for go-ipfs https://github.com/ipfs/go-ipfs/issues/7652 and https://github.com/ipfs/go-ipfs/issues/7653) and then for someone motivated by this to make a plugin for Tor support. If this approach interests any of you then feel free to comment on https://github.com/ipfs/go-ipfs/issues/7653 to voice support for making the libp2p transports pluggable.

[Jorropo]

I'm not sure of what is the current status about this issue, but @berty we are working on a tor transport.

This is a beta and will not protect your IP !

Basicaly I need to add a dns resolving service (all dns resolution are still made through your real ip, not through tor)

It's using a lot of CGO and CI magic to builtin a tor node for a seemless experience (the CI prebuild tor in go a package so it's CGO seemless (just import it and it works, other alternatives require manualy building a make file to produce a valid go package)).
Right now it's working on Linux and Android (I'm working on macos and it is in alpha (still need some ci stuff)) and IOS is planned in the "close" future (hopefully before next week).
A 1.0 should be available soon (it's lacking, dns resolving, IOS, some reffinment of the transport (such as the option to use an external tor node for server or the ability to use 0/1 hop mode)).
Using 0/1 hop mode it could also be a usefull replacement of circuit (basicaly instead of using libp2p's network to relay it would use the tor network (wich is way way bigger)) the problem with that is due to how tor work it would be really really hard (close to impossible) to make it work in an offline situation.

[david415]

If I recall correctly openbazaar uses our Tor transport we've written for IPFS:
https://github.com/OpenBazaar/go-onion-transport

You should not be making separate DNS queries. The Tor exit node handles this. If you do this the Tor integration will be meaningless for traffic analysis resistance because the network intermediaries such as the client's ISP will be able to identify the connection destinations via the DNS queries, obviously. Just my two cents.

I'm not clear on why exactly our work on the Tor integration hasn't been acceptable but recent comments suggest a plugin system is desired and is a prerequisite. This to me seems more like a political decision and it would've been nice to know this upfront from Juan instead of years later after our efforts have largely gone unnoticed and for the most part, ignored.

Additionally the fact that IPFS and the associated commercial money making enterprises have chosen not to pay for the Tor integration themselves speaks volumes about their priorities. (we make money not privacy)

Is it against your code of conduct to be frustrated with the way our volunteer code contributions have been treated?
Is it against your code of conduct to point out that there are several glaring issues that illustrate your project as a post-privacy software project? Lack of encryption for files at rest is another issue which I admit is rather off topic for this thread but offers additional support for my claim of post-privacy.

[balupton]

Really, until we have something like https://onionshare.org/ or https://www.tribler.org/ for ipfs, I wouldn't consider this issue of privacy solved, considering the ipfs dht doxes you. It seems all the big ipfs vc money is going to centralised ipfs cloud providers, and the imitations of them. Would be nice if some went to decentralised privacy. In the end, ipfs just seems a tool for big data companies to save some coin on automatic redundancy replication and deduplication, however the need to encrypt absolutely everything on ipfs for minimal privacy (still doxed by the dht though) nearly completely eliminates such benefit, making ipfs just a slower alternative to existing cloud providers.

For alternatives, Sia and Blockstack are worth keeping an eye on. IPFS marketing makes it out as if they are the only players in town.

[Jorropo]

considering the ipfs dht doxes you

@balupton
yes, it's a technical limitation of the DHT, we can't do better, I2P (a completely decentralized Tor equivalent) uses DHT instead of central server for rendez-vous meeting, but shabils attacks have been demonstred working against it, I guess you can use a blockchain instead of a DHT for rendezvous, but that costy and doesn't scale well.
It's just something really hard to do, DHT is scalable and undiriged / consensus free, relatively cheap but it's mutable, if you really want to avoid DHT doxing you can use DHT + central servers for rendez-vous (it's what we do @berty) so if the rendez-vous servers are DDOS or maitainer shutdown them / manipulate them, you still have DHT and reverse, you would need to control all rendez-vous servers and a part of the DHT to mute someone.

To conclude : DHT have advantages but have drawbacks like other solution (central server and blockchain), IPFS's devs thinks DHT is the best tradeoff but nothing stop you from hosting your own central solution (like we do) or write a blockchain rendezvous driver.

PS: I just noticed, the DHT is not used on berty right now because we were having problems with it, but it's gonna be bring up later.

[balupton]

Thinking about this more. What is actually needed here for adoption, is not a client that does tor/onion routing for all files, but for only the files/dirs one considers private.

For instance, an interface like ipfs desktop, onionshare, tribler, or dropbox - they are all equivalent in this proposal - which when you drag a file into, or select a file in a add file dialog, should prompt, before it is added, which privacy mode you want for the file:

1. Open + Public: the original file will be listed, shared, and readable via your ip address to the public: best for increasing availability for public domain files (eg. public goods)
2. Open + Confidential: the encrypted file will be listed, shared, and readable via your ip address to the public, however only those who you share it with will be able to decrypt it: best for copyrighted files (eg. documents and photos)
3. Private: the original file will be listed, shared, and readable via an anonymised tor IP address to the public: best for increasing availability for files you don't want to be publicly associated with (eg. private public files, eg. controversial media)
4. Private + Encrypted: the encrypted file will be listed, shared, and readable via an anonymised tor IP address to the public, however only those who you share it with will be able to decrypt it: best for files you want to share privately, only to a select audience (eg. private confidential files, eg. early stage whistleblowing)

So the transport should be specific to the file/directory's privacy setting. The UI can distinguish the modes via colours and symbols for the file's row in the file listing, similar to browser's incognito mode.

Perhaps a moniker between these can be used: private, public, open, confidential, transparent, closed. So private vs transparent for tor vs no-tor, and closed vs open for encrypted vs non-encrypted.

For the private mode, many may be suitable with a openvpn or wireguard or cloudflare-warp transport, rather than necessarily a tor transport. VPN transports could also be a source of revenue from affiliate sales.

[Kelketek]

@balupton The issue with opt-in privacy and encryption is the anonymity set is very small. It is better if privacy is opt-out. We don't want a file or user to be targeted as especially important due to the fact it's marked private. If only a small number of files are marked as private, then those will look especially delicious to those seeking compromising or otherwise personal information.

It also makes it a trap for people who are configuring data stores and either are unaware that all the files are public or else forget to toggle the correct setting, and invites for bugs and oversights that make things that were once private not so when some recent change to a script that forgets to explicitly enable privacy restores the default.

[Kelketek]

For clarification-- your idea does not preclude privacy by default, it just points out that not all files have to be encrypted. That's true, but I'd like to suggest it should be the default that they are.