Raspiblitz: New: Tor2IP (ip2tor) Bridge Subscriptions

[rootzoll]

As part of the Lightning Hackathons a tor2ip bdrige concept was developed to make it easy for a RaspiBlitz running behind TOR to buy a tunnel bridge giving it a clearnet IP+port to make services like the REST API reachable for Mobile Apps, POS systems or other apps.

The goal if this issue is to make it possible to make a subscription to a automated shop offering such bridges - running the open shop software: https://github.com/frennkie/django-ip2tor

• [x] retest when LND moved and because of https://github.com/rootzoll/raspiblitz/pull/1385

[rootzoll]

To create a subscription script its needed to easily parse JSON data from such a shop. This may get complicated with bash .. so I looked at python and to make user interaction using the following UI dialogs: http://pythondialog.sourceforge.net

This python dialog wrapper can be installed from the debian repos with:
apt-get install python3-dialog
To be tested.

[rootzoll]

Consider a new config valaue format for storing the list of subscription to bridges - see: #1185
The data for the subscriptions can be in a seperate file next to raspiblitz.conf - so we can maybe test a new config value format here.

[frennkie]

I noticed yesterday that a key part was missing in the API.. but now this workflow should work:

• Retrieve Host list: GET https://shop.ip2t.org/api/v1/public/hosts/
• Place Purchase Order: POST https://shop.ip2t.org/api/v1/public/order/ (store the resulting ID/URL)
• wait a few ms
• Retrieve Purchase Order: GET https://shop.ip2t.org/api/v1/public/pos/22a942b3-89de-48e4-841c-f15d4d21e69f/ (store both item_details[0] (e.g. in order to extend life time of bridge later) and ln_invoices[0] <- if empty repeat until a value shows up)
• wait a few seconds (this is a looping script running every 5-30 seconds)
• Retrieve LN Invoice : GET https://shop.ip2t.org/api/v1/public/po_invoices/1b7fe1ce-0ba6-4c74-81de-bbd304261fb4/
• Pay to payment_request
• status of LN Invoice should change to 2.
• Check implementation status (PO: item_details[0].object_id (ToDo: flatten this ..!): GET https://shop.ip2t.org/api/v1/public/tor_bridges/0f25a0b7-e261-44eb-a01b-0b2b25981c68/ status should change to "A".

[frennkie]

Extending is slightly different (shorter) steps:

• empty POST to https://shop.ip2t.org/api/v1/public/tor_bridges/0f25a0b7-e261-44eb-a01b-0b2b25981c68/extend/ (store po)
• Retrieve Purchase Order: GET https://shop.ip2t.org/api/v1/public/pos/a22843b6-a2dd-4742-a97e-15fcb395847a/
• wait a few seconds (this is a looping script running every 5-30 seconds)
  Retrieve LN Invoice : GET https://shop.ip2t.org/api/v1/public/pos/a22843b6-a2dd-4742-a97e-15fcb395847a/
• Pay to payment_request
• status of LN Invoice should change to 2.

[frennkie]

I included many/most of the data in the nested relations. So after POSTing the order all info should be available here: https://shop.ip2t.org/api/v1/public/pos/22a942b3-89de-48e4-841c-f15d4d21e69f/

[rootzoll]

TODO: after RC1 release change default shop to production server

[rootzoll]

Check again if Subscriptions can be canceld - got this error report:

There seems some problems with the permissions on the toml file, when scripts run from menus ...check sudo calls.

[rootzoll]

• [X] cannot reproduce on my test machines - test on a fresh RaspiBlitz install
  OK - added fix. Ready for final testing.

[rootzoll]

• [ ] Setup a FULMO IP2TOR shop for v1.5RC2

@frennkie let me know when you have time to setup the IP2TOR store

[openoms]

works great, this one exposes a test BTCpay:
https://3.127.188.50:50799

I think we should add https:// front of the link to make it clickable (now it needs to be typed by hand). Also no service should be exposed without SSL encryption so would make a good default.

[openoms]

Question here:
Would it worth to separate the HTTPS .onion services under an other Hidden Service address?
It would complicate the setup,but currently the port 80 and 443 is forwarded the the same Tor hostname.
The above example is ok since there is no HTTP access.
Is there a check in the shop that it does not expose cleartext data?

[rootzoll]

I am not sure. Having both on the same onion address is OK I think to recude to address confusion - its hard to keep overview with too many addresses.

To have it more seperated and prevent cleartext data spillage I think we should make nginx forward from http to https once a IP2TOR/lentsencrypt is activated for that service. Use this kind of nginx feature:
https://www.bjornjohansen.com/redirect-to-https-with-nginx

But this can be a feature for v1.7 when we get closer to automate the letsencrypt HTTPS part.

[frennkie]

To have it more seperated and prevent cleartext data spillage I think we should make nginx forward from http to https once a IP2TOR/lentsencrypt is activated for that service. Use this kind of nginx feature:
https://www.bjornjohansen.com/redirect-to-https-with-nginx

In the context of the IP2Tor bridges the HTTP (typically port 80) to HTTPS (typically port 443) would not work that easily. Normally nginx just changes the port from 80 to 443.. but with the bridges neither HTTP will have 80 nor HTTPS will have 443.

Would it worth to separate the HTTPS .onion services under an other Hidden Service address?
It would complicate the setup,but currently the port 80 and 443 is forwarded the the same Tor hostname.

My feeling is that it's ok, that we have 80 and 443 on the same .onion.

Is there a check in the shop that it does not expose cleartext data?

Not yet... I'd be happy for any help on this. I think doing an "inspection" of flowing data borders on evil. But the Shop could attempt a TLS handshake before accepting the order.

[rootzoll]

@frennkie I added more details in case that a IP2TOR order is not working. Here is an example ao an oder that was payed but then timed out ... let me know if this is good enough for debug:

###### ERROR DETAIL FOR DEBUG #######

Error Short:
timeout bridge not getting ready
Shop:
https://ip2tor.fulmo.org
Bridge:
{'id': 'de5f9e86-dd35-4a75-88e8-652f91cd2330', 'site': 'ip2tor.fulmo.org', 'created_at': '2020-07-03T10:07:55Z', 'modified_at': '2020-07-03T20:17:41Z', 'ip': '91.109.21.148', 'name': 'tondro01', 'is_testnet': False, 'offers_tor_bridges': True, 'tor_bridge_duration': 86400, 'tor_bridge_price_initial': 40000, 'tor_bridge_price_extension': 24000, 'offers_rssh_tunnels': False, 'rssh_tunnel_price': 1000, 'terms_of_service': '-', 'terms_of_service_url': '-', 'owner': 2, 'tor_bridge_duration_hours': 24, 'tor_bridge_price_initial_sats': 40, 'tor_bridge_price_extension_sats': 24}
Error Detail:
{'id': '2c150c15-eeb2-4de9-8dfe-d6374d113c0b', 'status': 'P', 'host_id': 'de5f9e86-dd35-4a75-88e8-652f91cd2330', 'port': 20828, 'suspend_after': '2020-07-16T13:00:55Z', 'comment': 'test', 'target': 'ak2oywmlr5m35igvm3vd4kraoxxtshqpuqhvn364nr24ufgkbfhbixqd.onion:8080'}

[grnqrtr]

I've been running a subscription for several days now and went to check in SUBSCRIBE. On the details of the subscription I set up it doesn't seem to be showing the details correctly. It says I've only paid 24 sats so far, but looking at RTL I can see I've been paying 24 sats a day for several days.

[frennkie]

@grnqrtr Thanks for reporting - I also noticed it and already fixed it: https://github.com/rootzoll/raspiblitz/commit/a403a7b27556f3764794949424c00fe97e0203e9

[frennkie]

@grnqrtr from "aws1" and the pruchase date I can look your subscription up in the database. As far as I see it you set up this subscription to port 80. Are you using this IP2TOR bridge for HTTP or HTTPS connections?

If you are using cleartext HTTP then the next renew should fail as I merged a change into the backend that rejects if you don't use encrypted HTTPS traffic. Would be great if you could give some feedback.

[grnqrtr]

I set it up to be able to access BTCPayServer over clearnet. I actually want it to be for HTTPS, but when I go to it in browser it's not secure. I'm not exactly sure how to specify that. I thought maybe having the Let's Encrypt Client enabled from SETTINGS would work, but seems like not. Is that something I should have specified when setting up the subscription?

EDIT: I'm happy to test & give more feedback. If this isn't the place to go back and forth over my HTTPS issue, feel free to contact me on Telegram or Keybase instead (same username, grnqrtr).

[rootzoll]

@grnqrtr v1.6 RC3 will give you more details on the IP2TOR in the mainmenu for BTCPay.

[rootzoll]

@frennkie trying to order a bridge I got the following timeout:

Error Short:
timeout on getting invoice
Shop:
https://ip2tor.fulmo.org
Bridge:
{'id': '58b61c0b-6d84-466e-9d56-bdf5d902ebdf', 'site': 'ip2tor.fulmo.org', 'created_at': '2020-06-28T19:35:56Z', 'modified_at': '2020-06-28T19:35:56Z', 'ip': '3.127.188.50', 'name': 'aws1', 'is_testnet': False, 'offers_tor_bridges': True, 'tor_bridge_duration': 86400, 'tor_bridge_price_initial': 40000, 'tor_bridge_price_extension': 24000, 'offers_rssh_tunnels': False, 'rssh_tunnel_price': 1000, 'terms_of_service': '-', 'terms_of_service_url': '-', 'owner': 2, 'tor_bridge_duration_hours': 24, 'tor_bridge_price_initial_sats': 40, 'tor_bridge_price_extension_sats': 24}
Error Detail:
{'url': 'https://ip2tor.fulmo.org/api/v1/public/pos/51b5fc57-65fe-4e28-83ac-7eeb1253c132/', 'status': 'T', 'message': None, 'item_details': [{'url': 'https://ip2tor.fulmo.org/api/v1/public/po_items/42c8485e-85a4-418a-8591-0085fbd8998e/', 'product_id': '81668a8e-997f-4ede-840d-54a92a30a64c', 'product': {'url': 'https://ip2tor.fulmo.org/api/v1/tor_bridges/81668a8e-997f-4ede-840d-54a92a30a64c/', 'id': '81668a8e-997f-4ede-840d-54a92a30a64c', 'comment': 'test', 'status': 'I', 'host': {'ip': '3.127.188.50', 'name': 'aws1', 'site': {'domain': 'ip2tor.fulmo.org', 'name': 'IP2Tor@Fulmo'}, 'is_testnet': False}, 'port': 39352, 'target': 'wckfqsu4sq3k6zw3nexixdrbfwmloz4vsh7iuwmnnjoyw6umd6ifzvid.onion:443', 'suspend_after': '2020-07-23T21:08:43Z'}, 'position': 0, 'price': 40000, 'quantity': 1, 'po': 'https://ip2tor.fulmo.org/api/v1/public/pos/51b5fc57-65fe-4e28-83ac-7eeb1253c132/'}], 'ln_invoices': [{'url': 'https://ip2tor.fulmo.org/api/v1/public/invoices/bf90b424-0ab7-4189-99c9-81c808caa7c3/', 'lnnode_id': '8051ccde-3d83-436d-a4cb-7a79fbcb3897', 'tax_currency_ex_rate': '8096.15', 'info_currency_ex_rate': '9367.00', 'price_in_tax_currency': '0.00 €', 'tax_in_tax_currency': '0.00 €', 'price_in_info_currency': 'US$0.00', 'created_at': '2020-07-22T20:58:50Z', 'modified_at': '2020-07-22T20:59:03Z', 'label': 'PO: 51b5fc57-65fe-4e28-83ac-7eeb1253c132', 'msatoshi': 40000, 'tax_rate': '16.00', 'tax_currency_ex_rate_currency': 'EUR', 'info_currency_ex_rate_currency': 'USD', 'payment_hash': None, 'payment_request': None, 'status': 1, 'pay_index': None, 'description': None, 'metadata': '', 'expiry': 900, 'creation_at': None, 'expires_at': None, 'paid_at': None, 'qr_image': None}], 'timestamp': '2020-07-22T21:01:02Z'}

[frennkie]

Autossh ist still not working for me... I have to change the - M 0 to - M 20000 every time and forgot after upgrading to RC3.

Invoices should work again.

[rootzoll]

@frennkie should we change something in the internet.sshtunnel.py script?

Autossh ist still not working for me... I have to change the - M 0 to - M 20000 every time and forgot after upgrading to RC3.

ExecStart=/usr/bin/autossh -M 0 -N -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=2 [PLACEHOLDER]

[frennkie]

I'm wondering whether I am the only person using this or if I misconfigured something on the sshd side.

I would say the safest thing would be that we stick with the default (-M 0) but allow for customization via raspiblitz.conf.

[rootzoll]

@frennkie can you open a seperate issue for that - marked for v1.7?

[frennkie]

Done.... I'm in parallel working on Django-IP2Tor to implement a heartbeat from the Hosts and a alive check for the Lightning Nodes..

Makes no sense to list stale hosts or hosts for which no LND backend is available to create invoices.

[grnqrtr]

After getting my other issues figured out, I got back to testing this again and just wanted to report that my new ip2tor bridge subscription for BTCPay is up and running fine with the Let's Encrypt subscription. Things are working well, very cool!

Just thought I'd mention one thing, after seeing this screen it took probably 5~10 minutes before I could access the new bridge:

I almost cancelled the subscription and was going to try with the other option tondro01 instead of aws1 because I thought it wasn't working. But give it a few minutes and things are working well :)

[frennkie]

Just thought I'd mention one thing, after seeing this screen it took probably 5~10 minutes before I could access the new bridge

From the server logs I have:

>>> b.created_at.isoformat()
'2020-07-25T01:51:43.347113+00:00'
>>> b.modified_at.isoformat()
'2020-07-25T01:52:22.458111+00:00'

Systemd: active (running) since Sat 2020-07-25 01:52:22 UTC; 9h ago

So it should not have taken more than one minute to be available.

[grnqrtr]

I don't know, maybe it was something on my end then. I kept refreshing the page, but didn't come up for a while.

[rootzoll]

OK tested again and making a subscription is still working. Closing this issue for v1.6.1 release.
For reporting operational problems to improve the service further new issues shoulbe be opened or added to existing like: #1662