Go: x/build: Scaleway builders offline

[bradfitz]

Our scaleway account has been locked for abuse.

Our 50 linux-arm builders are offline.

/cc @jessfraz @kevinburke @adams-sarah @cybrcodr @aclements @randall77

[kevinburke]

In hindsight, maybe BenchmarkBitcoinHash(b *testing.B) was not the best idea

[gopherbot]

Change https://golang.org/cl/52130 mentions this issue: dashboard: remove linux-arm-scaleway while we fix issues

[bradfitz]

@kevinburke, we still don't know what "abuse" they're talking about.

I opened a ticket to ask. They have an "Abuses" page to list abuse reports open against you, but there's nothing there:

[bradfitz]

Scaleway replied:

One of your server (163.172.128.145) was detected being part of a DDoS targeting 81.30.144.118/32.
Here is the log of the detection : DDOS from IP 163.172.128.145 (attack ID 640640): protocols : tcp, targets: 81.30.144.118/32, sports: Dynamic (1024-65535), dports: 58823
Your account was automatically locked 3 days ago, this should automatically have opened a ticket 3 days ago to let you know about the issue but it didn't.
This apparently didn't and your server were automatically suspended after 48 hours as you obviously didn't answer to the ticket.

We will further investigate about the root cause to understand why you didn't receive any notification.

We sincerely apologise for this issue and we've added a 20€ discount on your account.

Please make sure no server was hacked in your fleet.

I replied:

We had too many tickets open, so I had to close some old ones in order to open _this_ ticket.

Perhaps your automated system to open a ticket to tell us about abuse also failed to open a new ticket due to the ticket limit.

I'm very surprised that we'd be part of a DDOS. Our images are pretty recent (Xenial) and only listen on port 22 (ssh). I'm not aware of any recent OpenSSH vulnerability since Xenial's time.

But we'll investigate.

Thanks.

So, open questions, assuming our instances were actually 0wned:

• are our Xenial listening on anything besides port 22?
• are there recent OpenSSH vulnerabilities since Xenial?
• should we use the new Scaleway firewall feature to restrict network access from a trusted IP/range?

I leave this to @jessfraz and @adams-sarah.

[gopherbot]

Change https://golang.org/cl/52192 mentions this issue: dashboard: turn on arm-scaleway builders

[jessfraz]

nmap says only ssh is open, but I am doing a more full depth scan for due-diligence

[adams-sarah]

firewall sounds like a good idea. no harm, anyway.

On Mon, Jul 31, 2017 at 10:15 AM, Jess Frazelle [e-mail]
wrote:

nmap says only ssh is open, but I am doing a more full depth scan for
due-diligence

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/21237#issuecomment-319133659, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACnUbJAEn_mN-oxghvLWqIwFJ1Ae8Rtwks5sTguigaJpZM4Oomrw
.

[bradfitz]

firewall sounds like a good idea. no harm, anyway.

Well, there is harm: we have to maintain a bastion host, and we have the inconvenience of having to jump through it or VPN through it or copy files through it whenever we're trying to work. It's not completely free.

[jessfraz]

I turned the scaleway builders back on they should work now.

[bradfitz]

I logged into one of our now-back-up 50 ARM servers to check the image's OpenSSH version:

# dpkg -s openssh-server | grep ^Version
Version: 1:7.2p2-4ubuntu2.1

https://packages.ubuntu.com/xenial/openssh-server says 7.2p2-4ubuntu2.1 is the latest.

So, I'm starting to doubt the whole DDoS thing.

[adams-sarah]

I googled around to see if other people have run into this. there are a
bunch of other reports of ddos from scaleway.
i was thinking maintenance around firewall is probably better than all the
boxes coming down again?
but maybe not

On Mon, Jul 31, 2017 at 10:46 AM, Brad Fitzpatrick <[email protected]

wrote:

I logged into one of our now-back-up 50 ARM servers to check the image's
OpenSSH version:

dpkg -s openssh-server | grep ^Version

Version: 1:7.2p2-4ubuntu2.1

https://packages.ubuntu.com/xenial/openssh-server says 7.2p2-4ubuntu2.1
is the latest.

So, I'm starting to doubt the whole DDoS thing.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/21237#issuecomment-319142652, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACnUbGffzpIHBd0Xw4-XygsmIbA6z4xBks5sThMOgaJpZM4Oomrw
.

[bradfitz]

I think Scaleway's mapping from (time, source IP) => customer is flawed. But I also haven't logged the start/stop time of each machine's ephemeral IP addresses either, so I can't say. And they also didn't tell us a time.

[29e7e280-0d1c-4bba-98fe-f7cd3ca7500a]

you should not put all server open to public. you can have 1 or 3 jump gateway host. then make other only have private acess via private ip/net range. then add firewall for jump host limit acess ip.

_Sent from my Xiaomi Mi Note 2 using FastHub_

[bradfitz]

@netroby, yeah, that's what the "bastion host" referred to above is.

Like I said, there's a non-zero cost in maintaining that, and the machines we're defending against are stateless, can be reimaged & rebooted per builds, and are not valuable (open source code only), so the cost may not outweigh the benefits. It might be easier to just regularly rebuild the images we're using if there are OpenSSH exploits. But we've been running the latest Xenial LTS code with security updates, which is why I suspect Scaleway's finger pointing at our instance is flawed. I doubt somebody wasted their 0 day exploits on the Go continuous build system to do a DDoS attack.

[jhelbling]

• Use different port for SSH
• Us Firewall function from Scaleway. Very simple and if you have problems - it is scaleway support job.
• Use old function aka 'white listing'
• Use tunnels
• Use simple protection tools aka fail2ban or what ever.

Cheers

[dmitshur]

This outage has been resolved (2 years ago), closing.