rancher/agent never connects to server on custom host

[dnadave]

I have my rancher/agent running on the same host as the server. I have followed the instructions to set up the CATTLE_IP as instructed and I have verified that the host address is reachable. The rancher/agent is stuck at:

792492e71fd5 rancher/agent:v0.8.2 "/run.sh run" 12 minutes ago Up 12 minutes rancher-agent

How do I get this fixed? I have already tried the solutions proposed in the troubleshooting guide.

[dnadave]

16 HOURS NOW AND I'M STILL NOT ABLE TO USE ANYTHING!!!!!!!!!!!!! COME ON! THIS IS A COMPLETE BLOCKER FOR ME AND I'M STARTING TO LOSE MONEY!!!!!!!!!!!!!1

[deniseschannon]

Just to confirm, you did CATTLE_AGENT_IP and not CATTLE_IP, correct?

What version of Rancher and Docker are you running? Where is your host being run from?

Can you copy/paste the command you used to add the agent?

[dnadave]

Rancher:
Component Version
Rancher v0.56.1
Cattle v0.141.1
User Interface v0.86.0
Rancher Compose v0.7.1

Rancher-Agent: v0.8.2

Docker
Docker version 1.9.1, build a34a1d5

Command to start Rancher:
docker run -d --volumes-from rancher-data --restart=always -p 8080:8080 rancher/server:latest

Command to start Agent:
docker run -d -e CATTLE_AGENT=10.0.252.112 -e CATTLE_AGENT_IP=10.0.252.112 -v /var/run/docker.sock:/var/run/docker.sock rancher/agent:v0.8.2 http://axiohost13:8080/v1/scripts/DC0D4D659B0B3DAA8A6C:1454457600000:EpVwHOoRG1UuOeKA5P1oBOGyzg

Wiped all of my containers and volumes and accounts. Started over fresh and still no go.

We're using local authentication:

curl -i http://axiohost13:8080/v1
HTTP/1.1 401 Unauthorized
Content-Length: 158
Content-Type: application/json; charset=utf-8
Date: Wed, 03 Feb 2016 00:51:20 GMT
Server: Jetty(9.2.11.v20150529)
Www-Authenticate: Basic realm="Enter API access key and secret key as username and password"
X-Api-Schemas: http://axiohost13:8080/v1/schemas

{"id":"5348efce-7a12-4be2-a9be-3526952b6cf2","type":"error","links":{},"actions":{},"status":401,"code":"Unauthorized","message":"Unauthorized","detail":null}/P

[deniseschannon]

Before you try to re-add the agent, can you try to wipe out /var/lib/rancher/state?

[dnadave]

I've done the following and then tried to re-add the agent:

# rm -rf /var/lib/rancher/state
# docker rm -fv rancher-agent
# docker rm -fv rancher-agent-staterancher-agent-state

I tried all of this before posting.

[dnadave]

It seems stuck in this state:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
792492e71fd5 rancher/agent:v0.8.2 "/run.sh run" 18 hours ago Up 18 hours rancher-agent
30b97cb81318 rancher/server:latest "/usr/bin/s6-svscan /" 18 hours ago Up 18 hours 3306/tcp, 0.0.0.0:8080->8080/tcp stupefied_dubinsky

[deniseschannon]

Okay, I was able to look a little more closely at the command you have. It seems you're missing the --privileged. Also, you don't need the -e CATTLE_AGENT in the command. This is what comes from Rancher:

sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v0.8.2 http://<rancher-server>:8080/v1/scripts/B5E912F949EDAAAB6521:1454522400000:0DNPHdev1R0USnsIH9buDx1Wf1w

Note: you can add three backticks before and after your code make code blocks

[dnadave]

Still no go. Adding the --privileged and removing the -e CATTLE_AGENT doesn't fix the problem.

CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                              NAMES
5b843559e559        rancher/agent:v0.8.2    "/run.sh run"            4 minutes ago       Up 4 minutes                                           rancher-agent
30b97cb81318        rancher/server:latest   "/usr/bin/s6-svscan /"   19 hours ago        Up 19 hours         3306/tcp, 0.0.0.0:8080->8080/tcp   stupefied_dubinsky

[deniseschannon]

The container seems to be running, but it's just not showing up in your UI?

Can you post the rancher/agent logs? docker logs 5b8

[deniseschannon]

If it's super long, just add in a tail to it.

docker logs --tail=500 5b8

[dnadave]

INFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/apply.sh
bootstrap.Df3p10D/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/config.sh: line 222: ./apply.sh: Permission denied
bootstrap.Df3p10D/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/config.sh: line 222: ./apply.sh: Permission denied
INFO: Starting agent for 1F0BD9CA1CB9E23029EA
INFO: Access Key: 1F0BD9CA1CB9E23029EA
INFO: Config URL: http://axiohost13:8080/v1
INFO: Storage URL: http://axiohost13:8080/v1
INFO: API URL: http://axiohost13:8080/v1
INFO: IP: 10.0.252.112
INFO: Port:
INFO: Required Image: rancher/agent:v0.8.2
INFO: Current Image: rancher/agent:v0.8.2
INFO: Using image rancher/agent:v0.8.2
INFO: Downloading agent http://axiohost13:8080/v1/configcontent/configscripts
INFO: Updating configscripts
INFO: Downloading http://axiohost13:8080/v1//configcontent//configscripts current=
INFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/apply.sh
bootstrap.WPKPSfD/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/config.sh: line 222: ./apply.sh: Permission denied
bootstrap.WPKPSfD/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/config.sh: line 222: ./apply.sh: Permission denied
INFO: Starting agent for 1F0BD9CA1CB9E23029EA
INFO: Access Key: 1F0BD9CA1CB9E23029EA
INFO: Config URL: http://axiohost13:8080/v1
INFO: Storage URL: http://axiohost13:8080/v1
INFO: API URL: http://axiohost13:8080/v1
INFO: IP: 10.0.252.112
INFO: Port:
INFO: Required Image: rancher/agent:v0.8.2
INFO: Current Image: rancher/agent:v0.8.2
INFO: Using image rancher/agent:v0.8.2
INFO: Downloading agent http://axiohost13:8080/v1/configcontent/configscripts
INFO: Updating configscripts
INFO: Downloading http://axiohost13:8080/v1//configcontent//configscripts current=
INFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/apply.sh
bootstrap.Xx31C2b/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/config.sh: line 222: ./apply.sh: Permission denied
bootstrap.Xx31C2b/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/config.sh: line 222: ./apply.sh: Permission denied
INFO: Starting agent for 1F0BD9CA1CB9E23029EA
INFO: Access Key: 1F0BD9CA1CB9E23029EA
INFO: Config URL: http://axiohost13:8080/v1
INFO: Storage URL: http://axiohost13:8080/v1
INFO: API URL: http://axiohost13:8080/v1
INFO: IP: 10.0.252.112
INFO: Port:
INFO: Required Image: rancher/agent:v0.8.2
INFO: Current Image: rancher/agent:v0.8.2
INFO: Using image rancher/agent:v0.8.2
INFO: Downloading agent http://axiohost13:8080/v1/configcontent/configscripts
INFO: Updating configscripts
INFO: Downloading http://axiohost13:8080/v1//configcontent//configscripts current=
INFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/apply.sh
bootstrap.u9RdZJ7/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/config.sh: line 222: ./apply.sh: Permission denied
bootstrap.u9RdZJ7/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/config.sh: line 222: ./apply.sh: Permission denied

[dnadave]

This is repeated over and over and over...

[deniseschannon]

And just to confirm, can you do a docker ps -a. You should have a rancher-agent-statecontainer that was launched.

[dnadave]

CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                      PORTS                              NAMES
5b843559e559        rancher/agent:v0.8.2                 "/run.sh run"            49 minutes ago      Up 49 minutes                                                  rancher-agent
778aafbc4060        rancher/agent:v0.8.2                 "/run.sh state"          49 minutes ago      Exited (0) 49 minutes ago                                      rancher-agent-state
023a553f8995        rancher/agent:v0.8.2                 "/run.sh http://axioh"   49 minutes ago      Exited (0) 49 minutes ago                                      romantic_goodall
5fc006391bab        3a03c4bcb70a                         "/bin/sh -c 'git clon"   18 hours ago        Exited (2) 18 hours ago                                        awesome_meitner
e2e5414aed84        rancher/agent:v0.8.2                 "/run.sh http://axioh"   19 hours ago        Exited (0) 19 hours ago                                        hopeful_jones
4645f1e9690f        rancher/agent:v0.8.2                 "/run.sh http://axioh"   19 hours ago        Exited (0) 19 hours ago                                        elegant_fermat
915b8b56cff7        rancher/agent:v0.8.2                 "/run.sh http://axioh"   19 hours ago        Exited (0) 19 hours ago                                        happy_mahavira
cf14291f46a8        rancher/agent:v0.8.2                 "/run.sh http://axioh"   20 hours ago        Exited (0) 20 hours ago                                        jovial_jennings
30b97cb81318        rancher/server:latest                "/usr/bin/s6-svscan /"   20 hours ago        Up 20 hours                 3306/tcp, 0.0.0.0:8080->8080/tcp   stupefied_dubinsky
42c4efb3e67a        rancher/server:latest                "/usr/bin/s6-svscan /"   20 hours ago        Created                                                        rancher-data
2f2480dea172        rancher/agent:v0.8.2                 "/run.sh http://axioh"   20 hours ago        Exited (0) 20 hours ago                                        tender_bhaskara
91fea6834108        rancher/server:latest                "/usr/bin/s6-svscan /"   20 hours ago        Exited (0) 20 hours ago                                        happy_saha

[dnadave]

Since there's been no response in over 3 HOURS, I rebooted the server as I can't think of anything else to do. Now, I'm getting the following error in the rancher/agent log:

INFO: Running Agent Registration Process, CATTLE_URL=http://axiohost13:8080/v1
INFO: Checking for Docker version >= 1.6.0
INFO: Found Server version: 1.9.1
INFO: docker version: Client version: 1.6.0
INFO: docker version: Client API version: 1.18
INFO: docker version: Go version (client): go1.4.2
INFO: docker version: Git commit (client): 4749651
INFO: docker version: OS/Arch (client): linux/amd64
INFO: docker version: Server version: 1.9.1
INFO: docker version: Server API version: 1.21
INFO: docker version: Go version (server): go1.4.2
INFO: docker version: Git commit (server): a34a1d5
INFO: docker version: OS/Arch (server): linux/amd64
INFO: docker info: Containers: 4
INFO: docker info: Images: 286
INFO: docker info: Storage Driver: overlay
INFO: docker info: Backing Filesystem: extfs
INFO: docker info: Execution Driver: native-0.2
INFO: docker info: Kernel Version: 4.1.15-8-default
INFO: docker info: Operating System: openSUSE Leap 42.1 (x86_64)
INFO: docker info: CPUs: 32
INFO: docker info: Total Memory: 125.9 GiB
INFO: docker info: Name: AxioHost13
INFO: docker info: ID: 7MDY:2LCE:NMQ2:3INA:HL4A:LRTZ:VL43:TLBT:M5UN:PF7G:KKGN:AM6D
INFO: docker info: Http Proxy:
INFO: docker info: Https Proxy:
INFO: docker info: No Proxy:
INFO: Attempting to connect to: http://axiohost13:8080/v1
WARNING: No swap limit support
ERROR: http://axiohost13:8080/v1 is not accessible

This has to be some of the most flaky and delicate software I deal with. Nowhere near as STABLE and RELIABLE as Stackengine was.

[dnadave]

Now, the catalogs page is broken. WHAT IS GOING ON?!?!?!?

[dnadave]

OK. I stopped docker, shut down docker0, deleted the network bridge, and then recreated the bridge to docker0. Now, I have the catalog back, but I'm back to the permission denied error. NO HOST means NO WORK. THIS IS INCREDIBLY FRUSTRATING!!!!!

INFO: Starting agent for 9BC7075D49B5A4620E7B INFO: Access Key: 9BC7075D49B5A4620E7B INFO: Config URL: http://axiohost13:8080/v1 INFO: Storage URL: http://axiohost13:8080/v1 INFO: API URL: http://axiohost13:8080/v1 INFO: IP: 10.0.252.112 INFO: Port: INFO: Required Image: rancher/agent:v0.8.2 INFO: Current Image: rancher/agent:v0.8.2 INFO: Using image rancher/agent:v0.8.2 INFO: Downloading agent http://axiohost13:8080/v1/configcontent/configscripts INFO: Updating configscripts INFO: Downloading http://axiohost13:8080/v1//configcontent//configscripts current= INFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/apply.sh bootstrap.tglEzDs/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/config.sh: line 222: ./apply.sh: Permission denied bootstrap.tglEzDs/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/config.sh: line 222: ./apply.sh: Permission denied

[ibuildthecloud]

@dnadave I have never seen this permission denied error. There must be something peculiar about your setup. Honestly I've yet to encounter a user on openSUSE and we don't test that ourselves. Something in your system is preventing the execution of binary files. I can only think of two things that could cause that.

1) you have SELinux or AppArmor (I'm guessing AppArmor) enabled and that is somehow causing an issue. Try disabling AppArmor, rebooting and testing again
2) You have docker running on a mount that has noexec set which prevents execution of binaries.

[dnadave]

SELinux and AppArmor are off. We had issues in the past with them and docker playing nicely together, so we disabled them.

We are using docker on a shared RAID mount, but execution is enabled on that mount. We are also running docker in overlay mode versus devicemapper mode.

[deniseschannon]

Can you double check that the mount doesn't have noexec set? There was a similar issue that was just filed this morning (and I asked the same questions that Darren had suggested in this post). He was able to correct the host registration after updating his "/var FS".

https://github.com/rancher/rancher/issues/3521

[dnadave]

I remounted the partition that docker is on and now I can see that the host is added, but a new problem arises. I can no longer see the catalog. It was there before the restart of docker and remount of the partition, but, now it's not.

Here's the log from the rancher server:

time="2016-02-04T22:34:16Z" level=error msg="Unable to create EventRouter" Err="Get http://axiohost13:8080/v1: dial tcp: i/o timeout" 
time="2016-02-04T22:34:16Z" level=info msg="Exiting go-machine-service..." 
time="2016-02-04T22:34:16Z" level=info msg="Setting log level" logLevel=info 
time="2016-02-04T22:34:16Z" level=info msg="Starting go-machine-service..." gitcommit=5023ae4 
fatal: unable to access 'https://github.com/dnadave/rancher-catalog.git/': Could not resolve host: github.com
time="2016-02-04T22:34:22Z" level=fatal msg="Failed to clone the catalog from git exit status 128" 
time="2016-02-04T22:34:24Z" level=info msg="Starting Rancher Catalog service" 
time="2016-02-04T22:34:24Z" level=info msg="Using catalog library=https://github.com/rancher/rancher-catalog.git" 
time="2016-02-04T22:34:24Z" level=info msg="Using catalog Axio=https://github.com/dnadave/rancher-catalog.git" 
time="2016-02-04T22:34:24Z" level=info msg="Cloning the catalog from git url https://github.com/rancher/rancher-catalog.git" 
Cloning into './DATA/library'...
time="2016-02-04T22:34:42Z" level=fatal msg="Unable to create event router" error="Get http://axiohost13:8080/v1: dial tcp: i/o timeout" 
time="2016-02-04T22:34:44Z" level=info msg="Starting rancher-compose-executor" version=v0.7.1 
time="2016-02-04T22:34:46Z" level=error msg="Unable to create EventRouter" Err="Get http://axiohost13:8080/v1: dial tcp: i/o timeout" 
time="2016-02-04T22:34:46Z" level=info msg="Exiting go-machine-service..." 
time="2016-02-04T22:34:48Z" level=info msg="Setting log level" logLevel=info 
time="2016-02-04T22:34:48Z" level=info msg="Starting go-machine-service..." gitc

[dnadave]

Now, no host, no catalog, and no network! WHAT IS GOING ON HERE!?!?!?!?

```> docker logs --tail 500 fbd

INFO: Running Agent Registration Process, CATTLE_URL=http://axiohost13:8080/v1
INFO: Checking for Docker version >= 1.6.0
INFO: Found Server version: 1.9.1
INFO: docker version: Client version: 1.6.0
INFO: docker version: Client API version: 1.18
INFO: docker version: Go version (client): go1.4.2
INFO: docker version: Git commit (client): 4749651
INFO: docker version: OS/Arch (client): linux/amd64
INFO: docker version: Server version: 1.9.1
INFO: docker version: Server API version: 1.21
INFO: docker version: Go version (server): go1.4.2
INFO: docker version: Git commit (server): a34a1d5
INFO: docker version: OS/Arch (server): linux/amd64
INFO: docker info: Containers: 11
INFO: docker info: Images: 340
INFO: docker info: Storage Driver: overlay
INFO: docker info: Backing Filesystem: extfs
INFO: docker info: Execution Driver: native-0.2
INFO: docker info: Kernel Version: 4.1.15-8-default
INFO: docker info: Operating System: openSUSE Leap 42.1 (x86_64)
INFO: docker info: CPUs: 32
INFO: docker info: Total Memory: 125.9 GiB
INFO: docker info: Name: AxioHost13
INFO: docker info: ID: 7MDY:2LCE:NMQ2:3INA:HL4A:LRTZ:VL43:TLBT:M5UN:PF7G:KKGN:AM6D
INFO: docker info: Http Proxy:
INFO: docker info: Https Proxy:
INFO: docker info: No Proxy:
WARNING: No swap limit support
INFO: Attempting to connect to: http://axiohost13:8080/v1
ERROR: http://axiohost13:8080/v1 is not accessible```

[deniseschannon]

For the catalog issues, it looks like there is an issue with your personal catalog.

fatal: unable to access 'https://github.com/dnadave/rancher-catalog.git/': Could not resolve host: github.com
time="2016-02-04T22:34:22Z" level=fatal msg="Failed to clone the catalog from git exit status 128" 

That's a git error. There is a PR to fix the catalog service so that catalog will still show up if the user inputs an invalid catalog. https://github.com/rancher/rancher-catalog-service/pull/36

Can you try removing your personal catalog?

The error in the catalog logs

time="2016-02-04T22:34:42Z" level=fatal msg="Unable to create event router" error="Get http://axiohost13:8080/v1: dial tcp: i/o timeout" 

and the error in the agent logs

ERROR: http://axiohost13:8080/v1 is not accessible

indicate that there is some kind of network error. Have you tried exec-ing into the rancher-agent container to see whether or not you can curl -i http://axiohost13:8080/v1. You've tried this step and read about what you should expect in the docs?

http://docs.rancher.com/rancher/faqs/troubleshooting/#how-to-check-your-host-registration-is-set-correctly

Do you have auth on or off?

[ibuildthecloud]

I see in there errors

ime="2016-02-04T22:34:16Z" level=info msg="Starting go-machine-service..." gitcommit=5023ae4 
fatal: unable to access 'https://github.com/dnadave/rancher-catalog.git/': Could not resolve host: github.com
time="2016-02-04T22:34:22Z" level=fatal msg="Failed to clone the catalog from git exit status 128" 

It appears DNS is not working. Can you do ping github.com and docker run ubuntu ping github.com. Did you recently change anything regarding DNS that may have had an impact?

[dnadave]

In trying to fix the catalog error, I deleted the bridge and let docker create a new one via:

# rcdocker stop \# ip link set dev docker0 down \# brctl delbr docker0 \# rcdocker start

DNS was not working before I did this. Now, all networking is completely hosed. I've seen this before and it was a huge deal getting it fixed. I still don't really know what fixed that particular issue.

[ibuildthecloud]

Is it possible to just restart the server. That will reset the networking.

[dnadave]

Restarted it several times already.

[dnadave]

OK. I have networking again. On OpenSuSE Leap, rcdocker does not recreate the bridge at startup. You have to execute docker daemon to get the bridge recreated.

[smw355]

Dave this is Shannon - if you get stuck again, feel free to call me or send me a text and we'll do our best to help you through this demo tomorrow.

[dnadave]

OK, now how do I pull the rancher-agent-state container?

INFO: ENV: RANCHER_AGENT_IMAGE=rancher/agent:v0.8.2 time="2016-02-05T01:04:29Z" level=fatal msg="Error response from daemon: Could not find container for entity id b1ae1ee02453713bd0a4feee0133985f07372d43dec62d18b567275b3eec4063" Error: No such image or container: rancher-agent-state

[dnadave]

OK. Fixed this by deleting the entity ID from the sqlite database for docker:

sudo sqlite3 /var/lib/docker/linkgraph.db delete from edge where entity_id = '<conflicting-entity-id-sha-1>'; delete from entity where id = '<conflicting-entity-id-sha-1>';

[deniseschannon]

With networking fixed, does it have access to rancher/server when launching rancher/agent? Are you still getting the "not accessible" in the logs? Did you remove your personal catalog?

[dnadave]

Now, I'm getting this error about not authenticating. I've set up my registry with the correct username and password:

2016-02-05 02:27:44,131 ERROR [:] [] [] [] [cutorService-12] [i.c.p.e.e.i.ProcessEventListenerImpl] Unknown exception running process [instance.start:604] on [46] io.cattle.platform.eventing.exception.EventExecutionException: Image [dnadave/release_microarray:jan2016] failed to pull: Authentication is required. at io.cattle.platform.eventing.exception.EventExecutionException.fromEvent(EventExecutionException.java:53) ~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]

[dnadave]

I can start the containers by hand, but none of my catalog entries work from the web ui.

[dnadave]

I've been able to get some of the catalog entries to start, now, but they take forever to load. I had to create a custom entry in the registry using the URL for my docker hub to get past the authentication errors.

[dnadave]

It's doing a fresh pull of each catalog entry, despite that having already been done. Very strange.

[deniseschannon]

Are containers/catalog entries working for you but just slow?

[dnadave]

They seem to be working. What is strange is I pulled fresh versions of
each container this evening. It looks like rancher is pulling the images
again.

On Thu, Feb 4, 2016, 8:27 PM Denise [email protected] wrote:

Are containers/catalog entries working for you but just slow?

—
Reply to this email directly or view it on GitHub
https://github.com/rancher/rancher/issues/3503#issuecomment-180188327.

[ibuildthecloud]

@dnadave Depends on how your service is configured. There is a label io.rancher.container.pull_image (documented at http://docs.rancher.com/rancher/labels/) that controls if the image should be pull with each container start.

[deniseschannon]

There's also a checkbox that's automatically checked whenever you launch a service to re-pull the image.

[deniseschannon]

@dnadave Are you still having any issues? It seems like everything is working now?

[dnadave]

Hi Denise:

Sorry for the late reply. I've been traveling. We are no longer
experiencing any issues.

Dave H

On Tue, Feb 9, 2016, 12:31 PM Denise [email protected] wrote:

Closed #3503 https://github.com/rancher/rancher/issues/3503.

—
Reply to this email directly or view it on GitHub
https://github.com/rancher/rancher/issues/3503#event-544660956.

[sebbean]

So what was the problem??!!!!911 @dnadave? Did you get a refund or what?

[dnadave]

I do not know what the underlying problem was. We solved it the way we
have to every time something goes wrong with rancher: scrap our
configuration and start over from scratch.

On Mon, Dec 12, 2016, 11:04 AM Sebastian Bean [e-mail]
wrote:

So what was the problem??!!!!911 @dnadave https://github.com/dnadave?
Did you get a refund or what?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/3503#issuecomment-266520405,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABNSm9Bc1_YlMn7zHOpluNlxn9pWZ4NYks5rHZqjgaJpZM4HSCRz
.

[sebbean]

glad you found the help you needed! real team effort :-)

[TomHutter]

It helped me to:

• stop docker service
• flush all iptables
• start docker

IPTABLES="/sbin/iptables"
cat /proc/net/ip_tables_names | while read table; do
  $IPTABLES -t $table -L -n | while read c chain rest; do
      if test "X$c" = "XChain" ; then
        $IPTABLES -t $table -F $chain
      fi
  done
  $IPTABLES -t $table -X
done