Spinnaker: Fresh Spinnaker Install, Unable to create application. Spinner keeps spinning...

[ximbal]

I've just installed a fresh copy of Spinnaker, I can see that not all the services are up and running:

I am missing one of the key pieces, cloudriver.

So after navigating a bit on the other issues I decided to give it a spin manually and voilá, I believe I must have had missed something along the debianlocal installation, I am uploading the captured data with the hopes some one can point me on the right direction.

so here it is:

cloudriver-manual-start.log.txt

regards and thank you for the help.
EN

[ximbal]

So, I am finding an error that basically is related to the fact that AWS is not finding the credentials, I have created the credentials file with the actual credentials inside the root and spinnaker home folders and it does not seem to be able to find them there, any ideas?

[ximbal]

Went back to start from scratch, fresh copy of Ububtu 14.04,

the commands i"ve executed:

curl -O https://raw.githubusercontent.com/spinnaker/halyard/master/install/stable/InstallHalyard.sh
sudo bash InstallHalyard.sh
hal -v
hal config deploy edit --type localdebian
hal config storage edit --type redis

After this last command I get the following output.

LabSpinnaker:/opt/halyard/bin# hal config storage edit --type redis

• Get current deployment
  Success
• Get persistent storage settings
  Success
• Edit persistent storage settings
  Failure
  Problems in Global:
  ! ERROR Failure writing your halconfig to path
  "/root/.hal/config"

• Failed to edit persistent storage.

What in the world am I missing? is something broken? some steps that I skipped?

[lwander]

What user did you install halyard as? /opt/spinnaker/config/halyard-user, and what are the permissions on
/root/.hal/config?

[ximbal]

Answer First Question.

root@LabSpinnaker:/usr/local/java# cd /opt/spinnaker/config/
root@LabSpinnaker:/opt/spinnaker/config# ls
halyard-user halyard.yml
root@LabSpinnaker:/opt/spinnaker/config# ll
total 16
drwxr-xr-x 2 en root 4096 Jun 8 14:22 ./
drwxr-xr-x 3 root root 4096 Jun 8 14:22 ../
-rw-r--r-- 1 root root 3 Jun 8 14:22 halyard-user
-rw-r--r-- 1 root root 248 Jun 8 14:22 halyard.yml

Second question.

root@LabSpinnaker:~# ll
total 36
drwx------ 5 root root 4096 Jun 8 15:05 ./
drwxr-xr-x 22 root root 4096 Jun 8 14:06 ../
-rw------- 1 root root 353 Jun 8 14:28 .bash_history
-rw-r--r-- 1 root root 3103 Jun 8 14:50 .bashrc
drwxr--r-- 2 root root 4096 Jun 8 14:22 .hal/

But no

config

inside

.hal

[lwander]

What are the contents of /opt/spinnaker/config/halyard-user?

[ximbal]

root

[lwander]

config is written to .hal the first time an edit is made. Since the default deployment type is localdebian the first first edit was changing the storage type to redis.

Can you share any stack traces in the /var/log/spinnaker/halyard/halyard.log file during your failed edit operation?

[ximbal]

I'll send you the log,
halyard.log.txt

Just added the .txt extention for it to load here.

[lwander]

Thanks, this is very helpful.

Logged in as root in that machine, are you able to create files/folders in the .hal directory?

e.g. does touch /root/.hal/some-file work? What about mkdir /root/.hal/a-folder?

[ximbal]

I hope it is, I need help too.

And Yes, I can create folders and files inside .hal

root@LabSpinnaker:~# mkdir /root/.hal/x-fld
root@LabSpinnaker:~# ls /root/.hal
uninstall.sh  x-fld
root@LabSpinnaker:~# touch /root/.hal/someconfigfile
root@LabSpinnaker:~# ls /root/.hal
someconfigfile  uninstall.sh  x-fld

[lwander]

What user were you when you ran sudo bash InstallHalyard.sh?

[lwander]

Also, what's the output of ps -aux | grep halyard? One entry is going to have a very long classpath attached, you don't need to paste that here.

[ximbal]

I have always been root,

As mentioned the second entry on the ps -auxis enormous, here's the first line:

root 435 0.0 0.0 62112 3668 ? S Jun08 0:00 sudo -u en -g en /opt/halyard/bin/halyard
The big line however is run by my own user...which is a little bit odd. is this normal?

en 582 0.1 3.5 4014200 288384 ? Sl Jun08 1:56 java -Djava.security.egd=and then blah blah....long sequence...

[lwander]

Ah - looks like halyard is being run as user en but the files it needs are owned by root - let me read through the install script and see if I can supply a fix there. In the mean time - try running

chown en:en /root/.hal

and then attempt to use halyard again.

[ximbal]

There might be a little piece of information missing on the documentation, it is really not evident how I can specify the version I want to install, I got this message which is different (so I guess we've moved a bit forward).

root@LabSpinnaker:/home/en# hal deploy apply
+ Get current deployment
  Success
- Apply deployment
  Failure
Problems in default:
- WARNING You have not yet selected a version of Spinnaker to
  deploy.
? Options include:
  - 1.0.0

Problems in Global:
! ERROR Error creating the directory /root/.hal/default/staging:
  Error creating the directory /root/.hal/default/staging with unknown reason.

- Failed to deploy Spinnaker.

So my new question is How to specify which spinnaker version to deploy I do not seem to find any obvious command when adding the-hoption

[lwander]

This should do the trick:

https://www.spinnaker.io/setup/install/deploy/

[ximbal]

Ohhh boy, seems like I have a rights lockdown!, tried to run it as root. and I wasn't unable to create the config file.

root@LabSpinnaker:/home/en# sudo hal config version edit --version 1.0.0
+ Get current deployment
Success
- Edit Spinnaker version
Failure
Problems in Global:
! ERROR Failure writing your halconfig to path
"/root/.hal/config"

• Failed to update version.

[lwander]

You said earlier that the contents of /opt/spinnaker/config/halyard-user was root right? If that's the case, restart Halyard sudo service halyard restart so it will pick up that user, and try again.

[ximbal]

yeap the only way that I managed to create the config file was to go against the laws of nature....and perform achmod 777 rooton the root folder.

Then it suceeded on the config file creation, after I performed a restart, I checked the output of the ps - aux, and it launched the service as my username instead of root, however the service halyard restart was performed on the console with the root user login. I am going bananas right here.

[lwander]

Is en your user? And the contents of /opt/spinnaker/config/halyard-user are still root? Take a look at /etc/init/halyard.conf to see how the username is resolved.

[ximbal]

Yes en is the username,

Here's how/etc/init/halyard.confis resolving the username:

description "halyard"

start on filesystem or runlevel [2345]

expect fork

exec sudo -u $(cat /opt/spinnaker/config/halyard-user) -g $(cat /opt/spinnaker/config/halyard-user) /opt/halyard/bin/halyard 2>&1 > /var/log/spinnaker/halyard/halyard.log &

and the user in /opt/spinnaker/config/halyard-user is still root

It is odd, obviously whan I changed to 777 the root directory then the .hal directory inside was accesible and the config file could be created. It seems that Spinnaker has installed however, I do not seem to have available the CloudDriver Service...

[lwander]

I have (hopefully) a fix in the install behavior: https://github.com/spinnaker/halyard/pull/546, once that's merged, can you try on a fresh VM the same install path?

[ximbal]

how often does the merges take place?

[lwander]

It's merged

[ximbal]

OK, i will install in a fresh VM and get back to you after lunch time.

[ximbal]

Victorious!

Ok, so I got it to work, but not without your valuable help! @lwander .

So I'll leave the steps here for posterity.

Each line is a separate command in the command line, I've installed this on a virtual machine with a freshly installed Ubuntu 14.04 copy with nothing else than SSH. Then SSH as root, You will need to configure sshd on your console to allow root access.

https://askubuntu.com/questions/469143/how-to-enable-ssh-root-access-on-ubuntu-14-04

curl -O https://raw.githubusercontent.com/spinnaker/halyard/master/install/stable/InstallHalyard.sh

created a user account member of the adm and sudo groups (is this necessary???)

then Install Halyard:

bash InstallHalyard.sh

Verify that HAL is installed and validate its version.

hal -v

Tell Hal that the deployment type will be as a local instance (this will publish all services in localhost which will be tricky later in order to access them, but I have a turnaround so keep reading)

hal config deploy edit --type localdebian

Hal will complain that a version has not been selected, just tell HAL which version:

hal config version edit --version 1.0.0

The tell HAL which storage you are going to use, in my case and since it is local I want to use redis.

hal config storage edit --type redis

So now we need to add a cloud provider to HAL, we use AWS so we add it like this:

hal config provider aws edit --access-key-idXXXXXXXXXXXXXXXXXXXX--secret-access-key

I created a user on AWS and added access keys to the user inside IAM on the user security credentials tab. Obviously my access-key-idis not XXXXXXXXXXXXXXXXXXXX, I edited it. You do not need to enter the secret-access-key because the command will prompt for it.

Then you need to create a username relative or that will only concern you spinnaker installation however this will get related to you AWS Account-ID, so in MY spinnaker local installation I chose the username spinnakermaster you should choose yours!. And my AWS Account ID is not YYYYYYYYYYYY, I've edited too.
All the configurations and steps that you'll need to do inside AWS for this to work are really well documented here:

https://www.spinnaker.io/setup/providers/aws/

And to tell HAL of of the above here's the command:

hal config provider aws account add spinnakermaster --account-id YYYYYYYYYYYY --assume-role role/spinnakerManaged

And after all that and if everything went according to plan we can ask HAL to deploy our brand new spinnaker installation.

hal deploy apply

It will begin a long installation downloading and configuring all the services.
Once it has finished you may do whatever you like but in my case I created a monitoring script like the one described here:

https://github.com/spinnaker/spinnaker/issues/854

Which can be launched on a recursive manner as this:

watch -n1 spinnaker-status.shor until toctrl+Cit!.

then to be able to access your local VM spinnaker copy you can either setup a reverse proxy with the proxy server of your choice to forward all the requests to localhost or you can simply ssh the SH** out of this redirecting the ports;

ssh [email protected] -L 9000:127.0.0.1:9000 -L 8084:127.0.0.1:8084 -L 8083:127.0.0.1:8083 -L 7002:127.0.0.1:7002 -L 8087:127.0.0.1:8087 -L 8080:127.0.0.1:8080 -L 8088:127.0.0.1:8088 -L 8089:127.0.0.1:8089

Where obviously theZZZ.ZZZ.ZZZ.ZZZ is not an actual IP Address.

And finally to begin having fun with this cutie you have to go to your browser of choice and type into the address bar:

http://127.0.0.0:9000

Hope this helps and saves some time to everybody!.

Cheers.

EN

[lwander]

Thanks so much for the writeup :)

[KPREDDIE]

useful doc...thanks!

[cbonami]

VERY useful doc;

[durgeshworks]

@ximbal I have struggled long time to spin up a full working spinnaker. This article i have followed and it helped me to spin a compete spinnaker. Thank you so much and very helpful

[mononokehime]

Running on Ubuntu 14.04 on AWS, I run all the commands above as user ubuntu with sudo rights. The final install command I run: hal deploy apply. Further down the line I had problems running as root. This was with a fresh install, with hal version 0.35.0-20171002204641

[jawlitkp]

When I followed above steps , I was able to launch spinnaker UI but when I tried to create new application, its just hanging there, nothing is happening. did somone had the same issue ? please let me know solution

[ayushri21]

When I followed above steps , I was able to launch spinnaker UI but when I tried to create new application, its just hanging there, nothing is happening. did somone had the same issue ? please let me know solution

[techqu]

When I followed above steps , I was able to launch spinnaker UI but when I tried to create new application, its just hanging there, nothing is happening. did somone had the same issue ? please let me know solution

[ayushri21]

@techqu Are you using Spinnaker with AWS ?
This issue is because cloud driver service (with port 7002) is not configured properly, I resolved this issue by correcting AWS Role, which is assigned while adding AWS account.

[anuppoojari]

@ayushri21
I have followed the doc
https://www.spinnaker.io/setup/install/providers/aws/

Still cloud driver service (with port 7002) is not running because of this new application i cant create
Can you let me know what cahnges i have to do apart from this doc
Im running spinaker in EC2 ubuntu 14.04

[Hayyk]

@ximbal I have followed your doc for installation from scratch, but I get "WARNING No validation for the AWS provider has been implemented." every time. and if I deploy spinnaker clouddriver doesn't start. Manual start log is the same as yours, which you provided in the beginning of the issue. Can you provide steps to avoid this?

[Hayyk]

@ayushri21 what type of corrections have you made in AWS Role?

[ayushri21]

@Hayyk
Required EC2 Role: 'SpinnakerRole'

Attach following policies to the SpinnakerRole

1. Power user access

2. Inline policy - Inline policy through policy generator to provide
   custom “PassRole” role from IAM. For creating this policy, select “AWS
   Identity and Access Management” from the AWS Service pulldown and select
   “PassRole” for Actions. Apply it to all resources.

Attach this role to Spinnaker instance and in below command as well for
adding aws account:

hal config provider aws account add myacc --account-id ACCOUNT_ID
--assume-role role/SpinnakerRole

On Tue, Apr 10, 2018 at 3:38 PM, Hayyk [e-mail] wrote:

@ayushri21 https://github.com/ayushri21 what type of corrections have
you made in AWS Role?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spinnaker/spinnaker/issues/1729#issuecomment-380046734,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ai2DhJb7I2Sv1LdPtHA8KFzaiUFsiul-ks5tnISogaJpZM4N0C-s
.

[Hayyk]

@ayushri21 I've tried as you describe but without luck

[chrisevett]

I ran into this issue and was able to fix it. Hopefully this helps someone because I found this issue googling and couldnt find any relevant info anywhere else 😄 .

I installed spinnaker on my kubernetes cluster using halyard, my kubernetes is running on aws and we're using s3 as our storage.

Once the halyard process completed I changed the gate and deck services to be LoadBalancer instead of ClusterIP

I then logged into deck via my load balancer expecting it to work and instead i got the spinning icon when I went to create an application. I looked at the developer console in chrome and noticed that it was timing out when trying to access localhost:8084, presumably so that it could talk to gate. I re-read the docs and found that there is halyard config to change the base address of the ui and api components

hal config security ui edit --override-base-url http://<deck-loadbalancer-dns-entry>:9000
hal config security api edit --override-base-url http://<gate-loadbalancer-dns-entry>:8084

i did hal deploy apply and when it came back I noticed the developer console was throwing cors errors so I had to do the following.

echo "host: 0.0.0.0" | tee \ ~/.hal/default/service-settings/gate.yml \ ~/.hal/default/service-settings/deck.yml

Notice this is probably a poor security practice (also not using tls above like
I should) but I'm working on a test cluster and experimenting at this point. On a production system you'd want to make some better choices

Anyway hope this helps someone because it took me lots of time to figure out 😅

[ahadrez]

@chrisevett Thank you so much, I struggled so mcuh with it but the "hal config security ui edit configs" did the trick for me. thnx again!

[Deepak-Routray]

@chrisevett Thanks. I was struggling with this for a while. Fortunately, googling landed me here in no time and your suggestions helped me to resolve my issues. Thanks for that.
Below actually helped:
hal config security ui edit --override-base-url http://:9000
hal config security api edit --override-base-url http://:8084

Cheers!!