Docksal: Add ability to manually start and stop the Docksal system containers

[frederickjh]

Feature Request Description

Currently the docksal-dns, docksal-ssh-agent, docksal-vhost-proxy containers are all started with the restart policy of always. There also does not seem to be a fin command to stop them. I could only find the fin reset system that will stop, reload and restart them.

Reason Why
This feature request is to add a way to manually start and stop the Docksal system containers. This frees up system resources on the host machine when not doing web development work.

Ideas for a possible way this can be done

1. Add Docksal variable to allow setting the restart policy on the three system containers to no(or any of the allowed flags), but default to always if it is not set.
2. Add fin command to start and stop the system containers. Maybe fin docksal start and fin docksal stop.

Describe the results you expected:

1. Docksal and fin will have commands to allow manually starting and stop the system containers . .
2. and also allow setting a restart policy to one of the allowed docker restart flags (no, always, on-failure, or unless-stopped).

Closing Comments
Please let me know if this is something that you would consider incorporating into Docksal, as I might be willing to take the time to code and submit a pull request.

If so it would also be helpful if you could suggest a name for the environmental variable that could be placed in ~/.docksal/docksal.env that would set the restart policy for the system containers.

Thanks for your consideration of this feature!
Frederick

[lmakarov]

It is important to note, that this would be applicable to Linux users only, since Mac and Windows users use fin vm start and fin vm stop to start and stop the entire Docksal VM.

We can use fin docksal or fin system (TBD) as a command namespace to control Docksal system services (and other things). There, start|stop|reset [system service] would make sense. The existing fin reset [system service] command will have to be refactored.

As for the restart policy. I think, it's fine to add this as an override option for power users. Something like DOCKSAL_SYSTEM_RESTART=${DOCKSAL_SYSTEM_RESTART:-always}

@achekulaev your thoughts?

[achekulaev]

Resource saving from stopping those containers is close to neglectable. Those containers hardly consume any resources. RAM and CPU footprint is probably equal to a single tab in Chrome, that I'm sure everyone keeps a lot of, even when they are not needed.

I find the feature request to stop Docksal system containers a very edge case.

[lmakarov]

I find the feature request to stop Docksal system containers a very edge case.

For the purpose of feature parity, there is nothing wrong in having start and stop commands for the Docksal stack on Linux. Mac and Windows get that via fin vm start/stop, when we set/revert networks settings adjustments/etc. back to where they were before Docksal VM was started. Sounds fair to me to ask for a similar experience on Linux.

[lmakarov]

Regarding system resource - I agree with @achekulaev. Docksal system container hardly consume anything.

@frederickjh run fin diagnose. You will see something similar:

███  RAM Usage per container
CONTAINER             CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
docksal-dns           0.20%               4.914MiB / 1.952GiB   0.25%               198kB / 86.9kB      293MB / 0B          10
docksal-ssh-agent     0.00%               1.035MiB / 1.952GiB   0.05%               132kB / 0B          17.7MB / 0B         2
docksal-vhost-proxy   0.25%               9.324MiB / 1.952GiB   0.47%               109MB / 111MB       412MB / 119kB       12

[frederickjh]

Regarding the resource this is true. However Docksal is forcing the Linux users to either always use their system containers or find the Docker commands to set the Docker container restart policy and stop them all.

We need to consider that for Linux users who currently use another Docker Web Development system and want to try Docksal this issue would effectively force them to use Docksal until they can figure out how to stop the Docksal system containers or they uninstall it. The later is provided and better documented in Docksal. So which do you think that they will choose?

Currently the following commands will do this.
First set the restart policy on the containers to no with
docker update --restart=no docksal-vhost-proxy docksal-dns docksal-ssh-agent
then stop them with
docker stop docksal-dns docksal-ssh-agent docksal-vhost-proxy.

If you want to restart them you can use
docker start docksal-dns docksal-ssh-agent docksal-vhost-proxy
or
fin reset system.

[frederickjh]

Just testing fin reset system after adding the DOCKSAL_SYSTEM_RESTART environmental variable. In the feature request description say that this could be placed in ~/.docksal/docksal-local.env but this does not work. It should be placed in ~/.docksal/docksal.env. I have corrected this above.

[achekulaev]

@frederickjh I'm not sure I follow the forcing thing.

1. Think about Docksal containers like about regular Linux service than runs the in the background, but you don't need to think about it unless it consumes resources (and it doesn't).

2. We are not locking you in. You can still use Apache (or anything else) in parallel to Docksal if you want (just make sure Apache binds to 127.0.0.1 not 0.0.0.0).

A friendly reminder, that you can always automate the commands you use to stop Docksal system containers. Create custom global command(s) in ~/.docksal/commands. Let's say some docksal-suspend and docksal-resume

#!/usr/bin/env bash
## Docksal system containers suspend
fin docker update --restart=no docksal-vhost-proxy docksal-dns docksal-ssh-agent
fin docker stop docksal-dns docksal-ssh-agent docksal-vhost-proxy

#!/usr/bin/env bash
## Docksal system containers resume
fin reset system

And use them when needed fin docksal-suspend / fin docksal-resume.

@lmakarov yes we could implement something similar to fin vm stop for Linux to stop Docksal system containers and remove network interface. Say fin docksal start/fin docksal stop. That would be much more in line with current UX than a variable that prevent Docksal system containers restart.

[frederickjh]

@achekulaev Re: forcing, try installing any of the following Drupal Docker Development environments on top of a computer with Docksal already install and get them to run their system containers without first stopping Docksal's system containers. They all want ports 80 and 443.

Lando
Amazee.io pygmy
Docker4Drupal
Terra
Kalabox

[achekulaev]

Despite that other services trying to force you into using only them is not a Docksal issue the command that @lmakarov describes, that will stop Docksal system services and remove network interface should be what you need.

It will take some time to implement it as several additional checks with friendly messaging should be included to process different edge cases. Meanwhile you can use your local automation I mentioned above.

[lmakarov]

@frederickjh regarding forcing and port conflicts. Docksal makes every effort to play nicely with others:

$ fin docker ps --filter "label=io.docksal.group=system" --format "{{.Image}} {{.Ports}}"
docksal/dns:1.0 192.168.64.100:53->53/udp
docksal/ssh-agent:1.0 
docksal/vhost-proxy:1.1 192.168.64.100:80->80/tcp, 192.168.64.100:443->443/tcp

As you can see from the output above, we are using a dedicated virtual IP for Docksal (192.168.64.100) and bind all port to that. If there are port conflicts - that means that the other tool(s) are set up to bind to the wildcard 0.0.0.0 (all interfaces) and should be adjusted to not do so.

Now that I look back at this discussion, it looks to me, that we just need fin docksal start/fin docksal stop for Linux. Having that, I do not see a use case for the DOCKSAL_SYSTEM_RESTART variable.

[frederickjh]

Maybe my choice of the word "force" was a poor decision in hindsight. There are more resources than CPU cycles and RAM, ports are also resources on a computer. As long as the Docksal's system containers are running those resources (ports) are not available to anyone else. This is all I was trying to point out.

Do either one of you run Docksal on Linux on your web development computer? I am just wondering.

---

@lmakarov For the fin docksal stop command if we do not include the DOCKSAL_SYSTEM_RESTART variable should we use a docker restart policy of no? Like in the docker update command @achekulaev had above:
docker update --restart=no docksal-vhost-proxy docksal-dns docksal-ssh-agent
This would mean we assume that if they stop them they do not want them to start again until they manually do it.

I assume for the fin docksal start a restart policy of always would be used. This would mean that once they restart using fin we assume that they want docker to make sure they stay running.

[achekulaev]

fin docksal stop will stop and remove those containers so they will not restart unless fin docksal start is executed. I guess the latter will replace fin reset system.

[frederickjh]

Or should fin docksal start be an alias for fin reset system?

Is this the direction we want to go for these commands?
fin docksal reset same as current fin reset system
fin docksal start same as current fin reset system or do we need to do something different for startup?
fin docksal stop like what @achekulaev suggested for a custom command:

docker update --restart=no docksal-vhost-proxy docksal-dns docksal-ssh-agent
docker stop docksal-dns docksal-ssh-agent docksal-vhost-proxy

[achekulaev]

fin docksal start - add network interface for Docksal subnet, start system containers
fin docksal reset - restart system containers (alias of fin reset system)
fin docksal stop - stop system containers, remove system containers, remove network interface for Docksal subnet

p.s. just for the reference, there is still hidden commands fin stop proxy, fin stop dns and fin stop ssh-agent present. They do not change autostart policy for containers.

[lmakarov]

In my mind, fin docksal reset would be just an alias to fin docksal stop; fin docksal start.

Restart policy should, probably stay as-is (always) to keep things simple. If you want them (docksal system services) stoped - fin docksal stop.

[lmakarov]

@frederickjh

Do either one of you run Docksal on Linux on your web development computer? I am just wondering.

Macs on a regular basis. Additionally, we do testing in Windows 10 Pro and Ubuntu 16.04.

[frederickjh]

p.s. just for the reference, there is still hidden commands fin stop proxy, fin stop dns and fin stop ssh-agent present. They do not change autostart policy for containers.

Hidden in the sense that, they are disabled or don't show in the help? Those commands do not work here:
ERROR: Cannot detect project root.

[frederickjh]

fin docksal stop - stop system containers, remove system containers, remove network interface for Docksal subnet

A question:
Do we have code for "removing network interface for Docksal subnet"? I hunted through fin but could not find anything. I also looked at the uninstall instructions, but even these only tell the user to remove the VM and delete the files. I am guessing since a lot of people use Docksal in a VM this is why the uninstall Docker cleanup got missed for Linux Docker users.

In my mind, fin docksal reset would be just an alias to fin docksal stop; fin docksal start.

I have to agree with @achekulaev on this one. The current fin reset system in the comments says:

# Reset container(s) (stop, remove, up)
# @param $1 $2... containers names
reset ()

fin is stopping, removing or reconfiguring in the case of the network and bring it all back up.

[lmakarov]

The reset () function in fin does reset for project services, as well as system service. It will be best to move the system service part outside of this function.

I'm also starting to think that fin docksal may be a good replacement for fin vm so that we keep things consistent across the board. Users don't need to know about the VM layer. From their perspective it's just Docksal starting up or stopping. Just a thought for now. Also... with Docker for Mac/Windows we currently do not control the Docker/VM layer and I don't think we want to. So fin docksal start|stop in that mode makes less sense. Now it gets complicated...

[frederickjh]

@lmakarov I had the same thought about fin vm and fin docksal however I did not want to mess with the status quo, as changing that command most likely means a change for the largest number of Docksal users.

[frederickjh]

lmakarov commented on Nov 15
Restart policy should, probably stay as-is (always) to keep things simple. If you want them (docksal system services) stopped - fin docksal stop.

If we go with a restart policy of always the container will try to restart after being stopped. The two possible restart policies that make sense to use would be no or unless-stopped.

I think it should default to the later, which would mean that they would restart after a reboot of the computer.

I think this shows the use case for the DOCKSAL_SYSTEM_RESTART environmental variable for those that want to be able to switch between Drupal Docker development systems and don't want one that they have shutdown restarting after a reboot and interfering with another Drupal Docker development system.

[lmakarov]

@frederickjh I tested unless-stopped and it looks to work fine.

The docs are a little confusing though:

unless-stopped Restart the container unless it is explicitly stopped or Docker itself is stopped or restarted.

In reality, the services do start when I stopped and then started Docker for Mac. Need to test this in Linux as well.

[lmakarov]

@frederickjh after further review with @achekulaev we decided to just remove Docksal system services in fin system stop and leave the restart policy as always.

The "or Docker itself is stopped or restarted" part in the docs for unless-stopped policy is confusing and does not work the same way as described in the docs. So we better stick with something known and proven to work.

[frederickjh]

@lmakarov Let me see if I have this correct. First the variable to set the restart policy is rejected (that i worked and created a pull request for before it was rejected.) then the ability to set this in the start and stop command is also missing.

This means that this issue is closed and I still do not have the functionality that want. Mainly the ability to shutdown the Docksal service containers and not have them restart until I ask them to.

unless-stopped does work as described in the Docker documentation for me on Linux. If you manually stop them they do not restart. However if you restart the computer or Docker they will restart.

In any case I wanted the ability to set the restart policy to no so that after a reboot the Docksal service containers will not restart. Looks like I will need to go back to using my Fish functions to change the restart policy on the service containers. Too much to ask for that functionality to be baked into Docksal.

I am running Docksal on a development server as my main Drupal Docker development system, however I have a website that I am working that will be web hosted with a company that use Docker. They have a much more simplistic Drupal Docker development system, but the containers are the same as what they will be on the web.

This issue was a request to be able to shut Docksal down completely that it would not restart unless asked to, so that I could use the web hosting company's Docker development system without Docksal starting up each time we boot the server but want to use the other Docker development system.

[achekulaev]

@frederickjh fin system stop will stop and remove Docksal system containers. They will not restart until you run another fin command like fin project start (which will detect that they are not running and attempt to start them) or fin system start.

If you find that fin system stop is not doing what you need, feel free to reopen this issue and explain what it is not happening as expected.

[frederickjh]

Sorry @achekulaev when I run the command fin system stop I get

Unknown command system stop. See 'fin help' for list of available commands

I did fin update, but after that I get the same response. I am running fin version 1.43.2

I tried this inside a project and outside of any projects.

[lmakarov]

@frederickjh this issue was closed automatically when PR #432 was merged into the develop branch. It will be released in the next release of Docksal.

If you'd like to try it ahead of that, you will have to switch to the develop version of Docksal:

DOCKSAL_VERSION=develop fin update

[frederickjh]

OK, Thanks for the heads up.

[frederickjh]

For those that find this issue and want to stop the Docksal containers but do not what to remove them each time they stop them, I have create a fish function for the fish shell that you should place in ~/.config/fish/functions/docksal.fish

function docksal --description "Start or stop Docksal system services: dns vhost-proxy ssh-agent. Usage: docksal <flag>  where flag is either start or stop."
    switch $argv
    case stop
        docker update --restart=no docksal-vhost-proxy docksal-dns docksal-ssh-agent
        docker stop docksal-dns docksal-ssh-agent docksal-vhost-proxy
    case start
        fin reset system
    case '*'
        echo "Invalid flag! Try either start or stop."
    end
end

If you use bash you can create aliases using the commands from the case statement above.

If you want to be able to stop them but have the restart on reboot or Docker host restart then change --restart=no to --restart=unless-stopped in the function above.

[lmakarov]

@frederickjh

For those that find this issue and want to stop the Docksal containers but do not what to remove them each time they stop them

Do you have a case, where removing Docksal system service (vs stopping them) is not desirable?

[frederickjh]

When you plan to use the containers again.

Removing the containers with docker rm removes all traces of containers. Not sure this is really needed when I plan to use them again as they then will need to be rebuilt. A simple stop with a no restart policy seems adequate to me.

Why do you think it is necessary / better to remove them vs. stopping them? Maybe this is a lack of understanding on my part of Docker. The technology has been growing so fast of the last couple of years it is hard to keep up.

[achekulaev]

@frederickjh fin reset system removes and recreates system containers. Even if they exist, running or stopped. It's not something new, it was like this all the time. Stopping containers won't "save" them 😃 they are going to be mercilessly removed during next fin system reset/fin reset system.

[lmakarov]

The reason we opted for rm vs stop was simplicity, so we don't have to play with the restart policy. You do lose logs from system services, but do you care? You do lose any custom adjustments made to the system services, but should you be doing those in the first place?

The whole fin system start/stop epic is about getting Docksal entirely ON/OFF, with fin system stop pretty much reversing any system changes done by fin system start (network settings, system services, NFS/SMB shares, etc.).