Ddev: Deprecate the 'environment' concept

[tannerjfco]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Feature Request

ddev (at least w/ the legacy plugin) requires that you specify an environment for a site in order to interact with it. This environment was intended to reflect what hosting environment you are importing from (staging, production, theoretically others). I believe this concept should be completely removed for the following reasons:

• It doesn't apply to local file plugin, as you are not importing from an external environment.
• ddev itself represents an environment (local development), and the idea of managing other environments in this local environment is confusing.
• The feature was confusing to newmedia devs
• The only valid use case I can think of for it is if you wanted to provision the same site, at the same time, with different checkouts. If we were to do a working directory based approach this scenario would easily be addressed by the user having another checkout to spin up from.

[beeradb]

Strongly relates to #27 and #2 in that it could likely be fixed alongside those two tickets.

[frodopwns]

I disagree.

1) It does apply to local because people may want to have different versions of the same app to work on different branch at the same time ie. dev one and review the other.

2) Same as above the environment or whatever we want to call it makes it a unique local version of the site in question. If I want to have multiple running for some reason I need a unique identifier.

3) If the feature was confusing to newmedia devs then they have no clue because the environment situation came from the system they were using already.

4) I think this applies to any site and it applies to the local plugin just as easily. I don't think we should remove but reword to make it a more applicable word then environment.

[tannerjfco]

1) my last bullet defines that use case and how we could satisfy it w/ a working directory approach. e.g. you have checkout for sitename, spin up, make another checkout for sitename2, cd to it and spin up.

2) same as above

3) if it confused the one group that should have understood it, i think it speaks even more to the potential for confusion outside of that system

4) short of requiring the developer to manually define environment in a config file or something, I don't really see how you make this apply to an arbitrary site whose original source environment is in no way determinable

[frodopwns]

I suppose if we make the name be the unique value and then use a location flag then people could add as many as they want

[frodopwns]

I reject any claim that something so simple could be confusing. Sounds like a training problem.

[frodopwns]

@tannerjfco the drud hosting plugin uses environment too.

[frodopwns]

It is important that the plugins share most of the same inputs or things will start to break down.

[tannerjfco]

As a web developer using drud legacy, I myself found it confusing. There's simply no reason for me to be considering environment in any sense for the vast majority of work I'm performing. There's no reason I should think about it in typing a command, or seeing it in an url, or anything. If I ever had the one applicable use case, being able to just spin another copy of the checkout would satisfy it.

And again, ddev itself is an environment. If i spin up drudio production on my local machine, am I changing the production site? To us the answer is obviously no, but it's all too easy to see how confusion could arise. There is but one environment, it is the local environment, and that is what ddev is managing for me. I need not think of others.

[frodopwns]

First of all you had plenty of opportunity to register your issue about the drud tool long before you joined the team and I don't remember that ever coming up. I know a lot of devs would forget to specify env but not much more than that.

The environment value applied to both legacy and drud. It made perfect sense because environment is a known quantity in both systems. In legacy it determined which branch to deploy. In DRUD it determined which deployment of an app you meant.

There was a default for environment and a workon command that would set the default site and env so the user didn't have to type either.

Ddev is not an environment. Ddev is a tool for creating local dev environments. If you have a repo and you need two or more versions of it locally for any reason you will have to supply a unique identifier. You could append things to the site name like "site1", "site2", "sitemaster" to make it unique or you could pass them both in as arguments. Like I said before maybe we change the wording to be something other than "environment" but I think it fills a role right now that isn't necessarily replaced by anything with your proposal.

[rfay]

It seems this can relate to the link @beeradb shared this morning, https://betta.io/blog/2017/02/23/understanding-developers/ -

It's totally easy to understand why you accept this @frodopwns - but when you look around and find that others like @tannerjfco and I were baffled by it (and found it pretty rigid), I encourage you to accept that while it "makes sense" to you, it might not make sense to others. And really it's "others" we're after.

[beeradb]

It's also worth noting that we got a lot of feedback that could easily be construed as related to this. Prime examples were from Ben and other 1Fee folks that words like "production" and "staging" in the site URLs were very confusing to them, and lead to some developers (particularly inexperienced ones) mistakingly thinking they were actually editing staging or production sites.

[frodopwns]

Lets try to stay on topic guys. Think about why I have said I don't want to remove this value but am willing to rename it. Then tell me why it is not an issue or how you can solve it better.

@rfay You should stop to consider if I might be right. How much of what you saw here when you started were you not baffled by?

@beeradb ben and 1fee were all new....the drupal devs were the ones used to the legacy system...who trained the newbies?

This premise is preposterous. Tanner knew the legacy system better than any of us. He especially should know the environment value was inherited from the legacy system and chef. We didn't change it. We only implemented it in docker instead of vagrant. So how he was confused is confusing to me.

As I have said over and over. We need a unique value in order to keep multiple versions of the same site running. We can conflate both values into one and assume the user implements their own unique names or we can accept a unique id plus the name. I'm open to other options but rather than suggest them we are going on about nonsense.

[rfay]

Time to think in terms of "What can we communicate successfully to our prospective users" instead of "What is right and what is wrong". Thanks.

[frodopwns]

Success is usually gained through solutions that are right rather than wrong.

[beeradb]

Listening to feedback from your users also typically helps :)

Let's try to refocus slightly. Here's what we know:

1. Both pantheon and drud hosting have the concept of environments (pantheon calls them IDs, see https://github.com/drud/general/issues/71) . We probably cannot escape needing to reference that ID when pulling in site Archives. drud hosting also has the concept of environments, but we do control that and are able to make adjustments if we need.
2. Platform.sh also appears to use IDs. We don't plan on supporting them at this point, but it's good to see what language other people use. https://docs.platform.sh/overview/cli.html
3. We know from feedback both from members of our team, as well as feedback from end users at newmedia, that the concept of environment, as well as words like "production" and "staging" as applied to a local development are confusing.
4. The only time we really need any concept of remote environment is when pulling in an archive (I think?)

So, here's the question I feel like we should attempt to discuss:

How can we bridge the gap between what most hosting providers are offering while maintaining simplicity in the local dev experience? We need to allow people to do things like pull down archives etc. while abstracting away or removing confusing terms like "environment", "staging", and "production"

[frodopwns]

If I remember correctly they were confused because they went from http://localhost:1025 and only being able to run one website at a time to something a little harder to remember that allowed you to run multiple sites. There is no way around some things. Keep in mind guys that I was working hand in hand with Rich for several weeks getting his feedback multiple times a day. I know what the major complaints were.

[beeradb]

To me, a start to getting to the root of this would be to answer the following two questions:

1. Is it the long term goal of the ddev tool to continue reaching out to third parties (S3, SSH, Pantheon, etc.) to pull in site archives, or is that something we expect the user to handle.
2. If so, does the concept of environment/ID have value anymore once that archive is pulled down?

My personal thoughts are yes to #1, but probably no to #2.

[frodopwns]

The answer to #1 is an obvious yes.

[frodopwns]

2 is a yes too. Because the archive/local plugins are the only plugins that might be doable without it. We can always ignore the 2nd arg for some plugins.

[beeradb]

@frodopwns What specific scenarios would a pantheon/drudapi platform plugin use the site ID/environment in after pulling the initial archive?

[tannerjfco]

There has still only been one valid use case defined for this functionality - ability to spin up more than 1 copy of the same site. That is frankly an edge case that does not warrant its own functionality. The solution should be to simply allow another copy of the site to be made by the user and spin that up. It requires no functionality of its own to maintain, and it's a simple concept a developer should easily understand, without additional terminology, documentation, or change to the mental model of their workflow.

If I am pulling in an archive from an external environment, the majority use case is for getting the latest state of data from production. I am dealing with one external source, that I want represented as that site on my local machine to complete my task. I need no conception of environments for that case.

If there's going to be such a strong resistance to removing legacy cruft, I feel like an equally strong technical justification should be presented. So far I see no justification for it.

[frodopwns]

In the DRUD hosting plugin each deployment of an app is treated as a unique site. If you want the default deploy you have to specify that. You can't switch a currently existing site to be a different brach or deployment, they are managed independently. I don't think we will going away from the git clone in the drudapi plugin so I don't know how you can interact with a deploy without both the app name and the deploy name.

Pantheon let's you keep branches on the git repo they provide. If you wanted to pull down a dev branch or something you would have to specify that name explicitly. If while you were developing on teh dev branch someone found a bug in production you might like to fix the bug in anohter environment without disturbing the one you are working in.

[frodopwns]

OK guys I'm going to get back to development and you all can duke this out after I submit my PR.

[beeradb]

I'm not trying to be snarky, but I'm not sure I follow your argument beyond what would normally be controlled via git branching. AFAIK every site in Pantheon is controlled via a different branch, so you can easily check out a feature-1 branch, commit and push to that, and then move on to to work on feature-2. This doesn't feel like something that necessitates a life for the idea of environment (as applied to local dev at least) beyond pulling in file/db assets.

On the contrary, it feels like we should _assume_ devs will use normal git workflows and move away from the initially checked out branch, after which the idea of a local environment which maps to dev/staging/production gets severed (outside of saying where the initial import came from, anyway)

[frodopwns]

@beeradb except the system is built for concurrent sites not changing existing ones. If you use the local plugin and decide you want to cd into your repo and git checkout that is fine by me. My plan for any 3rd party plugins is to provide a fully automated setup which means we will be git cloning for any new site and git pulling for existing ones.

[beeradb]

My goal is always to be able to come to agreement on issues like this based on discussing the technical merits and/or user experience of the issue at hand. Until now I've avoided interjecting myself into architectural decisions because I'd much prefer an environment where the entire team contributes perspective and works together to find solutions to technical challenges. However, when consensus can't be reached and team members are unwilling to compromise then it falls to me, as the team lead, to make those decisions. There is no one else on the team that is empowered to break these deadlocks or act unilaterally.

In this case, several team members and end users have expressed that the way this is working now is contributing to confusion. Sometimes complexity or confusion cannot be avoided, but it actively does us harm to pretend it doesn't exist.

I'd love for us to focus this conversation on finding solutions to the issues raised here that still achieve our goals (multiple versions of a site running locally, dealing with remote archives) while removing complexity and avoiding developer confusion to the extent possible. If that's not something we're able to do, then I'll make a call as needed.

[frodopwns]

"In this case, several team members and end users have expressed that the way this is working now is contributing to confusion. Sometimes complexity or confusion cannot be avoided, but it actively does us harm to pretend it doesn't exist."

Link me to the list of complaints from our last end user meeting. If its there its there.

[beeradb]

https://newmediadenver.atlassian.net/browse/DRUD-444 would be the main one.

Anecdotally I've also heard other developers on the 1fee/templatefit side say they found the hostnames confusing.

[frodopwns]

Have anything else? That complaint is about hostnames which I don't think applies to this conversation.

[frodopwns]

Here is my current thinking as I'm prototyping this:

ddev add

• checks current directory for src and data
  
  
  
  • If none it creates an empty dir for src and data in the DRUD_WORKSPACE
  
  
• uses cwd name as site id in ddev workspace
• copies src and data into the DRUD_WORKSPACE/siteID

ddev add siteName

• checks current directory for src and data
• uses siteName as site id in ddev workspace
• copies src and data into the DRUD_WORKSPACE/siteID

ddev add siteName deployID

• checks current directory for src and data
• uses siteName + deployID as site id in ddev workspace
• copies src and data into the DRUD_WORKSPACE/siteID+deplyID

ddev add siteName --src=/path/to/src --data=/path/to/data

• checks for existence of src and data dirs
• uses siteName as site id in ddev workspace
• copies src and data into the DRUD_WORKSPACE/siteID

ddev add siteName --in-place

• checks current directory for src and data
• uses siteName as site id in ddev workspace
• mounts src and data from current dir

[beeradb]

I went ahead and took the time to put together a stab at the architecture and workflow for what we've been talking about on several issues. Parts of this discussion have taken place here, some in #8, and of course #27 #26 and #33 all pertain to this at various levels. Currently, we're just talking about isolated pieces, but not addressing workflow as a whole.

In an attempt to address that, I took the time to consolidate input and feedback from those discussions and put together a document outlining my interpretation of what the ddev workflow should be.

I attempted to be as detailed as possible and include sample commands, but as such I'm sure there are mistakes. I'm hoping (to start) we can look past those and talk about the workflow as a whole, before diving in and focusing too much on the minutia.

For the import related stuff, I took a stab at what a pantheon platform could potentially look like, and used the example output from https://github.com/drud/general/issues/71 to inform some of the commands. I'd also be happy to take a stab and what a local importer might look like for comparison if it would be helpful.

All of these commands assume that no prior configuration files exist. I strongly advocate documenting any configuration formats we develop so that configs can be hand generated and prompting can avoided by advanced users. I also feel strongly that we should avoid flags and arguments to the extent possible, in an attempt to keep commands as simple and straightforward as we can.

Brace yourself, this is long.

Creating a fresh ddev site without a local import

Check out the git repository and config a ddev site.

$ cd ~/Projects
$ git clone <git-url>/drud-d8.git
$ cd drud-d8 
$ ddev config
Name (drud-d8):
Type [drupal7, drupal8, wordpress]: drupal8
Docroot location: src

Your ddev configuration has been written to .ddev/config.yaml

Inspect contents of ddev.yaml and ddev-compose.yaml

$ pwd
/Users/username/Projects/drud-d8

$ # Renamed to config.yaml to avoid stuttering.
$ cat .ddev/config.yaml
---
version: 1
name: drud-d8
type: drupal8
docroot: src
webimage: drud/nginx-php-fpm7:0.1.0
dbimage: drud/mysql-local:5.6-0.1.0

$ # ddev-compose would use env vars to populate container names, as to preserve any user customizations.
$ cat .ddev/ddev-compose.yaml
version: '2'
services:
  drud-d8-db:
    container_name: drud-d8-db
    image: $DRUD_DBIMAGE
    volumes:
      - "./data:/db"
    restart: always
    environment:
      MYSQL_DATABASE: data
      MYSQL_ROOT_PASSWORD: root
    ports:
      - "3306"
  drud-d8-web:
    container_name: drud-d8-web
    image: $DRUD_WEBIMAGE
    volumes:
      - "./src:/var/www/html"
    restart: always
    depends_on:
      - drud-d8-db
    links:
      - drud-d8-db:db
    ports:
      - "80"
      - "8025"
    working_dir: "/var/www/html/docroot"
    environment:
      # I'm pretty sure we don't actually need DEPLOY_NAME here, but I've left it in case existing containers may rely on it.
      - DEPLOY_NAME=local
      - VIRTUAL_HOST=drud-d8.ddev.local
networks:
  default:
    external:
      name: ddev_default

Reconfigure an existing site

$ pwd
/Users/username/Projects/drud-d8

$ ddev config
Existing configuration detected. Using defaults from /Users/username/Projects/drud-d8/.ddev/config.yaml

Name (drud-d8):
Type [drupal7, drupal8, wordpress] (drupal8):
Docroot location (src): src

Your ddev configuration has been written to .ddev/config.yaml

Deal with bad configuration file.

$ pwd
/Users/username/Projects/drud-d8

$ ddev config
Existing configuration exists at /Users/username/Projects/drud-d8/.ddev/config.yaml but cannot be read. Please delete the file and re-run "ddev config".

Start Site

$ ddev start

Successfully added drud-d8
Your application can be reached at: http://drud-d8.ddev.local
You can run "ddev describe" to get additional information about your site, such as database credentials.

List Sites

$ ddev list
1 site found.
NAME        DOCROOT                                 TYPE    URL                                 STATUS
drud-d8     /Users/username/Projects/drud-d8/src    drupal8 http://drud-d8.ddev.local           running

Describe View

Global describe with sitename argument

$ pwd
/Users/username
$ ddev describe drud-d8

NAME        DOCROOT                                 TYPE    URL                                 STATUS
drud-d8     /Users/username/Projects/drud-d8/src    drupal8 http://drud-d8.ddev.local           running

MySQL Credentials
-----------------
Username:       root
Password:       root
Database name:  data

Other Services
--------------
MailHog:        http://drud-d8.ddev.local:8025
PHPMyAdmin:     http://drud-d8.ddev.local:8026

From within the working directory

$ pwd
/Users/username/Projects/drud-d8
$ ddev describe

NAME        DOCROOT                                 TYPE    URL                                 STATUS
drud-d8     /Users/username/Projects/drud-d8/src    drupal8 http://drud-d8.ddev.local       running

MySQL Credentials
-----------------
Username:       root
Password:       root
Database name:  data

Other Services
--------------
MailHog:      http://drud-d8.ddev.local:8025
PHPMyAdmin:   http://drud-d8.ddev.local:8026

Pull in data from an external source

Generate Import config

$ pwd
/Users/username/Projects/drud-d8

$ ddev import
No import configuration found, would you like to create one? (y/N): y
# This list gets defined by plugins
Import Source (local, pantheon, s3): pantheon

# Everything below here is defined by a plugin.
You have the following sites, please choose one to import from.

Selection   NAME            ID
1           techingyouout   test
2           techingyouout   dev
3           techingyouout   live

Please select the import source [1-3]: 2
Please select the import destination directory [data]: data

Import config written to ddev.yaml

Would you like to import now? (y/N): y
Getting Resources......
Importing Data.........
Import successful. Your application can be reached at: http://drud-d8.ddev.local

Inspect import config

$ cat ddev.yaml
---
name: drud-d8
type: drupal8
docroot: src
webimage: drud/nginx-php-fpm7:0.1.0
dbimage: drud/mysql-local:5.6-0.1.0
import-config:
    platform: pantheon
    sitename: techingyouout
    site-id: dev
    destination-dir: data

Import again

Thoughts: this could be used between feature branches, or just periodically as needed by developers.

$ pwd
/Users/username/Projects/drud-d8/src/modules

$ ddev import

SOURCE      NAME            ID
pantheon    techingyouout   dev

Local data and file assets will be overwritten if you choose to continue. Perform import now (y/N)? y
Getting Resources......
Importing Data.........
Import successful. Your application can be reached at: http://drud-d8.ddev.local

[frodopwns]

I was under the impression we were going to make it work basically the way it does now and release it. You are talking about other types of changes.

[TopherGopher]

Consider adding a ddev ps which would wrap docker ps

[rickmanelius]

The original intention of this ticket was to address the proposal to remove the "environment" concept. Ultimately, this surfaced many other fundamental issues and conversations regarding workflow, our roadmap, etc. These will likely get addressed or split off into different tickets, but we have a consensus/path forward on the "environment" discussion.

Ultimately, the concepts of "environments" helped address two needs:

1. The ability to run multiple instances of the same application.
2. The ability to specify the source files/code/db of the application.

Issues surrounding 1.

Unfortunately, the word "environments" was a bit restrictive (e.g. it's possible to pull a DB from a "staging" site while files from a "production" site) and confusing (e.g. end-users sometimes confused these labels with actually working on remote files on those environments). At a minimum, we need to use a different word (e.g. "identifier", "label", "instance") in order to still allow for users to run multiple copies. We could use a sane default (e.g. specifying nothing results in a "default" identifier or none at all) and simply using a naming convention of "project-identifier" when used. Actual implementation TBD and will be beyond version 0.5.

Issues surrounding 2.

When jumping between multiple projects, it's helpful to know where the code + db + files came from. And in the case of a PaaS like Pantheon, you can specify those components independently. Therefore, it makes sense to not necessarily call it an "environment" but to at least log the time and location of the last pull. Note that this only matters when tied into a provider plugin that allows you to pull these assets, because up until that time these syncs are done manually and there is no way to "log" the source. This would be a "beyond 1.0" feature.

Summarizing:

A. We're going to remove the requirement of an "environment" for local dev (0.1 - 0.2)
B. We'll need an optional identifier (actual name TBD) for specifying multiple versions (pre 1.0).
C. We'll need a means to log and surface the code/db/file sources both from the CLI and GUI versions (post 1.0).

We'll need to split up new tickets for each.

[rickmanelius]

A second issue that was surfaced in this ticket, but that needs a better home for milestone 0.2 (proposed #33). We created a flow diagram that specified a generic strategy for adding the archive by hand VERSUS a single button install that would likely need assumptions from a provider plugin (i.e. a one button install to Pantheon). In this scenario, we decided to simplify and require the user to git clone the repo and then specify a files and database location in order to fully initialize an application.

Later, we can use this generic/more flexible approach and create simplifications/assumptions that can make this a faster process. But for the purposes of the 0.2 milestone, we've agreed to adopt this approach.

[beeradb]

I suggest this be closed, as at this point I think the issue has run it's course and tickets have been broken out as appropriate. cc @rickmanelius

[rfay]

Environment concept is now officially deprecated.