Rails: When I run ENV=development rake db:create I don't expect it to try and create the test db too

[ioquatix]

Steps to reproduce

According to the following code: https://github.com/rails/rails/blob/3f279977ead48ace1acf53567fd258fe140e3b6f/activerecord/lib/active_record/tasks/database_tasks.rb#L286-L288

When running something like this:

ENV=development rake db:create

It also tries to create test.

I think this is violating the principle of least surprise. At a cursory glance, this also applies to drop_current which might be even more surprising.

Expected behavior

Only development environment is created.

Actual behavior

It tries to create test database too.

System configuration

Latest.

[ioquatix]

Here is some example output:

% RACK_ENV=development rake db:create                                                                       
{"adapter"=>"mysql2", "database"=>"vmail_development", "strict"=>true}
Database 'vmail_development' already exists
Created database 'vmail_test'

An example of where this is a bit odd, is when test is an sqlite3 in-memory database. Therefore, get this output every time.

[ioquatix]

So, here is the odd behyavior from db:drop

export RACK_ENV=development
rake db:drop
{"adapter"=>"mysql2", "database"=>"vmail_development", "strict"=>true}
Dropped database 'vmail_development'
Dropped database 'vmail_test'

I don't think that's good behaviour.

[ioquatix]

In the same vein, I'd like to point out the following inconsistencies:

task :environment do
    # This must be a symbol... or establish_connection thinks it's a URL.
    DATABASE_ENV = :development

    ActiveRecord::Base.configurations = {
        # This key must be a string or it will not be matched by ActiveRecord:
        'development' => {
            'adapter' => 'sqlite3',
            # This key must be a string or rake tasks will fail (e.g. each_current_configuration fails):
            'database' => 'db/development.db'
        }
    }

    # Connect to database:
    unless ActiveRecord::Base.connected?
        ActiveRecord::Base.establish_connection(DATABASE_ENV)
    end
end

I'd be happy to take a look at these issues, but I'd need to know if the Rails core team consider these to be problems or not.

[ioquatix]

What's even more.. inconsistent.. is the fact that once you actually connect, all the keys are symbols:

> Account.connection_config
=> {:adapter=>"mysql2", :database=>"vmail_development", :mail_root=>"/tmp/mail_root"}

[nfedyashev]

Looks like a violation of the least surprise principle to me as well. Experienced that issue many times

[ioquatix]

Honestly, the db:* tasks seem like a bit of a mess. For all the above reasons.. and the way they tie into Rails by default, which breaks unless you override a magical set of variables on "module DatabaseTasks" which in the documentation is described as a class.

[ioquatix]

What kind of feedback would you like?

[ioquatix]

I've been hacking up a solution. Something like this:

                def each_current_configuration(environment)
                    unless configuration = ActiveRecord::Base.configurations[environment]
                        raise ArgumentError.new("Cannot find configuration for environment #{environment}")
                    end

                    if configuration['database'].blank?
                        raise ArgumentError.new("Configuration for environment #{environment} doesn't specify database")
                    end

                    yield configuration
                end

[ioquatix]

So, I've been working on a gem which fixes all these issues (and more). https://github.com/ioquatix/activerecord-migrations

I'm not really sure if this is a solution, it's more of a monkey patch. But it solves a lot of pain points for us w.r.t. deployment with active record, which to be honest leaves a lot to be desired.

[ioquatix]

Just to clarify another inconsistency:

ActiveRecord::Base.configurations = {
    'production' => Database::APP_PRODUCTION,
    'development' => Database::APP_DEVELOPMENT,
    'test' => Database::TEST,
}

DATABASE_ENV = :production

# Access key.. via string:
ActiveRecord::Base.configurations[DATABASE_ENV.to_s]

# Actually connect.. need to use symbol:
ActiveRecord::Base.establish_connection[DATABASE_ENV]

If you try to use symbols in configurations, it breaks in other ways. If you try to use a string in establish_connection it thinks it's a URL.

[JCSHoosier]

Just chiming in here... We have been troubleshooting this issue today as well. It seems to just disregard our RAILS_ENV and tries to create the test db as well as our development db. This causes issues in our case since we have completely different setups for development and test, so it errors and says it can't create the test db.

[ioquatix]

@JCSHoosier take a look at https://github.com/ioquatix/activerecord-migrations it fixes those issues by monkey patch.

[JCSHoosier]

@ioquatix Yep, looking through it now! Thanks!

[stale[bot]]

This issue has been automatically marked as stale because it has not been commented on for at least three months.
The resources of the Rails team are limited, and so we are asking for your help.
If you can still reproduce this error on the 5-1-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

[SGospodinov]

This issue still can be occurred in rails 5.1.0.rc1

The ENV=development rails db:create execution produces the following result:
Created database 'db/development.sqlite3'
Created database 'db/test.sqlite3'

But it is more concerning that dropping the test database drops the development one as well.
ENV=test rails db:drop
Dropped database 'db/development.sqlite3'
Dropped database 'db/test.sqlite3'

Rails version: 5.1.0.rc1
Ruby version: 2.4.0

[edwardmp]

Can someone from the Rails core team explain the rationale behind adding test to the environments array if ENV is development?(https://github.com/rails/rails/blob/3f279977ead48ace1acf53567fd258fe140e3b6f/activerecord/lib/active_record/tasks/database_tasks.rb#L286-L288) ?

[rafaelfranca]

Yes. The reason is rails new foo; rails g scaffold user name; rails db:create db:migrate; rails test have to pass without having to rails db:test:create.

[rafaelfranca]

See also https://github.com/rails/rails/commit/6ca9031ba3c389f71366c3e6abf069c6924c5acf

[ioquatix]

Why wouldn't rails test just create the test database if required? Wouldn't that make more sense?

[rafaelfranca]

At first look yes, but that would also require to other tests frameworks to be changed and in this case I prefer to play safer and keep the behavior that is being like this for years.

[ioquatix]

How can we make this not break our staging/development environment which most certainly doesn't want a test database.

[rafaelfranca]

Maybe implementing the alternative suggested https://github.com/rails/rails/pull/24201/files#r56104919 and applications overriding the task to do what they want.

[ioquatix]

Yeah, well, I already did that in activerecord-migrations gem.

https://github.com/ioquatix/activerecord-migrations/blob/master/lib/active_record/migrations/tasks.rb#L43-L50

But I think it's a nasty hack and shouldn't be necessary.

[edwardmp]

How about optionally disable creating the test base, e.g. by setting an environment var (e.g. RAILS_SKIP_TEST_DATABASE_CREATION. This would keep backward compatibility but allow skipping the test database creation without using monkey patches or dirty hacks.

For instance:
ruby def each_current_configuration(environment) environments = [environment] environments << 'test' if environment == 'development' && !ENV.fetch('RAILS_SKIP_TEST_DATABASE_CREATION').present?

[ioquatix]

@edwardmp That's an interesting idea.

The real issue, here, IMHO, is broken behaviour to support existing testing infrastructure. Your suggestion just adds another layer to the onion.

[edwardmp]

@ioquatix
I agree but I also accept that any existing infrastructure must not be broken by any changes made here. Realistically this is a better solution than you monkeypatching stuff.

[ioquatix]

@edwardmp IMHO, the existing infrastructure was broken the moment it decided to run tests without setting up a database first, and then Rails some how worked around that (or was buggy from the start) and now we have this huge mess.

In these situations, my response is: fix the bug/issue correctly (whatever that may be), and then fix the software which was depending on that behaviour. You can implement a more elaborate solution, e.g. issuing deprecation notices.

Making layers on top of layers like this is simply not maintainable. If ENV=development, trying to make the test database is rediculous - I think we can all agree on that. So, the solution is clear, that function should be fixed.

Yes, this breaks backwards compatibility with existing tools. That's the nature of depending on buggy, unspecified, implicit, behaviour.

[edwardmp]

@ioquatix
I agree the original behavior is troublesome. But realistically I don't see this getting accepted as it would probably break a lot of existing apps.

[ioquatix]

@edwardmp Thanks for your continued input and it is a valuable discussion to have.

I'm glad we can agree on this behaviour being troublesome. Whatever way you slice this cake, it's got problems. We couldn't deploy our database on development postgres because it insisted on creating a "test" database.

But realistically I don't see this getting accepted as it would probably break a lot of existing apps.

Making a new release of ActiveRecord doesn't retroactively break existing apps. Semantic versioning, a decent changeling, etc, all help solve problems like this.

[rafaelfranca]

You are assuming that this behavior is a bug. It is not, it is the behavior that we (the Rails core) wants to the framework. I'm sorry that the behavior don't work for you but it is not going to change.

Thank you for the issue

[ioquatix]

@rafaelfranca Fair enough. It's odd, unexpected, behaviour though, and it's tripping people up.

[worldnamer]

Just ran into this today and was definitely flummoxed. Your desired behavior is unexpected to your users.

[ioquatix]

It’s not a bug it’s a feature :)

[srghma]

@rafaelfranca

this behavior is causing a log of troubles for those who using DATABASE_URL without config/database.yml

Example:

I'm using docker-compose with DATABASE_URL: postgres://some_u:some_p@postgres:5432/some_development and when running

recreate_db_dev:
    docker-compose \
        -f docker-compose.dev.yml \
        run --rm be bash -c '\
        bin/rails db:environment:set RAILS_ENV=development && \
        (rake db:drop RAILS_ENV=development || true) && \
        rake db:create && \
        rake db:migrate && \
        rake db:seed && \
        rake db:schema:dump'

I'm getting

Dropped database 'some_development'
could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Couldn't drop database 'some_test'
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
/usr/local/bundle/gems/pg-0.21.0/lib/pg.rb:56:in `initialize'
/usr/local/bundle/gems/pg-0.21.0/lib/pg.rb:56:in `new'
/usr/local/bundle/gems/pg-0.21.0/lib/pg.rb:56:in `connect'

it is trying to drop some_test but doesn't have password and username for some_test, only for some_development

[srghma]

Wow, actually I'm able to prevent rake from trying to find some_test by removing config/database.yml completely

In this case, all rake db:* commands are working only with database specified in DATABASE_URL

[MmKolodziej]

I do not understand why this is considered not an issue. It obviously is, it is super surprising that tasks ran in development environment try to touch the test database at all.

Just hit a wall when trying to get a correct docker-compose with MySQL and a non-root user, where only one MYSQL_DATABASE can be specified at a time.

Thanks for fighting the good fight @ioquatix .

[ioquatix]

@MmKolodziej I appreciate your sentiment but making this into a fight puts me (and us) at odds with the entire Rails development team who support this design decision. I'd like to think we can solve this problem more constructively. Unfortunately this issue has been closed, and the Rails core team have made a decision, so we have to respect that. Even if we disagree with it.

In that sense, a while ago, I did actually implement https://github.com/ioquatix/activerecord-configurations, which along with https://github.com/ioquatix/activerecord-migrations solves a lot of my own pain points with ActiveRecord. If you have the same pain points, feel free to invest that energy into using and maintaining the above gems. All the pain points outlined in this issue (and more) are resolved by those gems.

[thisismydesign]

I just published a post detailing this issue and showcasing the gem dotenv_rails_db_tasks_fix where I attempt to solve the issues this behaviour causes with dotenv.

[BobbyMcWho]

Bitten by this today. Super disappointing that I can't resolve this without a monkey patch gem.

[ioquatix]

@rafaelfranca @schneems any chance of revisiting this for Rails 6? If the tests create their own database and we add support for Minitest and RSpec by default, is there anything else holding this back?

[cyclotron3k]

Similar to @MmKolodziej, this is also causing issues for me and my docker-compose script.

I for one would appreciate a fix. I realise that's not going to happen in 5.x, but it would be great to see in 6.x

[tinco]

Just a --skip-test-database flag or a TEST_DATABASE_URL environment variable would be nice features that could be added even without breaking stable rails 5 releases

[thilonel]

Is this fixed in any newer versions? On Rails 5.1 it's still annoying like hell.

[fredericgermain]

Rails started as a great framework in 2005. And now we have this.

[bobwhitelock]

@fredericgermain Rails is still great, this is just a minor, if surprising, issue that can be mostly worked around as suggested above. It would be nice if this was less surprising and if there was a first class solution though, but no need to be dramatic.

[fredericgermain]

The workaround is removing the test config from config/database.yml, correct?
If it's the one, I don't call that a workaround.
Or I miss the one you're talking about.

[bobwhitelock]

@fredericgermain - the following are the suggested workarounds so far:

• https://github.com/rails/rails/issues/27299#issuecomment-266386876
• https://github.com/rails/rails/issues/27299#issuecomment-354320563
• https://github.com/rails/rails/issues/27299#issuecomment-394356237

[fredericgermain]

These are far too much pain for a problem that shouldn't exist in the first place, sorry to say.

[romansavrulin]

On Rails 6 it still exists. I've spent an hour trying to fix that for docker container unless found this ticket. Can this be fixed?

[iqbalhasnan]

well, I guess I'm not the only one facing this 'issue'.

[jlevnhv]

It's pretty strange to me there is no way to override this behavior.

I understand the Rails Core team expects this behavior, and wants new users to hit the ground running with simple commands, etc. That's fine.

But we're using AWS' ParameterStore to manage our credentials, so every time we make an unnecessary, and unexpected query, it costs more money and creates more issues for us.

Even if we accept that this behavior is "expected," despite it being surprising, I still believe there should be a way to override this behavior with a switch or something.

[barrkel]

If we have different database connections for test vs development, it appears this new scheme doesn't work at all. Whatever login we use for test (using a ramdisk for speed) doesn't work for development (using persistent disk for large amounts of test data). And there's no easy workaround that I can see, apart from patching the source.

[pustomytnyk]

this should be configurable.

[esteban8a]

I got bitten by this today as well... Wasted about 4 hours looking into this particular issue until I found this thread.

If you add up all the hours everyone has invested in this not so obvious behavior you might be impressed by the amount of money that is being wasted because of an ego.

Quick advise: Listen to your users.

[ioquatix]

@rafaelfranca Do you think there is any way we can revisit the decision here?

[rafaelfranca]

No. This is by design.

[rafaelfranca]

But, I'm open to an environment variable to disable that if that is what people want.

[JCSHoosier]

I think that would definitely help. When I ran into this years ago it was such a pain due to how we had everything configured. @tinco 's suggestion above would help. This also should be documented as an expected behavior somewhere (with the flag to disable).

[cyclotron3k]

I would find this useful. At the moment, we are copying around dummy database config files in our CI/CD pipelines.

[ioquatix]

Maybe when DATABASE_ENV or RAILS_ENV is specified explicitly, it can be respected (a flag as it were). The default with no ENV specified, which implies development, can also include test.

How about that?

[jmks]

I came across this yesterday too.

@rafaelfranca I tried the environment variable approach you described: https://github.com/rails/rails/pull/39027

[simi]

@JCSHoosier Actually this behaviour is well documented.

bin/rake -D db:create
rake db:create
    Creates the database from DATABASE_URL or config/database.yml for the current RAILS_ENV (use db:create:all to create all databases in the config). Without RAILS_ENV or when RAILS_ENV is development, it defaults to creating the development and test databases.

Anyway creating only specified database should be somehow supported. Adding ENV variables is not systematic solution. If we should to make this configurable, I think it should be added as config option for active record and you can tweak this config using ENV variable on your own the same we do for example for static files:https://github.com/rails/rails/blob/7ee8d7ea9c371a420cb180cef5061049e379ae62/railties/lib/rails/generators/rails/app/templates/config/environments/production.rb.tt#L25

[simi]

Maybe when DATABASE_ENV or RAILS_ENV is specified explicitly, it can be respected (a flag as it were). The default with no ENV specified, which implies development, can also include test.

How about that?

:thinking: That will make environment having no RAILS_ENV set and RAILS_ENV set to development (which is default) different and that will be breaking and also inconsistent change.

[rafaelfranca]

I don't want this to be configurable. I'm not against people using this, but I also don't want this becoming mainstream feature.

There are clearly some use cases like creating test database in paid machine where creating those databases are expensive but this is not something everyone needs.

For those applications an environment variable is enough.

[deivid-rodriguez]

Would it be possible to get the best of both worlds?

Keep rails new foo; rails g scaffold user name; rails db:create db:migrate; rails test passing by default without having to rails db:test:create, but also respect RAILS_ENV if passed explicitly to rails db:create.

It might require some internal refactoring, but from an end user point of view, it should be possible, right?

[JohnDBR]

I came across this yesterday too.

@rafaelfranca I tried the environment variable approach you described: #39027

This will be available in the next rails release?