Onpremise: Error on migrations

[Apollon77]

Version

latest github

Steps to Reproduce

install.sh crashes at

Running migrations:
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/__init__.py", line 196, in main
    func(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/upgrade.py", line 68, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/upgrade.py", line 24, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.6/site-packages/sentry/new_migrations/monkey/executor.py", line 74, in apply_migration
    self._check_db_routing(migration)
  File "/usr/local/lib/python3.6/site-packages/sentry/new_migrations/monkey/executor.py", line 56, in _check_db_routing
    f"Migration `{migration.app_label} {migration.name}` contains "
sentry.new_migrations.monkey.executor.MissingDatabaseRoutingInfo: Migration `sentry 0191_make_externalactor_integration_id_not_null` contains operation(s) that miss `hints={'tables':..}` argument for correctly selecting database connection/alias. 
Operations:
<RunSQL '\n                    ALTER TABLE "sentry_externalactor" ALTER COLUMN "integration_id" SET DEFAULT 1;\n                    UPDATE "sentry_externalactor" SET "integration_id" = 1 where "integration_id" is NULL;\n                    ' reverse_sql='\n                    ALTER TABLE "sentry_externalactor" ALTER COLUMN "integration_id" DROP DEFAULT;\n                    '>
<RunSQL '\n                    ALTER TABLE "sentry_externalactor" ALTER COLUMN "integration_id" SET NOT NULL;\n                    ALTER TABLE "sentry_externalactor" ALTER COLUMN "integration_id" DROP DEFAULT;\n                    ' reverse_sql='\n                    ALTER TABLE "sentry_externalactor" ALTER COLUMN "integration_id" DROP NOT NULL;\n                    '>
An error occurred, caught SIGERR on line 12
Cleaning up...
root@sentry:~/sentry/onpremise# 

Expected Result

complete successfully

Actual Result

crashes see above

[Apollon77]

Ok, worked around by using "latest" image ... so issue is between "Latest" and nightly

[Apollon77]

Hm ... not sure if it really worked ... Seems like I ended up in a mixed state now ...

I used "COMPOSE_HTTP_TIMEOUT=240 SENTRY_IMAGE=getsentry/sentry:latest ./install.sh" to do the install in the end ( before I did "COMPOSE_HTTP_TIMEOUT=240 ./install.sh" which failed like shown above!

Now when I open the sentry page I see "Sentry 21.7.0.dev 0181fcf6" in the footer which points to the current master build?!

Interestingly in first try he said:

<E2><96><B6> Fetching and updating Docker images ...
nightly: Pulling from getsentry/sentry
Digest: sha256:6c33d64092a84325627602f9c6640e9dff5116d6c1a856a948336b5e8dfe1712
Status: Image is up to date for getsentry/sentry:nightly
docker.io/getsentry/sentry:nightly

and on second install with "latest" it doid not changed the image

<E2><96><B6> Fetching and updating Docker images ...
latest: Pulling from getsentry/sentry
Digest: sha256:2fd9160a1fa8be1208da10added416ba420dad25a20bf775cac079ee5e60c71d
Status: Image is up to date for getsentry/sentry:latest
docker.io/getsentry/sentry:latest

but on last try the migrations where all executed ...

And now I have an processing error because of a missing dfatabase column ...

https://sentry.iobroker.net/share/issue/fc1fe39f03884006a9ae3f05835966dc/

[Apollon77]

Ok,I now did install again with SENTRY_IMAGE=getsentry/sentry:21.6.1 and this time he fetched that new image ... The Webpage still shows "Sentry 21.7.0.dev 0181fcf6" in the footer :-(

[Apollon77]

And I get many of tehse processing errors now - in fact three different. The one from above and additionally
https://sentry.iobroker.net/share/issue/29b69f2275ac41d9b7e81371c66ec376/
https://sentry.iobroker.net/share/issue/c9aa3729dd1b482c9980f0dfb195496c/

[Apollon77]

Hm ...ok further ... I remembered that I forgot to do a "git pull" on premise repo, so I did and tried again ... now it can not connect to kafka anymorrw because zookeeper tells

===> Configuring ...


===> Running preflight checks ... 


===> Check if /var/lib/zookeeper/data is writable ...


===> Check if /var/lib/zookeeper/log is writable ...


===> Launching ... 


===> Launching zookeeper ... 


[2021-07-01 08:59:42,714] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)


[2021-07-01 08:59:42,730] ERROR Unable to access datadir, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)


org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory /var/lib/zookeeper/log/version-2


    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:132)


    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:125)


    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:107)


    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:65)


Unable to access datadir, exiting abnormally


    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)


    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)

[Apollon77]

Ok, I now went down to onpremise 21.6.1 tag and yes now build works again ... and NOW I'm also on 21.6.1 in browser ... holy moly ;-)

[Apollon77]

Ok, Exceptions gone ... I think I'm now fully on 21.6.1 ... I hope he details and "Chaos" in this issue is helpful for you ;-)

[BYK]

Alright, closing this ticket as a documentary, thanks for sharing the journey @Apollon77 😀

[Apollon77]

@BYK But I think something is not correct i the current "master", so you might want to check the issues on migration and pot in application (if not happened because of failed migration)

[Apollon77]

And also the zookeeper start failure with "master" version of "onpremise repo"

[BYK]

And also the zookeeper start failure with "master" version of "onpremise repo"

This, we are tracking here: https://github.com/getsentry/onpremise/issues/1009

@BYK But I think something is not correct i the current "master", so you might want to check the issues on migration and pot in application (if not happened because of failed migration)

https://develop.sentry.dev/self-hosted/#hard-stops

We are working on making this warning more prominent (on the console itself)

[Apollon77]

https://develop.sentry.dev/self-hosted/#hard-stops
We are working on making this warning more prominent (on the console itself)

I think this was not the reason ... I had a 20.x/maybe even an 21.x before, but all good I will see on next update ... I decided to not go to master/latest anymore, but stay on latest release tags from now on :-)

[BYK]

I think this was not the reason ... I had a 20.x/maybe even an 21.x before,

Note that it says you need to go to 21.6.0 or 21.6.1 first :)

[Apollon77]

Then the page is buggy :-))

If you are coming from 20.6.0, you can directly go to latest:
Copied

20.6.0 -> latest

[BYK]

Wow, indeed that's a bug, it should have said 21.6.0. Thanks a lot!

Nope, I got confused. This is how it was supposed to work. I'll ping the folks, still, thanks a lot 😁

[Apollon77]

PS: I think really best would be a "update checklist" ... so like

• 1.) get your version and check if you need to do a certain in between version, make backup
• 2.) check latest releas eversion and checkout onpremise tag ()
• 3.) Check config and example config for changes that you should do manually
• 3.) install.sh

... Maybe this could be beneficial 8and install.sh could ask if you checked the page with instructions) :-) Just as idea

[BYK]

@Apollon77 if you have a rough draft in your mind, you might be in a better position to add this to our dev docs. Would you be interested in that?

[Apollon77]

Let me put it on my list ... effectively I would wait till next "official release" and then build it which really updating my system :-)