Grafana: Grafana Logs "database is locked"

[KZ-Lucas]

What happened:
At some point, Grafana gets an error.
Log out with the error "database is locked."
The following are some of Grafana's logs:
lvl=eror msg="failed to look up user based on cookie" logger=context error="database is locked"

Environment:

• Grafana version: grafana-6.1.4-1.x86_64
• Data source type & version: postgresql 10.3, graphite 9.x
• OS Grafana is installed on: Centos 6.9 amd64
• User OS & Browser: ANY
• Grafana plugins: piechart
• Others:

[torkelo]

Can you describe your setup more, options, number of users, alerts, dashboards, provisioned dashboards, alert evaluation frequency etc.

[seet61]

I have the same issue too.

t=2019-04-24T09:37:46+0300 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=2019-04-24T09:37:46+0300 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0
t=2019-04-24T09:37:55+0300 lvl=eror msg="Failed to get system stats" logger=metrics error="database is locked"
t=2019-04-24T09:37:56+0300 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=2019-04-24T09:37:56+0300 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0
t=2019-04-24T09:37:59+0300 lvl=dbug msg="auth token rotated" logger=auth affected=1 auth_token_id=46 userId=1
t=2019-04-24T09:37:59+0300 lvl=dbug msg="Updating last user_seen_at" logger=context userId=1 orgId=1 uname=admin user_id=1

Graphite, Grafana, Postgresql 9.6.

[seet61]

Another services working correctly with the same db.

[cuxcrider]

I am having a similar issue using influxdb. Grafana works well for a while then all of sudden crashes and logs out. Grafana v6.1.6 (commit: cf9cb45), Ubuntu 16.04.

t=2019-05-14T00:36:04-0600 lvl=eror msg="Failed to get system stats" logger=metrics error="database is locked"
t=2019-05-14T00:36:05-0600 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"

[aggieben]

Same issue here. Running on an EC2 instance: prometheus, grafana, and a couple of my own services. Browser open to grafana, and that's it. No other services hitting grafana. Everything is fine for a few minutes, then at some point grafana craps itself with "database is locked" errors, and then it doesn't work, _and_ it won't start back up.

[cuxcrider]

When it says "database" is locked, is it referring to the influxdb data source or to the grafana database which I believe contains information about the dashboard, settings, etc and is an SQLite database by default? I am running influxdb on a KVM VM and grafana on a separate KVM VM. It would be nice to know if I should be trying to tune influx or grafana.

[aggieben]

@cuxcrider This is a problem with grafana's default sqlite db.

[cuxcrider]

Any idea if switching to postgres or mysql solves the problem?

[aggieben]

No, and I don't really have any interest in that, because part of what I'm working on is zero-config (or near-zero-config) setup for dev environments. Having to configure a different DB as a backend is a non-starter for me.

[cuxcrider]

I hear ya there. Would much prefer it to just "work". I might give it a shot, I found some info here:
https://community.hortonworks.com/articles/33401/how-to-set-up-grafana-to-use-mysql-database-rather.html

[cuxcrider]

fyi I just set up grafana on a new VM and setup mysql using the guide I pasted above. I then exported my dashboards as JSON and inserted them into my new grafana and so far, so good. No crashing. You can maybe convert your SQLite to mysql if you have a ton of stuff already setup, but my grafana was pretty minimal so I just exported my dashboards as json and then manually reconnected to data sources.

[truealex81]

I have similar issue when garafana suddenly stop working, and restarting doesn't fix this problem :(

t=2019-05-17T05:40:30+0000 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=2019-05-17T05:40:40+0000 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=2019-05-17T05:40:50+0000 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=2019-05-17T05:40:59+0000 lvl=eror msg="failed to run garbage collect" logger=remotecache.database error="database is locked"
t=2019-05-17T05:40:59+0000 lvl=eror msg="Failed to get system stats" logger=metrics error="database is locked"
t=2019-05-17T05:41:00+0000 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=2019-05-17T05:41:10+0000 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=2019-05-17T05:41:20+0000 lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"
t=20

[gsrcobr]

Same problem here.
Grafana 6.1.6
CentOS 6.9

t=2019-05-17T11:41:00-0300 lvl=eror msg="failed to look up user based on cookie" logger=context error="database is locked"

Starts to happen when upgraded to Grafana 6.

[psby]

I've got same issue.
Grafana send me 2 alerts tonight with Error message "Could not find datasource database is locked".

Grafana 6.1.6 @ Debian 4.9.168 in docker (image grafana/grafana:6.1.6)
InfluxDB

[kremers]

Same issue here v.6.1.6

[DanCech]

This seems to be an issue with sqlite itself, one way to recover the sqlite databse is to follow the procedure here: https://community.grafana.com/t/database-is-locked-unable-to-use-grafana-anymore/16557/2

The “database is locked” error indicates a problem with your sqlite database. This can happen if the database is left in an inconsistent state after a crash, or if there are problems with the disk.

One thing you can try is to dump the data from your existing db file into a new one, then swap it out. From inside your Grafana data directory (after shutting down Grafana):

sqlite3 grafana.db '.clone grafana-new.db'
mv grafana.db grafana-old.db
mv grafana-new.db grafana.db

[roidelapluie]

We have that a lot sinci grafana 6.2 (we use auth proxy)

[marefr]

@roidelapluie see #17247 for auth proxy specific problems with database is locked

[roidelapluie]

Ok we will upgrade to master ;) sorry for the noise, my github skills failed me this time!

[aggieben]

@DanCech isn't there anything that could be done in Grafana to avoid or minimize this issue? It seems pretty unusable to me from this vantage point.

[magicrobotmonkey]

6.2.1 (9e40b07) with sqlite backend, can confirm that cloning the db does not appear to help. I'm able to login and save some work, but intermittently get logged out with
t=2019-06-04T15:42:33+0000 lvl=eror msg="failed to look up user based on cookie" logger=context error="database is locked".

I also see that my health monitor happened to scrape at the right timing and hit a 503:
curl: (22) The requested URL returned error: 503 Service Unavailable

fwiw, i seem to somewhat reliably be able to reproduce it by doing something that generates a lot of requests at once, like moving a bunch of dashboards between folders.

[juanvmedrano]

This is a problem that affects many ...

Grafana after these errors is closing the session alone.

This is a very bad mistake because I can not put the monitors and have to start the session every hour and a half.

[cuxcrider]

I agree, sqlite was simply not a usable solution. Grafana would log me out
after only a few minutes of use. It would remain running, but I would have
to log in again.

for what it is worth I switched to mysql for grafana over two weeks ago and
have not had a single error even with multiple users and multiple
datasources.

On Tue, Jun 4, 2019 at 10:25 AM juanvmedrano [e-mail]
wrote:

This is a problem that affects many ...

Grafana after these errors is closing the session alone.

This is a very bad mistake because I can not put the monitors and have to
start the session every hour and a half.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/grafana/grafana/issues/16638?email_source=notifications&email_token=AB7553DCDWT2JP3PVEOJX3LPY2JQ5A5CNFSM4HGQUIUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW5DVLQ#issuecomment-498743982,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB7553GPXYO2I7RLUVWBGILPY2JQ5ANCNFSM4HGQUIUA
.

[davewat]

The Grafana default for SQLite journaling is the Rollback Journal (DELETE mode by PRAGMA), which originates from go-sqlite3. In addition, go-sqlite3 forces it back to DELETE mode even if you manually set to WAL (take DB offline and change PRAGMA journal_mode=wal;)

Switching to WAL (Write Ahead Log) mode reduces many opportunities for lock contention on the DB, and would likely provide a performance increase(https://www.sqlite.org/wal.html). We are doing this with a fork to relieve lock errors that appeared with heavy Grafana Alert queries. Might be worth looking into moving to WAL mode.

[DanCech]

@davewat You can pass the connection string directly, so something like this should work:

[database]
type=sqlite3
connection_string=file:data/grafana.db?cache=private&mode=rwc&_journal_mode=WAL

[davewat]

@davewat You can pass the connection string directly, so something like this should work:

[database]
type=sqlite3
connection_string=file:data/grafana.db?cache=private&mode=rwc&_journal_mode=WAL

@DanCech That is excellent - thanks! That is much better than running a custom build just for WAL mode! I noticed you also used private cache in your example, but source shows shared cache (we also are running private cache). I think private would be a more stable setting. Unsure if the shared cache choice was to better support tests, but would seem unnecessary in production with only Grafana accessing the DB.

[DanCech]

In 6.1.6 it was made configurable and the default changed to private, but if you use the connection_string setting that is just passed directly to xorm and through to go-sqlite3 and anything else in the [database] block (besides type) is pretty much ignored.

[magicrobotmonkey]

yea switching to WAL journal made a big difference for me. Not only is it not logging me out, its much, much faster.

[juanvmedrano]

@davewat You can pass the connection string directly, so something like this should work:

[database]
type=sqlite3
connection_string=file:data/grafana.db?cache=private&mode=rwc&_journal_mode=WAL

Where can I apply this configuration?

I want to try this.

[DanCech]

@juanvmedrano you'd include it in your grafana configuration file

[EdisonRobertoJr]

@davewat You can pass the connection string directly, so something like this should work:

[database]
type=sqlite3
connection_string=file:data/grafana.db?cache=private&mode=rwc&_journal_mode=WAL

Nice, worked to me, but you have to change the "connection_string=file:data/grafana.db" to your grafana.db path.

[Georege]

Same issue here v.6.1.6 on nfs pv

[zegerius]

Same issue here on: version=6.2.5 commit=6082d19 branch=HEAD compiled=2019-06-25T17:56:19+0000

[sipicsg]

Same with docker image grafana/grafana:6.2.5 .

[brunetton]

Same here (sorry for the noise)
Grafana v6.1.1 (eff01d2)
I'm totaly new to Grafana, just added a new empty dashboard + one panel. This errors stops me from using grafana.

I tried Switching to WAL (Write Ahead Log) as suggested by @davewat and so far so good :) (using it for 2 hours now, created dashboards, panels, no problem !)

[Quicksy]

Same here, with grafana in kubernetes with 
image: 'grafana/grafana:6.2.2'

[AvinKumar]

Facing the same "database is locked" issue on my Grafana v6.4.2 setup installed in SUSE 12SP3.
Noticed this issue only after I upgraded the Grafana version from v6.0.x to v6.4.2

-Avin

[mbolek]

grafana 6.4.3 on amazon ami 2
fresh install, using default sqlite
t=2019-10-29T09:26:26+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=XXXX time_ms=0 size=29 referer= t=2019-10-29T09:26:38+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=XXXX time_ms=0 size=29 referer= t=2019-10-29T09:26:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2019-10-29T09:26:56+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=XXXX time_ms=0 size=29 referer= t=2019-10-29T09:27:08+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=XXXX time_ms=0 size=29 referer= t=2019-10-29T09:27:17+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=XXXX 1 time_ms=0 size=29 referer= t=2019-10-29T09:27:23+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2019-10-29T09:27:26+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=XXXX time_ms=0 size=29 referer= t=2019-10-29T09:27:38+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=XXXX time_ms=0 size=29 referer= t=2019-10-29T09:27:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2019-10-29T09:27:56+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=XXXX time_ms=0 size=29 referer= t=2019-10-29T09:28:03+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2019-10-29T09:28:03+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2019-10-29T09:28:08+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=XXXX time_ms=0 size=29 referer= t=2019-10-29T09:28:10+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=XXXX time_ms=0 size=29 referer=
It's rather annoying, since it'll often fail on listing all dashboards, displaying only a portion of the list, on the other occasion it'll display the list but then will fail on loading a dashboard with 404, upon page reload it's fine.
I have used the tip from https://github.com/grafana/grafana/issues/16638#issuecomment-498818278
but doesn't seem to change much in my case.
As for the setup:
~30 dashboards
~ 5 users (not simultaneous)
no alerts
I'll try with postgres but it has already been mentioned to also have issues

[marefr]

@mbolek please try to use WAL as suggested in https://github.com/grafana/grafana/issues/16638#issuecomment-537939123

[mbolek]

@marefr hey, I did and still had issues. Was able to work around this problem by removing the DB (so no migration etc) and start fresh

[tgranqvist]

I'm seeing this same error when deploying to Azure Container Instances and mounting a volume from Azure File Share (see issue linked by marefr). I've tried the solution from @DanCech [here]. This seem to have worked for several people in this thread. Unfortunately not working for me. I see in the container log that the default settings are overridden and the database file is created with the name from my ini file, but still get that same error: _"Service init failed: Migration failed err: database is locked"_.

My ini file in its entirety:

[database]
type=sqlite3
connection_string=file:/var/lib/grafana/mygrafana.db?_journal_mode=WAL

Any pointers on what to do? Azure Files volumes are SMB3, does that have anything to do with the issue?

My yaml deployment file: deploy-gvt-11.yaml.txt
Deployment log file: aci-grafana.log

Any help is much appreciated!

[Iziman127]

Hi everyone,

I have an issue where my dashboard provisioning is not working properly.
It load some of my .json files and doesn't load some other, which randomly cause a "Dashboard not found" error message. I have around 25 .json files.

In my investigation of this issue I have noticed that the grafana.log is full of "Database locked" log message. This message occur around 10 times every 10 seconds.
I am also noticing that in the grafana.db the dashboard_provisioning table is always updated but randomly missing some .json files.
I think the database lock is related to this issue. Am I wrong ? Is there a way to figure what cause the db lock ?

Environment:

Grafana version: Grafana v6.5.1
Grafana DB : sqlite
OS Grafana is installed on: RHEL 7.4
Users : less than 10

Any help is much appreciated

[marefr]

@Iziman95 could be related to using same uid in multiple dashboards?

[Iziman127]

@Iziman95 could be related to using same uid in multiple dashboards?

thanks but I have already checked this when I was searching for an answer here, all my uids are unique

[Iziman127]

@marefr The grafana documentation describes how provisioning is supposed to work :
"Dashboards will be reloaded when the json files changes".
But it is like they are permanently reloaded every 5-10 seconds while there is no change on these files. I can confirm this by watching the table "dashboard_provisioning" and seeing that the "id" increases indefinitely each time I perform a select (current value is 2005572) for the same rows.
Is there a way to take .json files only at the start of grafana server and not after ?

[marefr]

@Iziman95 you must have configured something wrong. Does any of your dashboard have an id with a numeric value? Are you by any chance multiple paths configured where one of them is a sub directory of the other? If that doesn't help I would suggest open a new issue with provided provisioning config and dashboards so that we can look at it separately from this issue..

[Iziman127]

@marefr ok so I have done many tests with fresh dashboards in different paths and checked all my configuration and I have found that the root cause was the property providers name which was the same in all of my yaml file.
After editing my yaml files to have an unique name my issue is solved and I have no more "database locked" in my logs. My database size is also huge now because the table dashboard_version was permanently updated with my +20 dashboard every 10 seconds.

Thank you for your help

[pirminjanka]

i had the same problem with grafana 6.5.1 on a container orchestarted by kubernetes. I did the following inside the container:

mv /var/lib/grafana/grafana.db /var/lib/grafana/grafana.db.old
rm /var/lib/grafana/grafana.db
mv /var/lib/grafana/grafana.db.old /var/lib/grafana/grafana.db

after this steps it worked fine.
Maybe my issue was because the grafana container switched to a different kubernetes node and therefore the database was locked

[dy-lan]

If you're trying to get Grafana deployed via an Azure App Service container, this might be of help.

This issue has also been brought up to Microsoft in https://github.com/MicrosoftDocs/azure-docs/issues/47130.

[lsoica]

I'm getting the same error logs during IO heavy workloads (I can reproduce it by running a dd on the same disk as grafana's). My disk is a remote disk (Ceph RBD) and during a dd I get 100% util and a very high latency (await > 10s).

lvl=eror msg="Failed to get system stats" logger=metrics error="database is locked" lvl=eror msg="failed to search for dashboards" logger=provisioning.dashboard type=file name=default error="database is locked" lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"

[lsoica]

I'm getting the same error logs during IO heavy workloads (I can reproduce it by running a dd on the same disk as grafana's). My disk is a remote disk (Ceph RBD) and during a dd I get 100% util and a very high latency (await > 10s).

lvl=eror msg="Failed to get system stats" logger=metrics error="database is locked" lvl=eror msg="failed to search for dashboards" logger=provisioning.dashboard type=file name=default error="database is locked" lvl=eror msg="Could not load alerts" logger=alerting.ruleReader error="database is locked"

_journal_mode=WAL didn't work for me. I had to add _busy_timeout in the connection string aswell.

[kuldeeps88]

Environment:
Grafana version: Grafana v6.3.5
Grafana DB : sqlite

With Grafana as kubernetes deployment, Without adding any data source or dashboards, Grafana Docker container itself is throwing msg="Database locked, sleeping then retrying" logger=sqlstore error=" database is locked" retry=0

This issue we observed after upgrading from 4.6.3 to 6.3.5.

**Note: Grafana Docker container itself is throwing error without any data source or dashboard being added.

Also, Grafana is deployed as a Kubernetes deployment so it's getting scheduled on numerous worker nodes with the same error.

t=2020-03-09T19:03:42+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=0.0.0.0:3000 protocol=http subUrl=/grafana socket= t=2020-03-09T19:03:42+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:42+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=1 t=2020-03-09T19:03:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=1 t=2020-03-09T19:03:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:43+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=1 t=2020-03-09T19:03:44+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:44+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:44+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:44+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:44+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:44+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:44+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 t=2020-03-09T19:03:44+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0

[f-ld]

Got that issue on my kubernetes deployment with grafana 6.0.1. It was almost systematic, with some alerts not loading (well appearing in dashboard but not in list of alerts and remaining as not active in dashboard). Mote: we are using provisioned dashboards, alerts, etc ...

Changed to latest version, grafana 6.7.1, and it look way better, issue not reproduced so far. I will update this comment if I see it again in the next future. But I would recommend all having this issue to update grafana

[andloh]

Same issue on the latest grafana docker image. Using Azure files with mountoption uid/gid 65534
msg="Server shutdown" logger=server reason="Service init failed: Migration failed err: database is locked"

Tried connection_string=file:/var/lib/grafana/grafana.db?cache=private&mode=rwc&_journal_mode=WAL, not working.

[beingyash]

@torkelo Any update?

[f-ld]

Almost 1 month and a half later after https://github.com/grafana/grafana/issues/16638#issuecomment-606605113, with grafana 6.7.1 on k8s cluster created with kops on AWS, still not reproduced even once.

[beingyash]

I have v6.7.3 and am still facing this issue.

[sukhan85]

is this issue resolved? We are facing the similar issue.

[KursLabIgor]

same issue EC2 + EFS mounted.

[sorriso93]

same problem here

[KursLabIgor]

i solved it temporary moved data folder to EBS storage, and now planning move to MySQL DB ad store it in EFS

[sorriso93]

i solved it temporary moved data folder to EBS storage, and now planning move to MySQL DB ad store it in EFS

what is EBS storage? I have the sqllite db of grafana on external hard disk with the raspberry, disabled login page and enabled anonymous user. no result, same problem!

[pimjansen]

Same here on Azurefile (7.0.1)

[cwrau]

It's happening to us as well, but we're not using persistent storage, just an emptyDir

On AKS

[ilietrasca]

I encounter the same problem with version 7.0.3.

t=2020-06-19T19:55:33+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0
t=2020-06-19T22:24:15+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0
t=2020-06-20T01:23:06+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0
t=2020-06-20T04:12:02+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0
t=2020-06-20T04:12:02+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=1
t=2020-06-20T07:42:16+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0

The error received in the browser:
A cookie associated with a resource at http://monitoring.sre.idemia.io/ was set withSameSite=Nonebut withoutSecure. A future release of Chrome will only deliver cookies markedSameSite=Noneif they are also markedSecure. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5633521622188032.

Can you please provide an workaround for this issue ?

[DSalvigni]

Hello Guys,
i am running the 7.0.3 as docker container directly from docker HUB in a Azure App Service but since some days it stopped to work. I have the database saved in Azure File Share but any time I tired to logon i receive the error of "Database Locked". More in details:

2020-06-24T11:20:41.558497227Z t=2020-06-24T11:20:41+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
2020-06-24T11:20:41.559484232Z t=2020-06-24T11:20:41+0000 lvl=info msg="Starting DB migration" logger=migrator
2020-06-24T11:20:46.689980794Z t=2020-06-24T11:20:46+0000 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: Migration failed err: database is locked"

Any workaround??

[tynor88]

@DSalvigni we are also experiencing the same issues the past couple of days... No workaround yet.

[DSalvigni]

So I fixed in this way, but before start please backup locally the sqlite3 database (grafana.DB).

1) GF_DATABASE_TYPE=postgresql (or you can do the same in MySQL)
2) Created a DB called grafana in a postgresql Server
3) Set up the following paramenters for the container (in App Service):
GF_DATABASE_HOST=
GF_DATABASE_NAME=
GF_DATABASE_PASSWORD=
GF_DATABASE_TYPE=
GF_DATABASE_USER=
3) Moved the mounted persistent storage "/var/lib/grafana" under Azure File Storage (and I got rid of the Blob COntainer).
5) Deployed from Docker HUB the new version of Grafana

Now you have a new deployed version of grafana clena up an running. You need to port the data from the old DB to the new one and for this I simply dumped the content of the tables of grafana,DB into the new created grafana DB in postgresql.

To access grafana.DB i used "SQLiteDatabaseBrowserPortable" and to move the data this "https://www.dbsofts.com/". The trial version limits the dumb to 10.000 lines but more than enough to move the following table:

alert*
org*
dashboard* (not necessary if you have dumped already your dashboard as JSON)
data_source (the most important)

Let me know if you need some more questions.

[kaisert]

Experiencing the same issue. Grafana randomly logs out and only shows its error page. In the logs it says

2020-07-07T02:04:03.442792383Z t=2020-07-07T02:04:03+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=8166

Restarting the container brings back Grafana for a while.

Using Grafana 7.0.5 docker image deployed as an app service on azure. /var/lib/grafana is connected to a file share.
GF_DATABASE_URL is set to sqlite3:///var/lib/grafana/grafana.db?cache=private&mode=rwc&_journal_mode=WAL

[DSalvigni]

GF_DATABASE_URL is set to sqlite3:///var/lib/grafana/grafana.db?cache=private&mode=rwc&_journal_mode=WAL

This did not worked properly for me: DB remain always locked, so I moved directly to an external DB and the problem is solved, plus finally i can connect BI to link SLA to Events backlog 👍

[urab0t]

"SQLite uses reader/writer locks to control access to the database. (Under Win95/98/ME which lacks support for reader/writer locks, a probabilistic simulation is used instead.) But use caution: this locking mechanism might not work correctly if the database file is kept on an NFS filesystem. This is because fcntl() file locking is broken on many NFS implementations. You should avoid putting SQLite database files on NFS if multiple processes might try to access the file at the same time. On Windows, Microsoft's documentation says that locking may not work under FAT filesystems if you are not running the Share.exe daemon. People who have a lot of experience with Windows tell me that file locking of network files is very buggy and is not dependable. If what they say is true, sharing an SQLite database between two or more Windows machines might cause unexpected problems." - this information from https://www.sqlite.org/faq.html

[sacrelege]

Displaying a dashboard with 10s refresh (graphite datasource). Logouts happen random, between 10min and 2-3 hours.

Fresh installation from "https://packages.grafana.com/oss/deb stable main" of Grafana v7.3.0-test (98b94d3824) on a fresh Ubuntu 20.04.1 LTS. Grafana is pretty much on default settings (sqlite).

hardware: 12core 3900x, 64GB ram, HP EX950 nvme (2.9GB/s read/write speed).

log entries: msg="Failed to look up user based on cookie" logger=context error="user token not found"

update: maybe the following error in the log is more related to the logout:

lvl=eror msg="Data proxy error" logger=data-proxy-log userId=2 orgId=1 uname=xxxxx path=/api/datasources/proxy/1/render remote_addr=xxx.xxx.xxx.xxx referer="http://xxx.xxx.xxx.xxx:3000/d/-G-7XccMz/xxxxxx?orgId=1&refresh=10s&from=now-12h&to=now" error="http: proxy error: context canceled"

Update: increasing token rotation (default 10min) to 10080 minutes (1 week) seems to be a working workaround.
--- grafana.ini ---
;token_rotation_interval_minutes = 10
token_rotation_interval_minutes = 10080
--- eof ---

[pimjansen]

In Azure i finally got it working by running this command on the database:

sqlite3 grafana.db 'pragma journal_mode=wal;'

After that i overwritten the database using the Azure Storage explorer and all is running fine from there

[lobocobra]

same issue here with grafana and influxdb

[KursLabIgor]

i solved it temporary moved data folder to EBS storage, and now planning move to MySQL DB ad store it in EFS

what is EBS storage? I have the sqllite db of grafana on external hard disk with the raspberry, disabled login page and enabled anonymous user. no result, same problem!

This is hard disk storage for VM (EC2) in AWS it is called like EBS

[petersohn]

There seems to be multiple issues here, actually. One is the database locking issue that Grafana sometimes fails to access its user database. There is an error handling issue too. If this error happens, the server returns with a 401 status code, for which the client does a logout. However, this should not be a 401 status code, but instead of an 500 or something similar, because the problem is not with the client's authentication, but with the server database. Then the client would occasionally fail a query, but it would not log out.

[cameleon77]

+1

same issue here, just upgraded to Grafana 7.3.4 from 5.x.x and started getting:
t=2020-12-05T14:00:00+0100 lvl=eror msg="Failed to look up user based on cookie" logger=context error="user token not found"
some time after logged in

Grafana default settings with sqlite3
installed on some old Ubuntu 16.04