Instapy: sqlite3.OperationalError: database is locked

[fgisslen]

@uluQulu any idea?
A new error:

Traceback (most recent call last):
File "/home/fredrik/InstaPy/instapy/util.py", line 524, in click_element
update_activity()
File "/home/fredrik/InstaPy/instapy/util.py", line 194, in update_activity
{"var":id})
sqlite3.OperationalError: database is locked

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/fredrik/InstaPy/5b86d4566c927700141ddf9e.py", line 23, in
session.like_by_tags(['xx','xx'], amount=90)
File "/home/fredrik/InstaPy/instapy/instapy.py", line 1505, in like_by_tags
self.logfolder)
File "/home/fredrik/InstaPy/instapy/like_util.py", line 570, in like_image
click_element(browser, like_elem[0])
File "/home/fredrik/InstaPy/instapy/util.py", line 532, in click_element
browser.execute_script("document.getElementsByClassName('" + element.get_attribute("class") + "')[0].scrollIntoView({ inline: 'center' });")
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py", line 143, in get_attribute
resp = self._execute(Command.GET_ELEMENT_ATTRIBUTE, {'name': name})
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: headless chrome=69.0.3497.100)
(Driver info: chromedriver=2.36 (7165af0ca9863529f28e493568ae3d105eddcdef),platform=Linux 4.15.0-34-generic x86_64)

[fgisslen]

a error in the same niche:

WARNING [2018-09-25 06:18:20] [xx] Heeh! Error occured while getting a DB profile for 'xx':
b'database is locked'
Traceback (most recent call last):
File "/home/fredrik/InstaPy/5b97e8915f0d2d0014004ccf.py", line 14, in
session = InstaPy(username=insta_username, password=insta_password, headless_browser=True, nogui=True, multi_logs=True, bypass_suspicious_attempt=True, proxy_address=proxy, proxy_port=port, page_delay=100)
File "/home/fredrik/InstaPy/instapy/instapy.py", line 212, in __init__
get_database(make=True)
File "/home/fredrik/InstaPy/instapy/database_engine.py", line 43, in get_database
id = get_profile(name, address, logger) if id is None or make else id
File "/home/fredrik/InstaPy/instapy/database_engine.py", line 129, in get_profile
profile = dict(profile)
UnboundLocalError: local variable 'profile' referenced before assignment

[uluQulu]

Hi @fgisslen

WARNING [2018-09-25 06:18:20] [xx] Heeh! Error occured while getting a DB profile for 'xx':
b'database is locked'

sqlite3.OperationalError: database is locked

It says that database is not operational [accessible] cos it is locked.

Either you have no read/write privilege to the instapy.db file OR you are running simultaneous sessions at the same time OF the same user AND there some session request a database access during the other uses it WHERE it will be remained locked for the rest.

In both cases, there are lots of solutions you can think of.

---

Cheers 😁

[fgisslen]

@uluQulu the problem is that I am not running more than one session of the same user.
I will change so that it is read/write for all users and try again

[sionking]

@uluQulu JFYTK

Either you have no read/write privilege to the instapy.db file OR you are running simultaneous sessions at the same time OF the same user AND there some session request a database access during the other uses it WHERE it will be remained locked for the rest.

mmm sqlite support simultaneous process read write access.

(5) Can multiple applications or multiple instances of the same application access a single database file at the same time?

Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.

SQLite uses reader/writer locks to control access to the database. (Under Win95/98/ME which lacks support for reader/writer locks, a probabilistic simulation is used instead.) But use caution: this locking mechanism might not work correctly if the database file is kept on an NFS filesystem. This is because fcntl() file locking is broken on many NFS implementations. You should avoid putting SQLite database files on NFS if multiple processes might try to access the file at the same time. On Windows, Microsoft's documentation says that locking may not work under FAT filesystems if you are not running the Share.exe daemon. People who have a lot of experience with Windows tell me that file locking of network files is very buggy and is not dependable. If what they say is true, sharing an SQLite database between two or more Windows machines might cause unexpected problems.

We are aware of no other embedded SQL database engine that supports as much concurrency as SQLite. SQLite allows multiple processes to have the database file open at once, and for multiple processes to read the database at once. When any process wants to write, it must lock the entire database file for the duration of its update. But that normally only takes a few milliseconds. Other processes just wait on the writer to finish then continue about their business. Other embedded SQL database engines typically only allow a single process to connect to the database at once.

However, client/server database engines (such as PostgreSQL, MySQL, or Oracle) usually support a higher level of concurrency and allow multiple processes to be writing to the same database at the same time. This is possible in a client/server database because there is always a single well-controlled server process available to coordinate access. If your application has a need for a lot of concurrency, then you should consider using a client/server database. But experience suggests that most applications need much less concurrency than their designers imagine.

When SQLite tries to access a file that is locked by another process, the default behavior is to return SQLITE_BUSY. You can adjust this behavior from C code using the sqlite3_busy_handler() or sqlite3_busy_timeout() API functions.

You need to catch BUSY state.

[fgisslen]

@sionking oh, is this something you can help with?

[sionking]

I don't use the sqlite .... sorry.

[fgisslen]

Okey, why are you not using it?

[uluQulu]

@sionking

mmm sqlite support simultaneous process read write access.

It does not support simultaneous process read [possible but not always] / write [never].
It's just you can just get to know when the database is busy and wait for it become unlocked.
As a result, my comment above is absolutely correct 😛

BTW, @sionking, I didn't know of there is a built-in capability for SQLite to return that information of busy-state. Thanks for the cap 🎉 Do you think it also is supported in python?
And I just learnt that MySQL supports simultaneous writes 😎 it's cool!

@fgisslen you have told you are using single session per user at a moment WHERE you must look for the problem elsewhere rather than simultaneous read/writes.

---

Cheers 😀

[jeremycjang]

https://www.sqlite.org/lockingv3.html

I experienced that error when running simulataneous sessions on different user accounts and messed with fixing it a lot, simplest workaround is to just install multiple instances of the program so they’re all writing to their own database file and you won’t have to worry about it

[fgisslen]

@jeremycjang problem is that I have like 200 accounts running on it....

[uluQulu]

Ouch!
I think I forgot one thing.
Cos of all of the users' data is stored at 1 database then in case of you are running more than one session (_it doesn't matter the same user or different users_) simultaneously WHICH are using the same database file will also make that data acquisition conflict I mentioned above.

I think you are having the problem in that case, @fgisslen.

My suggestion

Set a unique database name for every other username and they get their unique database file so that no conflict occurs.
Just make sure you always use the that same database file for those users, _so that_ the crucial data being used is acquired properly [record activity for _Quota Supervisor_ & follow restriction for _follow times_ usage mainly].

How to set a unique database per user?

Add this line to your quickstart script:
Settings.database_location = "./db/instapy_{}.db".format(insta_username)
_It must be located before instantiation of InstaPy _class__.
I have myself wrote the database engine and I can make you sure that it will make that database file wherever you point to.

Also, don't be scared of having lots of database files, actually, oppositely, having unique database files per user will gain in performance.
In future I will think of adding a new parameter to make that process easier- to have a unique database per user for the people who run simultaneous sessions at a time, etc.

Wow @jeremycjang it is scary 😄

---

Cheers 😁

[fgisslen]

@uluQulu is it possible to just set: Settings.database_location = "./db/instapy_{}.db".format(insta_username) or do I have to change anything more?

I get NameError: name 'Settings' is not defined
what do I have to import?

[uluQulu]

@fgisslen
from instapy import Settings

[fgisslen]

Even though I run with the new setting I only can see one instapy.db in Instapy/db . Is that correct @uluQulu

[uluQulu]

@fgisslen
It does have to open new specific database file for each unique user, such as: instapy_fgisslen.db, and from that time on it must use that specific database file as long as you keep that line in the quickstarts script.

If it behaves differently, then you have either not applied it correctly or your source code has extra modifications in that part or some other situation out of your usage [try to write more about your usage mechanism for that].

---

Cheers 😁

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. > If this problem still occurs, please open a new issue