peewee.OperationalError: Connection already opened. (peewee 3.3.4)

[salva-imm]

im running peewee with flask ,
everything was fine , till i deployed my app with "Gunicorn , 3 workers and gracefull timeout arg"

any idea about using multiple workers with peewee ?

[salva-imm]

i found problem
it was when i tried to "create" record , and because of "Not Availabe ForeignKey" , peewee raised exception ,, and left connection open , till restarting workers , it return error ,,

for solving this problem , we can use "Try , Except" , and "abort" request when this happen ,, or use gevent workers

[coleifer]

It sounds like you're opening an already-opened connection. You will want to add middleware to manage the connections: http://docs.peewee-orm.com/en/latest/peewee/database.html#flask

Peewee stores connection state in a thread-local so you can safely use peewee with multi-threaded (or gevent) web apps.

I'd suggest that you post the relevant code on stackoverflow if you have more questions.

[salva-imm]

@coleifer
i did exactly same thing , http://docs.peewee-orm.com/en/latest/peewee/database.html#flask
peewee exceptions cause workers crash ,and leave connection open,.

[coleifer]

Can you share the full traceback or some actual code?

[salva-imm]

ERROR in app: Exception on /course/form-save [POST]
Traceback (most recent call last):
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flasgger/marshmallow_apispec.py", line 79, in dispatch_request
    return super(SwaggerView, self).dispatch_request(*args, **kwargs)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/views.py", line 158, in dispatch_request
    return meth(*args, **kwargs)
  File "/home/user/Workspace_Py/my_app/app/course/api.py", line 131, in post
    Parse(data)
  File "/home/user/Workspace_Py/my_app/app/course/parser.py", line 10, in Parse
    Obj['course_price']=int(Obj['course_price'])
ValueError: invalid literal for int() with base 10: 'dsvvgfbgf'
[2018-07-01 13:00:10,859] ERROR in app: Exception on /course/form-save [POST]
Traceback (most recent call last):
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flasgger/marshmallow_apispec.py", line 79, in dispatch_request
    return super(SwaggerView, self).dispatch_request(*args, **kwargs)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/views.py", line 158, in dispatch_request
    return meth(*args, **kwargs)
  File "/home/user/Workspace_Py/my_app/app/course/api.py", line 131, in post
    Parse(data)
  File "/home/user/Workspace_Py/my_app/app/course/parser.py", line 10, in Parse
    Obj['course_price']=int(Obj['course_price'])
ValueError: invalid literal for int() with base 10: 'dsvvgfbgf'
[2018-07-01 13:00:17,098] ERROR in app: Exception on /auth/check [POST]
Traceback (most recent call last):
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 1811, in full_dispatch_request
    rv = self.preprocess_request()
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/flask/app.py", line 2087, in preprocess_request
    rv = func()
  File "/home/user/Workspace_Py/my_app/app/main.py", line 51, in MiddleWare
    db.connect()
  File "/home/user/Workspace_Py/my_app/lib/python3.5/site-packages/peewee.py", line 2512, in connect
    raise OperationalError('Connection already opened.')
peewee.OperationalError: Connection already opened.

this is hapening when i run my app with "gunicorn --workers 3 main:app"
and after crashing (On Peewee Create) , for some minutes it return "peewee.OperationalError: Connection already opened." for all requests ,

[coleifer]

Well, you have two tracebacks for this error in your code:

ValueError: invalid literal for int() with base 10: 'dsvvgfbgf'

The last one looks like the one you referenced in this ticket.

My best guess is that you're leaving some database connections open. Do you open a db connection as part of the app startup? i.e., to create tables or something? Do your views spawn other tasks that use the database and open a connection? You need to be sure to close your connections!

Note that you can also pass reuse_if_open=True to the db.connect() call, which will not report an error if you forgot to close a connection somewhere:

db.connect(reuse_if_open=True)

But this will only mask the problem.

[salva-imm]

thanks for reply ,
no , my "table creation" script is seperated from app and startup ,,
i really just added "connect and close" to "before and after request" ,

my main :

@app.before_request
def MiddleWare():
    db.connect()
        ..........
    pass

@app.after_request
def AfterReq(response):
    if not db.is_closed():
        db.close()
    return response

my db_init.py:

from peewee import *
db = PostgresqlDatabase('testdb', user='user', password='password',
                           host='localhost', port=5432)

class BaseModel(Model):
    class Meta:
        database = db

after these , im importing db and BaseModel to My Models : (I done think this make connection open):

from peewee import *
from db_init import db,BaseModel

class Tag(BaseModel):
    class Meta:
        table_name = 'tags_tag'
    tag = CharField(max_length=60,unique=True)
    en_tag = CharField(max_length=60,unique=True)

and using my model classes

[coleifer]

You want @app.teardown_request, not @app.after_request.

[Colaplusice]

I met the same problem. Because I run the code before app create:
db.database.create_tables([User, Note])
and I use from playhouse.flask_utils import FlaskDB as middleware.
when in debug mode. everything is ok.
but when i use gunicorn. it raised:
raise OperationalError('Connection already opened.')

[coleifer]

with db.database:
    db.database.create_tables([User, Note])