Fastapi: startup and repeat_every doesn't work, @app.get doesn't send real time data

[neothegod75]

I am using fastapi as a backend for a dashboard that shows technical details of stock in realtime. I use a third party (broker) API for to get the data and add technical analysis using pandas. I have tried passing the final result of my analysis as a list/ updating sqlite table and then connecting to db to show realtime data. But it doesn't work. Even if try browser refresh, it just refreshes old data. I have tried async and without async, neither of them work. It makes me wonder, does fastapi support realtime data?
I have searched everywhere but didn't get any help. @tiangolo it will be of great help if you can guide me in the right direction.

`@app.on_event("startup")
@repeat_every(seconds=60, logger=logger, wait_first=False)
def startup():

candle_df = dashboard_data()

connection = sqlite3.connect('stocknoteapi.db')  

candle_df.to_sql('livedata', connection, if_exists='replace', index = False)

connection.commit()
connection.close()

@app.get("/watchlist", response_model=Symbols, response_model_exclude_unset=False)
def watchlist(request:Request):

connection = sqlite3.connect('stocknoteapi.db')  
connection.row_factory = sqlite3.Row
cursor = connection.cursor()

cursor.execute("""
    Select * FROM livedata
""")

rows = cursor.fetchall()

connection.commit()
connection.close()

return templates.TemplateResponse("watchlist.html", {"request": request, "stocks": rows})`

[Mause]

Firstly, repeat_every isn't part of fastapi, it's part of fastapi_utils, but that project is abandoned, so you may not get support there.

Have you ensured that your task is actually running successfully? Either with a log statement at the end, or putting a breakpoint in it?

[neothegod75]

@Mause Thanks for your reply. The task runs only once but doesn't repeat. Unless there is a way to reload uvicorn, data doesn't refresh. Any ideas on how uvicorn can be forced to reload at a specific interval?

[ycd]

You can use a simple job queue library like RQ if you think that would be overkill, there are tools that can do this in-memory.

[neothegod75]

@ycd Can you please suggest a simple in-memory implementation to refresh the db and pull latest data from db. Thanks in advance!

[ycd]

You can check out the https://github.com/dbader/schedule (I have no experience with this but there are a lot of tools like this)

[neothegod75]

@ycd schedule won't work as long as unicorn is running. unless the server reloads, realtime data is not refreshed. Is there a way to run unicorn as a separate process and update the db through a separate process. I am in windows environment so running unicorn through guvicorn doesn't work.

[Mause]

Are you certain your task isn't blocking, and preventing a second or later run? Again, I would recommend that you debug the code as you have it before trying alternate scheduler implementations

[ycd]

@neothegod75 then use the schedule to run and stop uvicorn programmatically,

import schedule
import time
import os 

def restart_uvicorn():
    os.system("uvicorn my_app:app --port 8000")
    os.system("sudo lsof -t -i tcp:8000 | xargs kill -9")

schedule.every(1).minutes.do(restart_uvicorn)

while True:
    schedule.run_pending()

[neothegod75]

Thanks @ycd . I was able to use asyncio repeat task functionality to get around this roadblock. I had to add async task in startup event and make sure to use async for all methods and use await to retrieve data. The below 2 posts guided me to the solution.

https://github.com/tiangolo/fastapi/issues/543

https://stackoverflow.com/questions/54153332/schedule-asyncio-task-to-execute-every-x-seconds

I wished there was better documentation for asyncio usage with fastapi. If you are building an API, there should be a way to update and serve the data in realtime. @tiangolo please consider updating documentation for real-time data update and serving through uvicorn.

[Mause]

I think you'll find that that is exactly what repeat_every is doing under the hood for you, actually.

Regardless, if your problem is now solved, please close this issue.