Fastapi: [QUESTION] Customizing UploadFile

[haizaar]

Description
I found issue with Starlette's UploadFile where it actually always buffers file to RAM fully and never switches to disk.

To overcome this until new version lands, I want to use my own version of UploadFile. This is what I tried:

import logging
from fastapi import FastAPI, File, UploadFile

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
app = FastAPI()

class FixedUploadFile(UploadFile):
    def __init__(self, *args, **kwargs):
        logger.info("FixedUploadFile is initializing...")
        super().__init__(self, *args, **kwargs)
        self.file.rollover()

@app.post("/uploadfile/")
async def create_upload_file(file: FixedUploadFile = File(...)):
    logger.info("Upload file handler")
    return {"filename": file.filename}

Running this I see that "FixedUploadFile is initializing..." is not called. How can I supply a custom class for UploadFile? (Either FastAPI's or Starlette's)

[euri10]

Thinking out loud, shouldn't you implement __call__?

Le mar. 16 juil. 2019 à 9:14 AM, Zaar Hai [e-mail] a
écrit :

Description
I wound an issue https://github.com/encode/starlette/issues/579 with
Starlette's UploadFile where it actually always buffers file to RAM fully
and never switches to disk.

To overcome this until new version lands, I want to use my own version of
UploadFile. This is what I tried:

import loggingfrom fastapi import FastAPI, File, UploadFile

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
app = FastAPI()
class FixedUploadFile(UploadFile):
def __init__(self, args, *kwargs):
logger.info("FixedUploadFile is initializing...")
super().__init__(self, args, *kwargs)
self.file.rollover()
@app.post("/uploadfile/")async def create_upload_file(file: FixedUploadFile = File(...)):
logger.info("Upload file handler")
return {"filename": file.filename}

Running this I see that "FixedUploadFile is initializing..." is not
called. How can I supply a custom class for UploadFile? (Either FastAPI's
or Starlette's)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tiangolo/fastapi/issues/390?email_source=notifications&email_token=AAINSPUV436GBYKFUMBODNTP7VYL3A5CNFSM4ID5ZRXKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G7MRVYA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAINSPVM5L3TFGJ4EDF2MJLP7VYL3ANCNFSM4ID5ZRXA
.

[haizaar]

FastAPI.UploadFile inherits from Starlette.UploadFile and neither implements __call__() :)
And anyhow, if Starlette.UploadFile object is being used, then __init__ should run eventually...

[euri10]

you can hack it in the @classmethod, feels gross

from starlette.datastructures import UploadFile
class FixedUploadFile(UploadFile):

    @classmethod
    def __get_validators__(cls: Type["UploadFile"]) -> Iterable[Callable]:
        yield cls.validate

    @classmethod
    def validate(cls: Type["UploadFile"], v: Any) -> Any:
        if not isinstance(v, UploadFile):
            raise ValueError(f"Expected UploadFile, received: {type(v)}")
        v.file.rollover()
        return v

@app.post("/uploadfile/")
async def create_upload_file(file: FixedUploadFile = File(...)):
    logger.info("Upload file handler")
    return {"filename": file.filename}

[haizaar]

Yeah... gross... Thanks though. Here is another one. We can start code-ugly competition :)

from starlette.datastructures import UploadFile as StarletteUploadFile

def fix_starlette_upload_file():
    if hasattr(StarletteUploadFile, "_fixed"):
        return  # Occational double-call will chain initialization

    _init = StarletteUploadFile.__init__

    def new_init(self, *args, **kwargs):
        _init(self, *args, **kwargs)
        logger.info("Forcing rollover")
        self.file.rollover()

    StarletteUploadFile.__init__ = new_init
    StarletteUploadFile._fixed = True


fix_starlette_upload_file()

[euri10]

Yeah... gross... Thanks though. Here is another one. We can start code-ugly competition :)

:rofl:

I'd be interested nonetheless to get insights on why effectively the subclassing doesn't work, at first sight there's no reason, appart from the fact maybe that file: FixedUploadFile = File(...) does not instantiate the FixedUploadFile class but is just there for typing reasons ?
There might be a way, and I think that might serve others as subclassing some objects FastAPI or Starlette can sometimes be interesting, or there is another pattern that's more suitable ?

[haizaar]

Hehe, it's very tricky indeed.

First, starlette request's lands to FastAPI zone. It may contain a form with an uploaded file, but we don't know that yet :)

Then initial check is performed and body is cast to starlette.FormData.

Then app calls solve_devendencies to convert incoming request into values suitable for calling the handler.

The real magic happens here where received_body is actually starlette.FormData so calling .get(<form_field>), where is our argument name as defined in the handler, will actually return starlette.UploadFile as per starlette docs.

The type hint at this stage is only used to validate that extracted value is indeed of type starletted.UploadFile.

I'm still far away to say I understand FastAPI well, but couple of things are surfacing from this research:

• @euri10, your ugly hack will not work - by the moment validation code is reached, the form is already uploaded to server, meaning the file is already fully in RAM :)
• @tiangolo, why fastapi.UploadFile inherits from starlette.datastructures.UploadFile? I see it's being used only for validation, unless... it's only to satify mypy? (which is truly valid concern of course)

[euri10]

my bad effectively, I just checked the private _rolled argument without thinking too much.

let's wait for the master's explanations :)

[tiangolo]

Thanks for the discussion here!

@haizaar the instance of UploadFile is actually created in Starlette. The one from FastAPI is only to make it compatible with Pydantic, to be able to use it as an annotation in path operations. But the actual instance is the one from Starlette.

If the issue is still happening, it might be a bug in Starlette, or at least a feature request to support some extra config. Then the best thing to do would be to create a simple self-contained app that shows a way to reproduce it and post it on the issues in Starlette.

[github-actions[bot]]

Assuming the original issue was solved, it will be automatically closed now. But feel free to add more comments or create new issues.

[haizaar]

@tiangolo this ticket was about _customizing_ UploadFile in FastAPI, but the use case was a bug in then current UploadFile implementation. The bug was fixed at https://github.com/encode/starlette/issues/579 long time ago, but this ticket was left open since it was about _customizing_ UploadFile which is still not possible without monkey patching.

I think it's indeed resolved for now, because the only reason to customize was to work around a Starlette bug, hence the original use case became irrelevant and with it the whole reason to customize :)

[tiangolo]

Ah, great, thanks for reporting back @haizaar !