Kfserving: Makes KFServing supports batch inference

[axelborja]

/kind feature

Describe the solution you'd like

Makes KFServing supports batch inference.

It could take the following form:

BatchPredict POST /v1/models/<model_name>:batch_predict
Request: {instances_path, predictions_path}
Response: {}

[ellistarn]

Can you describe your usecase? This is currently identical to predict. I
expect batch predict will not be http. I'd love to hear any requirements
you have.

EDIT:

Re-reading your request payload, it's obviously different with the "paths" -- sorry! Either way, thank you for providing a detailed use case!

On Thu, Sep 26, 2019, 2:41 PM Axel Borja [e-mail] wrote:

/kind feature

Describe the solution you'd like

Makes KFServing supports batch inference.

It could take the following form:

BatchPredict POST /v1/models/:batch_predict
Request: {instances_path, predictions_path}
Response: {}

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kubeflow/kfserving/issues/369?email_source=notifications&email_token=AATXLCFE5ONRACNVBIADSDDQLUUCJA5CNFSM4I27MH22YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HOAD65Q,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AATXLCFPQGE6G434IVUQN3LQLUUCJANCNFSM4I27MH2Q
.

[axelborja]

Hello Ellis,

Use Case

Our use case is to request inference by batches of tens of millions of instances, and this a multitude of time a day.

The inference will probably take tens of seconds or even a few minutes.

Triggering

The triggering should be done on demand.

It could be event-driven. Triggered by an event, and triggering another event when done.

It could even be an HTTP request but if it is, we could not expect to get a response of completion on the triggering call. However, a second call on another route could be done to get the job status.

Inference

I think the logic of applying the model is quite similar than the current KFServing one. We expect to use pre/post processing and transformers.

Parameters

The requester should provide 2 parameters, which is the input/output file paths of the inference job:

• instances_path
• predictions_path

[cliveseldon]

One possibility would be a CRD that allows these requirements and wires things together with Knative eventing. Maybe the user can provide existing Knative source, sinks.

[yuzisun]

Looks like the use case here is batch file in and batch file out, somewhere on the pipeline we need to do micro batches since single event processing is not efficient in this case, matrix operations on batches would be much more performant especially on predictor.

I did a bit digging looks like on gcloud Apache Beam is used for batch prediction and under kubeflow there is also a repo batch-prediction(https://github.com/kubeflow/batch-predict) which uses Apache beam although seems no longer active.

So far kfserving has been targeting the use cases for online prediction, we’d love to support batch prediction but need to think about how would the architecture looks like.

These are good readings:
https://cloud.google.com/ml-engine/docs/tensorflow/online-vs-batch-prediction
https://cloud.google.com/solutions/comparing-ml-model-predictions-using-cloud-dataflow-pipelines

[wronk]

I'm also looking to do batched inference with kubeflow, so +1 for this feature and thanks @yuzisun for those resources. Our use case is running inference over very large sets (10s to 100s of millions) of satellite images. On-demand or event triggering would also be helpful for us as the frequency can vary quite a bit by project (but likely between once a week to once every 3 months).

[radcheb]

👍

[jtfogarty]

/area engprod
/priority p2

[apryiomka]

As i understand, the ask is similar to Sagemaker batch transform?

[radcheb]

As i understand, the ask is similar to Sagemaker batch transform?

Yes it's a bit similar but should be more efficient. Sagemaker batch transform is using regular inference endpoints to batch inference with HTTP as transfer protocol. It's not efficient for big datasets.

[axelborja]

Agree with @radcheb