Autogluon: predictor.predict hanging

[austinmw]

I'm trying to integrate autogluon with SageMaker training/deployment.

With the test script I'm currently using, sometimes fit/deploy/predict work fine, but about half of the time (without any modification of docker image or train/serve code, seemingly randomly) when I get to the point where I'm trying to call predictor.predict(data), I can see in my endpoint logs that it's just hanging there and timing out. I'm using the tabular predictor on a single datapoint, so it shouldn't be timing out due to model complexity.

What could be going wrong here, and how can I go about debugging this?

[zhanghang1989]

Looks like question on Tabular Predition. @Innixma @jwmueller

[Innixma]

Removing the SageMaker component of this, does your code fail on predict in the local setting?

Can you also provide a code sample of the failure? Ensure that the single data point you are passing into .predict is a DataFrame and not a Series.

[austinmw]

Hi @Innixma , looks like it does not fail in the local setting, so must be a sagemaker/environment issue. Having a bit of difficulty in teasing the difference between the two environments. Happen to know of a way to get more to the bottom of why predictor.predict() times out in SM? Dataframe input and model look to be the same. Going to get more environment info and minimum reproducible examples. Thanks.

[Innixma]

@zhanghang1989 would be the most knowledgable regarding SM with AutoGluon, but right now its largely an untested domain. Insights into this from your testing would be greatly appreciated!

[jwmueller]

Is this your first time using SM? Perhaps you can also try using a different python model (eg. scikit-learn) as well as autogluon, just to rule out that it is a pure SM issue?

[Innixma]

It is possible the issue stems from the path issue solved in #319. I'd recommend trying again after building from latest source once this PR is merged.

[austinmw]

@jwmueller been using SM for quite a while now. Will look into path issue, thanks! Still working on reproducible examples.

[Innixma]

319 has been merged, please try to see if it works with the latest build. Note that it requires retraining of the model, the model from the old version won't be fixed.

Best,
Nick

[austinmw]

Worked!

[Innixma]

Awesome!