Kfserving: Alibi income explainer example fails

[lsuanet]

/kind bug

What steps did you take and what happened:
I tried running the sample alibi income explainer using alibi v0.3.0.

When running the sample prediction, it works:

{'predictions': [0]}

However, getting an explanation fails. The explainer logs:

   HTTPServerRequest(protocol='http', host='x6f3fb4c2-86fb-4d45-9659-ee297b734bfd-explainer-default.models.svc.cluster.local', method='POST', uri='/v1/models/x6f3fb4c2-86fb-4d45-9659-ee297b734bfd:explain', version='HTTP/1.1', remote_ip='127.0.0.1')
2020-09-15T12:50:54.737122638Z     Traceback (most recent call last):
2020-09-15T12:50:54.7371264Z       File "/usr/local/lib/python3.7/site-packages/tornado/web.py", line 1701, in _execute
2020-09-15T12:50:54.737130478Z         result = method(*self.path_args, **self.path_kwargs)
2020-09-15T12:50:54.737133957Z       File "/kfserving/kfserving/handlers/http.py", line 61, in post
2020-09-15T12:50:54.737137617Z         response = model.explain(request)
2020-09-15T12:50:54.737141173Z       File "/alibiexplainer/alibiexplainer/explainer.py", line 74, in explain
2020-09-15T12:50:54.737144949Z         explanation = self.wrapper.explain(request["instances"])
2020-09-15T12:50:54.737148857Z       File "/alibiexplainer/alibiexplainer/anchor_tabular.py", line 89, in explain
2020-09-15T12:50:54.737152659Z         anchor_exp = self.anchors_tabular.explain(arr[0], **self.kwargs)
2020-09-15T12:50:54.737156486Z       File "/usr/local/lib/python3.7/site-packages/alibi/explainers/anchor_tabular.py", line 803, in explain
2020-09-15T12:50:54.737160777Z         for sampler in self.samplers:
2020-09-15T12:50:54.737164397Z     AttributeError: 'AnchorTabular' object has no attribute 'samplers'

It seems like it has something to do with the alibi version, but I am not sure.

What did you expect to happen:

I expected to get the sample explainer output.

Anything else you would like to add:

I am using the kfserving client and I am specifying runtime_version of my V1alpha2AlibiExplainerSpec to be v0.3.0. Anything other than v0.3.0 returns

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \"inferenceservice.kfserving-webhook-server.validator\" denied the request: Alibi RuntimeVersion must be one of v0.3.0","code":400}

This also influenced my choice to go with alibi version v0.3.0 when building the explainer.

Environment:

• KFServing Version: v0.3.0
• alibi: v0.3.0

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/engprod | 0.54 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[yuzisun]

/cc @cliveseldon

[yuzisun]

@larssuanet check this https://github.com/kubeflow/kfserving/pull/866/files, looks like this doc did not get patched in to the 0.3 release branch. I know it is a bit confusing but v0.3.0 is the KFS alibi image version not alibi library version

[lsuanet]

@yuzisun I retried on the 0.3 release branch.

After training the model and explainer and deploying them, I get the following error:

2020-09-18T08:05:32.610652094Z [I 200918 08:05:32 __main__:116] Extra args: {}
2020-09-18T08:05:32.611033532Z [I 200918 08:05:32 storage:35] Copying contents of /mnt/models to local
2020-09-18T08:05:32.61120532Z [I 200918 08:05:32 __main__:125] Loading Alibi model
2020-09-18T08:05:32.616445313Z Traceback (most recent call last):
2020-09-18T08:05:32.616478573Z   File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
2020-09-18T08:05:32.616485093Z     "__main__", mod_spec)
2020-09-18T08:05:32.616490891Z   File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
2020-09-18T08:05:32.616496213Z     exec(code, run_globals)
2020-09-18T08:05:32.616501804Z   File "/alibiexplainer/alibiexplainer/__main__.py", line 126, in <module>
2020-09-18T08:05:32.616507687Z     alibi_model = dill.load(f)
2020-09-18T08:05:32.616512779Z   File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 270, in load
2020-09-18T08:05:32.616517954Z     return Unpickler(file, ignore=ignore, **kwds).load()
2020-09-18T08:05:32.616523205Z   File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 472, in load
2020-09-18T08:05:32.616528697Z     obj = StockUnpickler.load(self)
2020-09-18T08:05:32.616533705Z   File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 462, in find_class
2020-09-18T08:05:32.616539134Z     return StockUnpickler.find_class(self, module, name)
2020-09-18T08:05:32.616544926Z AttributeError: Can't get attribute '_create_code' on <module 'dill._dill' from '/usr/local/lib/python3.7/site-packages/dill/_dill.py'>

Suspecting this was a dill version error, I downgraded dill from 0.3.2 to 0.2.7.1 and ran again. Then this error came up:

2020-09-18T08:17:43.197632385Z [I 200918 08:17:43 __main__:116] Extra args: {}
2020-09-18T08:17:43.198254324Z [I 200918 08:17:43 storage:35] Copying contents of /mnt/models to local
2020-09-18T08:17:43.19854656Z [I 200918 08:17:43 __main__:125] Loading Alibi model
2020-09-18T08:17:43.291990102Z [I 200918 08:17:43 explainer:47] Predict URL set to x451d5193-4db8-418e-bef7-0ea10c5024e9-predictor-default.models
2020-09-18T08:17:43.300944431Z Traceback (most recent call last):
2020-09-18T08:17:43.300987089Z   File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
2020-09-18T08:17:43.300993406Z     "__main__", mod_spec)
2020-09-18T08:17:43.300998605Z   File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
2020-09-18T08:17:43.30100391Z     exec(code, run_globals)
2020-09-18T08:17:43.301008666Z   File "/alibiexplainer/alibiexplainer/__main__.py", line 132, in <module>
2020-09-18T08:17:43.301014665Z     alibi_model)
2020-09-18T08:17:43.301019238Z   File "/alibiexplainer/alibiexplainer/explainer.py", line 51, in __init__
2020-09-18T08:17:43.301024294Z     self.wrapper = AnchorTabular(self._predict_fn, explainer, **config)
2020-09-18T08:17:43.301029145Z   File "/alibiexplainer/alibiexplainer/anchor_tabular.py", line 37, in __init__
2020-09-18T08:17:43.301034222Z     self._reuse_cat_map(self.anchors_tabular.categorical_names)
2020-09-18T08:17:43.301039095Z AttributeError: 'AnchorTabular' object has no attribute 'categorical_names'

Still seems like an Alibi version mismatch. Does the explainer server automatically load the latest Alibi version?

[lsuanet]

After downgrading alibi to 0.3.2 as I found this is the version the explainer server runs on , the server starts successfully. However, I am now back to the original error:

2020-09-18T09:41:52.107872365Z [I 200918 09:41:52 anchor_tabular:83] Arr shape ((2, 12),) 
2020-09-18T09:41:52.14564083Z [E 200918 09:41:52 web:1792] Uncaught exception POST /v1/models/x1b2182b7-3d83-4bbe-80ab-75942f59dba1:explain (127.0.0.1)
2020-09-18T09:41:52.145678077Z     HTTPServerRequest(protocol='http', host='x1b2182b7-3d83-4bbe-80ab-75942f59dba1-explainer-default.models.svc.cluster.local', method='POST', uri='/v1/models/x1b2182b7-3d83-4bbe-80ab-75942f59dba1:explain', version='HTTP/1.1', remote_ip='127.0.0.1')
2020-09-18T09:41:52.145688965Z     Traceback (most recent call last):
2020-09-18T09:41:52.14569476Z       File "/usr/local/lib/python3.7/site-packages/tornado/web.py", line 1701, in _execute
2020-09-18T09:41:52.145700201Z         result = method(*self.path_args, **self.path_kwargs)
2020-09-18T09:41:52.145705463Z       File "/kfserving/kfserving/handlers/http.py", line 61, in post
2020-09-18T09:41:52.145710753Z         response = model.explain(request)
2020-09-18T09:41:52.145715992Z       File "/alibiexplainer/alibiexplainer/explainer.py", line 74, in explain
2020-09-18T09:41:52.145734543Z         explanation = self.wrapper.explain(request["instances"])
2020-09-18T09:41:52.145742744Z       File "/alibiexplainer/alibiexplainer/anchor_tabular.py", line 89, in explain
2020-09-18T09:41:52.145748711Z         anchor_exp = self.anchors_tabular.explain(arr[0], **self.kwargs)
2020-09-18T09:41:52.145754241Z       File "/usr/local/lib/python3.7/site-packages/alibi/explainers/anchor_tabular.py", line 803, in explain
2020-09-18T09:41:52.145759784Z         for sampler in self.samplers:
2020-09-18T09:41:52.14576529Z     AttributeError: 'AnchorTabular' object has no attribute 'samplers'
2020-09-18T09:41:52.145771057Z [E 200918 09:41:52 web:2250] 500 POST /v1/models/x1b2182b7-3d83-4bbe-80ab-75942f59dba1:explain (127.0.0.1) 47.96ms

[yuzisun]

@larssuanet are you using the alibi image from docker.io/seldonio/kfserving-alibiexplainer ?

[lsuanet]

@yuzisun I was not. It seems like I installed my installation before the patch. I updated the image and everything seems to be working fine now. Thanks for the help!