Kfserving: Allow passing model parameters for PyTorch server

[yuzisun]

/kind feature

Describe the solution you'd like
Currently we can only pass ModelClassName on Pytorch spec, however most of time ModelClass requires additional parameters such as EmbeddingSize and Layers etc.

This is an example

class TorchTextClassifier(nn.Module):

    def __init__(self, vocab_size, embedding_dim, seq_length, num_classes, 
                 num_filters, kernel_size, pool_size, dropout_rate):
        super(TorchTextClassifier, self).__init__()

        self.embeddings = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embedding_dim)

        self.conv1 = nn.Conv1d(seq_length, num_filters, kernel_size)
        self.max_pool1 = nn.MaxPool1d(pool_size)
        self.conv2 = nn.Conv1d(num_filters, num_filters*2, kernel_size)

        self.dropout = nn.Dropout(dropout_rate)
        self.dense = nn.Linear(num_filters*2, num_classes)

Anything else you would like to add:
The proposal is to support passing additional parameters on pytorch spec

  default:
    predictor:
      pytorch:
        modelClassName: TorchTextClassifier
        modelClassKwargs: {'embedding_size': 128, 'n_layers': 6}

[yuzisun]

@cliveseldon @animeshsingh @ellis-bigelow @rakelkar what do you think ?

[cliveseldon]

Why do you need the parameters at inference time. The example seems very similar to https://cloud.google.com/blog/products/ai-machine-learning/ai-in-depth-serving-a-pytorch-text-classifier-on-ai-platform-serving-using-custom-online-prediction

Are the parameters not just needed for training?

[alexcoca]

@yuzisun The current assumption of the server is that the user would save their model using torch.save(model, PATH) at the end of the training loop, as opposed to passing the model state dictionary to torch.save. This means that the deserialisation procedure should already know the state of the model and passing the arguments for the constructor is not necessary.

Does this make sense?

[yuzisun]

@alexcoca currently we are doing torch.save(model.state_dict(), PATH) since this is suggested way on pytorch website https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference. Saving the entire model seems unsafe.

[alexcoca]

In this case, the parameters have to be passed, as you suggest, so that the model instance can be created and instantiated as opposed to unpickled, which is the current assumption.

[WangAllen851018]

@yuzisun In the example of pytorchserver, the modelClassName is 'Net', but if i will use the ResNet to make inference, the initialization of modelClassName 'ResNet' __init__(self, ResidualBlock, num_classes=10) must contain the other Class 'ResidualBlock' , how can i solve the problem ？Thank you ~

[yuzisun]

is ResidualBlock the actual class ? that can be a bit tricky.. my idea is to pass json which then can be turned into a python dict and pass to the model class constructor.

[WangAllen851018]

is ResidualBlock the actual class ? that can be a bit tricky.. my idea is to pass json which then can be turned into a python dict and pass to the model class constructor.

Thanks @yuzisun, your idea is brilliant, i will try it ~~

[yuzisun]

@WangAllen851018 are you planning to contribute this? If this is case I can point you where I left off.

[WangAllen851018]

@WangAllen851018 are you planning to contribute this? If this is case I can point you where I left off.

Yes, i have the plan to contribute this part of pytrochserver, do you have some other suggestions for the problem?

[jtfogarty]

/area engprod
/priority p2

[nlarusstone]

@WangAllen851018 are you still planning to contribute to this? This is a feature we are interested in as well

[zhangrongguo]

@yuzisun @WangAllen851018 @nlarusstone

Hi, WangAllen851018 and I are in the same team. Now we have the solution of this issue, and we want to contribute our code to kfserving project. But we don't know how to do that (We already have the CLA).

[WangAllen851018]

@WangAllen851018 are you still planning to contribute to this? This is a feature we are interested in as well

@nlarusstone Yes，the work will be contributed to this by the member of our research program @zhangrongguo in #727

[yuzisun]

Would you guys like to join our WG meeting on 3/18 9am PST to discuss this? If it is too late for you we can schedule a special topic meeting on this.

[zhangrongguo]

@yuzisun @WangAllen851018 @Iamlovingit
We will attend the WG meeting. By the way, what materials should we prepare? And could you send us the meeting link?

[yuzisun]

@WangAllen851018 @Iamlovingit @zhangrongguo Write a google doc with the proposal would be helpful, the zoom link is https://zoom.us/j/920703993 and the coming WG meeting is on 3/18 9am-10am PST. Please also join the KFServing slack so that we can chat there also. We are excited to talk to you guys!

[zhangrongguo]

@yuzisun
Hi, I want to know how to join the slack channel.
The searching result of kfserving is empty.

[zhangrongguo]

@yuzisun
When I clicked the link in your mail, it let me fill the slack url. But I am not sure how to fill it.

[yuzisun]

@zhangrongguo my bad, can you join kubeflow slack and then find the #kfserving channel ?

[zhangrongguo]

@yuzisun @WangAllen851018 @Iamlovingit
If you guys have any requstions about our proposals, please feel free to tell us. And if you have the discussing results, please also let us know. Thanks very much!

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.