Handson-ml: Chapter 2. End-to-End Machine Learning Project > Launch, Monitor, and Maintain Your System
I have participated in kaggle. But, I have found that many of the machine learning books rarely describe this part: Launch, monitor, and maintain your system.
Most of machine learning project description are end with a predict score.
I hope to know more about the startup, monitoring and maintenance of machine learning projects.
Could you help me?
shuiyuejihua
>All comments
Hi @shuiyuejihua ,
Great question! It really depends on your project and its requirements. For example, if you build a recommender system for a video hosting web site, you may train a model to predict the next video based on the previous N videos viewed by a user (and other information, such as the country and so on). Next, whenever the user's home page is requested, the web server needs to get the list of recommendations to display on that page. It will typically send a request to a webservice (e.g., send a REST request and receive a response containing a JSON object). This webservice hosts your latest ML model: in the request, the web server sends some details about the user (e.g., the list of N videos the user watched before, the user's country, etc.), and the webservice must apply the machine learning model based on this data to make a prediction: for example, it will predict a probability for each video. The webservice may return, say, the top 10 probabilities, and perhaps it will try to remove near duplicates (e.g., videos with the same name). It's up to you to decide whether the webservice will remove duplicates, or whether the web server will do that, or some other service (many factors may come into play to choose the architecture that best fits your users' needs). Finally the web server displays the home page with the recommended videos to the user.
In this scenario, launching a model means replacing the models currently executed by the webservice. There are many ways to do that. For example, you could manually connect to the server, copy the latest model to some directory and restart the webservice. Or (more likely) you may use automation tools such as saltstack to do this. Or you could use a system such as TensorFlow Serving that takes care of everything for you (including wrapping your TensorFlow model in a gRPC or REST webservice and gracefully switching to the latest model when it is available).
Regarding monitoring, you probably want your production system to log a bunch of things (as long as it does not hinder performance and your costs are under control), so that you can have systems to analyze the data. For example, you often want to monitor the number of requests per second, the latency (how long each request takes to process), and perhaps use an anomaly detection algorithm (see the notebook 8) to trigger an alarm if things look very unusual, and you could monitor anything else you care about (for example, if the churn is too low, meaning the recommender system keeps recommending the same videos to the same user, you probably want to know about it). You need to define which alerts should wake people up, and which ones can wait until the next business day. You need all the procedures and documentation to ensure that whoever receives the alarms knows what to do with them. And so on.
Regarding maintenance, you probably want to train your model regularly, or else it will not know about the latest videos. If your model must be updated every day (which would probably make sense for a recommender system), then you must build a cron job that will train it. For that you also need a cron job to gather the training data (i.e., get the list of videos, and all the user sessions of the day, etc). Once a model is trained, you need to automatically evaluate it and decide whether it is launchable or not. If the performance is lower than some threshold, you need to send an alert (and possibly cancel the launch). Or else, you need to automatically launch the model.
There's a lot more I could say, but hopefully this gives you an idea of what all this implies. It's not really that specific to machine learning, which is why most ML books don't really talk about it. Moreover, it depends a lot on the specific applications, your requirements, etc., so it's impossible to list every possible use case. Perhaps it may help to add a couple paragraphs (such as the ones above) to the book so that readers can get a sense of some of the things they will need to think about for these steps.
I hope this helps,
Aur茅lien
ageron
on 3 Sep 2018
Related issues
eliaperantoni
路
3Comments
nitml
路
5Comments
shaminder1
路
3Comments
kusur
路
4Comments
qinhanmin2014
路
5Comments
Most helpful comment
Hi @shuiyuejihua ,
Great question! It really depends on your project and its requirements. For example, if you build a recommender system for a video hosting web site, you may train a model to predict the next video based on the previous N videos viewed by a user (and other information, such as the country and so on). Next, whenever the user's home page is requested, the web server needs to get the list of recommendations to display on that page. It will typically send a request to a webservice (e.g., send a REST request and receive a response containing a JSON object). This webservice hosts your latest ML model: in the request, the web server sends some details about the user (e.g., the list of N videos the user watched before, the user's country, etc.), and the webservice must apply the machine learning model based on this data to make a prediction: for example, it will predict a probability for each video. The webservice may return, say, the top 10 probabilities, and perhaps it will try to remove near duplicates (e.g., videos with the same name). It's up to you to decide whether the webservice will remove duplicates, or whether the web server will do that, or some other service (many factors may come into play to choose the architecture that best fits your users' needs). Finally the web server displays the home page with the recommended videos to the user.
In this scenario, launching a model means replacing the models currently executed by the webservice. There are many ways to do that. For example, you could manually connect to the server, copy the latest model to some directory and restart the webservice. Or (more likely) you may use automation tools such as saltstack to do this. Or you could use a system such as TensorFlow Serving that takes care of everything for you (including wrapping your TensorFlow model in a gRPC or REST webservice and gracefully switching to the latest model when it is available).
Regarding monitoring, you probably want your production system to log a bunch of things (as long as it does not hinder performance and your costs are under control), so that you can have systems to analyze the data. For example, you often want to monitor the number of requests per second, the latency (how long each request takes to process), and perhaps use an anomaly detection algorithm (see the notebook 8) to trigger an alarm if things look very unusual, and you could monitor anything else you care about (for example, if the churn is too low, meaning the recommender system keeps recommending the same videos to the same user, you probably want to know about it). You need to define which alerts should wake people up, and which ones can wait until the next business day. You need all the procedures and documentation to ensure that whoever receives the alarms knows what to do with them. And so on.
Regarding maintenance, you probably want to train your model regularly, or else it will not know about the latest videos. If your model must be updated every day (which would probably make sense for a recommender system), then you must build a cron job that will train it. For that you also need a cron job to gather the training data (i.e., get the list of videos, and all the user sessions of the day, etc). Once a model is trained, you need to automatically evaluate it and decide whether it is launchable or not. If the performance is lower than some threshold, you need to send an alert (and possibly cancel the launch). Or else, you need to automatically launch the model.
There's a lot more I could say, but hopefully this gives you an idea of what all this implies. It's not really that specific to machine learning, which is why most ML books don't really talk about it. Moreover, it depends a lot on the specific applications, your requirements, etc., so it's impossible to list every possible use case. Perhaps it may help to add a couple paragraphs (such as the ones above) to the book so that readers can get a sense of some of the things they will need to think about for these steps.
I hope this helps,
Aur茅lien