3 Crucial Steps for operating ML-based systems in production
Machine learning systems are complex and require careful monitoring to ensure they continue to perform as expected and avoid potential risks. In this blog, we’ll discuss the importance of monitoring ML systems in production, Avoiding Bias in ML Systems: Strategies for Fair and Ethical AI, and best practices for managing model versioning and deployment.
The Importance of Monitoring ML Systems in Production
As a machine learning engineer, you know that the deployment of ML systems in production is a complex process. It requires careful monitoring to ensure that your models continue to perform as expected and avoid potential risks. In this blog, we’ll explore the importance of monitoring ML systems in production and share best practices for managing model versioning and deployment.
Potential Risks and Pitfalls of Monitoring ML-Based Systems in Production
One of the most significant risks of operating ML-based systems in production is data drift, where the distribution of input data changes over time, leading to model degradation and performance deterioration. Additionally, bias can be introduced into ML systems, leading to unfair or unethical decision-making. We discuss bias based monitoring seperately, here we tend to focus on things for which we prioritize real time alerting usually involving devops/mlops/ml/ai teams.
An Ops person may be mostly interested in cpu/gpu alerts and the AI/ML teams are more interested in real time model performance, requests served while also monitoring GPU metrices along with Ops Teams. Business stakeholders may also have access to certain grafana views showcasing all models of the business and their performances. Top leadership can make better decision making for offering their ML Services offerings or requirements.
Real-Time Monitoring and Automated Alerting
To mitigate these risks, it’s essential to monitor ML systems closely. Real-time monitoring enables quick issue detection, while automated alerting notifies you when your model’s performance drops below a certain threshold.
There are several tools available for monitoring and alerting ML models. Some popular tools for ML model monitoring include Neptune.ai, Arize AI, Fiddler, and Pachyderm. These tools provide features such as tracking model performance in production, monitoring input/output distribution, evaluating model performance, and alerting when performance deviates from expected ranges. For example, Arize AI integrates with prevailing alerting tools such as email, Slack, OpsGenie, and PagerDuty to send alerts when a model deviates from its expected ranges.
Best Practices for Monitoring ML Systems
Start by defining clear metrics for evaluating your model’s performance. Then, establish a system for tracking those metrics and set up alerts to notify you when they fall outside an acceptable range. Continuously monitor your input data to detect any signs of data drift and validate your models regularly to ensure they’re still performing as expected.
Case Study
DoorDash is one example of the importance of monitoring ML systems. In the past, their models became out-of-date and began making incorrect predictions, negatively impacting the business and customer experience. Finding this kind of model drift took a long time because they did not have a way to monitor for it.
Avoiding Bias in ML Systems: Strategies for Fair and Ethical AI
Strategies for Developing Fair and Ethical AI Systems: Mitigating Bias in Machine Learning
In the development of machine learning (ML) systems, it is crucial to avoid bias as it can affect decision-making and lead to negative consequences for individuals and society. To prevent bias, developers should take a proactive approach by implementing strategies such as data preprocessing, algorithmic fairness techniques, and diverse training data selection. In this blog post, we will discuss these strategies in detail and provide examples of recent cases where bias in AI has caused harm.
Preprocessing Data to Remove Potential Sources of Bias
One way to mitigate bias in ML systems is to preprocess data to remove any potential sources of bias. This involves identifying and addressing any systematic errors or inconsistencies in the data that may lead to biased outcomes. For example, if a dataset contains demographic information, it is important to ensure that it is representative of the population and does not over-represent or under-represent certain groups.
Using Algorithmic Fairness Techniques
Another strategy for avoiding bias in ML systems is to use algorithmic fairness techniques. These techniques aim to ensure that the algorithms used in ML systems are fair and do not discriminate against any particular group. Some examples of algorithmic fairness techniques include counterfactual fairness, which tests whether a model’s predictions would have been the same if an individual had belonged to a different demographic group, and individual fairness, which aims to treat similar individuals similarly.
Diversifying Training Data
To ensure that ML models are unbiased, it is essential to have diverse training data that represents a wide range of demographics and experiences. This helps to prevent models from being skewed towards a particular group or perspective. By including a variety of data points and perspectives, developers can create more accurate and inclusive models that are less likely to perpetuate biases.
Ethical Considerations in ML Development
In addition to technical strategies for mitigating bias, ethical considerations such as transparency, accountability, and privacy must be taken into account when developing and deploying ML systems. By prioritizing these values throughout the development process, developers can ensure that their models are fair and ethical in their decision-making.
Examples of Bias in AI
Recent cases have highlighted the negative impact of bias in AI, such as a job search platform that offered higher positions more frequently to men of lower qualification than women and an algorithm designed to predict recidivism that was found to be biased against certain demographic groups. These examples illustrate the importance of avoiding bias in ML systems and the potential harm it can cause.
To develop fair and ethical AI systems, it is essential to mitigate bias at every stage of the ML pipeline. This includes preprocessing data, using algorithmic fairness techniques, diversifying training data, and considering ethical considerations throughout the development process. By doing so, developers can create models that are accurate, inclusive, and unbiased, and that benefit individuals and society as a whole.
Managing Model Versioning and Deployment Strategies in ML Systems
When it comes to managing model versioning and deployment in ML systems, there are several best practices that you should follow. These practices include implementing version control, using containerization and CI/CD pipelines, conducting A/B testing, optimizing compute resources, and deploying multiple models using greedy or Thompson sampling-based deployment strategies. Let’s take a closer look at each of these.
Implementing Version Control for Model File Versioning
Version control is crucial for ensuring that your models are up-to-date and performing as expected. By implementing version control for model file versioning, you can optimize compute billing and standardize data versioning, which brings sanity to experimentation in any typical ML project that involves multiple iterations. Additionally, versioning becomes even more important if you are trying to ship for multiple devices, as you need to make sure to build different artifacts targeting different devices.
Using Containerization and CI/CD Pipelines
Containerization and CI/CD pipelines can help streamline the deployment process by automating the building, testing, and deployment of your models. This ensures that your models are deployed consistently across different environments and reduces the risk of errors or inconsistencies.
Conducting A/B Testing
A/B testing enables you to test new models before rolling them out to production. This helps you identify potential issues or performance gaps and allows you to fine-tune your models before deploying them to a wider audience.
Optimizing Compute Resources
Optimizing your compute resources is crucial for proper utilization and justifying the billing properly. Your versioning system should ensure to move or create only the necessary data changes or model changes, which is especially important for large model sizes that reach GBs. This helps optimize data movement and avoid duplicate data entry.
Deploying Multiple Models Using Greedy or Thompson Sampling-Based Strategies
If you have many models, consider deploying them using greedy or Thompson sampling-based strategies. This helps avoid conflicts within the team or with vendors and ensures that the most traffic is routed to the best serving algorithm over time. To implement these strategies, you can use an MLOps tool like Kubeflow or Seldon Core & Enterprise.
Conclusion
As more and more use cases are now ML based it becomes more crucial to implement the ML pipelines with more monitoring at every stage for effectively utilising the AI for their specific use case. Although automation is being deployed for better productivity and various stages can be prioritised for automation. However, prioritising the real time monitoring and ensuring further automation by utilising information from those triggers. Further extending your MLOps capabilities for unbiased, ethical and responsible AI inttegration followed by Versioning and deployment strategies are the areas which are recommended to be priortized further for integration. These 3 backbones are considered as necessary for qualifying ML models for a mature stage operationalization and leads to fast true time-to-market.