MLOps: Streamlining Machine Learning Development and Deployment
Prologue -
One of my favourite and most informative newspapers is the Financial Times. Lucy Kellaway, a brilliant journalist, used to have a regular column there. Lucy was one of the most remarkable and insightful journalist on the corporate world but, alas, she has now decided to leave the FT and work as a teacher. This alone should be enough to understand her moral depth and intelligence. Luckily, she left us with a brilliant character that appeared on the columns of the Financial Times whose life was described by the flurry of e-mails sent and received by him. Martin Lukes, the CEO of a-b glöbâl and the child of Lucy Kellaway’s genius, has got his own Wikipedia page in which he is described as “a pastiche of every management fad and is an antidote to the orthodox management style taught in most MBA programmes and routinely lambasted by Lucy Kellaway in her business life column”.
I am sure that Dilbert’s boss was most likely one of the most fervent and ardent fan of Martin Lukes and felt he had much to learn from him.
Agile Management -
Agile management has been around for a while, yet questions persist about whether it is just a passing fad. Forbes raised this same query over 8 years ago, but Agile remains a prominent approach. A couple of years ago, the Harvard Business Review pondered whether we have taken it too far. However, despite its enduring presence, it is sometimes surprising how few people can accurately define what Agile truly means and how to effectively put it into practise. This may be partly due to the many incarnations of Agile project management, encompassing methodologies as diverse as Scrum, Kanban and Lean, leading to varied perspectives and criticisms.
(Widely circulated satirical meme on Agile Development)
Agile methodologies focus on the iterative and incremental development of software, emphasising flexibility, customer collaboration and responsiveness to change. Agile frameworks like Scrum or Kanban are specific methodologies used by development teams to manage their work.
While not typically classified as an Agile methodology, DevOps aligns well with Agile principles, emphasising not only development but also delivery and implementation.
Agile for Machine Learning -
The rapid evolution of machine learning (ML) has transformed countless industries, bringing unprecedented opportunities for data-driven decision-making and automation. Nonetheless, the complexities of building, deploying, and managing ML models in production remain a significant challenge. Traditional workflows often suffer due to manual processes, isolated teams and limited visibility, resulting in inefficiencies, errors and hindered scalability.
Machine learning projects are notoriously resistant to Agile management and people often say that implementing Machine Learning and AI projects following the Agile approach is like fitting a square peg into a round hole. To start they are heavily reliant on data availability and quality which frequently remains uncertain until the project work commences. A Machine Learning project encompasses more than software delivery - it involves a blend of data retrieval, analysis, model implementation, labelling, training and accuracy comparison between developed models. Hypotheses may go unconfirmed and require new approaches.
However, although there is no direct equivalent of Agile methodologies in Machine Learning, an analogous concept to DevOps exists, aptly named MLOps.
MLOps, short for Machine Learning Operations, concentrates on transitioning machine learning models into production while maintaining and monitoring them.
The machine learning lifecycle involves data ingestion, cleaning, training, tuning and deployment. Implementing a stringent operational structure can enhance efficiency across these processes and ensure synchronisation. Moreover, an optimal process implementation facilitates swift project scalability.
(Image from Databricks site at https://www.databricks.com/glossary/mlops)
Challenges of Traditional Workflows -
Traditional workflows present well-defined challenges, including:
Lack of Automation: Manual model building and deployment are prone to human error and consume valuable time and resources.
Siloed Teams: Data scientists and engineers work independently, hindering communication, knowledge sharing, and collaboration.
Difficulty with Reproducibility: Recreating and deploying models in production can be challenging due to undocumented processes and lack of version control.
Limited Monitoring and Governance: Model performance is not effectively monitored, and potential risks associated with bias and fairness are not identified.
MLOps to the Rescue -
Enter MLOps, a collection of principles and practises designed to bridge the gap between data science and operations. Inspired by DevOps, MLOps aims to streamline the entire ML lifecycle, from development to deployment and beyond. By automating tasks, breaking down silos, and fostering collaboration, MLOps empowers organisations to unlock the full potential of their ML models.
MLOps addresses these challenges by introducing a set of guiding principles and powerful tools:
Continuous Integration/Continuous Delivery (CI/CD): Automates the process of building, testing, and deploying models, ensuring consistency and reliability.
Version Control: Tracks changes in model code and data, enabling rollbacks and facilitating collaboration.
Infrastructure as Code (IaC): Defines and manages model environments using code, promoting consistency and scalability.
Monitoring and Logging: Tracks key performance metrics and detects potential issues, enabling proactive problem-solving.
Governance and Compliance: Establishes guidelines and processes for responsible AI development, ensuring ethical and compliant use of ML models.
Benefits of Implementing MLOps -
Among the benefits of implementing an MLOps-type workflow, we list:
Increased Efficiency and Productivity: Automating tasks frees up valuable time for data scientists to focus on innovation and problem-solving.
Improved Model Quality and Reliability: CI/CD and version control ensure consistency and reproducibility, leading to more robust and reliable models.
Reduced Risk of Errors and Deployments: Automated testing and monitoring identify and address issues early on, preventing costly errors in production.
Scalable and Sustainable Model Deployment: MLOps tools facilitate managing and scaling models across diverse environments efficiently.
Collaborative and Transparent Development Process: Breaking down silos fosters better communication and collaboration between teams.
We should however emphasise that MLOps represents more than a mere set of guiding principles - it embodies an extensive ecosystem comprising diverse tools and technologies.
Conclusion -
MLOps has become an essential ingredient for success in the age of machine learning. By embracing MLOps principles and adopting the right tools, organisations can unlock the full potential of their ML models, drive innovation, and achieve competitive advantage.
What would Martin Lukes think of MLOps? I am sure Martin Lukes would be 110 percent behind MLOps, minimum. No doubt it is better than the bestest and it is HUGE.