290
views
1
recommends
+1 Recommend
1 collections
    0
    shares
      scite_
       
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Machine Learning in Production: From Experimented ML Model to System

      Preprint
      In review
      research-article
      Bookmark

            Abstract

            Production ML pipeline refers to a complete end-to-end workflow of a machine learning product ready for deployment. In recent years, companies have vastly invested in Machine Learning research; developers are developing new tools and technologies to make ML more flexible. Now, we can experience AI in most devices around us, from home appliances to cars. When we want to develop an AI-powered product, it is vital to understand the crucial workflows of the ML. Academic research to develop an ML model and a production ML pipeline are entirely different scenarios. From business problems, data collection to deploying the model is an acutely iterative process. Most of the time, Data scientists and Machine Learning Engineers need to deal with issues like data shift, concept shift, model decay, etc. Sometimes, there are need to change the complete ML architecture or how the features are engineered in the dataset. It will become tedious if someone is working in such an environment and lacks an understanding of the entire workflow of the ML pipeline. Though every ML project is different, a data scientist/ ML engineer/ data engineer must understand the end-to-end workflow of the ML pipeline for the product they are developing. The challenge starts with a business problem. We may face different domain problem statements that need to be solved with Machine Learning. How the data will be collected is also a big concern. Data pre-processing, data validation, data monitoring, feature engineering, Model Selection, hyperparameter tuning, model optimization, model performance analysis, performance evaluation, detecting bias, model deployment, post-deployment analysis & monitoring are the crucial processes to make your model production-ready. The main contribution of this research paper is to present a complete picture of the end-to-end workflows of a production-ready ML pipeline.

            Content

            Author and article information

            Journal
            ScienceOpen Preprints
            ScienceOpen
            8 June 2022
            Affiliations
            [1 ] Computer Science & Engineering, IEM, Kolkata, India
            Author notes
            Author information
            https://orcid.org/0000-0003-4585-6024
            Article
            10.14293/S2199-1006.1.SOR-.PPKHXND.v1
            89844cc0-4424-4a6e-bf9d-bc33f1744f65

            This work has been published open access under Creative Commons Attribution License CC BY 4.0 , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Conditions, terms of use and publishing policy can be found at www.scienceopen.com .

            History
            : 8 June 2022

            All data generated or analysed during this study are included in this published article (and its supplementary information files).
            Artificial intelligence
            Machine Learning Pipeline, Neural Architecture Search, Principal Component Analysis, Model optimization, Dimensionality Reduction, Directed Acyclic Graph, Data Orchestrators, Model Decay

            Comments

            Comment on this article