What has worked fairly well so far: Models: - Models are structured as python packages, each model inherits a base class - base class has define how to trai

What has worked fairly well so far: Models: - Models are structured as python packages, each model inherits a base class - base class has define how to train, and how to predict (as well as a few other more specific things) - ML engineer can override model serialization methods, default is just pickle Infra: - Code is checked in to github, Docker container built each merge into master - Use Sagemaker BYO container to train models, each job gets a job_id that represents the state produced by that job (code + data used) Inference / deployment: - Deploy model as http endpoints (SageMaker or internal) using job_id - Have a service that centralizes all score requests, finds correct current endpoint for a model, emits score events to kinesis, track health of model endpoints - A/B test either in scoring service or in product depending on requirements - deploy prediction jobs using a job_id and a data source (usually sql) that can be configured to output data to S3 or our data warehouse So far this has been pretty solid for us. The tradeoff has been theres a step between notebook and production for ML engineers which can slow them down, but it forces code review and increases the number of tests checked in.