Event-Driven Kubernetes pipelines for high performant at-home patient health monitoring systems

• Pipelines in existing public cloud were not production ready, lacking automation and robustness, preventing ability to rapidly improve camera runtimes
• Lack of system automation for adding new data, running new experiments, and retraining machine learning models
• Adhoc commands issued to run models without device registration and automation
• No effective method to label new camera images in high volume setting was leading to poor camera accuracy and inability to collaborate as a team
• Lack of infrastructure portability and DevOps automation was limiting ability to migrate or adopt multi-cloud strategy
  

Bitstrapped re-architected an event driven system on GCP that enabled greater degree of automation and high volume of data ingestion and processing events. The chosen architecture for pre-processing and data ingestion was serverless, powered by IoT core with enablement of automated deployment of models to the edge. As images were collected, the architecture would facilitate data augmentation and preprocessing of the images in order to speed up downstream training pipelines.

For the data science team, the new infrastructure on Google Kubernetes Engine and use of production KubeFlow, enabled automation of ML pipelines to pre-process, train, and deploy the models. This included flexible compute for their experimentation dynamic and GPU enablement as needed to accelerate training time. 

Once the event-driven pipeline automation was in place, focus shifted to the performance of the camera solution. Implementing a semi-automated labelling solution would automatically detect failures and allow for humans in the loop as needed. This way, the team could be efficient in labelling images, while also improving the ability to do meta learning and make use of previously trained models to label new datasets. Label automation was achieved by integrating a custom instance of Label Studio into the architecture and MLOps process. 

Event driven automation enabled:

• Limitless data ingestion scale
• Real-time automation of raw data ingest, data pre-processing, data lake ingest, ETL process
• Semi-automated labelling pipelines reduced labelling activities from 1 week to <24hrs

DevOps and Orchestration enabled:

• Reduction of model training time by 75%

Kubernetes Engine adoption enabled:

• Increased performance through preemptible GPU node pools
• Clusters with greater horizontal scalability
• Terraform configurable Kubernetes CronJobs
• Kubeflow adoption for ETL pipelines

Improved Security with:

• Data uploads using Short-lived signed urls
• Cloud IotCore
• Adoption of service accounts
• Least privilege IAM and bucket access

Best Practices adopted:

• Data partitioning
• Terraform IaS for disaster recovery and portability
• Highly reliable event processing with Pub/Sub

IoT Core enabled:

• Registration of limitless devices
• Auto deployment of model updates to whole device ecosystem
  

Customer was equipped with productive grade, practically fully automated process, that freed back all the time previously spent manually executing steps to trigger events, label, and upload data. As a result, the same teams could focus on the accuracy of the models and new model development with a self sufficient cloud environment.