Metaflow: The ML Infrastructure at Netflix

Metaflow was started at Netflix to answer a pressing business need: How to enable an organization of data scientists, who are not software engineers by training, build and deploy end-to-end machine learning workflows and applications independently. We wanted to provide the best possible user experience for data scientists, allowing them to focus on parts they like (modeling using their favorite off-the-shelf libraries) while providing robust built-in solutions for the foundational infrastructure: data, compute, orchestration, and versioning.
Today, the open-source Metaflow powers hundreds of business-critical ML projects at Netflix and other companies from bioinformatics to real estate.
In this talk, you will learn about:
- What to expect from a modern ML infrastructure stack.
- Using Metaflow to boost the productivity of your data science organization, based on lessons learned from Netflix.
- Deployment strategies for a full stack of ML infrastructure that plays nicely with your existing systems and policies.
Speaker: Ville Tuulos
Website: www.aicamp.ai/event/eventdeta...
Discussion group on slack: bit.ly/3iLe40y

[@1993JosephS]

You need to use a lot of partitioning to be able to hit S3 with thousands of instances in parallel. There are per-prefix rate limits.

[@AICamp]

slides: www.slideshare.net/BillLiu31/metaflow-the-ml-infrastructure-at-netflix