Scaling Uber's Metric System from Elasticsearch to Pinot | Uber

ABOUT THE TALK:
Uber has been using realtime system to support time-sensitive critical use cases for years, including Gairos, which was initiated in the Marketplace Org and then widely used across the company since 2014, and uMetric, which has emerged rapidly since 2020.
Continuous effort has been spent toward the reliability and performance of these realtime platforms, to cope with traffic growth, increasing number of users, different varieties of use cases, and following work such as operation cost, resource planning, and optimization feature development. This presentation shares the things done right to solve these challenges, including fully replace Elasticsearch with Apache Pinot as the realtime storage of our ecosystem.
ABOUT THE SPEAKERS:
Yupeng Fu is a Principal Engineer at Uber and he leads the Real-time Data platform and Search platform at Uber. Yupeng Fu is also an Apache Pinot committer.
Nan Ding is a staff engineer at Uber, and leads data platform reliability and performance of Marketplace uMetric team.
ABOUT DATA COUNCIL:
Data Council (www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.
Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.
FOLLOW DATA COUNCIL:
Twitter: / datacouncilai
LinkedIn: / datacouncil-ai

[@theaugmenter]

Awesome talk, thanks!

[@YevgenSh]

I did not understand what the sense of supporting Flink ETL in parallel with Spark ETL? Why don't just use Spark Streaming framework instead of Flink ETL? Thank you.

[@MahmoudSabry-wr2im]

Spark streaming is more used for micro batch processing, not like flink which is more real-time processing engine