If Streaming Is the Answer, Why Are We Still Doing Batch?
3.9 هزار بار بازدید -
2 سال پیش
-
cnfl.io/podcast-episode-242 | Is real-time data streaming the future, or will batch processing always be with us? Interest in streaming data architecture is booming, but just as many teams are still happily batching away. Batch processing is still simpler to implement than stream processing, and successfully moving from batch to streaming requires a significant change to a team’s habits and processes, as well as a meaningful upfront investment. Some are even running dbt in micro batches to simulate an effect similar to streaming, without having to make the full transition. Will streaming ever fully take over?
In this episode, Kris talks to a panel of industry experts with decades of experience building and implementing data systems. They discuss the state of streaming adoption today, if streaming will ever fully replace batch, and whether it even could (or should). Is micro batching the natural stepping stone between batch and streaming? Will there ever be a unified understanding on how data should be processed over time? Is the lack of agreement on best practices for data streaming an insurmountable obstacle to widespread adoption? What exactly is holding teams back from fully adopting a streaming model?
Recorded live at Current 2022: The Next Generation of Kafka Summit, the panel includes Adi Polak (Vice President of Developer Experience, Treeverse), Amy Chen (Partner Engineering Manager, dbt Labs), Eric Sammer (CEO, Decodable), and Tyler Akidau (Principal Software Engineer, Snowflake).
EPISODE LINKS
► dbt Labs: www.getdbt.com/
► Decodable: www.decodable.co/
► lakeFS: lakefs.io/
► Snowflake: www.snowflake.com/en/
► The Data Streaming Revolution: Rise of the Kafka Heroes: cnfl.io/45SHTlj
► Stream Processing vs. Batch Processing - What to Know: cnfl.io/blog-streaming-vs-batch-episode-242
► From Batch to Real-Time: Tips for Streaming Data Pipelines with Apache Kafka ft. Danica Fine: cnfl.io/from-batch-to-real-time-streaming-data-pip…
► Kris Jenkins’ Twitter: twitter.com/krisajenkins
► Streaming Audio Playlist: • Streaming Audio Podcast | Apache Kafk...
► Join the Confluent Community: cnfl.io/confluent-community-episode-242
► Learn more with Kafka tutorials, resources, and guides at Confluent Developer: cnfl.io/confluent-developer-episode-242
► Live demo: Intro to Event-Driven Microservices with Confluent: cnfl.io/event-driven-microservices-demo-episode-24…
► Use PODCAST100 to get an additional $100 of free Confluent Cloud usage: cnfl.io/try-cloud-episode-242
► Promo code details: cnfl.io/podcast100-details-episode-242
TIMESTAMPS
0:00 - Intro
2:58 - Is the Lambda Architecture here to stay?
6:27 - What is preventing streaming adoption today?
10:00 - Is streaming a semantic model?
20:53 - Should we push for stream processing?
26:15 - When should we use streaming vs. batch processing?
37:10 - What is the future of stream processing?
41:48 - It's a wrap!
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#streamprocessing #current2022 #apachekafka #kafka #confluent
2 سال پیش
در تاریخ 1401/09/10 منتشر شده
است.
3,930
بـار بازدید شده