Streaming Data Pipelines with Kafka

Streaming data pipelines let you integrate data from multiple systems in real time, with instantaneously updating and processing from data source to data sink. In Streaming Data Pipelines with Kafka you'll build the kind of streaming pipelines that hold up modern data infrastructure, all with the industry-standard Apache Kafka platform.

Inside this practical guide, you'll learn how to:

- Serve real-time data to business departments of your organization
- Understand streaming data pipeline concepts such as change data capture
- Troubleshoot common challenges when building and deploying streaming data pipelines
- Setup open-source connectors with Kafka Connect and develop custom connectors yourself
- Implement stateless and stateful data processing with Kafka Streams
- Tune pipeline performance for low-latency and high-throughput requirements
- Scale pipelines both manually and automatically to cope with performance requirements
- Debug and monitor streaming data pipelines in production
- Decide when to use streaming data pipelines over batch pipelines

Data streaming doesn't have to be complex! Kafka Connect and Kafka Streams have made it possible for any developer to start building a data streaming pipeline without needing to fiddle with low-level APIs. This practical guide empowers you to utilize the full ecosystem of Kafka to implement your first streaming data pipelines.

About the book

Streaming Data Pipelines with Apache Kafka teaches you to build the kind of rapid, reliable data pipelines that can deliver real-time insights from your data. You'll follow along with an extended case study as Excellent Toys Corporation's data team migrates from batch processing to their very first streaming pipelines. Dive into custom connector development, extracting real-time changes from an HTTP-based Analytics API, and delve into event-driven, real-time processing with Kafka Streams. With guidance on packaging, deploying, and error handling, you'll soon be equipped to build and deploy streaming data pipelines in production environments.

About the reader

For developers and data scientists who know the basics of Java and database systems. No experience with Kafka required.

About the author

Stefan Sprenger has more than 15 years of experience in software engineering and specializes in building real-time data architectures. He has a PhD in computer science, is a frequent speaker at technical conferences, co-founded a startup in the data streaming space, and has contributed to various open-source projects.

Get a free eBook (PDF or ePub) from Manning as well as access to the online liveBook format (and its AI assistant that will answer your questions in any language) when you purchase the print book.