Building Modern Data Applications Using Databricks Lakehouse: Develop, optimize, and monitor data pipelines on Databricks
商品資訊
ISBN13：9781801073233
出版社：PACKT PUB
作者：Will Girten
出版日：2024/10/31
裝訂：平裝
商品簡介
Get up to speed with the Databricks Data Intelligence Platform to build and scale modern data applications, leveraging the latest advancements in data engineering
Key Features:
- Learn how to work with real-time data using Delta Live Tables
- Unlock insights into the performance of data pipelines using Delta Live Tables
- Apply your knowledge to Unity Catalog for robust data security and governance
- Purchase of the print or Kindle book includes a free PDF eBook
Book Description:
With so many tools to choose from in today's data engineering development stack as well as operational complexity, this often overwhelms data engineers, causing them to spend less time gleaning value from their data and more time maintaining complex data pipelines. Guided by a lead specialist solutions architect at Databricks with 10+ years of experience in data and AI, this book shows you how the Delta Live Tables framework simplifies data pipeline development by allowing you to focus on defining input data sources, transformation logic, and output table destinations.
This book gives you an overview of the Delta Lake format, the Databricks Data Intelligence Platform, and the Delta Live Tables framework. It teaches you how to apply data transformations by implementing the Databricks medallion architecture and continuously monitor the data quality of your pipelines. You'll learn how to handle incoming data using the Databricks Auto Loader feature and automate real-time data processing using Databricks workflows. You'll master how to recover from runtime errors automatically.
By the end of this book, you'll be able to build a real-time data pipeline from scratch using Delta Live Tables, leverage CI/CD tools to deploy data pipeline changes automatically across deployment environments, and monitor, control, and optimize cloud costs.
What You Will Learn:
- Deploy near-real-time data pipelines in Databricks using Delta Live Tables
- Orchestrate data pipelines using Databricks workflows
- Implement data validation policies and monitor/quarantine bad data
- Apply slowly changing dimensions (SCD), Type 1 and 2, data to lakehouse tables
- Secure data access across different groups and users using Unity Catalog
- Automate continuous data pipeline deployment by integrating Git with build tools such as Terraform and Databricks Asset Bundles
Who this book is for:
This book is for data engineers looking to streamline data ingestion, transformation, and orchestration tasks. Data analysts responsible for managing and processing lakehouse data for analysis, reporting, and visualization will also find this book beneficial. Additionally, DataOps/DevOps engineers will find this book helpful for automating the testing and deployment of data pipelines, optimizing table tasks, and tracking data lineage within the lakehouse. Beginner-level knowledge of Apache Spark and Python is needed to make the most out of this book.
Table of Contents
- An Introduction to Delta Live Tables
- Applying Data Transformations Using Delta Live Tables
- Managing Data Quality Using Delta Live Tables
- Scaling DLT Pipelines
- Mastering Data Governance in the Lakehouse with Unity Catalog
- Managing Data Locations in Unity Catalog
- Viewing Data Lineage Using Unity Catalog
- Deploying, Maintaining, and Administrating DLT Pipelines Using Terraform
- Leveraging Databricks Asset Bundles to Streamline Data Pipeline Deployment
- Monitoring Data Pipelines in Production
購物須知
外文書商品之書封，為出版社提供之樣本。實際出貨商品，以出版社所提供之現有版本為主。部份書籍，因出版社供應狀況特殊，匯率將依實際狀況做調整。
無庫存之商品，在您完成訂單程序之後，將以空運的方式為你下單調貨。為了縮短等待的時間，建議您將外文書與其他商品分開下單，以獲得最快的取貨速度，平均調貨時間為1~2個月。
為了保護您的權益，「三民網路書店」提供會員七日商品鑑賞期(收到商品為起始日)。
若要辦理退貨，請在商品鑑賞期內寄回，且商品必須是全新狀態與完整包裝(商品、附件、發票、隨貨贈品等)否則恕不接受退貨。