Principal Data Engineer (Databricks)

Overview

DAI Consultancy is seeking a Principal Data Engineer with deep expertise in Databricks platform technologies to join their team in Doha, Qatar. The ideal candidate will have a proven track record designing and managing robust data pipelines and complex workflows within large-scale production environments.

Key Responsibilities and Role Expectations

• Develop and maintain highly reliable ETL pipelines and intricate data workflows.
• Utilize advanced capabilities of Apache Spark, Delta Lake architecture, and Databricks tools to architect optimized data infrastructure.
• Implement best practices surrounding Delta Lake core architecture including Medallion Architecture layers: Bronze, Silver, and Gold.
• Leverage Databricks features such as Unity Catalog, Photon Engine, and Delta Live Tables to enhance data processing efficiency.
• Deploy workflow orchestration solutions involving Databricks Workflows, dbt (Data Build Tool), and Airflow.
• Perform performance tuning techniques including Z-Ordering, Liquid Clustering, Data Skipping, and Spark UI debugging to optimize pipeline execution.

Technical Requirements

• Demonstrated expert proficiency in Apache Spark and Spark SQL development.
• Strong understanding of Delta Lake architecture aligned with Medallion design patterns.
• Hands-on experience with Databricks components: Delta Live Tables (DLT), Unity Catalog, and Photon Engine.
• Familiarity with popular orchestration tools such as Airflow and dbt for workflow management.
• In-depth knowledge of performance optimization strategies for large-scale data engineering pipelines.

Certification

Possession of the Databricks Certified Data Engineer Professional certification is required.