Databricks Data Engineer

Position Summary

We are looking for a mid-level Databricks Data Engineer to join our team and take on a critical role in designing, constructing, and managing large-scale data pipelines. This role involves working with modern data engineering tools and collaborating with various teams to drive data solutions that empower business decisions and enhance data infrastructure.

Key Responsibilities

• Create scalable data pipelines leveraging Databricks, Apache Spark, and Delta Lake to ingest, process, and analyze vast datasets.
• Partner with data scientists and analysts to capture data requirements and build appropriate data engineering solutions.
• Write and maintain Python scripts to automate data workflows and facilitate integration with other systems.
• Optimize pipeline efficiency, resolve issues promptly, and ensure high standards of data quality and accuracy.
• Collaborate with stakeholders to establish and enforce data governance policies and procedures.
• Maintain and author SQL scripts to analyze and manage data stored in Delta Lake.

Essential Qualifications and Experience

• Proven experience in designing and constructing data pipelines using Databricks, Apache Spark, and Delta Lake.
• Proficient in Python programming, including use of data engineering libraries and frameworks.
• Solid knowledge of SQL, data modeling, data warehousing, and ETL processes.
• Deep understanding of data pipeline architectures and fundamental data engineering concepts.
• Experience managing data pipelines and governance in dynamic, fast-paced environments.
• Excellent interpersonal skills for collaborating with cross-functional teams.

Preferred Credentials

• Familiarity with data engineering best practices and current industry trends.
• Experience with cloud-based data infrastructure and understanding of data security standards.
• Certifications related to Databricks or Apache Spark are an advantage.