Databricks Engineer

Role Overview

As a Databricks Engineer, you will be responsible for designing, building, and maintaining large-scale ETL/ELT data pipelines utilizing Databricks and PySpark technologies. This role involves transforming and processing significant datasets through Python and Spark to support scalable and efficient data workflows. You will engage with complex SQL query development for data extraction, transformation, validation, and reporting purposes. Collaborating with various stakeholders, including data analysts and business teams, to gather requirements is a key aspect of the position.

Key Responsibilities

• Develop and sustain scalable ETL/ELT pipelines using Databricks and PySpark.
• Implement data processing solutions using Python and Apache Spark for large-scale transformations.
• Create advanced SQL queries for data extraction, validation, and reporting.
• Enhance data workflows focusing on performance, scalability, and reliability.
• Handle structured and semi-structured data from diverse sources.
• Collaborate with analysts and stakeholders to understand and fulfill data needs.
• Conduct data quality assessments, troubleshoot issues, and perform root cause analyses.
• Optimize Spark jobs and SQL queries to improve execution times.
• Adhere to coding standards and best practices including documentation and version control.
• Engage in code reviews and assist with production environment deployments.

Required Skills and Experience

• At least 3 years of experience in data engineering roles.
• Proficient hands-on experience with the Databricks platform.
• Strong working knowledge of PySpark and Apache Spark frameworks.
• Solid programming capability in Python.
• Advanced SQL proficiency, including complex joins, window functions, and query performance optimization.
• Experience in constructing ETL/ELT pipelines.
• Understanding of data warehousing principles and data modeling techniques.
• Familiarity with Git or comparable version control systems.
• Excellent analytical skills along with problem-solving and debugging expertise.

Eligibility Criteria

Candidates holding any graduate degree are eligible to apply.