Senior Data Engineer

About Numerator

Numerator is transforming the market research landscape by providing leading brands and retailers with unparalleled insights into consumer behaviors and key influencers. Join us as we pioneer the future of market intelligence.

Role Overview

We seek a highly experienced Senior Data Engineer specialized in constructing and refining robust data pipelines capable of managing large-scale data volumes. The ideal individual will possess advanced Python programming capabilities, deep knowledge of Databricks within the Azure Cloud ecosystem, and proficiency in DevOps alongside continuous integration/continuous deployment (CI/CD) tools. Familiarity with AI/ML methodologies and big data processing technologies such as Apache Spark and PySpark is essential.

Key Responsibilities

• Comply with coding protocols and technological standards established by Numerator.
• Create and maintain automated testing suites through Azure DevOps.
• Perform manual, load, and exploratory testing as necessary.
• Collaborate effectively with Business Analysts and Senior Data Developers to meet sprint objectives.
• Provide estimates for user stories and tasks on a sprint-by-sprint basis.
• Demonstrate proactive responsibility for the delivery of quality products.

Required Qualifications and Experience

• 7 to 10 years of hands-on experience in data engineering roles.
• Strong proficiency in Python programming.
• Experience with Microsoft Azure Cloud services.
• Working knowledge of Agile frameworks such as Scrum and Kanban.
• Competency with Apache Spark, PySpark, and Databricks platforms.
• Familiarity with DevOps pipelines, preferably Azure DevOps.

Preferred Credentials

• Bachelor's or master's degree in Computer Science, IT, Data Science, or related disciplines.
• Experience supporting operational data environments.
• Understanding and/or experience with AI and machine learning techniques.
• Relevant certifications in Data Engineering or allied areas.