Lead Data Engineer / Engineering Manager

Role Overview

We are seeking a Lead Data Engineer / Engineering Manager to design, develop, and maintain scalable data engineering solutions using Databricks and supporting technologies in a dynamic IT MNC environment. This position is located in Pune, India, and involves both technical leadership and stakeholder management responsibilities.

Data Engineering Responsibilities

• Create and maintain scalable data pipelines leveraging Databricks platform capabilities.
• Build comprehensive ETL/ELT workflows with PySpark, Python, and SQL for batch and streaming data processes.
• Enhance performance and cost-efficiency of Spark jobs and Databricks workloads.
• Apply Delta Lake best practices and implement Medallion Architecture to improve data reliability.
• Develop reusable engineering frameworks to uphold high coding and design standards.
• Modernize legacy ETL processes by transitioning them into Databricks-centric architectures.
• Monitor production environments proactively to ensure system stability and manage alerting and troubleshooting procedures.

Technical Leadership Duties

• Set and enforce technical standards and best engineering practices within the team.
• Conduct code and design reviews to maintain quality and consistency.
• Collaborate closely with Architects, Product Owners, Business Units, and Platform Teams to align on goals and implementations.
• Mentor junior engineers, guiding their technical growth and problem-solving skills.
• Champion continual improvement of engineering methodologies and workflows.

Delivery and Stakeholder Management

• Take ownership of data engineering project deliveries from planning to production release.
• Manage project schedules, sprint planning, and identification of technical risks.
• Maintain transparent and regular communication with stakeholders regarding project progress.
• Prepare environments for production readiness ensuring smooth and successful deployments.

Preferred Leadership Experience

• Lead and nurture Data Engineering teams, facilitating hiring, onboarding, and professional development.
• Oversee technical reviews and mentor engineering personnel toward excellence.
• Prior experience managing engineering teams is advantageous but not essential.

Technical Skills and Experience

• Proficiency in Databricks Workspace, Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, Structured Streaming, and Databricks Workflows/Jobs.
• Strong programming skills in PySpark, Python, SQL, and Spark SQL.
• Hands-on expertise in designing ETL/ELT processes, building data pipelines, data warehousing, and data modeling for both batch and streaming data.
• Significant knowledge and experience with cloud platforms such as Azure or AWS; familiarity with GCP is a bonus.
• Experience with cloud storage services like ADLS and S3 and DevOps tools including Git, CI/CD pipelines, Azure DevOps, or GitHub.
• Familiarity with complementary technologies such as MLflow, Terraform, Photon Engine, Kafka/Event Hubs, AI/GenAI, and concepts like Data Mesh/Data Fabric is desirable.

Required Experience

• Minimum of 8 to 15 years in Data Engineering roles.
• At least 4 years of hands-on experience working with Databricks.
• Extensive experience with PySpark, Python, SQL, and Spark for enterprise-scale data platform projects.
• Demonstrated expertise in modern Data Lakehouse architectures and ETL/ELT toolsets.
• Solid background in Agile delivery methodologies and cloud platform usage (Azure or AWS).
• Strong understanding of Delta Lake optimizations and performance tuning.

Preferred Qualifications

• Previous experience managing teams or engineering management responsibilities.
• Proven ability to handle multiple projects and diverse stakeholder groups.
• Experience providing mentorship to technical teams.

Mandatory Certifications

• Databricks Certified Data Engineer Associate or Professional certification is required.

Preferred Certifications

• Azure Data Engineer Associate, Azure Solutions Architect, AWS Certified Data Analytics, or Google Professional Data Engineer certifications are advantageous.

Soft Skills

• Excellent communication capabilities for clear stakeholder engagement.
• Strong analytical and problem-solving mindset.
• Effective stakeholder management and team collaboration skills.
• Leadership ability with focus on mentoring and ownership mentality.
• Experience in client-facing roles and agile work environments.

Eligibility

Candidates should hold a graduate degree to apply for this position.