Data Migration Developer

Role Overview

This position is for a data migration specialist who will help modernize enterprise data platforms by moving workloads from Amazon Redshift and AWS Glue into Databricks. The role focuses on rebuilding and improving ETL/ELT processes, protecting data quality, and ensuring the migrated solution performs reliably at scale.

Key Responsibilities

• Review existing data pipelines, ETL processes, and warehouse objects currently running in Amazon Redshift and AWS Glue.
• Plan and execute migration approaches for transferring data and workloads to Databricks.
• Build, tune, and support ETL/ELT pipelines using Databricks, Apache Spark, and Python.
• Rework SQL scripts, stored procedures, and transformation logic so they function effectively in Databricks-based solutions.
• Check data accuracy, completeness, and consistency before migration and after cutover.
• Improve Databricks data models, partitioning strategy, and overall processing performance.
• Apply Delta Lake practices for storage efficiency, version control, and optimization.
• Work closely with architects and business stakeholders to ensure the solution aligns with requirements.
• Investigate migration issues and resolve performance bottlenecks.
• Prepare technical documentation, migration playbooks, and handover material.
• Maintain compliance with security, governance, and regulatory standards during the migration process.

Requirements

• A bachelor’s degree in Computer Science, Information Technology, Engineering, or a similar discipline.
• At least 4 years of experience in data engineering or data platform migration work.
• Practical expertise with Amazon Redshift, AWS Glue, Databricks, Apache Spark, Python, and SQL.
• Working knowledge of cloud environments, ideally AWS.
• Strong understanding of data warehousing principles and dimensional modeling.
• Hands-on experience with Delta Lake and Lakehouse architecture.
• Exposure to ETL/ELT frameworks and workflow orchestration tools.
• Experience using Git and CI/CD pipelines.
• Good analytical thinking and troubleshooting ability.
• Strong written and verbal communication skills, including documentation.
• Preferred exposure to Databricks Workflows, Unity Catalog, and Delta Live Tables.
• Familiarity with Apache Airflow or similar orchestration tools is an advantage.
• Experience with Infrastructure as Code tools such as Terraform or CloudFormation is a plus.
• AWS and/or Databricks certifications are beneficial.
• Background in large-scale enterprise data migrations is preferred.
• Minimum overall experience required: 4 to 8 years in data engineering.
• Relevant migration experience required: at least 2 years moving cloud data platforms to Databricks.

Technical Focus

• Cloud platform: AWS
• Data warehouse: Amazon Redshift
• ETL tool: AWS Glue
• Target platform: Databricks
• Programming languages: Python and SQL
• Processing frameworks: Apache Spark and PySpark
• Storage layer: Delta Lake
• Version control: Git
• CI/CD tools: Azure DevOps, GitHub Actions, or Jenkins

Success Measures

• Move Redshift databases and AWS Glue ETL jobs successfully into Databricks.
• Deliver accurate reconciliations with very low data mismatch rates.
• Improve performance and scalability of pipelines on the Databricks Lakehouse platform.
• Provide complete migration documentation and support a smooth operational transition.
• Finish the migration within the planned schedule while keeping business disruption to a minimum.

Employment Details

This is a full-time or contract opportunity depending on project needs.

The position is based in Hyderabad, India.

Additional Information

This opportunity is best suited to a data engineer who has worked on cloud data migrations and modernized enterprise platforms using Databricks.