AWS Data Engineer

Job Summary

We are looking for a skilled AWS Data Engineer proficient in Databricks, PySpark, and cloud data engineering technologies. The candidate will be tasked with designing, developing, and managing scalable data pipelines, data lakes, and data warehouse solutions within the AWS ecosystem. This role demands extensive experience handling extensive datasets, enhancing data workflows, and supporting analytics and business intelligence functions.

Key Responsibilities

• Develop, maintain, and optimize scalable ETL/ELT data pipelines utilizing PySpark and Databricks.
• Create and enhance data ingestion processes for diverse structured and unstructured data sources.
• Manage and develop data lakes and warehouse architectures on AWS.
• Perform data transformation, cleansing, validation, and enrichment activities.
• Handle large datasets efficiently and fine-tune Spark jobs for improved performance and scalability.
• Integrate data from APIs, database systems, streaming platforms, and external systems.
• Collaborate with Data Scientists, Analysts, and business teams to comprehend data needs.
• Monitor, debug, and improve data pipeline functionality and reliability.
• Apply best practices for data quality, governance, security, and regulatory compliance.
• Engage in code reviews, architectural planning, and technical design discussions.

Essential Skills

• In-depth experience with AWS services such as S3, EMR, Glue, Lambda, IAM, Redshift, Athena, and CloudWatch.
• Proficient with Databricks and Apache Spark platforms.
• Advanced programming skills in PySpark and Python.
• Experience constructing ETL/ELT workflows and data integration methods.
• Strong expertise in SQL and relational database systems.
• Knowledge of Delta Lake, Spark SQL, and performance optimization techniques.
• Understanding of data modeling and data warehousing fundamentals.
• Familiarity with version control tools like Git.
• Awareness of CI/CD pipelines and automated deployment workflows.

Preferred Qualifications

• Working knowledge of Apache Kafka or similar streaming technologies.
• Experience with workflow orchestration tools such as Airflow or AWS Step Functions.
• Exposure to Snowflake or advanced Redshift implementations.
• Understanding of Infrastructure as Code concepts using Terraform or similar tools.
• Experience with machine learning data pipelines and MLOps practices.
• AWS certification is advantageous.

Education

• Bachelor's or Master's degree in Computer Science, IT, Engineering, or related disciplines.

Core Competencies

• Strong analytical thinking and effective problem resolution skills.
• Excellent communication and ability to manage stakeholder relationships.
• Capability to work well both autonomously and as part of a team.
• Dedicated focus on optimizing performance and maintaining high data quality standards.