Pyspark Data Engineer

About the Role

We are looking for an experienced Pyspark Data Engineer to join our team in Pune. The ideal candidate will have between 6 to 15 years of experience working on Big Data platforms and relevant technologies, contributing to data engineering projects with a strong focus on Pyspark and related tools.

Responsibilities

• Handle Big Data platforms including Hadoop, Spark, and Scala for data processing and engineering tasks.
• Work extensively with Hive, Impala, and SQL for querying and managing large datasets.
• Deploy and optimize Spark applications using Spark UI and debugging techniques.
• Develop and maintain Python scripts and applications as part of the data workflow.
• Execute data engineering activities involving Unix and HDFS command usage.
• Contribute effectively to data analytics projects utilizing Pyspark.
• Use intermediate SQL skills including subqueries, joins, and Common Table Expressions (CTEs).
• Leverage database technologies to manage and store data effectively.
• Work with AWS services such as EMR, S3, IAM, Lambda, SNS, and SQS for cloud data operations.
• Maintain clear and efficient communication within the team.

Requirements

• Proven expertise in Big Data platforms like Hadoop, PySpark, Scala, Hive, Impala, along with SQL and Python programming.
• Strong Python scripting abilities coupled with hands-on experience in Pyspark development.
• Familiarity with Spark performance tuning, debugging, and optimization techniques.
• Intermediate SQL capabilities encompassing subqueries, join operations, and CTEs.
• Knowledge of database technologies and cloud services including AWS EMR, S3, IAM, Lambda, SNS, and SQS.
• Comfortable with Unix operating system and HDFS commands.

Eligibility

Any graduate can apply for this position.