- 経験
- 4–6 yrs
- 給料
- 年間150万～250万インドルピー
- 求人情報
- 1
- 投稿済み
- 5時間前
- 作業モード
- 在任中
- 教育
- B.Tech / B.E. in Any Specialization
- 資格
- Candidates holding a B.Tech or B.E. degree in any specialization are eligible to apply.
- 再開する
- 応募必須
勤務地
仕事内容
About Nineleaps
Nineleaps applies artificial intelligence strategies to empower global organizations by transforming data into impactful outcomes across product and platform engineering domains. The company focuses on resolving client challenges through the creation of high-performance business applications, leveraging Cloud, AIOps, the NineX development platform, and proprietary frameworks. Its clientele includes global startups and Fortune 500 firms. Nineleaps fosters the growth of talented professionals into industry leaders and offers a stable, rewarding career environment that supports continuous evolution and employee satisfaction.
Role Overview
We seek a seasoned Data Engineer with between four and six years of practical experience to architect, develop, and sustain scalable, fault-tolerant batch and real-time data pipelines. The role involves utilizing Databricks, PySpark, Python, SQL, Apache Airflow, and Apache Hive to manage complex datasets and produce enterprise-grade data products.
Key Responsibilities
- Develop, implement, and maintain production-level ETL/ELT data pipelines using PySpark and Python on the Databricks platform.
- Create, schedule, and manage intricate Directed Acyclic Graphs (DAGs) with Apache Airflow to automate and monitor data integration workflows.
- Oversee Delta Lake and Hive Metastore architectures applying Medallion Architecture principles (Bronze, Silver, Gold layers) for optimal data structuring and governance.
- Identify and resolve performance issues in Apache Spark jobs through tuning partition strategies, caching, memory settings, and custom Spark SQL query optimization.
- Construct complex, efficient SQL queries for data transformations, aggregation analysis, and validation tasks.
- Implement automated data quality frameworks, including alerting and exception management, employing tools such as Great Expectations or Delta Live Tables (DLT).
- Collaborate closely with Data Analysts, Data Scientists, and Business Stakeholders to collect requirements, enforce version control using Git, and facilitate CI/CD deployment pipelines.
Qualifications and Experience
- Possesses 4 to 6 years of relevant experience in roles like Data Engineer, Big Data Engineer, or Analytics Engineer.
- Holds a bachelor's degree in Computer Science, Information Technology, Software Engineering, or a similar discipline.
Technical Expertise
- Advanced programming skills in Python and extensive experience with distributed data processing using PySpark (including DataFrames and RDDs).
- Proficient with Databricks features such as Workspaces, Delta Lake, Auto Loader, Unity Catalog, and Delta Live Tables (DLT).
- Expert-level command over SQL including creating, debugging, and tuning sophisticated queries with CTEs, window, and analytical functions.
- Hands-on experience building custom DAGs, operators, sensors, and managing dependencies with Apache Airflow.
- Deep understanding of Apache Hive architecture encompassing HQL, metastore and schema management, and partitioning techniques.