नौकरी का विवरण

Overview

Join a motivated and results-driven team at a leading financial institution committed to fostering an inclusive, high-performing culture. We seek a skilled Data Engineer to architect and optimize scalable data pipelines using Apache Spark technologies, primarily running on Azure Databricks.

Key Responsibilities

Design, develop, and maintain expansive Spark applications utilizing Python, Scala, and Java.
Create and refine batch and streaming data workflows within distributed computing environments.
Develop reliable, production-grade Spark code emphasizing high performance and scalability.
Tune Spark jobs including adjusting partitioning, caching, shuffle operations, memory settings, and execution strategies.
Build reusable Spark utilities, frameworks, and libraries.
Handle structured and semi-structured data formats such as Parquet, Delta, CSV, and JSON.
Collaborate closely with platform teams, data analysts, and data scientists supporting downstream analytics and data product needs.
Identify and resolve production issues including job failures and performance bottlenecks.
Adhere to best practices for coding standards, testing methodologies, logging, and technical documentation.

Candidate Qualifications

Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.
Minimum of 3 years’ professional experience as a Data Engineer or a related position.
Expertise in working hands-on with Apache Spark.
Proficiency in one or more programming languages: Python, Scala, or Java.
Strong knowledge of distributed systems and data processing principles.
Experience with SQL for complex data transformations and analysis.
Familiar with data lake technologies and columnar storage formats like Parquet and Delta Lake.
Understanding of Git version control and software engineering workflows.
Excellent analytical problem-solving abilities and meticulous attention to detail.
Experience running Spark workloads on Azure Databricks is a plus.
Background migrating Spark jobs from on-premises Hadoop/Cloudera ecosystems to cloud platforms is advantageous.
Familiarity with orchestration tools such as Airflow or Azure Data Factory is preferred.
Knowledge of cloud storage solutions like ADLS Gen2 and fundamentals of cloud security is desirable.

Benefits and Inclusion

Commitment to Diversity, Equity, Inclusion, and Allyship ensuring a respectful, inclusive workplace encouraging every employee’s growth.
Accessibility initiatives and accommodations for individuals requiring support during recruitment and employment.
Opportunities for professional development including online learning, cross-department exposure, and tuition assistance.
Competitive compensation with bonus schemes, flexible vacation, personal and sick leave, and benefits starting on the first day.
Work environment featuring amenities such as complimentary tea and coffee, universal washrooms, and ample collaborative space.
Engagement programs promoting community involvement regardless of workplace location.

Additional Information

This position is located in Toronto, Ontario, Canada. The company values inclusivity and encourages candidates who require accommodations during the recruitment process to inform the recruitment team. Only shortlisted applicants will be contacted.

Data Engineer

आप कहाँ काम करेंगे