Engineering Lead / Azure Solution Architect

Role Overview

We are looking for a Senior Azure Solution Architect / Engineering Lead to lead the design and implementation of cutting-edge data platforms on Azure. This role demands deep expertise in cloud-native big data solutions, real-time and batch data processing, and scalable analytics architectures along with significant leadership experience guiding large teams.

Technical Expertise

• Expertise in Azure Databricks lakehouse architecture, including Delta Lake and Unity Catalog.
• Proficiency with Azure Synapse Analytics including both dedicated and serverless pools.
• Experience orchestrating ELT/ETL pipelines using Azure Data Factory (ADF).
• Advanced skills in PySpark, Apache Spark (Core, SQL, Structured Streaming) performance tuning and optimization.
• Development experience with Python and Java, particularly focused on data engineering, Spark internals, and Kafka consumer/producer implementations.
• Familiarity with Apache Kafka for real-time data ingestion and streaming workflows.
• Strong knowledge of Lakehouse, Lambda, and Kappa data architectures, dimensional modeling, data governance, security practices, and data lineage.
• Competence in building scalable, fault-tolerant, and cost-efficient big data systems.

Responsibilities and Achievements

• Architect and deliver enterprise-scale Azure Lakehouse platforms managing petabytes of data.
• Design real-time streaming solutions using Kafka integrated with Spark Structured Streaming.
• Lead migrations of on-premises data warehouses to modern Azure platforms, enhancing performance and reducing operating costs.
• Create reusable PySpark frameworks for streamlined data ingestion, transformation, and validation processes.
• Increase Spark job efficiency through strategic partitioning, caching mechanisms, and tuning memory usage.
• Implement comprehensive ADF data pipelines deploying CI/CD practices and dynamic orchestration parameters.
• Mentor technical teams in architecture and development best practices, overseeing code reviews and quality standards.
• Collaborate cross-functionally with business stakeholders to craft scalable technical designs that meet analytics needs.

Leadership and Soft Skills

• Provide technical leadership and make key architectural decisions.
• Coordinate effectively across teams and maintain clear communication with stakeholders.
• Review designs and enforce high code quality, promoting best practices.
• Participate in Agile/Scrum methodologies for project execution within large enterprises.
• Manage team recruitment, governance, and mentorship of over 40 staff to drive innovation and achieve business goals.
• Function as the Module Lead for operations in India, collaborating closely with North American teams.

Additional Qualifications

Hands-on experience with CI/CD tools such as Jenkins and GitHub and observability solutions like Dynatrace. Exposure to Java, Kafka, Cassandra, microservices, APIs, Apache Flink, ZeroMQ, and Protocol Buffers is advantageous. Experience in capital markets and securities processing domains will be a plus.

Eligibility

Open to any graduate candidates qualified for this role.