Data Engineering Consultant

Role Overview

The Data Engineering Consultant is responsible for designing, constructing, and maintaining robust data pipelines, integrations, and data services to support client analytics solutions. This role translates approved architecture, business needs, and governance requirements into well-tested technical elements that handle data transfer, transformation, and delivery while ensuring quality, metadata accuracy, lineage, security, and operational tracking.

Key Responsibilities

• Review and analyse approved requirements including source systems, destination data models, data volumes, scheduling, control mechanisms, and service standards.
• Create batch, streaming, or API-based data ingestion and transformation pipelines.
• Develop source-to-target mapping, apply business logic transformations, reference data handling, and standardize data according to rules.
• Incorporate validation steps, reconciliation procedures, exception management, and data quality checks within engineering workflows.
• Document technical metadata, data lineage, schedule dependencies, pipeline ownership, and operational details for all built pipelines.
• Enforce security rules including privacy, classification, access control, encryption, and logging as per guidelines.
• Develop automated tests including unit, integration, regression, and data validation with evidence of test runs.
• Enhance pipeline performance, scalability, reliability, and cost-effectiveness within approved platform and architecture constraints.
• Manage deployment automation, version control systems, configuration management, and controlled release procedures.
• Monitor pipeline processes, diagnose failures, and assist in resolving defects throughout implementation and transition phases.
• Prepare comprehensive technical documentation including specifications, code comments, operational procedures, runbooks, and handover documentation.
• Guide client engineering teams through design rationales, operational workflows, and system limitations prior to handover.

Education and Experience Requirements

• Bachelor's degree in computer science, software or data engineering, or a related discipline; or equivalent education combined with relevant consulting or implementation experience.
• Between 3 and 8 years of professional experience in data engineering, integration, ETL/ELT, software development, or cloud-based data platform environments.
• Proven expertise building and testing production-level data pipelines, APIs, or data-processing services.
• Hands-on experience with version control systems, deployment automation, monitoring, and operational support.
• Experience collaborating in multidisciplinary teams interpreting architecture, security, and data quality requirements.

Technical Skills and Knowledge

• Understanding of data ingestion, transformation, and orchestration techniques.
• Proficiency in SQL and programming or scripting languages relevant to data engineering.
• Familiarity with batch processing, streaming, API, and integration technology stacks.
• Experience with cloud-based or enterprise data platforms and storage solutions.
• Expertise in data quality assurance, reconciliation methods, exception handling, and test automation.
• Knowledge of metadata management, data lineage, source-to-target mapping, and thorough documentation practices.
• Implementation of security protocols including access control, encryption, logging, and secrets management.
• Competency in version control, continuous integration and delivery (CI/CD), configuration management, and operational monitoring tools.

Core Competencies

• Strong skills in SQL and programming.
• Expertise in developing and managing data pipelines.
• Experience with APIs and data integration.
• Knowledge of cloud and data platform environments.
• Skills in testing methodologies and system observability.
• Enforcement of security and data quality measures.
• Engineering discipline with focus on reliability and problem-solving.
• Automation-oriented mindset.
• Aptitude for clarifying technical requirements and identifying decisions early in projects.
• Ability to estimate engineering efforts, dependencies, and project risks.
• Capability to communicate technical designs and issues to diverse audiences.
• Proficiency in producing maintainable documentation and verifiable test results.
• Collaboration with client engineers, vendors, and cross-functional teams.
• Experience working under controlled client access and change management processes.
• Skills in transferring engineering knowledge and supporting operational handover procedures.

Preferred Certifications

• Google Cloud Professional Data Engineer or equivalent cloud data certification.
• Certified Data Management Professional (CDMP) at Associate or Practitioner level.