Associate Data Engineer

About the Role

The New York Power Authority (NYPA) is dedicated to leading New York toward a sustainable, resilient, and economically thriving future through clean energy solutions. We are seeking an Associate Data Engineer to join our expanding Data Engineering team as we enhance our data modernization, artificial intelligence deployment, and cloud-centric product development initiatives. This position involves hands-on creation and maintenance of scalable data pipelines that support NYPA’s AI platforms, enterprise analytics, and data products. Working collaboratively with senior engineers and business teams, you will translate needs into production-ready data solutions.

Key Responsibilities

• Design, build, improve, and troubleshoot scalable data pipelines and system integrations.
• Convert business and technical requirements into robust data engineering implementations.
• Partner closely with lead data engineers and interdisciplinary teams, including AI specialists.
• Support the creation of data products aligned with NYPA's analytics and AI goals.
• Utilize cloud-based and distributed data processing tools to ensure dependable, high-quality data delivery.
• Develop elastic, secure, flexible, and reliable large-scale data solutions.
• Guide project teams to adhere to coding standards and data engineering best practices alongside lead engineers.
• Work with Data Governance to manage metadata and implement data quality rules.
• Construct and manage data pipelines, data products, integrations, and drive production releases.
• Create application integrations, APIs, and microservices utilizing hybrid cloud infrastructure.
• Stay current with trends in data/application integration, cloud technologies, containerization, and related fields.
• Assist stakeholders across product, data, and business units with technical data challenges and infrastructure needs.
• Adhere to cybersecurity standards to safeguard data privacy and security.
• Build batch pipelines for structured and semi-structured data and support ingestion of unstructured data.
• Implement fundamental data quality validations such as schema and null checks.
• Help prepare datasets suitable for AI, machine learning, and generative AI analytics.
• Support implementation of data contracts through schema validation and integrity checks.

Required Qualifications and Skills

• Bachelor’s degree in Management Information Systems, Computer Science, Engineering, or a related field is mandatory.
• At least two years of experience in data engineering roles.
• Proficient in building data pipelines with tools such as Python, PySpark, Azure Data Factory, and Databricks.
• Hands-on experience with traditional and cloud data management systems like MS SQL, RDS, Athena, or equivalents.
• Working knowledge of metadata-driven ingestion frameworks and data pipeline construction.
• Familiarity with DevOps practices and Agile methodology.
• Strong analytical aptitude and understanding of cloud security principles.
• Exposure to data governance practices and quality assurance tools.
• Experience with ETL/ELT orchestration, data integration using APIs, messaging, or services.
• Basic comprehension of AI/ML data requirements and data formats including structured, semi-structured, and unstructured.
• Understanding of streaming data concepts, parsing, and text processing.
• Knowledge of data quality and observability frameworks.
• Experience developing microservices, serverless components, or distributed data platforms.
• Practical experience scripting and programming for data engineering tasks.
• Cloud certifications are advantageous.

Additional Information

This role may require occasional travel. NYPA offers a competitive salary range from $102,000 to $128,000 annually, dependent on experience and qualifications. The organization is strongly committed to creating an inclusive and diverse workforce, encouraging applications from underrepresented groups and providing reasonable accommodations as needed during the hiring process.