Data Catalogue Specialist (Microsoft Purview)

Role Overview

The Data Catalogue Specialist is tasked with deploying and overseeing Microsoft Purview as the enterprise-wide metadata management and data governance platform for ADC. This position involves establishing metadata cataloguing, glossary management, classification, lineage tracking, and data governance protocols over all included data assets, supporting formal requirements of DGE and DDA governance.

Key Responsibilities

• Administer Microsoft Purview by configuring the environment specifically for ADC's requirements.
• Develop collection hierarchies that correspond to ADC's business domains and the Government Information Model (GIM).
• Register and maintain all relevant data sources, including IBM DB2 Warehouse, Databricks Unity Catalog, and IBM DataStage.
• Manage roles such as asset owners and stewards, along with maintaining governance records.
• Conduct automated metadata scans using Purview connectors to gather technical metadata like asset identifiers, schemas, data types, nullability, column statistics, and refresh intervals.
• Integrate Databricks Unity Catalog for continuous metadata harvesting.
• Create and sustain the Microsoft Purview Business Glossary aligned with GIM, administering business terms, definitions, and stewardship assignments.
• Associate every pertinent data element with the appropriate glossary term and synchronize this information with the central Purview environment through APIs.
• Design and enforce classification rules and apply Microsoft Information Protection sensitivity labels, categorizing assets as public, internal, confidential, or secret.
• Oversee the exclusion processes for secret data and maintain necessary audit trails and notifications.
• Architect and validate comprehensive data lineage across ADC platforms including landing, staging, IDM, semantic layers, and Databricks Bronze, Silver, and Gold zones.
• Implement lineage integrations for Databricks using Purview features and utilize Atlas Process entities to configure DataStage lineage if applicable.
• Highlight Databricks data quality metrics within Purview, customizing asset attributes such as DQI Score, Rule Pass Rate, and Last Profiled Date to support quality assessments directly from the catalogue.
• Develop and maintain metadata management artifacts for DGE and DDA governance review, including templates, classification registers, lineage diagrams, and governance documents, managing review iterations and version controls.

Essential Qualifications and Expertise

• Proven expertise in the implementation and administration of Microsoft Purview.
• Experience in enterprise metadata management, data cataloguing, and managing business glossaries.
• Familiarity with Microsoft Information Protection and frameworks for data classification.
• Competence in configuring and validating enterprise data lineage.
• Understanding of Databricks, Unity Catalog, IBM DB2 Warehouse, and DataStage platforms.
• Knowledge of REST APIs and metadata integration methods.
• Strong skills in stakeholder communication and data governance methodologies.
• Desirable knowledge of government data governance and information management standards.

Key Deliverables

• A fully configured Microsoft Purview environment tailored for ADC.
• Comprehensive metadata catalog encompassing all designated assets.
• Business glossary aligned with GIM terminology.
• Accurate classification and sensitivity labelling registers.
• Complete end-to-end data lineage documentation.
• Integrated data quality metadata within the catalogue.
• Prepared artefacts for DGE and DDA reviews and approvals.