Azure Data Tech Lead

Role Overview

We are seeking a skilled Azure Data Engineer to support client’s data initiatives on the Microsoft Azure platform. The role involves building and maintaining data pipelines, creating data marts, and enabling data quality checks using Azure Data Factory and Azure Databricks.

Works in the area of Software Engineering, which encompasses the development, maintenance and optimization of software solutions/applications. 1. Applies scientific methods to analyse and solve software engineering problems. 2. He/she is responsible for the development and application of software engineering practice and knowledge, in research, design, development and maintenance. 3. His/her work requires the exercise of original thought and judgement and the ability to supervise the technical and administrative work of other software engineers. 4. The software engineer builds skills and expertise of his/her software engineering discipline to reach standard software engineer skills expectations for the applicable role, as defined in Professional Communities. 5. The software engineer collaborates and acts as team player with other software engineers and stakeholders.

Responsibilities

•    Data Ingestion & Processing: Ingest structured/unstructured data (batch/real-time) into the Azure Data Lake and perform data cleaning and aggregation.
•    Data Mart Development: Collaborate with business stakeholders to design and create data marts for reporting and analytics.
•    ETL Pipeline Management: Build and maintain data pipelines to transform and load data from Azure Databricks into Azure Synapse Data Warehouse models.
•    Data Governance: Create and manage data marts, data catalogs, and data lineage documentation.
•    Data Quality & Timeliness: Implement data quality checks and schedule data pipelines to ensure timely and accurate data delivery.

Requirements

•    Data Engineering: Minimum 3 years of hands-on experience as a Data Engineer.
•    Azure Expertise: 2+ years of experience with Microsoft Azure data tools: 
o    Azure Data Factory (ADF): ETL/ELT pipeline creation.
o    Azure Databricks: Data processing with PySpark/Scala.
o    Azure Synapse Analytics/Data Warehouse: Data modeling and querying.
•    Programming: Proficiency in Python and Spark (Scala or Python) for batch and real-time processing.
•    Data Modeling: Strong understanding of data warehouse concepts (Star and Snowflake schema) and dimensional modeling techniques.
________________________________________
Preferred Skills (Good-to-Have):
•    Experience with Data Cataloging and Data Lineage tools.
•    Knowledge of Azure Data Lake Storage (ADLS).
•    Familiarity with CI/CD pipelines for data workflows.
•    Understanding of SQL Server Integration Services (SSIS) or other ETL tools.

Skills (competencies)