Senior Data Engineer

Long Description

Job Description – Senior Data Engineer

Overview

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and platforms across On-Premise and Cloud ecosystems (Microsoft Azure preferred). The role requires hands-on expertise in modern data engineering tools, frameworks, and best practices.

Key Responsibilities

• Migrate data pipelines from the existing data acquisition framework to the new GDP data acquisition framework.
• Configure, develop, and deliver data ingestion scripts for loading data into the T1 data layer.
• Develop and manage ETL/ELT workflows, ensuring high standards of data quality, integrity, and reliability.
• Integrate and automate data quality checks and validation processes within data pipelines.
• Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.
• Work with modern data lake and data warehouse technologies, including Apache Iceberg.
• Implement real-time streaming solutions using Kafka.
• Orchestrate complex workflows using Apache Airflow.
• Integrate data pipelines with data catalog and governance tools, such as DataHub and Ranger.
• Collaborate with cross-functional teams to understand business requirements and deliver robust data solutions.
• Ensure security, compliance, and best practices in data management and governance.

Key Skills Required

• Strong proficiency in Linux, Python, and Shell scripting.
• Hands-on experience with Docker, Kubernetes, and container orchestration.
• Experience with MinIO and Azure Data Lake Storage (ADLS) using S3-compatible protocols.
• Expertise with:
• Apache Iceberg
• Kafka
• Apache Airflow
• DataHub
• Trino
• Ranger
• Proficiency in Java for data engineering tasks.
• Solid understanding of data modeling, data warehousing, and big data technologies.

  ---

Prior Experience

• Extensive experience building and maintaining data pipelines and ETL processes.
• Proven background implementing and integrating data quality frameworks.
• Strong experience executing migration projects, specifically transitioning legacy pipelines to a modernized tech stack.