Data Engineer

Role Overview

We are seeking a hands-on Data Engineer to design, build, and maintain scalable, reliable, and high-performance data pipelines on Microsoft Azure and big data ecosystems (Hadoop/Spark). You’ll work across ETL/ELT, dimensional modeling, data warehousing, data quality, and DevOps for data, enabling business intelligence, advanced analytics, and data science use cases at production scale.

Key Responsibilities

Data Architecture & Modeling

• Design dimensional models (star/snowflake), data vault (nice-to-have), and canonical data models for analytics and BI.
• Build and maintain enterprise data warehouses and data lakes/lakehouses with partitioning, indexing, and cost/performance optimization.

Pipelines & Orchestration

• Develop robust ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, Spark, and SQL.
• Implement batch and streaming data ingestion (e.g., Kafka/Event Hubs) with parallel computing and checkpointing.
• Build reusable frameworks for ingestion, transformation, and data quality validations.

Big Data Engineering

• Optimize Apache Spark jobs (in Scala/Python/SQL): caching, partitioning, join strategies, AQE, and cost-aware design.
• Manage Hadoop ecosystem components (HDFS, Hive/HCatalog, Oozie/Airflow, Ranger/Atlas basics).

Cloud, DevOps & Reliability

• Implement data solutions on Azure (ADLS Gen2, ADF, Databricks, Synapse, Azure SQL/MI, Functions, Key Vault, Monitor/Log Analytics).
• Apply DevOps practices: CI/CD (e.g., GitHub Actions/Azure DevOps), IaC (Bicep/Terraform), version control (Git), and configuration management.