Azure Databricks Engineer

Job Description

Azure & Databricks Platform Support Engineer (SSr to Sr)

Job Summary

We are seeking an experienced Azure & Databricks Platform Support Engineer to manage, monitor, and optimize enterprise data platforms built on Azure and Databricks. The role involves ensuring platform reliability, performance, security, and operational excellence while supporting data engineering workloads, analytics platforms, and business-critical data pipelines. We are looking for a strong candidate who can work independently with customer.

Key Responsibilities

• Monitor and support Azure Databricks workspaces, clusters, jobs, workflows, and Delta Lake environments.
• Provide L2/L3 support for Azure Data Platform services including Azure Databricks, Azure Data Factory (ADF), Azure Synapse, Azure Data Lake Storage (ADLS), Azure Key Vault, and Azure DevOps.
• Troubleshoot and resolve production incidents, job failures, performance bottlenecks, and data pipeline issues.
• Monitor scheduled workflows, triggers, pipelines, and batch processes ensuring SLA compliance.
• Implement proactive monitoring, alerting, logging, and observability using Azure Monitor, Log Analytics, and Application Insights.
• Manage platform security, RBAC, service principals, secrets, and data access controls.
• Support deployment automation through CI/CD pipelines using Azure DevOps.
• Perform platform maintenance activities including upgrades, patching, cluster optimization, and configuration management.
• Collaborate with upstream and downstream application teams, business users, and data engineering teams to resolve data integration issues.
• Create and maintain operational runbooks, knowledge articles, and support documentation.
• Drive continuous improvement initiatives focused on platform stability, performance tuning, and cost optimization.
• Support disaster recovery, backup strategies, and high-availability requirements.

Required Skills

• Strong hands-on experience with Azure Databricks, PySpark, Spark SQL, and Python.
• Experience supporting Azure Data Factory, Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL.
• Knowledge of Delta Lake, Unity Catalog, Databricks Workflows, and cluster management.
• Experience with monitoring and troubleshooting distributed data processing environments.
• Hands-on experience with Azure DevOps, Git, CI/CD pipelines, and Infrastructure as Code.
• Understanding of cloud security, identity management, networking, and governance in Azure.
• Experience with ETL/ELT frameworks and large-scale data processing.
• Strong analytical and problem-solving skills.

Preferred Qualifications

• Azure Data Engineer Associate (DP-203).
• Databricks Certified Data Engineer Associate/Professional.
• Microsoft Azure Administrator Associate.

Experience

• 3+ years of Data Platform  experience.
• 3+ years of hands-on Azure Databricks support and administration.
• Experience supporting production environments in a 24x7 or managed services model.

Job Description - Grade Specific

Has more than a year of relevant work experience. Solid understanding of programming concepts, software design and software development principles. Consistently works to direction with minimal supervision, producing accurate and reliable results. Individuals are expected to be able to work on a range of tasks and problems, demonstrating their ability to apply their skills and knowledge. Organises own time to deliver against tasks set by others with a mid term horizon. Works co-operatively with others to achieve team goals and has a direct and positive impact on project performance and make decisions based on their understanding of the situation, not just the rules.