The role requires strong expertise in developing data ingestion and transformation pipelines using Databricks notebooks and Azure Data Factory
Job Summary
The role requires strong expertise in developing data ingestion and transformation pipelines using Databricks notebooks and Azure Data Factory.
Candidates must demonstrate the ability to manage and mentor a team of data engineers while explaining complex technical concepts to non-technical stakeholders.
The position involves building end-to-end analytics solutions by integrating Databricks with Azure Synapse, ADLS Gen2, and Power BI.
Matching Summary
Match Score: 85
The role requires strong expertise in developing data ingestion and transformation pipelines using Databricks notebooks and Azure Data Factory.
Skills & Requirements
Must-have
Azure Databricks cluster management
Delta Lake and Spark proficiency
Azure Data Factory and Synapse experience
ETL/ELT pipeline development
CI/CD and Git version control
Cloud security best practices
Nice-to-have
Power BI dashboarding and reporting
Microsoft Fabric integration tasks
Spark Streaming for real-time data
ARM templates for infrastructure as code
Leadership and team mentoring skills
Excellent communication with stakeholders
Key Requirements
Demonstrated expertise in Databricks/Synapse notebook development
Hands-on experience with Delta tables and Auto Loader
Proficiency in optimizing query layers using Databricks SQL
Experience deploying modern CI/CD practices in cloud-native environments