Utilizes software engineering principles to deploy and maintain fully automated data transformation pipelines that combine a large variety of storage and computation technologies
Job Summary
Utilizes software engineering principles to deploy and maintain fully automated data transformation pipelines that combine a large variety of storage and computation technologies.
Designs data products and data pipelines that are resilient to change, modular, flexible, scalable, reusable, and cost effective.
Collaborates with data scientists, data analysts, data architects and other stakeholders to understand data requirements and deliver high-quality data solutions.
Matching Summary
Utilizes software engineering principles to deploy and maintain fully automated data transformation pipelines that combine a large variety of storage and computation technologies.
Skills & Requirements
Must-have
Azure Data Factory
Azure Synapse
Azure Databricks
Azure Fabric
Azure Data Lake Storage Gen 2
Azure Blob storage
SQL DML
Big data technologies (Spark)
Nice-to-have
Learning agility
Technical Leadership
Consulting and managing business needs
Python programming
PySpark applications
File formats (Parquet, Delta, Avro)
Querying API endpoints
Azure DevOps pipelines
Ansible in Azure DevOps
Key Requirements
3-5 years experience
At least 2 years as Data Engineer or similar
Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience)
Proficiency in programming fundamentals
Strong knowledge of Microsoft Azure services
Experience utilizing SQL DML
Strong understanding of Software Engineering principles