ITCAN PTE. LIMITED is seeking a mid-to-senior level Data Engineer for a proof of concept (POC) project focused on Databricks on AWS. The ideal candidate should have extensive experience with Databricks, Apache Spark, and AWS infrastructure, particularly in building data pipelines and ensuring data governance
Job Summary
The role requires a mid-to-senior level Data Engineer to spearhead a Proof of Concept on AWS using Databricks.
You will be responsible for configuring the workspace integration, designing a scalable Medallion Architecture, and building automated ingestion pipelines.
The position demands hands-on expertise in optimizing Spark jobs for performance while managing costs through cluster configuration.
Matching Summary
Match Score: 85
ITCAN PTE. LIMITED is seeking a mid-to-senior level Data Engineer for a proof of concept (POC) project focused on Databricks on AWS. The ideal candidate should have extensive experience with Databricks, Apache Spark, and AWS infrastructure, particularly in building data pipelines and ensuring data governance.
Skills & Requirements
Must-have
3-5+ years Data Engineering experience
Expert Databricks and Delta Lake knowledge
Medallion Architecture implementation
Optimized PySpark and SQL coding
AWS S3, IAM, and VPC configuration
Databricks Autoloader or Unity Catalog usage
Nice-to-have
Experience with AWS Glue or EMR
VPC networking understanding
Scala or R programming skills
Databricks Certified Professional credential
POC success metric evaluation
Key Requirements
3-5+ years in Data Engineering
Strong experience with PySpark and SQL
Databricks Certified Data Engineer Professional preferred