Senior Data Engineer – Databricks

IDC TECHNOLOGIES (SINGAPORE) PTE. LTD.

Singapore, Singapore
Not specified
Pyspark development experience
Databricks platform expertise
Delta lake optimization skills
IDC Technologies (Singapore) is seeking a Senior Data Engineer with extensive experience in Databricks, PySpark, and data pipeline development. The ideal candidate will have a strong background in data engineering principles, a commitment to quality, and the ability to work collaboratively in a cross-functional environment

Job Summary

  • The role involves designing and operating scalable data pipelines on the Databricks platform with a focus on performance and reliability.
  • Candidates will lead the modernization of legacy ETL processes into modern ELT patterns using PySpark and Delta Lake.
  • This position requires implementing robust error handling, monitoring mechanisms, and ensuring data integrity during complex migrations.

Matching Summary

Match Score: 85

IDC Technologies (Singapore) is seeking a Senior Data Engineer with extensive experience in Databricks, PySpark, and data pipeline development. The ideal candidate will have a strong background in data engineering principles, a commitment to quality, and the ability to work collaboratively in a cross-functional environment.

Skills & Requirements

Must-have

  • PySpark development experience
  • Databricks platform expertise
  • Delta Lake optimization skills
  • Legacy code modernization
  • Production pipeline reliability

Nice-to-have

  • Cloud platform knowledge Azure AWS GCP
  • Streaming data processing experience
  • DevOps and CI/CD familiarity
  • Mentoring junior engineers
  • Cross-functional collaboration skills

Key Requirements

  • Minimum 8 years in data engineering
  • 2-3 years hands-on Databricks experience
  • Databricks Certified Data Engineer Associate or Professional
  • Strong Python programming skills
  • Experience refactoring legacy codebases

Work Rights

Not specified

Tailored Resume

Cover Letter