Design, develop and maintain data pipelines using Python and SQL programming language on GCP, including API development and integration, and near real-time streaming data
Job Summary
Design, develop and maintain data pipelines using Python and SQL programming language on GCP, including API development and integration, and near real-time streaming data.
Implement CI CD pipeline using GitHub Action, monitor and troubleshoot data pipelines, and ensure team collaboration using Jira, Confluence, and other tools.
The company offers a flexible scheme with benefits including best-in-class leave policy, gender-neutral parental leaves, childcare assistance, industry certifications sponsorship, and comprehensive insurance.
Matching Summary
Design, develop and maintain data pipelines using Python and SQL programming language on GCP, including API development and integration, and near real-time streaming data.
Skills & Requirements
Must-have
GCP Data Engineer
Python and SQL programming
API development and integration
streaming data pipelines
near real-time services (Pub Sub/Kafka)
Docker (GCP Cloud Run)
Terraform/Hashicorp
CI CD pipeline using GitHub Action
Nice-to-have
willingness to accept failures
Agile methodologies
continuous learning
team collaboration
problem-solving skills
prompt engineering
Key Requirements
10+ years of IT experience
Proficient in Python
Proficient in SQL
Hands-on experience on GCP Cloud Composer, Data Flow, Big Query, Cloud Function, Cloud Run, Google ADK
Proficient in Terraform/ Hashicorp
Experienced in GitHub and Git Actions
Experienced in CI-CD
Professional Google Cloud Data engineer certification (added advantage)