Manage, monitor, and optimize cloud data platform environments including workspaces, clusters, pools, and jobs in Databricks or other big data systems
Job Summary
Manage, monitor, and optimize cloud data platform environments including workspaces, clusters, pools, and jobs in Databricks or other big data systems.
Configure role-based access control (RBAC), audit logging, and ensure compliance with data governance and security standards.
Develop CI/CD pipelines for notebooks, jobs, and libraries, and automate cluster lifecycle management and job scheduling.
Matching Summary
Manage, monitor, and optimize cloud data platform environments including workspaces, clusters, pools, and jobs in Databricks or other big data systems.
Skills & Requirements
Must-have
Cloud-based data platforms
Apache Spark and distributed computing
Databricks or other big data systems
CI/CD pipelines for notebooks
Python, SQL, and shell scripting
Role-based access control (RBAC)
Nice-to-have
Passion for enabling data teams
Collaborative and creative culture
Commitment to sustainability
Continuous improvement mindset
Key Requirements
Proven experience administering big data platforms
Experience with cloud platforms (Azure preferred)
Familiarity with Unity Catalog, Delta Lake, Lakehouse architecture
Experience with CI/CD tools and infrastructure-as-code
Excellent problem-solving and communication skills