Develop and maintain the data infrastructure for Data AI platforms, including hands-on development and data pipeline creation
Job Summary
Develop and maintain the data infrastructure for Data AI platforms, including hands-on development and data pipeline creation.
Collaborate with stakeholders across technology and business units to understand data needs and translate them into technical solutions.
Contribute to the firmwide Artificial Intelligence (AI) Development Platform, aligning with technology principles to drive efficiency, consistency, and innovation.
Matching Summary
Develop and maintain the data infrastructure for Data AI platforms, including hands-on development and data pipeline creation.
Skills & Requirements
Must-have
Develop and maintain data pipelines
Optimize data systems for performance
Implement data quality and governance
Proficiency in Python programming
Experience with Apache Spark or Hadoop
Knowledge of SQL and NoSQL databases
Experience with Snowflake and Databricks
Familiarity with cloud platforms (AWS, Azure)
Experience with message queues and streaming platforms
Experience with version control systems (Git)
Nice-to-have
Familiarity with data visualization tools
Familiarity with data governance and security
Experience with Agile methodologies
Familiarity with data catalog and metadata management
Familiarity with CI/CD pipelines and DevOps practices
Key Requirements
8 years+ of experience in data engineering
Knowledge of data modeling and architecture
Experience with data warehousing concepts
Experience using Jupyter notebooks
Excellent communication and collaboration skills
Ability to work independently and in a distributed team