Data etl (apache airflow, aws step functions, apache nifi)
The Data Engineering role will be responsible for building and maintaining data processing pipelines and also the transformation and optimisation of data for analytical use
Job Summary
The Data Engineering role will be responsible for building and maintaining data processing pipelines and also the transformation and optimisation of data for analytical use.
We have opportunities for Data Engineers at every level within a team, so your Recruiter will discuss the great opportunities for development or challenges we offer based off your professional profile.
Benefits include a contributory pension scheme, life assurance, 25 days holiday plus statutory public holidays, and a company bonus scheme.
Matching Summary
The Data Engineering role will be responsible for building and maintaining data processing pipelines and also the transformation and optimisation of data for analytical use.
Skills & Requirements
Must-have
Python (PySpark, Pandas, PyArrow)
Distributed data processing (Apache Spark)
Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi)
Cloud services (AWS, Azure or GCP)
Messaging / Streaming (Kafka, AWS SQS)
SQL and NoSQL Storage (HDFS, Iceberg, Elastic, S3, Data Lake)
Containerisation and orchestration (Docker / Kubernetes / Openshift)