The role involves developing complex data pipelines for ingestion, processing, and transformation of large data volumes within the Cyber & Security Solutions area
Job Summary
The role involves developing complex data pipelines for ingestion, processing, and transformation of large data volumes within the Cyber & Security Solutions area.
Candidates will implement real-time stream processing applications using Apache Flink and Kafka with exactly-once semantics and state management.
The position requires orchestrating workflows with Apache Airflow and integrating streaming data into a unified lakehouse architecture for analytics.
Matching Summary
The role involves developing complex data pipelines for ingestion, processing, and transformation of large data volumes within the Cyber & Security Solutions area.
Skills & Requirements
Must-have
Apache Spark PySpark Scala
Apache Kafka event streaming
Apache Flink stream processing
Apache Airflow workflow orchestration
Advanced SQL and dbt transformations
Delta Lake or Iceberg lakehouse
Data modeling dimensional star schema
Nice-to-have
Tableau or PowerBI integration
CI/CD DataOps practices
Prometheus Grafana monitoring
Docker Kubernetes containerization
Confluent Schema Registry
Great Expectations data quality
Continuous learning mindset
Key Requirements
Master's degree in Engineering, Math, Statistics, Physics, or Computer Science
2 to 5 years of experience in Data Engineering roles