Data Engineer (gen Ai)

Interactive Brokers Group, Inc.

New York, NY, United States
Base: $150,000 to $200,000 py; bonus/equity: annua...
On-site
Scalable data crawlers and etl/elt pipelines
Data lake platform infrastructure (s3, iceberg, aws glue)
Real-time streaming data pipelines using kafka
Design, build, and maintain scalable data pipelines, data lake platforms, and analytics solutions that support enterprise-wide AI initiatives

Job Summary

  • Design, build, and maintain scalable data pipelines, data lake platforms, and analytics solutions that support enterprise-wide AI initiatives.
  • Develop Python-based applications for data ingestion, processing, and integration supporting Gen AI RAG workflows and knowledge base systems.
  • Collaborate with internal development teams, data scientists, and stakeholders to architect appropriate data solutions and create comprehensive technical documentation.

Matching Summary

Design, build, and maintain scalable data pipelines, data lake platforms, and analytics solutions that support enterprise-wide AI initiatives.

Salary

Base: $150,000 to $200,000 per year; Bonus/Equity: annual performance-based bonus and stock grant; Benefits: health and wellness benefits, 401(k) with match, paid time off, parental leave, education reimbursement

Skills & Requirements

Must-have

  • Scalable data crawlers and ETL/ELT pipelines
  • Data lake platform infrastructure (S3, Iceberg, AWS Glue)
  • Real-time streaming data pipelines using Kafka
  • Python for data ingestion and Gen AI RAG workflows
  • AWS cloud services (S3, Glue, Athena, EMR, Lambda)
  • PySpark on EMR for large-scale data processing
  • CI/CD practices with Git and Docker

Nice-to-have

  • Cutting-edge data technologies
  • AI knowledge bases
  • Cloud-native data platforms
  • Emerging data engineering trends

Key Requirements

  • 6+ years of data engineering experience
  • Strong SQL skills
  • Experience with Kafka
  • Knowledge of data lake architectures
  • Experience with CI/CD practices
  • Understanding of data modeling concepts

Work Rights

Not specified

Tailored Resume

Cover Letter