Lead Engineer Bigdata - Pyspark

Workforcity

**
Pyspark for large-scale data processing
Python programming proficiency
Apache spark architecture
** The job posting is for a Lead Engineer specializing in Big Data and PySpark at Workforcity, seeking an experienced candidate to design and optimize data pipelines. The position requires extensive knowledge of Python and Spark technologies, along with strong problem-solving and communication skills. **

Job Summary

  • Design, develop, and maintain efficient, scalable, and reliable data pipelines using PySpark.
  • Collaborate with multiple stakeholders to understand data requirements and translate them into technical specifications.
  • Mentor junior developers and contribute to the continuous improvement of the team's technical capabilities and processes.

Matching Summary

Match Score: 75

** The job posting is for a Lead Engineer specializing in Big Data and PySpark at Workforcity, seeking an experienced candidate to design and optimize data pipelines. The position requires extensive knowledge of Python and Spark technologies, along with strong problem-solving and communication skills. **

Skills & Requirements

Must-have

  • PySpark for large-scale data processing
  • Python programming proficiency
  • Apache Spark architecture
  • Spark SQL and DataFrame API
  • distributed file systems
  • SQL and relational databases
  • version control systems (Git)

Nice-to-have

  • cloud platforms and services
  • workflow orchestration tools
  • streaming data processing
  • data warehousing concepts
  • containerization technologies
  • data governance and security

Key Requirements

  • 8-12 years of relevant experience
  • 5+ years of software development with Big Data
  • 5+ years of hands-on PySpark experience
  • Bachelor's or Master's degree

Work Rights

Not specified

Tailored Resume

Cover Letter