**
KAISHI PARTNERS PTE. LTD. is seeking a Web Crawling Engineer to join their team focused on building advanced AI infrastructure for web crawling and indexing. The ideal candidate will have experience in scalable backend systems and knowledge of modern web technologies, particularly in distributed systems and browser automation.
**
Job Summary
The role involves designing and building internet-scale crawling infrastructure capable of processing hundreds of millions of webpages daily.
Engineers will work on complex problems including intelligent crawl scheduling, JavaScript-heavy site rendering, and anti-bot detection.
This position sits at the intersection of distributed systems, performance engineering, and large-scale data infrastructure for AI-native search.
Matching Summary
Match Score: 75
**
KAISHI PARTNERS PTE. LTD. is seeking a Web Crawling Engineer to join their team focused on building advanced AI infrastructure for web crawling and indexing. The ideal candidate will have experience in scalable backend systems and knowledge of modern web technologies, particularly in distributed systems and browser automation.
**
Skills & Requirements
Must-have
Distributed web crawling systems
High-performance backend languages
Browser automation tooling
Nice-to-have
Chrome DevTools Protocol experience
Anti-bot detection evasion
Kubernetes cloud infrastructure
Key Requirements
Strong experience in scalable backend or distributed systems
Prior experience with web crawlers or scraping infrastructure