E
Exa
Software Engineer (Web Crawling)
Skills
RustTypescriptPlaywrightWeb DesignJavascript
2 months ago
Share this opportunity
Share this opportunity to other talents of your network:
✓ Offer them a visibility boost with clients.
✓ Help your contacts find their next job.
Important information
Contract type:
Permanent
Salary:
$150,000 – $300,000 / year
Location:
San Francisco, CA, USA
Work mode:
Onsite
Published on:
29 August 2026
What they need
- As a Web Crawler engineer, you’d be responsible for crawling the entire web. Basically build Google-scale crawling!
- Build a distributed crawler that can handle 100M+ pages per day
- Optimize crawl politeness and rate limiting across thousands of domains
- Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
- Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency- You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
- You care about the problem of finding high quality knowledge and recognize how important this is for the world
- You have experience with some high performance language (C++, Rust, etc.)
- You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
- You’re comfortable optimizing a system to an exceptional degree
Other offers great for you!
These companies are also looking for great profiles
N
NGC x OPTIMA
Ingénieur Logiciel embarqué
Freelance
Cergy, France
Hybrid, Onsite
Skills
AUTOSARBSW
1 month ago
Quick apply
S
Salutech
Ingénieur Logiciel IA
Freelance
Paris, France
Onsite, Hybrid
Skills
DockerPythonTesting & QAKubernetesSoftware refactoringCI/CDObservabilityAsynchronous programming
7 days ago
Quick apply
A
Agile Partner
Full-Stack Software Engineer Java-Angular
Permanent contract
Luxembourg
Onsite, Hybrid
Skills
SQL QueriesAngularJavaObject-oriented programmingSoftware testingCloud computing
20 hours ago
Quick apply
No company information available.