Senior Scraper Engineer

Key Responsibilities

• Design, build, and maintain scalable Python-based scraping systems (e.g., using requests, BeautifulSoup, Selenium, Playwright, Scrapy), architecting for reliability and maintainability rather than one-off scripts
• Design and implement anti-detection strategies at scale, rotating proxies, CAPTCHA-bypass solutions, user-agent/device fingerprint randomization, to sustain high scraping success rates across changing site and app defenses
• Reverse-engineer and extract data from mobile apps (Android/iOS) via API traffic interception, SSL pinning bypass, and emulator/device automation, in addition to standard web scraping
• Architect data pipelines that ingest and normalize structured and unstructured data from web APIs, mobile app APIs, HTML, JSON, and XML sources
• Own orchestration and scheduling of scraping workflows (Airflow, Prefect, n8n, or similar), including retry logic, alerting, and failure recovery
• Lead integration of pipelines into downstream systems (Snowflake, Google Sheets, S3, GCS, SharePoint), ensuring data quality and consistency
• Establish monitoring, logging, and observability practices for scraping infrastructure to catch breakages proactively rather than reactively
• Evaluate and prototype new scraping targets (web and mobile), data enrichment sources, and tooling; make build-vs-buy recommendations
• Anticipate and adapt to web/app structure, anti-bot, and API changes ahead of failures; mentor junior engineers on scraping best practices

Requirements

• Bachelor's degree in Computer Science, Information Systems, or related field
• At least 5 years of experience in web scraping, crawling, or automation scripting, including at least 1-2 years designing scraping systems at scale (not just writing individual scripts)
• Strong proficiency in Python and its scraping ecosystem (requests, BeautifulSoup, Selenium, Playwright, Scrapy)
• Hands-on experience with mobile scraping/reverse engineering, tools like Frida, mitmproxy, Charles Proxy, Objection, or Appium; experience bypassing SSL pinning and root/jailbreak detection
• Proven experience with headless browser automation (Puppeteer, Playwright) and defeating modern anti-bot measures (proxy rotation, CAPTCHA handling, rate-limiting, fingerprinting)
• Hands-on experience with containerization (Docker) and Git-based CI/CD in production environments
• Track record of scraping high-defense targets such as social media or e-commerce platforms (web and/or mobile app) at scale
• Experience owning end-to-end data pipelines into a warehouse (Snowflake preferred) or cloud storage
• Prior experience mentoring or leading other engineers is a plus