Python Developer - Web Scraping

We are hiring a Python Developer specialising in web scraping to build and run the systems that gather data.

You will design collection pipelines, run them against real targets to a deadline, and own the quality of what comes out. This is a hands-on ownership role rather than a maintenance one: you will build the system, not inherit a queue of tickets.

This is a fully remote role and we are open to candidates based anywhere.

Working hours

Monday to Friday: 9am to 6pm UK time

Saturday: 9am to 1:30pm UK time

You will be working against large social and professional platforms and other dynamic, JavaScript-heavy sites that employ active bot detection. Experience with that class of target is the core of this role.

What you will do

• Build web scraping and data collection systems that take a target and a set of parameters and produce a structured dataset, end to end.
• Work with sites that employ bot detection: fingerprinting, behavioural analysis, device checks and challenge pages. Handle CAPTCHAs and verification challenges, and know when to solve, when to back off and when to stop.
• Manage authenticated sessions and account pools at scale: session and cookie persistence, 2FA, rotation across accounts, and detecting when a session has silently degraded rather than failed outright.
• Manage proxy infrastructure: residential and datacentre pools, IP rotation, and keeping personas and egress isolated from one another.
• Set and document a sustainable request rate. Understand the difference between a soft throttle with a known reset and a hard restriction, and respond to each differently.
• Build reliable block detection: tell being restricted apart from receiving a genuinely sparse or empty page. These look identical to naive code and need completely different responses.
• Make every run resumable, so an interrupted job continues rather than restarting.
• Extract structured fields from inconsistent records: missing sections, partial dates, other languages, varying layouts, and DOM changes that break selectors without throwing an error. Degrade gracefully, and never write a value into the wrong field.
• Resolve entities: recognise when two differently written names refer to the same organisation, and when two similar names do not.
• Instrument your pipelines with run logs, per-field coverage rates and alerts that fire on data quality drift rather than only on errors.
• Scale a single job into a larger programme: parallel workers, containerised personas, a central work queue, and realistic planning for account attrition.
• Report clearly on coverage: what was attempted, what was retrieved, what was excluded and why.

Requirements

• 2+ years operating web scraping or data extraction systems in production over months, not building one once. We are interested in what you have kept running, and what you learned when it stopped.
• Hands-on experience of session and account management at scale:
• cookie and session persistence, 2FA, rotation, and recognising a degraded session before it corrupts your output.
• Practical knowledge of anti-bot systems, rate limiting, CAPTCHA and verification challenges, and browser fingerprinting.
• Proxy and IP rotation across residential and datacentre pools.
• Sound judgement about sustainable request rates and account safety, with real examples of what you have lost and what you changed.
• Strong Python.
• Browser automation and headless or stealth browsers: Playwright, Selenium, Puppeteer or equivalent.
• Clear written English. You will be sending findings to people who are not engineers.

Nice to have

• Large-scale extraction from authenticated sources, particularly social or professional platforms.
• Distributed crawling: Celery, Redis, queues, workers.
• Entity resolution or record linkage.
• Docker, cloud infrastructure, monitoring and alerting.
• OCR, or computer vision applied to extraction.
• Experience handling personal data responsibly.
• Working with AI
• You are expected to use AI tools as a normal part of how you work. You should also be able to explain and defend any code you ship, including how you diagnosed the problems you hit along the way.

Working with AI

You are expected to use AI tools as a normal part of how you work. You should also be able to explain and defend any code you ship, including how you diagnosed the problems you hit along the way.