Guide
Job vacancy scraping for recruiters: build or buy?
A job vacancy scraper collects job postings from websites automatically and turns them into searchable data. Building one is a weekend project. Keeping it running is a permanent job, because the sites it depends on change without warning, and that maintenance cost is what decides build against buy.
That is the short answer. The rest of this guide covers what actually breaks, what the real cost looks like, and when building genuinely is the right call.
Recruiters scrape vacancies for a different reason than jobseekers read them. You are not looking for a role to apply to. You are looking for evidence that a company has a hiring need and a budget, so you can start a conversation. The vacancy is a signal, not a product.
That changes what good looks like. A jobseeker wants the most relevant roles. You want the earliest sight of the widest set of hiring companies, which is a much harder problem.
The two things people mean by scraping
Almost every discussion of this conflates two activities with very different economics.
Scraping job boards
Technically the easy version. A handful of large sites, consistent page structures, predictable pagination. A competent developer can pull structured listings from a major board fairly quickly, and plenty of off-the-shelf tools already do it.
The catch is commercial rather than technical. Everything on a public job board has already been published to the whole market, and hundreds of agencies are running the same alerts. You will get the data. So will everyone else, at the same moment. Board terms of service also frequently prohibit automated collection, which is worth reading before building anything.
Scraping employer career pages
Technically much harder and commercially far more useful. Roles usually appear on an employer's own site before they are advertised anywhere, and a good share never get advertised at all.
The difficulty is that there is no such thing as a standard careers page. Some are plain HTML. Some are embedded applicant tracking system widgets. Many are JavaScript applications that return an empty shell to anything that does not run a browser. Every site is effectively its own small project.
This is the whole trade-off. The easy data is contested and the contested data is easy for a reason. Anything a single developer can scrape in an afternoon is being scraped by everyone else too.
What actually breaks
Scrapers do not fail loudly. They fail by quietly returning less than they should, which is considerably worse, because nobody notices until the pipeline has been thin for a month.
- Layouts change. A site redesign moves the element your parser depends on and the scraper starts returning nothing for that employer. Multiply by the number of sites you cover.
- Bot protection appears. Cloudflare, rate limits, and challenge pages get added without notice. Your requests start returning a challenge page, which your parser dutifully processes into zero results.
- JavaScript rendering. A site moves to a front-end framework and the raw HTML no longer contains the jobs. Now you need a headless browser, and your infrastructure cost multiplies.
- Soft failures. The worst category. A page returns a valid looking response with no jobs on it, so the scraper records zero vacancies and reports success. Nothing alerts, and you conclude the employer is not hiring.
- Duplicates and drift. The same role appears with slightly different titles across sources, and company names never quite match, so your database slowly fills with near-duplicates.
Counting the real cost
The build is the part people estimate, and it is the smaller part. A scraper covering a few dozen sites is genuinely a short project for a capable developer.
Then it needs monitoring, because silent failure is the default. It needs someone to fix breakages, which arrive at a rate proportional to how many sites you cover and at times that suit the sites rather than you. It needs deduplication and company matching, which is a data problem rather than a scraping one and is usually the piece nobody budgeted for. It needs hosting, and headless browsers are not cheap to run at volume.
And it needs to keep doing all of that indefinitely, because the day it stops being maintained is the day it starts quietly under-reporting.
The honest way to frame the decision is not "can we build this" but "are we willing to own this permanently". Plenty of agencies can build it. Very few want a developer permanently assigned to keeping it alive.
| Building your own | Buying the data | |
|---|---|---|
| Time to first results | Weeks, longer with career pages | Immediate |
| Ongoing effort | Permanent, and grows with coverage | None |
| Realistic coverage | The sites you can afford to maintain | Whatever the provider covers |
| Silent failure risk | High unless you build monitoring too | The provider's problem |
| Decision makers included | No, that is a separate build | Depends entirely on the provider, so ask |
| Best when | You have permanent developer capacity and unusual requirements | You want the output rather than the system |
When building is the right answer
It genuinely is, sometimes, and it is worth saying so plainly.
If you have in-house developers with capacity you can commit permanently, if your requirements are unusual enough that no provider covers them, or if the scraped data is going into a product of your own rather than a desk, building makes sense. Some agencies also build deliberately narrow scrapers over twenty or thirty key accounts, which is a small enough surface to maintain by hand and is a perfectly reasonable approach.
Where it goes wrong is the middle case. An agency wants broad market coverage, assigns a developer for a few weeks, gets something working, and then discovers that broad coverage and low maintenance are in direct conflict. The scraper degrades, nobody notices, and eighteen months later the conclusion is that the data was never any good.
The half that scraping does not solve
Even a perfectly working scraper gives you a list of vacancies. A vacancy is not a lead. To act on it you need the person who owns the hire and a way to contact them, and that is a second and largely unrelated problem.
This is where in-house projects most often stall. The scraping gets solved, the contact side does not, and consultants end up doing manual research on every role anyway. The bottleneck moved rather than disappeared.
How Elephant approaches it
Elephant covers the harder side deliberately. It monitors employers' own careers pages rather than job boards, across 1.1 million UK companies and more than 500 data sources, holding over 11,000 live direct site vacancies at any time.
Each vacancy arrives with the hiring decision maker mapped to it and their contact details verified, so the second problem is solved alongside the first rather than left to the consultant. The maintenance, the silent failures and the company matching are ours to worry about rather than yours.
Elephant is the buy option
Career page coverage across the UK market, with the decision maker mapped to every vacancy and nothing for your team to maintain.
See how it works