@levelsio@levelsio✨ Okay with lots of help from @javilopen and his Spanish scraping friends I've managed to vibe code my own @Scrapingbee alternative and replace my $249/mo Scrapingbee plan with my own $1/mo scraper running on my VPS!
First I tried OpenSERP, an open source project that seems freshly maintained but it didn't work on my VPS IP or laptop IP and got blocked fast
Then I tried to use residential IP proxies from IPRoyal but they were already quite tainted IPs meaning Google already knew they were scrapers and just blocked about 60% of them already, not so great, also didn't work!
Then I discovered OpenSERP uses a vanilla headless Chrome, Google instantly detects that and blocks that, so literally the entire repo is useless cause it won't even work? I don't get why they don't say that in the docs? OpenSERP is even worse than just basic CURL, as my AI said: "plain curl got 7% blocked but OpenSERP's headless browser got 100% blocked"
So then I just built my own with Playwright and a real Google Chrome on my VPS, headful and persistent context, and slow pacing like every 30 seconds or so
And I got some better residential IP recommendations (which I can't tell you because thenhey t stop working!)
Now I'm at about 90% success rate, which is similar or better than Scrapingbee
And it costs me ~$1/mo (measured by GBs, obviously for text scraping I barely use a lot) instead of the $249/mo I paid Scrapingbee!
My only remaining SaaS now are:
- Cloudflare Domains
- Cloudflare Email Sending
- Cloudflare R2 Storage
- Backblaze B2 for backups
- Hetzner for all my servers for all my sites (VPS)
- xAI for all LLMs for all my sites (so many things like moderation, writing emails, anything)
@levelsio@levelsioI've tried for weeks now to build my own @ScrapingBee and repeatedly AI has now told me that it can't do it
80% of my scraping is Google SERP results for Hotelist and even with residential IP proxies and solve CAPTCHA services and OpenSERP scraper it won't do it well for