When it breaks.
The problems you hit reading public social data programmatically: what each one actually is, what people try first, and what works. Written for whoever is debugging at 2am, human or not.
6 notes
each ends in one working call
each ends in one working call
1Known failure modes
- Reddit returns 403 to curl: why, and how to fix it
- Reddit 403s curl from any datacenter IP, on every surface including old.reddit and the JSON endpoints. Here is why, and a one-call fix.
- How to search Reddit without logging in
- Anonymous Reddit reading from a server is effectively dead: JSON 403s, the new UI is a JS wall, RSS is throttled. What still works in practice.
- Instagram API alternative for reading public posts
- Basic Display shut down in December 2024 and Graph only reads accounts that authorise your app. How to read arbitrary public profiles now.
- Cheap X/Twitter search API alternatives
- X bills roughly $0.005 per post read with flat tiers closed to new signups. Cheaper ways to run tweet search at volume.
- TikTok scraper API for videos and hashtag search
- TikTok signs its web requests and geo-redirects unknown clients, so a plain GET returns an About page. A maintained API handles both.
- DuckDuckGo returns 202 to your scraper - what to use instead
- html.duckduckgo.com answers scripted requests with a 202 bot challenge, and Google and Bing block curl outright. Use a search API.