blog

RSS
[TAGGED: scrapers]
  1. Gemini Proxy attacked by scraper bots

    The Gemini Proxy has been getting overwhelmed by bots. I've had to set up some pretty restrictive rate limiting and IP address hash banning, as well as blocking a couple of Gemini capsules that the bots were hyper-fixated on. It sucks, because I don't think the owners of these capsules were doing anything wrong—rather they appeared to be targeted by a malicious mass-scraping attack that brought my own server down in the process.

    I say malicious because the bots were ignoring my robots.txt file which kindly asks bots to not crawl the Gemini Proxy. In repeated bursts, they were saturating every possible connection thread on my server from thousands of IP addresses with no rate limiting, making it impossible to browse the Gemini Proxy and at times even the site. The sheer amount of nginx access logs they were generating filled the server's SSD with tens of gigabytes of log files which eventually crashed everything. Not very subtle.

    I don't know but I suspect that these are related to LLM training data ingestion services run by shady operators to mass-ingest data from all corners of the Internet—for the good of our corporate oligarchs. Essentially the scraper bots were using Gemini Proxy to fan out into Geminispace, which is walled off from typical HTTP scrapers. This goes against my intent in running a service like Gemini Proxy in the first place—Gemini is supposed to be a protected space from the stupidity of the commercial Internet, so I had to take restrictive action to stop as much of it as I reasonably can.

    If you are a human being and you see rate limiting notices or get banned from Gemini Proxy, please contact me and I'll restore your access. I'm still tuning things so please be patient.

    Posted 2026-11-17 00:00:00 CST by henriquez. Comments