How the FSF Sysadmins are Blocking Botnets with reaction
Published on 2026-07-11T21:47:00Z
For nearly two years the Free Software Foundation has been fighting web crawlers (including many aggressively scraping training data for AI models). A botnet controlling about five million IPs hit one system for six months in 2025. Their systems administrator wrote this week that they view these as distributed denial-of-service attacks. How are they fighting back? We noticed patterns in the scrapers that were abnormal, which gave us material for writing regular expressions. Searching for the regular expression then gave us a large lists of IP addresses. Looking up the origin of those IP addresses revealed that some of the crawlers were using botnets of residential IP addresses to scrape faster and avoid detection. We looked for what kinds of botnets might be generating the kind of traffic that we were seeing, and one that we suspected was called the "Vo1d" botnet, comprised of smart TVs running some sort of compromised app... We got confirmation that at least some of the botnet traffic hitting GNU Savannah was originating through the Vo1d/Popa botnet. We placed our regular expressions in fail2ban, and found that we were hitting the maximum rules that could be added to UFW firewall rules on our systems which showed degradation around 65,000 rules... We learned about ipset and configured fail2ban to add IP addresses that it found to IP sets. Using ipset, we kept building larger IP sets and did not find instability with as large as five million rules... We eventually found a promising project on Framasoft's forge Framagit called reaction written by ppom... After we ran into scaling issues with our initial implementation, we developed a much faster implementation where the reaction shutdown process would export the IP sets to disk and the reaction startup process would restore the IP sets. This allowed us to have nearly instantaneous restarts of the service to apply new rules. We published both of our configurations upstream to reaction's wiki so that everyone can benefit from it. reaction's getting started documentation now leads to the method that we proposed... Many sysadmins know about fail2ban, but not enough people know about reaction. I am very grateful to ppom for the help they have provided and for the tremendous project they have released to the world with reaction. We have implemented other defenses as well, but reaction is doing the majority of the automated work keeping our sites online.
Published on 2026-07-11T21:47:00Z
For nearly two years the Free Software Foundation has been fighting web crawlers (including many aggressively scraping training data for AI models). A botnet controlling about five million IPs hit one system for six months in 2025. Their systems administrator wrote this week that they view these as distributed denial-of-service attacks. How are they fighting back? We noticed patterns in the scrapers that were abnormal, which gave us material for writing regular expressions. Searching for the regular expression then gave us a large lists of IP addresses. Looking up the origin of those IP addresses revealed that some of the crawlers were using botnets of residential IP addresses to scrape faster and avoid detection. We looked for what kinds of botnets might be generating the kind of traffic that we were seeing, and one that we suspected was called the "Vo1d" botnet, comprised of smart TVs running some sort of compromised app... We got confirmation that at least some of the botnet traffic hitting GNU Savannah was originating through the Vo1d/Popa botnet. We placed our regular expressions in fail2ban, and found that we were hitting the maximum rules that could be added to UFW firewall rules on our systems which showed degradation around 65,000 rules... We learned about ipset and configured fail2ban to add IP addresses that it found to IP sets. Using ipset, we kept building larger IP sets and did not find instability with as large as five million rules... We eventually found a promising project on Framasoft's forge Framagit called reaction written by ppom... After we ran into scaling issues with our initial implementation, we developed a much faster implementation where the reaction shutdown process would export the IP sets to disk and the reaction startup process would restore the IP sets. This allowed us to have nearly instantaneous restarts of the service to apply new rules. We published both of our configurations upstream to reaction's wiki so that everyone can benefit from it. reaction's getting started documentation now leads to the method that we proposed... Many sysadmins know about fail2ban, but not enough people know about reaction. I am very grateful to ppom for the help they have provided and for the tremendous project they have released to the world with reaction. We have implemented other defenses as well, but reaction is doing the majority of the automated work keeping our sites online.
Read more of this story at Slashdot.
Comments
Post a Comment