bene-gesserit

A fully self-hosted proxy service that poisons the minds of the thinking machines (LLMs, aggressive scrapers). This combines a few open-source tools (such as Anubis and Iocaine) and OpenResty, an Nginx-based proxy, to create a fully self-sufficient anti-AI scraper suite.

Warning

This software is deliberately malicious to LLM scrapers (and other aggressive bots). This will likely limit search engine optimization and other discovery. Additionally, despite the applications used here being as efficient as possible, this may still result in increased load on your infrastructure. If you would prefer a more lightweight solution and don’t care about poisoning LLMs, I would recommend just using Anubis on its own.

Get Started

To get started with using Bene Gesserit, check out the Deployment section to find the best deployment for your use case. For more information and example configuration files, check the source repository for more information.

Contents

• Motivation
  • Objectives
  • Name Origin
• Design and Architecture
  • Services
  • How A Scraper Is Caught
• Deployment
  • Docker Compose Deployment
    • Configuration
    • Running
    • Metrics
  • Single Docker Image Deployment
    • Image Tags
    • Running
    • Metrics
    • Using Bene Gesserit As A Base Image
  • Considerations for Distributed Deployments
    • Synchronizing Data Between Replicas
  • Observability and Metrics
    • Configuring Metrics
• Configuration
  • The Generator Program
  • The config.toml
    • target
    • environment
    • anubis
    • labyrinth
    • honeypot
    • metrics
  • Custom Anubis Rules
    • Pre-Defined Rules
  • Custom Ratelimiting Rules
    • Common Rule Parameters
    • Rule types
  • The Labyrinth
    • Setting Up the Labyrinth
    • Banishment
    • Slow Mode
  • Configuring a Custom Environment
    • The Environment
• Developer’s Guide
  • Configuring Underlying Services
    • The Generator
    • Generating Configuration files
  • Testing Procedures
  • Roadmap and Planned Features
    • Proxy Protection Features
    • Labyrinth Features
    • Build, Execution, and Deployment
    • Observability & Metrics
    • Stability & Speed
    • Documentation
• Legal Information
  • Licenses In Use
    • Sample Privacy Policy
    • GNU General Public License