Crawler defense

Crawler defense for protected sites without enterprise WAF complexity.

A practical crawler defense layer that starts in monitor mode and grows into challenge or block controls by route.

Start setupOpen quickstart
crawler defensecrawler protectioncrawler blocking APIwebsite bot defense

Defend content before it leaves your app

Voidek runs a decision step before protected routes return valuable HTML or API content, giving site owners a way to log, challenge, or block crawler traffic.

Separate useful bots from unwanted crawlers

Search-engine crawlers can be tagged differently from AI crawlers, scraper libraries, headless browsers, and high-rate path probes.

Built for gradual rollout

Use monitor mode first, review decision reasons, then raise friction for specific crawler classes without surprising normal users.

Detection proof

Crawler Defense signals Voidek can separate before content is served.

Use these examples as rollout checks: start in monitor mode, confirm the log rows, then enforce only the classes that match your protected routes.

Route boundaryDocs, pricing, API, content feedProtect selected routes only
Mixed bot trafficAI crawler plus search crawlerSeparate wanted and unwanted automation
Rollout settingMonitor then enforceReduce false-positive risk

Rollout path

Start with one protected site, one test request, and one Analytics row.

Voidek is meant to start quietly: save the runtime key, run a controlled test, then raise friction only where live logs support it.

01

Add one protected site

Create a personal account on the Free plan, add one protected site, and save the runtime key before turning on crawler defense.

02

Run the test decision

Copy the test request, export VOIDEK_API_KEY, then confirm the GPTBot row appears in Analytics with path, action, and score.

03

Enforce by route

Add the decision API at middleware or backend boundaries, then tune crawler blocking API after live rows look clean.

Related crawler defense pages
ProofPublic proof matrixReplay evidence, decisions, and limits.GuideAI Crawler BlockerAI bot blocker · block AI crawlersGuideGPTBot Blockerblock GPTBot · GPTBot robots.txt alternativeGuideScraper ProtectionPython requests blocker · Selenium detectionGuideWebsite Crawler Blockerblock website crawlers · crawler blocker for website

FAQ

Questions site owners ask before turning on crawler defense.

What is crawler defense?

Crawler defense is runtime detection and policy for automated clients. Voidek combines user-agent, header, browser, route, and rate signals.

Can Voidek protect a single site?

Yes. Voidek is designed for protected site owners who want crawler visibility and blocking without a full CDN or enterprise WAF migration.

First setup

Set up one protected site before the next crawl spike.

Create a personal account, add one protected site, copy the runtime key, and run one test decision before enforcing crawler defense.

Start protected site setupOpen quickstart