Website crawler blocker

Block unwanted website crawlers while allowing normal browsers.

Website crawler blocking with route logs, install snippets, and search-bot separation.

Start setupOpen quickstart
website crawler blockerblock website crawlerscrawler blocker for websiteanti scraping for websites

Know which crawlers are hitting your site

Dashboard logs summarize user agents, client types, scores, decisions, and paths so site owners can see crawler behavior before enforcing blocks.

Challenge suspicious browser automation

The browser snippet collects lightweight signals that help identify webdriver, empty plugin surfaces, missing language, and static HTML clients.

Protect selected routes

Middleware can protect pricing, content, docs, search, and API routes while leaving public pages open during rollout.

Detection proof

Website Crawler Blocker signals Voidek can separate before content is served.

Use these examples as rollout checks: start in monitor mode, confirm the log rows, then enforce only the classes that match your protected routes.

Website routeprotected page, docs, content endpointLog path-level crawler activity
Path probe.env, .git, wp-login scansBlock noisy probes early
Search trafficKnown search crawlersAvoid broad SEO-breaking blocks

Rollout path

Start with one protected site, one test request, and one Analytics row.

Voidek is meant to start quietly: save the runtime key, run a controlled test, then raise friction only where live logs support it.

01

Add one protected site

Create a personal account on the Free plan, add one protected site, and save the runtime key before turning on website crawler blocker.

02

Run the test decision

Copy the test request, export VOIDEK_API_KEY, then confirm the GPTBot row appears in Analytics with path, action, and score.

03

Enforce by route

Add the decision API at middleware or backend boundaries, then tune crawler blocker for website after live rows look clean.

Related crawler defense pages
ProofPublic proof matrixReplay evidence, decisions, and limits.GuideAI Crawler BlockerAI bot blocker · block AI crawlersGuideGPTBot Blockerblock GPTBot · GPTBot robots.txt alternativeGuideScraper ProtectionPython requests blocker · Selenium detectionGuideCrawler Defensecrawler protection · crawler blocking API

FAQ

Questions site owners ask before turning on crawler defense.

Will a website crawler blocker hurt SEO?

It can if configured badly. Voidek classifies known search engines separately so Googlebot can stay allowed while unwanted automation is challenged.

Does Voidek block all crawlers automatically?

No. Start in monitor mode, then choose which crawler classes should be logged, challenged, or blocked.

First setup

Set up one protected site before the next crawl spike.

Create a personal account, add one protected site, copy the runtime key, and run one test decision before enforcing website crawler blocker.

Start protected site setupOpen quickstart