Skip to content

Show bots and AI crawlers seen on the site before any account; lead the listing with card testing - #120

Merged
cport1 merged 2 commits into
mainfrom
feat/value-before-signup
Nov 10, 2026
Merged

cport1 merged 2 commits into
mainfrom
feat/value-before-signup

Conversation

@cport1

@cport1 cport1 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Why

Plugins that grow on WordPress.org show value before anyone signs up. Ours already protects a site with no account, but a fresh install had little to show: the dashboard widget stayed at zero until something tripped a trap, and the crawlers a site allows (search engines, AI crawlers) were identified and then dropped without a trace.

What changes

Bot visit counters (no account needed). In early_check(), before the good-bot exit, every request that names a known agent from the shared registry adds one to a counter keyed by (hour, agent_id, verification):

  • verified: the detector's reverse and forward DNS check passed (verified_good_bot)
  • failed: the address did not belong to the bot it claimed (bot_verification_failed), unless the site is behind a proxy the plugin has not been told about, where it counts as claimed
  • claimed: identified by User-Agent only

Verification is read from the detector's flags, not worked out a second time. No IP, User-Agent, path or anything else about a request is stored, and requests naming no known agent (every human visitor) are not counted. Nothing is sent anywhere.

Surfaces

  • Dashboard widget: "Bots seen on your site in the last 24 hours" (known bot visits, AI crawlers and agents, checkout bot signals when WooCommerce is active, otherwise search engines), the verified/claimed/failed split, top 5 bots, then the existing 7-day detection stats. The connect prompt states only what connecting adds (cloud dashboard and monthly report free; cross-site address intel and webhook alerts paid).
  • Statistics page: per-bot table for 7 days (operator, type, visits, verified, claimed, failed).

Storage. New table webdecoy_crawler_visits, DB_VERSION 2.3.0 to 2.11.0 (created on activation or the next admin load; recording waits for it), pruned after 30 days by the existing hourly cleanup, dropped on uninstall. The version bump also creates webdecoy_ai_referrals on sites that upgraded past 2.10.0 without reactivating, since that table was only ever created on activation.

Copy fixes found along the way

  • "Indefinite data retention / history" removed (Statistics upsell, Cloud tab, readme).
  • "Block AI crawlers" never blocked PerplexityBot or other AI search crawlers (they are in the search engine category). The settings label and FAQ said it did; now they say what happens.
  • Good-bot count "60+" is now "more than 130" (registry categories treated as good bots).

readme.txt

  • Title: WebDecoy Bot Detection - Stop Card Testing, Fake Orders & AI Crawlers
  • Short description (141 chars) leads with WooCommerce card testing, fake orders and checkout bots, then AI crawler visibility.
  • Opening sections: card testing and fake orders first, then a new "See which bots and AI crawlers read your site" section, then the existing ones.
  • Tags (only the first 5 count): card testing, fake orders, woocommerce, ai crawlers, bot detection. Two narrow symptom tags where a small plugin can actually appear on the tag page, woocommerce for association, ai crawlers for the visibility angle, and bot detection to keep the name match. Dropped security and spam protection: both are crowded and the readme body still covers spam.
  • No version bump and no changelog entry. A suggested entry for the release PR:
    * New: the dashboard widget and Statistics page show bots and AI crawlers seen on your site, verified, claimed or failed, with no account. Only counts are kept.

Tests

  • php tests/run.php: 224 passed, 0 failed. New tests/CrawlerVisitsTest.php covers classification (browsers not counted, DNS verdict, proxy downgrade, real BotDetector flags), summary folding, labels for every registry category, UTC hour buckets, and source-reading guards: the table has no address/UA/path column, the only write is the four-column upsert, and observe() runs before the good-bot exit. That last guard was checked to fail when the call is moved after the exit.
  • PHPStan (level 3): no errors. PHPCS: no new warnings. PHPCompatibilityWP 7.4+: clean.
  • tests/wp-compat: smoke.sh SMOKE OK and woo.sh WOO OK (WP 7.1, PHP 8.3, MariaDB, latest WooCommerce), debug.log empty. Requests sent as GPTBot, ClaudeBot and Googlebot (from a non-Google address) produced claimed, claimed and failed rows, and the widget and Statistics page rendered them. A browser request produced no row.

Follow-ups (not in this PR)

  • A one-time post-activation notice pointing to the widget.
  • Optionally send the counters to the cloud when connected. This would need an External Services disclosure.
  • Readme claims not verified here: "less than 5ms of latency", the client-side list naming PerplexityBot, and the "Who builds WebDecoy" FAQ.
  • Regenerate languages/webdecoy.pot at release.

A fresh install had nothing to show until something tripped a trap: the
crawlers a site allows (search engines, AI crawlers) were identified and
then dropped without a trace.

Every request that names a known agent from the shared registry now adds
one to a counter keyed by hour, agent id and verification state
(verified by DNS, claimed by User-Agent only, or failed verification).
Verification is read from the bot detector's own flags. Behind an
unconfigured proxy a failed check is counted as a claim. Nothing else
about the request is stored, and requests that name no known agent are
not counted.

- Dashboard widget: known bot visits in the last 24 hours, AI crawlers
  and agents, checkout bot signals when WooCommerce is active, the
  busiest bots, and a connect prompt that states what connecting adds
- Statistics page: per-bot table for the last 7 days
- New table webdecoy_crawler_visits (DB_VERSION 2.11.0), pruned after
  30 days, dropped on uninstall; the version bump also creates the AI
  referrals table on sites that upgraded past 2.10.0
- AgentRegistry::find() to look up a stored agent id
- Cloud upsell copy no longer promises indefinite retention
…ility

- Short description, title and opening sections lead with card testing,
  fake orders and checkout bots, then the new bot visibility
- Tags: card testing, fake orders, woocommerce, ai crawlers, bot detection
- Good-bot count matches the registry (more than 130, not 60+)
- Block AI crawlers: PerplexityBot and other AI search crawlers are
  treated as search engines and are not blocked; the FAQ and the settings
  label no longer say otherwise
- Remove the "indefinite history" claim
- Privacy section describes the bot visit counters
@cport1
cport1 merged commit 08f567a into main Nov 10, 2026
4 checks passed
@cport1
cport1 deleted the feat/value-before-signup branch November 10, 2026 15:59
@cport1 cport1 mentioned this pull request Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant