Monitoring & Analytics crawler

    CROtrustifyBot crawler policy

    Identity, scope, crawl controls, and data-handling information for the automated storefront auditor operated by CROtrustify.

    Last updated: July 18, 2026

    Bot identity

    CROtrustifyBot is operated by CROtrustify. It identifies itself with a stable User-Agent and signs eligible storefront requests using Web Bot Auth.

    User-Agent
    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 CROtrustifyBot/1.0 (+https://www.crotrustify.com/bot)
    Signature-Agent header value
    "https://api.crotrustify.com"

    Purpose and crawl scope

    The bot performs user-requested and scheduled conversion-trust monitoring of public e-commerce storefronts. A bounded audit can render the homepage and selected same-store product, policy, support, cart, checkout-entry, and critical-link pages needed to evaluate public trust, accessibility, policy, link-health, and conversion-friction signals.

    It does not access Shopify Admin, customer accounts, password-protected areas, or private store APIs. A paid flow may add an item to an isolated anonymous cart to inspect the pre-purchase journey, but the bot never enters customer or payment credentials and never places an order.

    Robots, pacing, and failures

    • The bot retrieves the applicable robots.txt before the storefront homepage.
    • It applies the matching CROtrustifyBot group, including Allow, Disallow, wildcards, and Crawl-delay.
    • Disallowed same-store URLs are not requested. Top-level redirects to an origin whose policy was not verified are not followed.
    • Same-store document checks are sequential. A declared Crawl-delay is applied between top-level page navigations. Hard caps are 1,200 same-store requests, 800 third-party rendering requests, and 2,000 total requests per scan.
    • Any same-store HTTP 403, 429, or 430 response stops the store scan. A failed scheduled scan is not retried automatically.
    • Configured monitoring normally runs about once every 30 days, while user-requested audits run on demand.

    Data handling

    CROtrustify processes public storefront HTML and text, URLs, response metadata, screenshots, and derived audit results to create reports. Selected public text and screenshots may be processed by Google Gemini for paid analysis, so that subset can leave CROtrustify's hosting infrastructure.

    Scan outputs are stored with CROtrustify's contracted hosting and database providers. See the Privacy Policy for more information.

    Opt out and contact

    A storefront operator can opt out through the store's public robots.txt. The following group prevents CROtrustifyBot from crawling the store:

    User-agent: CROtrustifyBot Disallow: /

    For identification, policy, or opt-out assistance, email [email protected].