# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / # Training crawlers User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: omgili Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: MistralAI-Training Disallow: / User-agent: meta-externalagent Disallow: / User-agent: GoogleOther Disallow: / User-agent: Baiduspider Disallow: / User-agent: PetalBot Disallow: / User-agent: AwarioSmartBot User-agent: AwarioRssBot Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: QualifiedBot Disallow: / User-agent: Cotoyogi Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: atlassian-bot Disallow: / User-agent: FishBot Disallow: / User-agent: BorderxBot Disallow: / User-agent: NavuBot Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: WARDBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: KimiBot Disallow: / User-agent: CitibotSiteCrawler Disallow: / # END Cloudflare Managed Content # stevensbooks.com # # NOTE ON INTERNAL SEARCH RESULTS — read before adding a Disallow here. # # "Disallow: /search" below covers spree_storefront's own /search route. It does # NOT cover where this store's search results actually live, which is # /products?keywords= (Spree::ProductsController#index via # MeilisearchFilterBuilder#query, which also accepts ?q= and ?filter[name]=). # That URL space was fully crawlable and fully indexable, and # LegacyRedirectMiddleware's search fallback 301s every unrecoverable legacy # Shopify 404 into it, so it grew without bound. GA4 for 2026-08-10..2026-09-07 # shows those URLs as the top landing pages after the homepage, at 99-100% # bounce. # # They are handled by SearchNoindexMiddleware, which sets # "X-Robots-Tag: noindex, follow" on them, NOT by a rule in this file. That is # deliberate and the order matters: # # * Those URLs are already in Google's index. A Disallow stops the crawl, # Google therefore never sees the noindex, and the already-indexed URLs # persist as URL-only "Indexed, though blocked by robots.txt" entries. The # block would lock the bloat in rather than clear it. # * noindex requires the page to STAY crawlable. So /products must remain # allowed until Search Console shows the ?keywords= URLs dropped out — weeks, # because every one has to be re-crawled first. # # Once they are gone, adding "Disallow: /products?keywords=" here is a # reasonable crawl-budget saving on a 1.3M-page catalogue. Do not add it before # then. Do not add it for /products generally: the faceted browse # (?taxon=, ?condition=, ?price=, ?page=) is legitimate and must stay crawlable. # # CORRECTION, 2026-09-08 — the sentence that used to end the paragraph above # said the faceted browse "must stay crawlable so Googlebot can reach product # pages through it". That is not true and it mattered, because it was the stated # reason for believing product pages had an internal link path at all. # # * The category facets are FORM CHECKBOXES, not links # (themes/default/spree/products/filters/_taxons.erb renders f.check_box # inside a form). Googlebot does not submit forms. The faceted browse # contributes ZERO crawlable links to product pages. STILL TRUE. # * Pagination is a prev/next ARROW PAIR with no numbered links # (themes/default/spree/products/_pagination_arrow.html.erb), so page 40 of # a category is 40 sequential fetches deep. FIXED 2026-09-09, see below. # * ~1.2M products / ~50 top-level BISAC taxons / 24 per page is roughly 1,000 # pages per category. # * And until 2026-09-08 every one of those paginated pages carried # pointing at page 1, because # Spree::CanonicalUrlHelper#canonical_path drops the query string # unconditionally. See Spree::CanonicalUrlHelperDecorator. FIXED. # # So product pages were reachable, in practice, only from the sitemap — which is # a discovery mechanism and carries no ranking signal. Nothing in this file can # fix that; it is a link-architecture problem. It is recorded here because this # is where the wrong version was written down. # # UPDATE, 2026-09-09 — the link-architecture problem above has been addressed. # The last paragraph is no longer an accurate description of the site: # # * Listing pages now emit NUMBERED page links as well as the arrows # (Seo::PaginationLadder, spree/shared/_crawlable_pagination). First, last, a # window around the current page, and exponentially spaced rungs, so a run of # N pages is O(log N) deep rather than O(N). Capped at 200 pages # (SEO_MAX_CRAWLABLE_PAGE) because these are OFFSET paginators over Postgres # and a link to page 50,000 is a link to the most expensive query the app can # issue. # * /categories (PagesController#categories) is an HTML index of every taxon # that has products, linked from the footer on every page. Before it, ~5,000 # taxons shared a few dozen crawlable links between them. # * Product pages link to all of their own taxons, not just the one in the # breadcrumb (themes/default/spree/products/_details.html.erb). # # What this does NOT change: the facets are still checkboxes, and this file # still cannot fix anything about it. Do not read the update above as a reason to # add "Disallow: /products?*keywords=" earlier than the paragraph before it says. # # Note also: robots.txt cannot express "Disallow: /products?keywords=" — the # standard matches paths, and query strings only via wildcards # ("Disallow: /products?*keywords="). Whoever acts on the paragraph above should # check the syntax before trusting it. # CRAWLERS THAT BRING NO CUSTOMERS AND WALK THE 1.2M-URL CATALOGUE # # ingram:analyze_access_log on 2026-09-13 (200,000 requests, 4 hours, after # every application fix of the day had shipped) put product-page renders -- # each one a full Rails render, because a crawler is by construction always # on an uncached page -- at: # # browser 44.6% 133/min # AhrefsBot 21.1% 63/min # other bots 14.6% 44/min (ExaSearchBot, Amzn-SearchBot, # KeenableBot, Reflectionbot ...) # Applebot 14.0% 42/min # GPTBot 2.4% 7/min # Googlebot 1.8% 5/min # bingbot 1.5% 4/min # # AhrefsBot alone rendered twelve product pages for every one Googlebot # rendered. SEO-tool crawlers (Ahrefs, Semrush, Moz, Majestic) index the # site for THEIR customers' backlink reports; none of it reaches a shopper. # The scrapers listed with them feed private datasets. Each of these # honours robots.txt by its own documentation; Bytespider and PetalBot do # not and are not listed here -- Cloudflare is the lever for those. # # Deliberately NOT blocked: Googlebot, bingbot (Search), GPTBot / # OAI-SearchBot / ClaudeBot / Claude-SearchBot (AI answer engines are a small # but real referral channel; blocking them trades CPU for reach). # # PerplexityBot was on that list until 2026-09-16 and is not any more; it has # its own section below, off /products/ only. Applebot likewise. The principle # did not change -- the volume did. Read the per-crawler sections before # adding another name here or taking one off. # # This is a request, not a wall. Crawlers re-read robots.txt on their own # schedule (Ahrefs documents up to 24 hours), and a Cloudflare WAF rule on # the same user-agents takes effect immediately -- docs/deploy-nginx.md, # section "Crawlers", has the expression. # ShapBot and MathPicDatasetCrawler added 2026-09-16 20:26 UTC, from the same # task: 5,025 and 607 requests in a 199,993-request window. Neither is a search # engine and neither can send a reader here — MathPicDatasetCrawler says # "(+research)" in its own user-agent, which is to say it is collecting a # dataset, and it is welcome to collect it from somebody else's Puma workers. User-agent: AhrefsBot User-agent: SemrushBot User-agent: DotBot User-agent: MJ12bot User-agent: ExaSearchBot User-agent: Amzn-SearchBot User-agent: KeenableBot User-agent: Reflectionbot User-agent: DataForSeoBot User-agent: BLEXBot User-agent: ShapBot User-agent: MathPicDatasetCrawler # SERankingBacklinksBot added 2026-09-26. It is a backlink crawler for an SEO # suite — the same category as AhrefsBot, SemrushBot, DotBot, MJ12bot and # DataForSeoBot above, indexing this site for somebody else's backlink report. # # It ARRIVED BETWEEN 09-20 AND 09-25, and that is measured rather than assumed: # analyze_access_log's top-user-agent list over a 200,000-request window on # 2026-09-20 does not contain it at all, and the same read on 09-25 has it at # 3,455 requests. So it is new, it scaled to thousands within days, and the # older SEO crawlers were blocked before this list was last touched — which is # why it is not already here rather than any decision to allow it. User-agent: SERankingBacklinksBot Disallow: / # Applebot: after the SEO crawlers were blocked at the edge (2026-09-14 # 04:00 UTC) it was the largest crawler left by a wide margin -- 103 # product-page renders a minute, 36% of all product-page renders, more # than browsers and Google combined -- on an 8-core box already at a load # of 7-9. It feeds Siri and Spotlight suggestions, which have never sent a # measurable customer here. Kept off the 1.2M-URL product surface only; # the homepage, categories and reading lists stay crawlable. Cloudflare's # rate limiting would have been the gentler lever and is a paid feature. User-agent: Applebot Disallow: /products/ Disallow: /rails/active_storage/ # PerplexityBot: the same story as Applebot, one crawler later. After Ahrefs # went to the edge (09-14 04:00) and Applebot came off /products/, this filled # the vacuum at twice the volume either ever had. # # ingram:analyze_access_log, 2026-09-16, 199,993 requests over 371 minutes # (08:26-14:37 UTC): # # PerplexityBot 83,487 41.7% of ALL requests ~225/min # all identified bots 64.3% of ALL requests # product pages 46.3% of ALL requests # of those, this bucket 63.7% 159/min # # Nine product-page renders for every one Googlebot rendered. Every one is a # full Rails render -- a crawler walking 1.2M unique URLs is by construction # never in the fragment cache -- and page_weight measured a render at 1,979ms # the same afternoon. That is ~5.3 of 8 cores held continuously, which is the # load of 8.94 the owner asked about. # # This REVERSES the "deliberately not blocked" line above, with the owner's # decision on 2026-09-16 and not silently. The reasoning there still holds and # is why this is /products/ and not "/": AI answer engines cite editorial # pages, not the 1.2M supplier-feed pages that are identical to every other # Ingram reseller's. The homepage, /categories, reading lists, collections and # /posts stay fully crawlable, so everything actually worth citing still is. # # If the load does not fall within ~24h, this crawler is ignoring the file and # Cloudflare is the lever -- see docs/deploy-nginx.md, section "Crawlers". # COVERS TOO, added 2026-09-16 20:26 UTC. Disallowing /products/ stops the HTML # and not the images: analyze_access_log put "other bot" at 49.9% of the 60,384 # /rails/active_storage requests in that window, 84/min against browsers' 82.6. # Every one of those was a Puma worker streaming a cover to something that will # never buy a book, and 49 of them returned 500. An AI answer engine citing an # editorial page does not need the 1.2M covers to do it. # # NOT added to the "*" group, deliberately: Googlebot-Image and Merchant Center # still need covers, the feed's image_link points at storage.googleapis.com via # Catalog::PublicImageUrl rather than at this path, and blocking the path for # everyone would cost Google Images for no gain here. User-agent: PerplexityBot Disallow: /products/ Disallow: /rails/active_storage/ User-agent: * Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /orders Disallow: /search # THE TWO PRICE SORTS COST 30+ SECONDS OF DATABASE CPU EACH. # ingram:diagnose_price_sort, 2026-09-29, returning THREE rows: # # price-low-to-high 16,880 ms + a 13,802 ms count query # price-high-to-low 24,351 ms + 5,086 ms # best-selling 1,196 ms # newest-first 34 ms # name-a-z 13 ms # manual 10 ms # # Only the price sorts are slow, and the reason is structural rather than a # missing index: the query is `min(spree_prices.amount) ... GROUP BY # spree_products.id` across 1.27M products, 1.55M variants and 1.49M prices. # ORDER BY an aggregate cannot be served by an index — Postgres aggregates the # whole catalogue, sorts it, and then counts it again for the paginator. No # index will fix that; only a denormalised min_price column on spree_products # would, which is a schema change and a separate decision. # # diagnose_db_performance the same day found this query shape had consumed # 208,699,727 ms — 58 HOURS of database time — across 3,948 calls, averaging 53 # seconds and peaking at 860. It is the single largest consumer on an instance # costing ~$690/month, and no human sorts a 1.2M-title catalogue by price four # thousand times. It is crawlers walking every facet combination. # # ONLY THE PRICE SORTS ARE BLOCKED. The others are milliseconds and stay # crawlable, as does the faceted browse — see the long note at the top of this # file about why /products must remain reachable. A sorted listing is also pure # duplicate content: identical products, different order, nothing for Google to # index that the unsorted page does not already carry. Disallow: /products?*sort_by=price-low-to-high Disallow: /products?*sort_by=price-high-to-low Allow: / Sitemap: https://stevensbooks.com/sitemap.xml