Crawlability of regulatory sources
Each tracked source is requested twice — once with a desktop browser user-agent and once with an AI-agent user-agent — and the responses are recorded. 102 of 132 return the document to a browser; 0 of 132 return it to an AI agent.
Those two counts are reported separately because a single "crawlable" figure answered neither question: a source returning 200 to a browser and 404 to a crawler was counted as crawlable, which is true of a person and false of the machine the word describes.
The two figures above count tracked sources, requested with two client profiles. A wider probe — 12 client profiles including a no-user-agent control, counted over probe requests across the measurement window — is recorded in the measurement dataset, and the paper reports it over that window and over its own current design, which are different populations and so different counts. Two agents can find an inversion; they cannot describe a landscape.
This records observed HTTP responses. It expresses no view about any publisher's choices and draws no conclusion about why a source responds as it does.
Answer a browser, refuse an AI agent (9)
- ASTM E2363 — browser 200, agent
- ASTM E2500 — browser 200, agent
- IEC 62443 — browser 200, agent
- IEEE 7000 — browser 200, agent
- GAMP AI Guide — browser 200, agent
- GAMP 5 2nd Ed — browser 200, agent
- ISPE Pharma 4.0 — browser 200, agent
- WHO AI health — browser 200, agent
- WHO computerised systems — browser 200, agent
Full dataset
| Instrument | Authority | Browser | AI agent | Browser accessible | Agent accessible | Render | robots.txt |
|---|---|---|---|---|---|---|---|
| ANVISA | Agência Nacional de Vigilância Sanitária | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | client | no AI-specific disallow |
| ASTM E2363 | ASTM International | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | disallows Google-Extended |
| ASTM E2500 | ASTM International | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | disallows Google-Extended |
| ASTM E2537 | ASTM International | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| CEN/CLC JTC 21 | CEN/CENELEC | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| CISA CPG 2.0 | Cybersecurity and Infrastructure Security Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EMA AI reflection paper | European Medicines Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| Annex 11 concept paper | European Medicines Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EMA CT computerised systems | European Medicines Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EMA data quality framework | European Medicines Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| Annex 15 revision concept paper | European Medicines Agency and Pharmaceutical Inspection Co-operation Scheme | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| DI revision concept paper | European Medicines Agency and Pharmaceutical Inspection Co-operation Scheme | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EMA work programme | European Medicines Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU AI Act | European Parliament and Council | 202 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | no robots.txt published |
| EU Cyber Resilience Act | European Parliament and Council | 202 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | no robots.txt published |
| EU Cybersecurity Act | European Parliament and Council | 202 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | no robots.txt published |
| Regulation (EU) 2026/1744 | European Parliament and Council | 202 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | no robots.txt published |
| EU GMP Annex 1 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 10 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 11 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 11 (draft) | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 12 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 13 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 14 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 15 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 16 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 17 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 19 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 2 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 21 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 22 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 3 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 6 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 7 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 8 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Annex 9 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Chapter 1 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Chapter 2 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Chapter 3 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Chapter 4 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Chapter 4 (draft) | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Chapter 5 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Chapter 6 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Chapter 7 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Chapter 8 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU GMP Chapter 9 | European Commission | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| EU IVDR | European Parliament and Council | 202 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | no robots.txt published |
| EU Machinery Regulation | European Parliament and Council | 202 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | no robots.txt published |
| EU MDR | European Parliament and Council | 202 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | no robots.txt published |
| EU NIS2 Directive | European Parliament and Council | 202 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | no robots.txt published |
| FDA AI guidance | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA AI device list | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA CDRH AI/ML | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA clinical decision support | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA CSA | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA data integrity | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA device cybersecurity | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA device cybersecurity (2026) | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA electronic systems (clinical) | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA/EMA AI principles | US Food and Drug Administration and European Medicines Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA ETP | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA GMLP guiding principles | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA guidance agenda | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA Part 11 scope | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA PCCP | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA postmarket cybersecurity | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA process analytical framework | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| FDA software validation | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| Health Canada ML | Health Canada | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| ICH E6(R3) | International Council for Harmonisation | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | client | no robots.txt published |
| ICH Q12 | International Council for Harmonisation | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | client | no robots.txt published |
| ICH Q13 | International Council for Harmonisation | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | client | no robots.txt published |
| ICH Q14 | International Council for Harmonisation | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | client | no robots.txt published |
| ICH Q2(R2) | International Council for Harmonisation | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | client | no robots.txt published |
| ICH Q9(R1) | International Council for Harmonisation | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | client | no robots.txt published |
| IEC 62304 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| IEC 62443 | IEC / ISA | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · robots.txt refuses this agent User-agent: GPTBot Disallow: / | server | disallows GPTBot |
| IEC 81001-5-1 | IEC | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| IEEE 7000 | Institute of Electrical and Electronics Engineers | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · robots.txt refuses this agent User-agent: GPTBot Disallow: / | server | disallows GPTBot |
| IMDRF AI/ML WG | International Medical Device Regulators Forum | TimeoutError | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| IMDRF SaMD | International Medical Device Regulators Forum | TimeoutError | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO 13485 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO 14971 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO 81001-1 | ISO | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 22989 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 23894 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 24028 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 27001 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 27002 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 27017 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 27017:2026 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 27018 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 27018:2025 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 38507 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 42001 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 42005 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC 5259 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| ISO/IEC SC 42 | ISO/IEC | 403 | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| GAMP AI Guide | ISPE | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · robots.txt refuses this agent User-agent: GPTBot Disallow: / | server | disallows GPTBot |
| ISPE GAMP RDI | International Society for Pharmaceutical Engineering | no | no | not observed | |||
| GAMP 5 2nd Ed | ISPE | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · robots.txt refuses this agent User-agent: GPTBot Disallow: / | server | disallows GPTBot |
| ISPE Pharma 4.0 | International Society for Pharmaceutical Engineering | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · robots.txt refuses this agent User-agent: GPTBot Disallow: / | server | disallows GPTBot |
| MHRA AI | Medicines and Healthcare products Regulatory Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| MHRA AI Airlock | Medicines and Healthcare products Regulatory Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| MHRA data integrity | Medicines and Healthcare products Regulatory Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| NIST agent standards | US National Institute of Standards and Technology | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| NIST AI 600-1 | US National Institute of Standards and Technology | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| NIST AI RMF | US National Institute of Standards and Technology | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| NIST AI RMF Playbook | US National Institute of Standards and Technology | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| NIST CSF 2.0 | US National Institute of Standards and Technology | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| NIST SP 800-82 | US National Institute of Standards and Technology | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| NISTIR 8183 | National Institute of Standards and Technology | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| NMPA portal | National Medical Products Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| PDA TR 80 | Parenteral Drug Association | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| PIC/S GMP Guide | Pharmaceutical Inspection Co-operation Scheme | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| PIC/S Annex 11 | Pharmaceutical Inspection Co-operation Scheme | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| PIC/S PI 011 | Pharmaceutical Inspection Co-operation Scheme | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| PIC/S PI 041 | Pharmaceutical Inspection Co-operation Scheme | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| PMDA AI | Pharmaceuticals and Medical Devices Agency | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| prEN 18229-1 | CEN/CENELEC JTC 21 | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| prEN 18229-2 | CEN/CENELEC JTC 21 | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| prEN 18281 | CEN/CENELEC JTC 21 | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| prEN 18282 | CEN/CENELEC JTC 21 | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| Swissmedic | Swissmedic | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | not asked · no readable robots policy | server | no robots.txt published |
| TGA SaMD | Therapeutic Goods Administration | TimeoutError | not requested — this record does not present a borrowed identity (DECISION #20) | no | not asked · no readable robots policy | not observed | robots.txt unreachable |
| 21 CFR Part 11 | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| 21 CFR Part 211 | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| 21 CFR Part 820 (QMSR) | US Food and Drug Administration | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| WHO AI health | World Health Organization | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | disallows CCBot |
| WHO data management | World Health Organization | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
| WHO computerised systems | World Health Organization | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | disallows CCBot |
| WHO data integrity guideline | World Health Organization | 200 | not requested — this record does not present a borrowed identity (DECISION #20) | yes | no | server | no AI-specific disallow |
Machine-readable at /api/v1/crawlability.json.
Method: a source counts as yes when at least one of the two probes returned
HTTP 200 with 600 or more characters of extractable text.
Last updated: