Machine accessibility of published regulatory guidance: a multi-agent probe of pharmaceutical and standards authorities

Preprint draft — not yet submitted.

Revision note — 2026-08-10. Populations are now named where they are used: source URLs ever probed against URLs on the final date against store rows; and profiles across the window against profiles in the current design. A design-version table was added to §2.2, with the 2026-08-03 change stated as what it is — a constraint this archive placed on itself. Every percentage now names its denominator in its own sentence. No figure changed value. Each was re-derived against the record and each survived; what was missing was the population, not the arithmetic.

Abstract#

Regulatory authorities publish the guidance that governs computerised systems and artificial intelligence in medicine manufacture. Increasingly, that guidance is read by automated clients rather than only by people. This paper reports a repeated measurement of whether those documents are retrievable by such clients.

107 source URLs ever probed (96 on the final date), published by 28 authorities whose documents were probed, were requested with 12 client profiles across the window — 11 in the current design — on 12 observation dates, yielding 28,127 requests under a design that changed with the corpus and with a constraint the archive placed on itself; see §2.2. For each request the HTTP status, redirect chain, response headers indicating bot management, and the directive that the host's robots.txt declares for that client were recorded.

Retrieval rates differed by client, each rate taken over that profile's own requests: 80.5% of the 947 requests carrying a browser user-agent returned a document (2026-07-30 to 2026-08-02, before that profile was withdrawn), against 66.4% of the 2,727 no-user-agent control requests and 18.4% of GPTBot's 2,734. In 72.9% of all requests the host's robots.txt permitted the client while the server did not return the document.

The data, its codebook and the collection script are published openly.

1. Introduction#

Guidance issued by medicines regulators and standards bodies is a primary input to compliance work. Automated clients now retrieve such documents routinely: as inputs to retrieval-augmented systems, as sources for search indexes, and as training corpora.

Whether a given authority's guidance can be retrieved by a given automated client has not, to our knowledge, been measured systematically. Accessibility is usually treated as a property of a website in general rather than as a property of a specific document with respect to a specific client, and the two are not the same: a host may serve one client and refuse another at the same URL.

This paper reports such a measurement. It is descriptive. It records what happened when each request was made, and makes no recommendation about what any authority should publish or how any system should be built.

2. Method#

2.1 Sources#

107 URLs were probed, drawn from a tracked corpus of instruments concerning artificial intelligence, computerised systems and data integrity in regulated manufacture. URLs were deduplicated: accessibility is a property of a URL, so a URL named by several documents was probed once.

2.2 Client profiles and design versions#

Eleven profiles are used in the current design; twelve appear across the window. The roster and the URL set both changed during the observation period, so the design is reported as versioned rather than as fixed:

The request column sums to 28,127, which is the record total.

The design change of 2026-08-03 is a constraint this archive placed on itself. On that date the browser profile was withdrawn — this archive stopped sending a user-agent it is not — and three named agents were added. Both are visible in the table above. The browser-profile figures reported in §3 therefore describe 2026-07-30 to 2026-08-02 only, four dates, and are so labelled where they appear.

The current roster:

The no user-agent profile is a control. Without it, a difference between two named clients cannot be attributed to the name rather than to the shape of the request headers generally.

Applebot-Extended is declared in robots.txt but is never sent as a user-agent; it governs the use of content that Applebot has fetched. It is probed with Applebot's user-agent so that the transport is comparable, and the robots.txt columns are the measurement for that profile.

2.3 Requests#

One request per URL per profile per observation date, issued sequentially with a delay between requests. Redirects were followed and the chain recorded. Response bodies were read to a cap of 2,000,000 bytes.

For each host, robots.txt was retrieved once per run and parsed by group: the most specific matching User-agent group applies, and a named group replaces the wildcard group rather than adding to it. Within the matching group the longest matching path prefix wins, with Allow taking precedence over Disallow at equal length.

2.4 Limitations#

Measurements were taken from a single network vantage point; results may differ by origin. Bot-management layers may respond differently to a repeat visitor than to a first request, and this design cannot distinguish a policy applied to a client name from one applied to a request's other characteristics beyond the no user-agent control. A robots.txt directive is a declaration and is not enforced by the mechanism that returns a status code; this paper measures both and does not treat either as ground truth for the other.

Cross-window comparisons compare designs as well as hosts. The roster and the URL set both changed during the observation period, so a figure spanning the whole window is a measurement over a versioned design; §2.2 states which version applied on each date.

Unmeasured is a third outcome, and it is neither blocked nor open. A host may return HTTP 200 and serve no text a reader can extract, because the document is assembled in the client. That is not a refusal and it is not access: nothing was learned about what the URL serves. Counting such a surface on either side would be a fabricated measurement, so this study counts it on neither.

The discriminating question is whether the reader found zero of the sought thing, or zero of anything. Two surfaces met that test during this collection. picscheme.org/en/publications returned 200 over 66,741 bytes with 0 extractable characters, sought for two specific identifiers and yielding none of any form — the signature of client rendering rather than of a listing omitting two documents. who.int returned 200 with 0 extractable characters likewise. A reader that read nothing has not measured an absence; it has failed to read.

Sources this archive does not fetch, named with their reasons. Four rows are frozen by the archive rather than by the publisher: ispe-gamp-ai, ispe-gamp5-2e and ispe-pharma-4-0 republish their last honest read of 2026-08-03, each carrying that date and no request made since. The fourth, ispe-gamp-rdi, has never been read at all — it entered the corpus after the freeze, so it carries an explicit absence rather than a date, and appears in the dataset as unmeasured with no read_on. It is the sharpest case in this section: an entry about which this archive knows nothing, saying so, rather than inheriting a neighbour's reading.

A freeze is a constraint this archive placed on itself and is recorded as such; it quotes no publisher and carries no citation, because attributing our decision to their evidence would misstate whose constraint it is.

3. Results#

3.1 Retrieval by client#

3.2 Response by host and client#

Figure 1 shows the most recent observation for each host–client pair.

3.3 Declared directive against observed response#

In 20492 of 28,127 requests (72.9%), the host's robots.txt permitted the requesting client and the server did not return the document.

3.4 Render mode#

Render mode is computed only where a body was returned. A render mode reported for a refused request would describe a document that was never received.

4. Discussion#

The measurement distinguishes three outcomes that are often treated as one.

A refusal — HTTP 401 or 403 — tells a client that access was denied. A client can record that the document exists and was withheld.

A reported absence — HTTP 404 — tells a client that there is nothing at the URL. A client that records this outcome faithfully records that the guidance does not exist. Where a host returns 404 to a named crawler and 200 to a browser at the same URL, an automated index built from the former will be silently incomplete rather than visibly incomplete.

No HTTP response at all — a refused or reset connection — is distinct from both, and cannot be attributed to a policy about the client without further evidence, since it is also what a network failure looks like.

The gap between declared directive and observed response is the paper's central observation. robots.txt is a statement of intent maintained by one part of an organisation; the response is produced by infrastructure that may be configured by another, or by a vendor. Where the two disagree, the declared policy is not the operative one.

5. Data availability#

The dataset, its codebook and the collection script are published alongside this report. Each release is versioned and immutable; a published version is never rewritten.

Three populations appear in this report and they are not interchangeable. 107 source URLs were ever probed; 96 were probed on the final date, the corpus having grown during the window; and the crawlability store carries 106 rows, one per tracked instrument, which is a different unit from a URL because instruments may share a watch surface.

The frozen edition supporting this report is 2026-08-10, 106 rows, published as both JSON and CSV and cited by URL:

https://gxplex.com/papers/crawlability/export-2026-08-10.json
https://gxplex.com/papers/crawlability/export-2026-08-10.csv

No DOI is asserted. This archive has registered none, and an identifier it does not own would be a claim it cannot support. The edition is the store's own checked date, and the export reads no clock: generated twice over the same corpus it is byte-identical, including with its input rows reversed, and it differs in exactly one line when exactly one source moves. The dataset records publicly available regulatory information; the underlying documents carry their own terms and none is reproduced here.

6. Conclusion#

Machine accessibility of regulatory guidance varies by client at the same URL, and declared crawler policy does not reliably predict the response. Both are measurable, and this paper reports the measurement over 107 sources and 12 client profiles.

Observation window: 2026-07-30 to 2026-08-10. Generated from dataset v78.