# Field Test 001 · *Do They Say the Same Thing?* · the rules as fixed in advance

*Published with the results. Section 1 was written and approved on 2026-10-31 and frozen on 2026-11-01, before any public response. Amendments 1 and 2 were added on 2026-11-05, each before any answer had been opened, and each says why. Nothing here was changed after the results were seen. The analysis that applies these rules is `analyze_001.py`; the answer key is `001-key.json`; the anonymous data is `001-data.csv`.*

*Note: this test was planned under the working title Inquiry IV. It was renumbered before publication, and is reported in Inquiry V, *Sunlike*.*

---

## 1. Preregistration (approved by Brandon 2026-10-31; 🔒 frozen 2026-11-01)

**The question.** Do the passage pairs the Map connects across traditions that never met make the same point more often than random passage pairs from the same books? And do unconnected traditions agree as often on furniture as on structure?

**What each rater sees.**
- A short consent and instructions screen, including the reading instruction (*"You don't need to understand every word. Read for the gist…"*).
- One self-description question: *"When it comes to questions like these, do you lean more empirical, more contemplative, or neither?"*
- **12 passage pairs** drawn at random for that person (4 from each pile, in random order). *(History: 24 / 8 each in v1; 18 / 6 each in v2; 12 / 4 each in v3, 2026-11-01, after the trial. All are length changes.)*
- For each pair:
  1. ***"Do these two passages make the same point about how we experience existence?"*** on a 1–5 scale (1 = not at all, 5 = the same point);
  2. ***"Have you read either passage before?"*** (yes / no / not sure).

**The three piles** (§2):
- **A:** pairs the Map already connects.
- **B:** random pairs from the same books.
- **C:** random pairs from the same books where both passages concern furniture (the afterlife, the cosmos, ritual).

**Primary measure.** For each rater, the mean rating of A minus the mean rating of B, then averaged across raters. **95% interval** by bootstrap over raters and items.
- **Supports the Map:** A − B ≥ 1.0 point, with an interval that excludes 0.
- **Small effect:** 0.5–1.0, interval excluding 0. Reported as small.
- **Against the Map:** below 0.5, or an interval that includes 0. Reported as a loss: the Map's agreement is mostly what a random draw from these books produces.

**Secondary measures** (reported whatever they show):
- **C against B and A against C:** is agreement on furniture lower than agreement on structure?
- **Results by lean:** empirical, contemplative and neither.
- **Results with and without pairs the rater recognised.** The primary analysis drops recognised pairs, rater by rater.
- **Agreement between raters.**

**Which submissions count:** only `version: 3`, `phase: "public"`. Trial submissions (`phase: "trial"`) are never counted.

**Exclusions, fixed now:**
- completed in under 4 minutes;
- every rating identical;
- incomplete responses.

**Sample size and stopping.**
- **Target:** at least 60 complete responses (raised from 40 on 2026-11-01 with the cut to 12 pairs).
- **The survey closes on a date fixed when it opens** (21 days after it opens; Brandon posts the same day), whatever the count.
- **No peeking:** results aren't looked at until it closes. Checking that submissions are arriving (a count) isn't peeking.

**What doesn't change after freezing:** the questions, the piles, the measures and these criteria.

**Reporting.** Inquiry IV §III reports the primary result in plain words, with its interval, whatever it is. The data (anonymous) and the analysis script are published with the results.

## 1a. Amendment 1: a paid recruitment channel (Prolific). Dated 2026-11-05.

*Drafted and ✅ **approved by Brandon on 2026-11-05**, before E.3 deployed and before any Prolific response existed. It adds a recruitment channel. It doesn't touch §1.*

**When it was written.** On 2026-11-05 the public run had 2 complete responses (plus 6 trial submissions, which never count), out of a target of 60. That is a count. **No answer has been looked at**, so this change can't have been shaped by the results.

**Why.** The free posts and the paid feed ad brought visitors but almost no completions:
- About 88 ad visitors stayed an average of 18 seconds and produced 1 response.
- About 17 organic Reddit visitors on Nov 4 produced none.

Without another channel the test would close on Nov 23 far short of 60.

**What's added:**
- **A channel: Prolific**, a research-participant platform that pays people for their time.
  - **Places:** about 58, enough to bring the public total to 60 complete responses.
  - **Pay:** $3.00 for an estimated 15 minutes ($12 an hour, Prolific's recommended rate).
  - **Eligibility:** residents of the US, Canada, the UK, Ireland, Australia and New Zealand, fluent in English. No other screeners: the test wants general readers, and lean is self-reported inside the survey.
- **A tag on every submission:** `src`. It is `prolific` for raters who arrive through Prolific's link, and empty for everyone else.
- **Payment doesn't depend on the exclusions.** Every Prolific rater who finishes and returns the completion code is approved and paid. §1's exclusions are applied in the analysis only, exactly as for everyone else.
- **No Prolific ID is collected.** Raters stay anonymous to this site. Prolific approves submissions on the completion code.

**How Prolific responses are counted.** They are `version: 3`, `phase: "public"` submissions like any other. They count toward the 60, and §1's exclusions apply to them unchanged.

**One new secondary analysis (reported whatever it shows):**
- The primary measure (A − B) and its interval, computed three ways:
  1. **All public responses.** This is the primary result, unchanged.
  2. **Without Prolific responses.**
  3. **Prolific responses only.**
- Each is reported with its number of raters. If a subset is too small to estimate anything, it is reported as descriptive only and says so.
- **The method section also reports:** how many responses came through each channel, and the exclusion rate in each.

**Stated limits:**
- **Prolific raters are paid, experienced survey-takers from six English-speaking countries.** They may read differently from Reddit volunteers. The channel split and the lean question show part of that difference, not all of it.
- **The completion code is visible in the page's code.** Someone could claim payment without taking the test. The check: the number of `src = prolific` submissions in Netlify is compared with the number of completions Prolific approves, and any gap is reported.
- **Exposure to the site is the same as before.** Prolific raters, like Reddit raters, land on the site's own domain. The test page links nowhere else.

**What doesn't change:** the question, the piles, the items (version 3), the measures, the criteria, the exclusions, the target of 60, and the closing date (**2026-11-23**). If the Prolific places haven't all filled by then, the test still closes on Nov 23. *(The closing date is superseded by Amendment 2.)*

## 1b. Amendment 2: an early close, a cutoff, and the analysis written down. Dated 2026-11-05.

*Decided by Brandon on 2026-11-05 (about 1:15pm Denver), while the Prolific places were still filling. **No answer had been opened**: the only things known were counts and Prolific's completion times. The decision depends on the count, not on any result.*

**Why.** Prolific filled the pilot within hours, and the remaining 53 places were filling the same afternoon. The target of 60 complete public responses was about to be reached 18 days early. Brandon chose to analyse and write now, rather than leave the test open until Nov 23. Leaving it open would also have kept the site's planned ad campaign on hold, because ad visitors could find the test.

**What changes:**
- **Data collection for the result ends at the cutoff.** The cutoff is the moment Brandon confirms that Prolific shows all 58 places complete and stops the study. Its time is recorded here: **cutoff = 2026-11-05 13:47 Denver (20:47 UTC)**, when Brandon reported Prolific complete. At that moment Netlify held 66 `field-test` submissions (8 before Prolific, 5 pilot, 53 main), the last received at 13:42:47 Denver. Every one of them is inside the cutoff.
- **Which submissions count:** every `version: 3`, `phase: "public"` submission received at or before the cutoff, after §1's exclusions. Nothing else about §1 changes.
- **After the cutoff:** anything that arrives before the page closes is counted and reported as a separate late group. It never enters the main result.
- **The closing date** moves from Nov 23 to the deploy that publishes the results. The results page says plainly that the test closed early, on reaching its sample, and why.

**The analysis, fixed before the data.** `analyze_001.py` (in `Inquiry IV texts\field-test-001\`) codes §1 and both amendments. It was written and tested on synthetic data only before any public answer was opened. It makes these decisions explicit:
- **"Recognised" means "yes".** "Not sure" is kept.
- **Raters who can't give a difference:** a rater left with no A pair or no B pair, after their recognised pairs are dropped, is left out of the A − B measure and counted in the report.
- **The bootstrap:** 10,000 draws (seed `20261105`) resampling raters and, at the same time, items within each pile.
- **Agreement between raters:** Krippendorff's alpha (interval).
- **Reported as well:** the late group, if any, and every channel and lean breakdown, whatever they show.

**Stated limit.** The Reddit post, the mod messages and the test page all said "closes November 23". People told that date who come later will find the test closed. The results page and the Reddit follow-up say why.
