How the legal lexicon clusters by domain, and whether models agree on the shape of the map.
Experiment §3.1 — Distance structure
364 legal terms drawn from the Hong Kong
ordinances co-drafted under the Bilingual Laws Project (§2.2), spread
across seven domains (administrative, civil, constitutional, criminal,
international, labour, procedure). Ten language models process each
term in two ways: attested — the average of the four or more
real ordinance passages in which the term appears — and bare,
the term in isolation. The attested encoding is the result the
experiment analyses; the bare encoding stays alongside as a baseline
for the control-pool subtraction discussed on the Robustness page.
The section answers four nested questions. Within a single model, are
terms in the same legal domain closer to each other than to terms in
other domains (§3.1.1)? What is the shape of the seven-by-seven map of
inter-domain distance, and does it recur across models (§3.1.2)? When
two models produce distance maps of the same 364 terms, do they agree
more when they share a language tradition than when they do not
(§3.1.3)? And do the models recover the relative ordering of legal
categories that the doctrine defines as graded (§3.1.4)?
§3.1.1 · Distances within and between legal domains
Legal scenarioA lawyer groups terms by legal domain — civil, criminal, constitutional — without thinking about it. Asked the same of the lexicon, do language models reproduce the grouping? And do they keep the legal lexicon, taken as a whole, distinct from the everyday vocabulary against which it is supposed to specialise?
Result in words8 of 10 models place the legal lexicon closer together than the everyday-language control vocabulary, with rank-biserial r above zero and p below 0.05. The two non-conforming readings are diagnostic rather than failures: FreeLaw-EN is fine-tuned on a legal corpus and therefore applies its legal prior to ordinary words as well; Qwen3-0.6B-EN is small and multilingual, with English representations not resolved enough to support the term-class contrast. Within the three Western-trained models, the same pattern holds for intra-domain versus inter-domain distance.
Rank-biserial r for the legal-versus-control Mann-Whitney test, one bar per model. Positive r means legal-legal distances sit below legal-control distances. The two negative bars belong to FreeLaw-EN and Qwen3-0.6B-EN, the two diagnostic cases of §4.2 of the thesis.
Intra-domain versus inter-domain distance on the three Western-trained models. Positive r marks intra-domain compactness.
Take-homeThe legal lexicon is internally domain-organised in every model of the panel and externally distinguishable from everyday vocabulary in eight of ten. The two exceptions delimit the encoder regimes on which the diagnostic cannot be relied upon (§4.2).
Technical apparatusOpen technical detail
r = 1 − 2U / (nx ny)
test = Mann-Whitney U, one-sided · legal pairs = 66 066 · control pairs = 36 400 · models passing = 8 / 10
Rank-biserial r is the transform of U from [0, nx ny] to [−1, +1]: positive r means the first distribution sits below the second.
§3.1.2 · Maps of distance between legal domains
Legal scenarioWhat does the geometry of the seven domains of Hong Kong law look like, when a model projects them as a 7 × 7 map of average inter-domain distance? Do procedure and administrative sit at the centre, criminal and international at the periphery, as doctrinal intuition would predict? Do models agree on the shape of the map?
Result in wordsEvery domain is, on average, closer to itself than to any of the other six. Procedure and administrative anchor the centre of the map (lowest mean off-diagonal distance to the rest); criminal and international sit at the periphery; labour and constitutional take intermediate positions. The qualitative pattern recurs across the ten models; the §3.1.3 measurement that follows turns the visual recurrence into a number.
The 7 × 7 inter-domain map. Each cell is the mean cosine distance between terms in the row domain and terms in the column domain (attested encoding). Use the dropdown above the chart to switch between the ten models — the same diagonal-darkest pattern recurs in every reading.
Take-homeThe inter-domain map has a stable qualitative shape across the ten models. The recurrence is the geometric premise on which the §3.1.3 measurement of agreement between pairs of models rests.
Technical apparatusOpen technical detail
M[i,j] = meanterms a ∈ domain i, terms b ∈ domain j cosine(emb(a), emb(b))
Diagonal entries are upper-triangle means, excluding self-distances; off-diagonal entries are full-block means.
§3.1.3 · Agreement between pairs of models
Legal scenarioWhen two models trained on the same language tradition produce distance maps of the same 364 terms, do they agree on the shape of the map? When they belong to different traditions, does their agreement drop? The answer is the principal quantitative reading of the chapter.
Result in wordsOn attested encodings, the average rank correlation between pairs of Western-trained models is 0.712, the average between pairs of Chinese-trained models is 0.868, and the average across nine pairs that span the two traditions is 0.246. The symmetric within-versus-cross gap, computed as the average of the two within-tradition means minus the cross-tradition mean, is 0.543. The two bilingual readings (a single model embedding both languages of input) sit at 0.316, in the same band as the cross-tradition pairs and well below either within-tradition floor: holding the model identity fixed and varying only the language of input does not close the cross-tradition gap.
Seventeen pre-registered model pairs. Error bars are 95% confidence intervals from term-level block bootstrap (B = 10 000). All seventeen Mantel p-values lie at the permutation floor; the Holm-adjusted maximum is 0.0017. Use the toggle above the chart to switch between the attested and the bare encoding — the within-tradition bands climb steeply under attestation; the cross-tradition band barely moves.
Bare-to-attested ρ trajectory for each of the seventeen pairs. Within-tradition pairs rise steeply under contextualisation on Hong Kong ordinance passages; cross-tradition pairs trace nearly flat lines.
The 0.543 figure is not the legal-meaning gap on its own. The same construction on bare encodings of the 364 terms returns 0.165, and the same construction on 100 everyday-language control terms returns 0.156 — statistically indistinguishable from the bare gap on the legal core. The bare gap is therefore shaped by the models themselves, not by legal vocabulary. The legal-meaning contribution is the difference that attestation adds on the legal core: 0.378 = 0.543 − 0.165. See the Robustness page for the full decomposition.
Take-homeWithin-tradition agreement on the attested 364-term core is high and tight; cross-tradition agreement is markedly lower. The gap is real but its interpretable size is the 0.378 attested-bare difference, not the 0.543 attested absolute. §4.1 of the thesis reads §3.1.3 as the principal cross-tradition finding; §4.2 reads the control-pool subtraction as its primary limit.
Technical apparatusOpen technical detail
Δρsym = (ρ̄W + ρ̄S) / 2 − ρ̄cross
Δρ<sub>sym</sub> attested = 0.543 · Δρ<sub>sym</sub> bare = 0.165 · Mantel B = 10 000 · Bootstrap B = 10 000 · Holm K = 17 · p<sub>max</sub> (Holm) = 0.0017
Spearman ρ on the upper triangle of each model's 364 × 364 cosine distance matrix; 95% confidence intervals from term-level block bootstrap (Nili et al. 2014). See §2.4 of the thesis for the full statistical apparatus.
§3.1.4 · Ordered legal categories and meaningful breakpoints
Legal scenarioLegal doctrine grades concepts along ordered continua marked by doctrinally significant transitions: criminal law marks the doli incapax threshold between infancy and imputability; contract law marks the onset of capacity at the age of majority; procedure marks the summary / indictable divide and the determinate / indeterminate disposal boundary. If embedding geometry tracks legal meaning, the largest cosine gap along the principal component of an ordered sequence should fall at the doctrinally expected break, not at a linguistic midpoint.
Result in wordsFive pre-registered ordinal probes, each templated across eleven legal categories and five paraphrase variants per language, are evaluated against 10 models. The ensemble mean Spearman ρ on the age-and-contractual-capacity probe is 0.935; on disposal severity 0.778; on offence severity 0.659; on the doli incapax threshold 0.937, a borderline test whose expected break sits close to the eleven-position midpoint. A negative-control probe on contract value returns 0.651, confirming that the positive signal is doctrinal, not generic to ordered sequences. The probe operates on templated sentences rather than on the curated lexicon, so its reading is independent of pool curation.
Mean PC1 projection of the eleven ordered categories across the five paraphrase templates, one line per model. The dashed vertical line marks the doctrinally expected break for the selected test; use the test dropdown to switch between the five probes. Western-trained models in blue, Chinese-trained in red, bilingual in grey. The eleven categories and the five paraphrase templates for each test are listed verbatim under Inside the inputs.
Ensemble mean Spearman ρ per test, averaged across ten models. The orange bar marks the borderline doli incapax probe, where the expected break sits within one position of the linguistic midpoint.
Take-homeWhere a legal threshold marks a discontinuity that is also linguistically marked — by a calque, by a morphological boundary, by a lexicalised opposition — the embedding registers the threshold as the geometric breakpoint. Models do not discover orderings autonomously, and they do not validate a breakpoint that the lexicon does not carry.
Each probe encodes eleven templated category sentences in both languages, projects the resulting vectors onto their first principal component, and asks whether the cosine gaps peak at the doctrinally expected position.