Five readings drawn from the inferential discipline of §4.2 of the thesis.
Robustness & caveats
The §3.1.3 cross-tradition reading of 0.543 is the measured quantity,
not the interpretable one. This page collects the five passages of
inferential discipline that §4.2 of the thesis declares as the form
in which the measurement is usable: the subtraction that isolates the
legal-meaning contribution from the model-tradition baseline; the
behaviour of the principal quantity under pool perturbation; the
term-level reading on same-lemma pairs; the two diagnostic regimes
on which the §3.1.1 instrument cannot be relied upon; and the
bilingual control that rules out a model-identity confound.
The control-pool subtraction
Legal scenarioThe §3.1.3 reading of 0.543 measures the symmetric within-versus-cross tradition gap as the pipeline actually computes it, on the attested encodings of the 364-term legal core. It does not, by itself, isolate the contribution that the legal attestation adds. What part of the 0.543 is owed to the legal contextualisation on Hong Kong ordinances, and what part is a baseline that any two cross-tradition models would carry into the comparison before the legal context has been applied?
Result in wordsThe same construction repeated on the bare encodings of the 364 legal terms returns 0.165: a model-tradition baseline with no legal content. The same construction repeated on 100 everyday-language control terms returns 0.156, statistically indistinguishable from the bare baseline on the legal core. The bare gap is therefore model-tradition-shaped, not legal-tradition-shaped. The legal-meaning contribution is the difference that attestation adds on the legal core: 0.378 = 0.543 − 0.165.
0.378
Legal-meaning signal · attested − bare on the 364 curated terms · 0.543 − 0.165. This is the share of the §3.1.3 number that can be attributed to contextualisation on Hong Kong ordinances, against a model-tradition baseline of approximately 0.16.
Reading
Δρsym
Interpretation
Bare, 364 legal terms
0.165
model-tradition baseline with no legal content
Bare, 100 everyday-language control terms
0.156
indistinguishable from the baseline above: confirms the bare gap is shaped by the models, not by legal vocabulary
Attested, 364 legal terms
0.543
the §3.1.3 result the experiment analyses
Legal-meaning contribution
0.378
attested − bare = 0.543 − 0.165, the share attributable to contextualisation on Hong Kong ordinances
Take-homeThe 0.543 figure and the 0.378 figure belong together. The thesis cites the absolute 0.543 as the principal §3.1.3 measurement and the 0.378 as the interpretable legal-meaning contribution. §4.2 of the thesis carries this decomposition as the primary methodological limit of the experiment.
Control terms are everyday vocabulary — pronouns, deixis, basic common nouns — that have no Hong Kong ordinance attestation by design, so the only comparable reading on them is bare-on-bare.
Robustness under pool perturbation
Legal scenarioHow robust is the §3.1.3 reading if the curated 364-term pool is partly replaced by uncurated background legal vocabulary that the corpus does include but that the manual curation did not vet?
Result in wordsThe symmetric within-versus-cross tradition gap moves from 0.538 with no background injected to 0.590 with 75% of the pool replaced by background legal terms. The trajectory drifts upward, not downward: the gap does not decay with contamination from neighbouring legal vocabulary, it strengthens slightly. The cross-tradition reading is therefore a property of legal vocabulary at large, not an artefact of the particular 364-term selection on which §3.1.3 was computed.
Symmetric within-versus-cross tradition gap as a function of the background share of the pool. Mean ± 95% confidence interval across ten pool replicates per injection level.
Take-homeThe principal §3.1.3 reading survives heavy curation perturbation. The §4.1 synthesis cites this as the robustness condition under which the cross-tradition claim travels beyond the curated pool.
Technical apparatusOpen technical detail
Δρsym(p) = (ρ̄W(p) + ρ̄S(p)) / 2 − ρ̄cross(p)
at 0% background = 0.538 · at 25% background = 0.542 · at 75% background = 0.590 · replicates / level = 10
Confidence intervals across pool replicates at each injection level. No level whose interval excludes the no-injection baseline.
Same-lemma divergence at term level
Legal scenarioConsider a legal term and its direct Chinese translation — say, specimen signature and 簽名樣本 (literally signature sample): the same legal object, a reference exemplar of an authorised signer's handwriting deposited with a bank or court. When one model trained on English legal corpora and one model trained on Chinese legal corpora encode the pair, do they recognise the two strings as the same concept? And does a single model that handles both languages at once agree? If the two tradition-specialised models disagree but a single bilingual model agrees, the disagreement cannot be a calibration mismatch between models. It must lie in what each tradition-specialised model has internalised.
Result in wordsOn the same set of same-lemma legal pairs, two models trained one on English legal corpora and one on Chinese legal corpora (BGE-EN-large × BGE-ZH-large) report cosine similarity in the band −0.11 to +0.10: at best mild alignment, often outright anti-correlation. A single bilingual model that places both languages in one semantic space (BGE-M3-EN × BGE-M3-ZH) reports cosine in the band +0.27 to +0.87 on the very same pairs. The disagreement is not a property of the model architectures: when the tradition layer is removed by a shared model, the lemmas align.
Each point is one same-lemma pair (English headword paired with its canonical Chinese translation in the Hong Kong DOJ bilingual glossary). Cosine under the two tradition-specialised models on the x-axis, cosine under the single bilingual model on the y-axis. The cloud sits consistently above the y = x diagonal.
English term
Chinese term
Cross-tradition cosine
Bilingual cosine
specimen signature
簽名樣本
-0.048
+0.839
pharmacist
註冊藥劑師
-0.045
+0.809
non-identifiable
非可識辨身分資料
-0.027
+0.823
central bank
中央銀行
+0.006
+0.853
Lead example: specimen signature / 簽名樣本. The two tradition-specialised models place the pair at cosine -0.048 (near-orthogonal or weakly anti-aligned). The single bilingual model places it at +0.839 (strongly aligned). The instrument is not failing: it is reporting a difference that the tradition-specialised training itself has internalised.
Take-homeCross-tradition divergence on the very same legal lemma is a property of how each tradition's training corpus encodes the concept, not a calibration mismatch between model architectures. When both languages flow through a single model, the divergence resolves. §4.1 of the thesis reads this as the term-level proof of the §3.1.3 agreement claim.
Technical apparatusOpen technical detail
eligible same-lemma pairs = 4 156 · attestation filter = at least two ordinance contexts per term, each language · tradition-specialised models = BGE-EN-large × BGE-ZH-large · bilingual model = BGE-M3-EN × BGE-M3-ZH · cross-tradition cosine range = [−0.106, +0.100] · bilingual cosine range = [+0.273, +0.867]
Same-lemma pairs are English headwords paired with their canonical Chinese translation in the Hong Kong DOJ bilingual glossary, restricted to terms with at least two ordinance attestations in each language. Cosine is computed on the mean of the attested context vectors.
Expected failure modes
Legal scenarioTwo readings in Chapter 3 do not align with what the rest of the panel reports. §3.1.1 of the thesis identifies two model regimes on which the legal-versus-control diagnostic cannot be relied upon; §3.1.4 reports a negative-control probe whose result is the absence of the very signal the probe was designed not to find. Both are limits of the instrument, declared explicitly by §4.2 of the thesis.
Result in wordsFreeLaw-EN, the model fine-tuned on a United States legal corpus, fails the §3.1.1 legal-versus-control test in the sign-reversed direction: rank-biserial r = −0.121, with the legal-control median sitting below the legal-legal median, so the alternative hypothesis of the one-sided test is rejected. The reason is structural: a model steeped in legal English applies its legal prior to ordinary vocabulary as well, treating I, you, here under the same representational regime as trustee, lien, registration. The diagnostic operates on general-purpose models, not on already-fine-tuned ones.
Per-model rank-biserial r on the §3.1.1 legal-vs-control test, bare encoding. The two muted-grey bars (FreeLaw-EN at r = -0.121, Qwen3-0.6B-EN at r = -0.044) are the two diagnostic regimes that §4.2 of the thesis declares as outside the instrument's reliable scope. Hover for the p-value of each model.
Result in wordsThe §3.1.4 negative-control probe on contract value finds no doctrinal break, as the law prescribes: common law imposes the writing requirement on contracts for the sale of land regardless of consideration, so the sequence from symbolic to massive contract value should not register a stable structural break. It does not. The English-side readings cluster the modal break at the linguistic midpoint (the generic artefact the pre-registration anticipated for a uniform sequence); the Chinese-side readings disaggregate without converging on any alternative break. The probe is not finding a signal where the law expects none — which is, on this design, exactly the success criterion.
Take-homeBoth readings make the instrument's negative space visible: the §3.1.1 diagnostic does not apply to legally-fine-tuned models, and the §3.1.4 probe behaves as the law prescribes when the law prescribes no threshold. §4.2 of the thesis carries these as the explicit limits on the affirmative reading of §4.1.
Technical apparatusOpen technical detail
FreeLaw-EN, legal-vs-control r = −0.121 · FreeLaw-EN, p (one-sided) = 1.0 · Negative control, ensemble Spearman ρ = 0.651 · Negative control, modal break = at the linguistic midpoint, no doctrinal anchor
FreeLaw-EN passes the §3.1.1 within-domain test (r = +0.214 bare, +0.258 attested) but fails the legal-versus-control test; the fine-tuning collapses the term-class boundary the second test is designed to detect.
The bilingual control
Legal scenarioA sceptic might argue that the cross-tradition gap of §3.1.3 is an artefact of model identity: nine cross-tradition pairs, however symmetrically chosen, are still nine pairs of distinct models, trained by different teams on different curations. To rule out the model-identity confound, two bilingual models (BGE-M3 and Qwen3-Embedding-0.6B) embed the entire 364-term lexicon twice — once in English, once in Chinese — and the same agreement statistic is computed on the pair (English side × Chinese side) of each.
Result in wordsThe two bilingual readings cluster at ρ̄ = 0.316 — within the cross-tradition band of 0.246 and statistically indistinguishable from it, well below the within-tradition floors of 0.712 (Western-trained) and 0.868 (Chinese-trained) by more than four times the typical confidence-interval width. Holding model identity fixed and varying only the language of input does not close the cross-tradition gap. The factor that explains the §3.1.3 reading therefore cannot be model identity; it must be something the models have absorbed from the corpora on which they were trained.
Pair
Spearman ρ
Reading
Within Western-trained (3 pairs)
0.712
models agreeing within tradition
Within Chinese-trained (3 pairs)
0.868
models agreeing within tradition
Cross-tradition (9 pairs)
0.246
models from the two traditions, compared
Bilingual control (2 pairs)
0.316
same model, two languages of input
The four group means side by side. The bilingual control bar sits firmly in the cross-tradition band, not in the within-tradition one.
Take-homeThe bilingual control is the causal counterfactual of §2.3 of the thesis: by holding model identity fixed, it isolates the corpus-tradition component of the cross-tradition gap. The reading delivers: the gap survives the manipulation.
Technical apparatusOpen technical detail
within Western-trained = 0.712 · within Chinese-trained = 0.868 · cross-tradition (9 pairs) = 0.246 · bilingual control (2 pairs) = 0.316
Bilingual control pairs: BGE-M3 read on its English side against itself read on its Chinese side, and the same for Qwen3-Embedding-0.6B. Source: §3.1.3 of the thesis.