Fine-Tuning an AI Guardrail for European Portuguese

There is a small, persistent failure that quietly erodes trust in AI-written copy for a Portuguese brand: the model writes Brazilian Portuguese. Not obviously, not every time — but a Brazilianism slips in, an accent goes missing, or a sentence drops into the informal "tu" register. To a European reader it lands as foreign, and foreign copy from a brand that claims to be local breaks trust instantly.

The obvious fix is to tell the model "write European Portuguese." We tried that. It is unreliable. Prompting nudges the odds but does not close the gap — the model still leaks PT-BR intermittently, and always at the moment you have stopped proofreading. A request is not a guarantee.

So we stopped asking and built a guarantee instead. This is how we fine-tuned a small guardrail model whose only job is to rewrite any text into clean European Portuguese while preserving the meaning — and, just as importantly, what it still gets wrong.

Turn the style guide inside out

The hardest part of fine-tuning is usually the data. You need matched pairs — bad input, good output — and thousands of them. Hand-writing those pairs would take forever.

We already had half the answer sitting in our own tooling. For years we have maintained a strict clean PT-PT style guide and a marker-based checker (check-pt-pt.sh) that scans text for known Brazilianisms. That checker exists to catch dirty text. The trick was to run it backwards.

Instead of correcting bad Portuguese into good, we took our clean PT-PT corpus and programmatically corrupted it into Brazilian Portuguese: introducing Brazilianisms, stripping accents, and swapping neutral phrasing into the "tu" register. Each corruption produced a (dirty → clean) training pair — the corrupted text as input, the original clean text as the target. The generator does the messy half; the pristine original is the answer key.

The guarantee lives in a cleanliness gate. Every target output must pass the checker's markers before it enters the dataset, so we can promise that the thing the model is learning to produce is genuinely clean — no accidental leakage in the labels. The result was 1,572 training pairs from 509 clean units, split into train, validation, and test with a group-aware method and verified to have zero leakage between the splits. (Group-aware matters: if two pairs derive from the same source unit, they must land in the same split, or your test score is quietly measuring memorisation.)

A 3B model, 12 minutes, on a 2018 GPU

The model is deliberately small. This is a guardrail, not a general assistant — it does one narrow rewrite task, so it does not need to be large.

We fine-tuned Qwen2.5-3B using QLoRA (4-bit) via Unsloth, at fp16, for roughly 2 epochs. The whole run took about 12 minutes on an RTX 2080 Ti — the same 11GB gaming card from 2018 we use for the rest of our local work. This is the same old-hardware theme as the rest of our stack: capability we own, on a card that cost a fraction of a cloud training bill.

Once trained, the guardrail is served locally through Ollama as a roughly 1.9GB quantized model. In the pipeline it sits as a deterministic slot at the very end: whatever the upstream model generates, the guardrail rewrites into clean European Portuguese before anything reaches a human. It is not a suggestion the writing model can ignore — it is a step that always runs.

The results, honestly

Here is where most write-ups round up. I am not going to.

On a held-out evaluation set — the no-leakage test split, text the model never saw in training — the base model's PT-PT clean-rate was already respectable, sitting in the high-80s to low-90s. The headline figure we cite is ~87% clean on the base model rising to ~100% after fine-tuning on that held-out set. Near-perfect, on that set.

But "near-perfect on the held-out set" is not "perfect everywhere," and the honest number depends on how strictly you score. When we later added a stricter, tu-aware scorer (more on why below), the same comparison read closer to 90% → 98%. That is still a large, real improvement — but it is 98%, not a flat 100%, and the two percent that remains is exactly the edge cases a synthetic test is worst at seeing.

So the honest summary is: the guardrail closes almost the entire gap on the language patterns it was trained on, and the residual is small but real. Anyone who tells you a fine-tune hit 100% everywhere is either not measuring strictly or not telling you the whole test.

Two lessons that cost us

The interesting part of this project was not the training. It was two bugs that taught us something general.

Lesson 1 — gendered nouns need article handling, not word swaps. Our first corruption logic treated the problem as a dictionary: find the Brazilian word, swap in the European one. That breaks on gender. Take tela (Brazilian, feminine) versus ecrã (European, masculine). A naive swap turns "na tela" into "no ecrã"… except it did the opposite — it produced grammatically wrong output like "no tela," because it swapped the noun but left the determiner agreeing with the old gender. The fix was gender-aware corruption that also flips the determiner along with the noun (o↔a, no↔na, do↔da, num↔numa, and so on). That generalises to any masculine↔feminine PT/BR pair, not just this one word.

Lesson 2 — the only honest test is running on real output. This one is the reason for the tu-aware scorer above. Our synthetic evaluation was blind to the informal "tu" register — because the checker did not list tu-forms as markers. That single omission cascaded: tu-register training examples passed the cleanliness gate (the gate did not know to reject them), and the eval never measured them (the scorer did not know to look). By both metrics the guardrail looked perfect on the "tu" problem, while doing nothing about it.

We only caught it by running the guardrail on a real generation rather than another synthetic test. On live downstream output it left "alinhada ao teu foco" — informal "tu" — completely uncorrected. The fix was to add a curated set of tu-markers so the gate now purges tu-forms from targets and the eval actually measures them; then we retrained. The takeaway is worth stating plainly, because it applies to any AI quality system:

A defect that isn't in your checker is neither trained away nor measured. So run the model on real downstream output, not just on synthetic tests — the synthetic tests can only see the failures you already thought to encode.

What it still gets wrong

A guardrail that claimed to be complete would be repeating exactly the mistake Lesson 2 just described. So here is what still slips through.

Two residual weaknesses stand out. First, a transport and rural lexicon the training corpus is thin on — the ônibus / pedágio / carona family of words that rarely show up in our source material, so the model has seen too few examples to reliably correct them. Second, rare "tu" verbs buried in relative clauses, where the informal form is far enough from the surface that the current markers do not always catch it.

It is a guardrail, not magic. It catches the high-frequency 95%-plus of PT-BR leakage that actually shows up in day-to-day copy, and it misses a long tail of rarer cases. That is a genuinely useful tool — as long as you know which is which and do not switch off the human read for the tail.

Why this matters if you sell in Europe

Step back from the mechanics and the point is simple. European-language quality at this level is something you cannot reliably get from a generic US cloud model — it will keep leaking the wrong dialect, and no prompt fully stops it. A small, fine-tuned guardrail does, for the common cases, deterministically.

And because it is a small self-hosted model, it delivers that quality without the usual trade. The client's text is rewritten on infrastructure you control; it never leaves the building for a third-party model. This is the same "sovereign and verifiable" method we apply to the rest of our stack — own the model, own the data path — applied here to language itself. You get EU-quality Portuguese and you keep the data. For a European brand, that is not a nice-to-have. It is the whole reason to build it this way.

The takeaway

Prompting a model to respect a dialect is a request. Fine-tuning a guardrail to enforce it is a guarantee — at least for the patterns you remembered to test. We got there by turning our own checker inside out to generate training data, training a 3B model in twelve minutes on a seven-year-old GPU, and then being honest about the two percent it still misses.

The most durable lesson had nothing to do with Portuguese: your AI quality system can only catch the defects you have already taught it to see. Run the thing on real output, watch where it fails, and add that failure to the checker. Then do it again.