Sauerkraut, Not Burgers: Why German LLMs Need German Data

TL;DR

For pre-training, we decided to implement our own Common Crawl filtering pipeline for German and complement it with open datasets and synthetic data. Our goal was to reach 4T German tokens, which couldn’t be met with the 1.94T tokens that open datasets provide. Openly available datasets also wouldn’t allow us to continuously get up-to-date data. That’s why we complemented those datasets with more than 2T German tokens that we curated and generated ourselves.

Introduction

At Aleph Alpha, we want to train language models that, alongside English, perform well in German language tasks. This means that the models should produce grammatically correct German, should have a solid understanding of German cultural context and be able to fulfill our customers’ use cases. To train such models we need appropriate training data. Early on, we determined that existing open datasets would not suffice, at least not by themselves. We have also considered generating data synthetically but learned that this isn’t as easy as it sounds. This is why we decided to curate our own organic data and supplement it with open datasets and synthetic data.

The limitations of existing open datasets

Training a state-of-the-art LLM requires upwards of 20T tokens (see e.g. Nemotron 3 Super). And our ablations have shown that a ratio of 20% of German data is optimal for our German benchmarks. This leaves us with a lower bound of 4T German tokens needed for training. Note that this refers to training tokens, not unique tokens: since datasets are commonly upsampled during training, the underlying data pool can be smaller than 4T, though not arbitrarily so, because repeating data stops paying off after roughly four epochs (Muennighoff et al.).

In the following table, we list some of the existing open German datasets. You can see that they add up to around 1.94T tokens. We list those datasets to illustrate the gap between available open datasets and the amount of German data we need for training our models, not implying that we trained our models on these open datasets.

Open German datasets, their sizes and the year of the most recent crawl or source version they contain. Token sizes in this table have been estimated by dividing their uncompressed byte size by 4 to enable a fair comparison between datasets.

Two caveats make the 1.94T an upper bound. First, KletterMix and MT-Nemotron-CC, about 600B tokens together, are machine translations of English corpora and carry the problems we discuss in the next section. Second, HPLT, FineWeb2 and German Commons are all derived from Common Crawl and overlap considerably, so their sizes are not additive after deduplication.

Our own curation pipeline yielded around 1.3T tokens, which, together with our own synthetic data, brings us a lot closer to our goal. This is illustrated in the following figure.

The other issue with open datasets is that they are snapshots. Each one is built from crawls up to a fixed date and then frozen; FineWeb2, for instance, contains nothing crawled after April 2024, so a model trained on it will, without retrieval or tool access, confidently answer “Olaf Scholz” when asked who the German chancellor is. Newer releases push the date forward, but always with a lag of many months and only when their maintainers decide to publish again. If we want our models to have up-to-date world knowledge, we cannot depend on that cadence. By building our own Common Crawl curation pipeline, we can extend our dataset with each new crawl as it appears.

The limitations of synthetic data generation

Synthetic data is a natural complement to organic data, and we do use it. However, synthetic data is only as good as the model that produces it. Whatever that model knows, doesn’t know, or was aligned to say ends up in the training data, and from there in our own models. This is why we are careful about which models we use for this. We prefer open-weights models that we can run on our own infrastructure. Within these constraints, we use synthetic data in two ways: to rephrase existing German documents and to translate documents from other languages. Both approaches have their limitations, which we discuss in the following.

Let’s start with rephrasing. Here, an LLM rewrites an existing organic document – for instance as an encyclopedia entry, a Q&A dialogue, or a textbook passage – while preserving its content. Because the model only reformulates what is already there, the requirements on it are much lower: it does not need deep German world knowledge, only good German. Rephrasing is valuable because it teaches the model the same facts in many surface forms, which improves generalization, and because it effectively multiplies scarce organic data – exactly our problem for German. Open-weights models fit this bill, and we used them to augment our data: in Aleph-Alpha-GermanWeb, we produced roughly 1T tokens by rephrasing German FineWeb2 data. Rephrasing only multiplies what already exists, though: it creates no new knowledge, and the pool of German source texts remains finite.

LLMs can also be used to translate data from English or other languages to German. We found that LLM-based translation of datasets can work with the right choice of model and the right combination of prompts and chunking strategy, i.e. how a long document is split into pieces that the model translates one at a time. However, synthetic translation suffers from two problems. The first is that, while most of its output adheres to the correct grammar, it might still occasionally suffer from what is commonly referred to as “translationese”, the tendency to literally translate idioms from one language to another (e.g. “Drive safe!” might get translated to “Fahre sicher!” instead of the more fitting “Komm gut an!”).

The other shortcoming of synthetic translations is that cultural context is not translated: if an English website discusses buying houses in Miami or ordering burgers, the translated document will not discuss flats in Berlin or ordering “Sauerkraut and Schupfnudeln”. This is not a problem in itself. A German model benefits from knowledge about Miami, Jakarta or Buenos Aires just as much as an English one does, and a corpus that carries many cultural perspectives is a strength. The problem is one of distribution. A corpus translated from English inherits the geographic, demographic and institutional skew of the English web, so a model trained predominantly on it learns to speak German about a world that looks American. Recent work on synthetic persona data makes the same point from the other direction: Pleias and NVIDIA ground their Nemotron-Personas datasets in national census statistics precisely because a dataset that misrepresents the population a model will serve produces systematically skewed outputs, and the US edition shows how strongly such data stays anchored to its origin.

There’s also the argument (e.g., put forward by the ÜberWeb authors) that knowledge transfers between languages. But both their work and the more recent work by Gaber et al. suggest that careful multi-lingual curation and potentially targeted translation are needed to get optimal language transfer without losing on individual languages’ performance.

Taken together, this leaves synthetic data in a supporting role. Rephrasing works and gave us about 1T tokens, but it can only multiply organic German data that already exists. Translation is technically feasible but imports the distribution of the source language. Neither approach replaces organic German web data; both depend on it. This is why we decided to build our own curation pipeline based on Common Crawl.

Our own curation pipeline

Our starting point was a web data pipeline similar to the one described in the Olmo 3 tech report, which was used to curate an English-only dataset. However, it turns out that such a pipeline recipe cannot be literally applied to another language without losing too much valuable data. Also, if you can recall the knowledge cutoff problem we discussed earlier in this post, we want to be able to ingest new crawls as they arrive. We will focus on why and how to adapt our Common Crawl pipeline to German in this blog post. In a future blog post we will shed light on the engineering challenges that come with building a web-scale data curation pipeline.

The figure above shows the eleven stages our data passes through, from raw Common Crawl snapshot to the final training mix. After downloading and extracting text from the WARC files, a URL blocklist and a set of language-specific heuristic filters remove blocked domains and obvious noise. Three deduplication stages – exact, fuzzy, and substring – then remove redundant content at increasing granularity. The surviving documents are annotated with quality and topic scores, PII is replaced with special tokens, and documents are sorted into quality buckets that serve as the basis for how they are mixed and upsampled in the training data.

Beyond the individual stages, a few design principles shape the pipeline as a whole. First, it is incremental: new Common Crawl snapshots can be added as they arrive, without reprocessing everything that came before – the answer to the knowledge-cutoff problem discussed earlier. Second, it is storage-efficient: each stage records only what changed rather than materializing a full copy of the data, so that a dozen stages do not mean a dozen copies. Third, it is traceable: every document in the final mix can be traced back to its source URL, crawl, and the decisions made at each stage. If we later find problems in our training data, we can identify where they originated and fix them at the root rather than patch them downstream. Fourth, it is parameterized by language: thresholds, word lists, and classifiers are configuration, which is what allowed us to support German alongside English and will allow us to add further languages. Finally, privacy and quality are built into the pipeline: PII replacement and URL filtering are fixed stages, quality annotation informs the final mix, and filtering thresholds are chosen based on ablations on our German benchmarks.

There are already more than 120 Common Crawl snapshots out there and new ones arrive roughly every month. And we want to be able to add them to our datasets as they arrive, keeping in mind that every new crawl contains roughly 100M German documents, or about 50B tokens. Note that these are raw numbers: deduplication and heuristic filtering happen downstream and will shrink this count considerably. This is one of the central challenges of data curation: every filtering step trades volume for quality.

Deduplication is a case in point. We apply three levels: exact deduplication removes byte-identical documents, fuzzy deduplication (MinHash) removes near-duplicates such as the same article with different boilerplate, and substring deduplication removes long repeated passages within otherwise distinct documents, e.g. shared disclaimers or navigation text. For our organic web data in pre-training, we do this partly for efficiency: every duplicate is a training step spent on nothing new. For mid-training, we deduplicate all datasets - including open ones. Some open datasets turn out to contain more than 50% exact duplicates (the “high quality” subset of Nemotron-Math-v2 being one example), which silently distorts the data mix. Muennighoff et al. showed that repeating data is beneficial only up to a point, with returns diminishing after roughly four epochs. If we want to upsample high-quality data deliberately, we need to know how often the model actually sees each document – and that requires starting from a deduplicated baseline.

Our overarching goal is to extract as much high-quality data from every dump as possible, so every heuristic filter has to be re-tuned for German. Two examples show the range of what that means: the word length filter, where the English thresholds cost a modest but avoidable 300,000 documents per crawl, and the stop-word filter, where they would throw away most of the German web.

The word length filter keeps a document when its mean word length falls inside [word_len_lower, word_len_upper]. Here is an example of why German and English need different thresholds for this filter, an excerpt from a municipal notice about registration of residence:

The German snippet is a typical piece of administrative prose, yet its mean word length (11.7 characters) is more than twice that of the English translation (5.7 characters) and falls outside Gopher’s English bounds of 3 to 10. With the [0, 15] interval proposed by FineWeb2 for deu_Latn, the rejection rate on German data drops from 0.3% to 0.02%. Small numbers, but at roughly 100M German documents per crawl, that is 300,000 ordinary German documents per crawl that we keep instead of throwing away.

The stop-word filter keeps a document that contains at least two words from a short list of very high-frequency function words. The English list is the Gopher one: {the, be, to, of, and, that, have, with}. The German list has been proposed by FineWeb2 and looks like this:

der, die, das, den, dem, des, und, in, im, von, mit, als, er, wurde, für

The effect here is far more drastic: applied to German documents, the English stop-word list rejects more than 87% of the documents reaching this stage of the pipeline, compared to 0.6% with the German list – roughly 150 times as many. In other words, without this single adjustment we would discard seven out of eight German documents.

What does a German document that gets past the English list look like? Below is one that survived: a CORDIS project page whose English project abstract supplies the stop words the filter looks for. The text is reproduced as it left the extraction stage, navigation boilerplate included.

Skip to main content
(öffnet in neuem Fenster)
de
CORDIS - Forschungsergebnisse der EU
CORDIS

Flow physics and interaction of laminar-turbulent transition and flow separation studied by direct numerical simulations

Projektbeschreibung

DE EN ES FR IT PL

Fortgeschrittene Simulationen könnten Aufschluss über veränderte Strömungsablösung geben

Die Forschenden des vom Europäischen Forschungsrat finanzierten Projekts TRANSEP werden sich mit den Problemen der numerischen Strömungsmechanik bei der Vorhersage der komplexen Strömungsphysik im Zusammenhang mit dem Übergang von Laminar- zu Turbulenzströmungen und der Strömungsablösung befassen. Durch den Einsatz von Datenverarbeitung im Multi-Peta-Maßstab werden die Forschenden rechnergestützte Methoden erarbeiten, mit denen diese komplexen Strömungsphänomene vorhergesagt, verstanden und gesteuert werden können. Bei den vorgeschlagenen Forschungsarbeiten werden das unerwartete Verhalten laminarer Bereiche an Tragflächen, die dramatischen Veränderungen in laminar-turbulenten Übergangs- und Ablösungsregionen an instationären Tragflächen und die komplexe Interaktion mehrerer Trennungs- und Ablösungsregionen an Hochauftriebsflügelkonfigurationen beleuchtet.

Ziel

The vision spelled out in this proposal is to overcome the failure of Computational Fluid Dynamics to tackle one of the central unsolved fluid physics problems, namely predicting the sensitive flow physics associated with laminar-turbulent transition and flow separation. A recent, highly influential report by NASA (Slotnick et al., 2014) clearly states that the major shortcoming of CFD is its “… inability to accurately and reliably predict turbulent flows with significant regions of separation”, most often associated with laminar-turbulent transition.

The research proposed here will address this shortcoming and develop and utilize computational methods that are able to predict, understand and control the sensitive interplay between laminar-turbulent transition and flow separation in boundary layers on wings and other aerodynamic bodies.

We will be able to understand enigmas such as the recent results from the experiments of Saric et al. at the Texas A&M Univeristy where the laminar area of a wing grows after a smooth surface have been painted (increased roughness), or the drastic changes of laminar-turbulent transition and separation locations on unsteady wings, or the notoriously difficult interaction of multiple separation and transition regions on high-lift wing configurations. For such flows there have been little understanding of flow physics and few computational prediction capabilities. Here we will perform simulations that give completely new possibilities to visualize, understand and control the flow around such wings and aerodynamic bodies, including the possibility to compute and harness the flow sensitivities.

We will tackle these outstanding flow and turbulence problem using the new possibilities enabled by multi-peta scale computing.

Wissenschaftliches Gebiet (EuroSciVoc)

CORDIS klassifiziert Projekte mit EuroSciVoc, einer mehrsprachigen Taxonomie der Wissenschaftsbereiche, durch einen halbautomatischen Prozess, der auf Verfahren der Verarbeitung natürlicher Sprache beruht. Siehe: Das European Science Vocabulary.

Sie müssen sich anmelden oder registrieren, um diese Funktion zu nutzen

Programm/Programme

Mehrjährige Finanzierungsprogramme, in denen die Prioritäten der EU für Forschung und Innovation festgelegt sind.

Thema/Themen

Aufforderungen zur Einreichung von Vorschlägen sind nach Themen gegliedert. Ein Thema definiert einen bestimmten Bereich oder ein Gebiet, zu dem Vorschläge eingereicht werden können. Die Beschreibung eines Themas umfasst seinen spezifischen Umfang und die erwarteten Auswirkungen des finanzierten Projekts.

Finanzierungsplan

Finanzierungsregelung (oder „Art der Maßnahme“) innerhalb eines Programms mit gemeinsamen Merkmalen. Sieht folgendes vor: den Umfang der finanzierten Maßnahmen, den Erstattungssatz, spezifische Bewertungskriterien für die Finanzierung und die Verwendung vereinfachter Kostenformen wie Pauschalbeträge.

ERC-ADG - Advanced Grant

Alle im Rahmen dieses Finanzierungsinstruments finanzierten Projekte anzeigen

Aufforderung zur Vorschlagseinreichung

Verfahren zur Aufforderung zur Einreichung von Projektvorschlägen mit dem Ziel, eine EU-Finanzierung zu erhalten.

(öffnet in neuem Fenster) ERC-2015-AdG

Alle im Rahmen dieser Aufforderung zur Einreichung von Vorschlägen finanzierten Projekte anzeigen

Gastgebende Einrichtung

KUNGLIGA TEKNISKA HOEGSKOLAN
Netto-EU-Beitrag

Finanzieller Nettobeitrag der EU. Der Geldbetrag, den der Beteiligte erhält, abzüglich des EU-Beitrags an mit ihm verbundene Dritte. Berücksichtigt die Aufteilung des EU-Finanzbeitrags zwischen den direkten Begünstigten des Projekts und anderen Arten von Beteiligten, wie z. B. Dritten.
€ 2 097 520,00
Adresse
BRINELLVAGEN 8
100 44 STOCKHOLM
Schweden

Auf der Karte ansehen
Region
Östra Sverige Stockholm Stockholms län
Aktivitätstyp
Higher or Secondary Education Establishments
Links
Gesamtkosten

Die Gesamtkosten, die dieser Organisation durch die Beteiligung am Projekt entstanden sind, einschließlich der direkten und indirekten Kosten. Dieser Betrag ist Teil des Gesamtbudgets des Projekts.
€ 2 097 520,00

Begünstigte (1)

Herunterladen Den Inhalt der Seite herunterladen

Letzte Aktualisierung: 27 Juli 2023

Mein Booklet

Permalink: https://cordis.europa.eu/project/id/694452/de

European Union, 2026

Mein Booklet 0 0

The heuristic filters are not the only stage that needed German-specific attention. The quality classifier models, which attach scores for quality and topic categories to each document, also had to be adjusted to account for German grammar and cultural specificity. Our Aleph-Alpha-GermanWeb paper touches upon this; a full treatment is beyond the scope of this post.

Compliance and privacy

To prevent training on illegal, infringing or pirated content, we filter our data based on a URL filter list.

To protect personal data, we replace personally identifiable information with special tokens before training. This covers, among other things, personal contact information such as email addresses and phone numbers, IP addresses and bank account numbers. For email addresses, this includes not only standard ones with the usual “@” separator, but also obfuscated variants such as “name [at] domain”. People often use this notation precisely to keep their address from being scraped by bots, and we saw it as an additional reason to respect that intention. Both the URL filtering and the PII replacement reflect our legal and compliance commitments, in particular under the EU AI Act.

These measures may come at a cost, and we wanted to make that trade-off consciously. We therefore designed dedicated benchmarks that test how well a model handles PII-like entities. For example, it should still understand which country a phone number is from, or whether something is a valid IBAN. On these PII benchmarks, our model performs 5 percentage points worse compared to a model trained on the same data without PII replacement – an expected consequence of never having seen real PII during training – while all other benchmarks were unaffected. We consider this an acceptable price for a model that does not memorize personal data.

Evaluation

Benchmarks face a similar scarcity problem as training data, since they are essentially small, specialized datasets themselves. We use them both to compare data recipes against each other and to check whether the final model handles our customers’ use cases. Open-source German benchmarks alone don’t provide enough coverage of realistic German tasks, nor enough sensitivity to demonstrate the value of adding our German data. We therefore built a dedicated German evaluation suite, both hand-crafted and model-generated, to measure downstream performance with confidence.

These benchmarks are only meaningful, however, if their contents don’t leak into the training data. Web-scale crawls almost inevitably contain copies of popular benchmarks, and a model trained on them will post impressive scores while underperforming on the real-world use cases that we and our customers actually care about. We therefore decontaminate our datasets against all benchmarks we evaluate on. Following the approach taken by AI2 for Olmo 3, we currently apply decontamination only to our mid-training (second-stage) data, but plan to extend it to pre-training in the future.

Conclusion

This blog post has described why and how we adapted our web data pipeline to German. In the future, we will apply the same rigor and attention to detail to other languages. Our pipeline is already language-parameterized: German was the first language we added beyond English, and our ablations show that the resulting data provides the value our customers need. In summary, we have curated and generated more than 2T high-quality German tokens to date, which we supplement with existing open datasets and which our ablations show can be confidently upsampled to improve downstream performance.

Acknowledgements

The pipeline and datasets described here are a team effort. Our thanks go to the whole data team at Aleph Alpha Research, in particular Ivo Schaper, Peter Rodenkirch Nfissi, Fabian Gebhart, Bastian Harren, Tom Burns, Mohamed Soliman, Denis Waßmann and Lukas Schott, whose work is in every stage of this pipeline. We are grateful to Yasser Jadidi, Fabien Benureau, Michael Hofmann, Noé Beckerle Vallejo, Helena Treeck and Paige Reddington for their careful reviews and valuable feedback on this blog post, to Svenja Fahlisch for coordinating it, and to Finn Kiesinger for publishing it on the website.