So what does the DataRater learn? It automatically identifies and down-weights data that aligns with human intuitions of low quality, such as incorrect text encodings, OCR errors, and irrelevant content.

The result? The DataRater is highly effective at filtering data, leading to significant compute efficiency improvements. In our experiments, we observed up to a 46.6% net compute gain while often improving final model performance.

We introduce the DataRater, a meta-learning method that learns to rate the value of each data point for training. Instead of manually specifying filtering rules, we train the DataRater to optimize for a simple goal: improving the training efficiency on a held-out dataset.