kandpal-2023-dbpedia-spotlight-2-1tb-128cpu-3weeks

IN premise — summaries/2026/08/24/kandpal-2023-long-tail-knowledge-s2-identifying-relevant-pre-training-data.md

Created 2026-08-25T02:58:06+00:00

The entity-linking pipeline used DBpedia Spotlight on a 128-CPU-core machine and processed 2.1 TB of pre-training data in approximately 3 weeks.

Summary

Even with a 128-core machine, linking entities across 2.1 TB of training data takes roughly three weeks, making this step a major time and compute bottleneck in the pipeline. This sets the practical cost floor for any re-run, scaling, or replacement of the entity-linking stage.