Sign inSign in

Deduplication

One word, taught differently in 2 fields. Each use is given apart.

In AI & ML

The process of identifying and removing redundant content from a dataset to improve compute efficiency and prevent overfitting.
as Percy Liang defines itStanford University

In Systems & CS

Storing a non-unique key once in a leaf node and maintaining a posting list of corresponding values.
as Andy Pavlo defines itCarnegie Mellon University

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Dataset management1Hash table1Leaf node1MinHash1Pretraining data1SQL joins1Window functions1

In AI & ML

2

Each in their own terms, from the easiest level.

  • Loubna Ben Allal

    Stanford University · Undergraduate

    The process of removing exact and near-duplicate files from training datasets to prevent model memorization and maintain creativity.
    • 25:21Behind the Scenes of LLM Pre-training: StarCoder Use Case

In Systems & CS

2

Each in their own terms, from the easiest level.

  • Martin Ladecký

    Devoxx · Undergraduate

    Removing duplicate transactions when a pending transaction is already booked in the system.
    • 22:2110 Billion Records vs. The Regulator: An Architectural Post Mortem of an 11TB Exadata Failure

Connections

6

The lectures that define it or assume it, and the concepts next to it.

4 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.