prh-binary-nce-optimal-equals-kpmi-plus-constant

Summary

Minimizing the binary NCE contrastive loss is, at its best possible, just learning how much two items co-occur beyond what you'd expect by chance, plus a fixed offset that depends on your label ratio. In other words, this result tells the system that the contrastive objective is fundamentally a mutual-information estimator, so any analysis of its behavior can be reframed in terms of how well it captures real co-occurrence structure versus random independence.