Tag: nearest-neighbor

• In-Context Pretraining Language Modeling Beyond Document Boundaries (18 Oct 2023)
  

  This is my reading note for In-Context Pretraining: Language Modeling Beyond Document Boundaries. This paper proposes to group relevant instead of random documents in each batch to improve Long text learning. The relevant docs are found by performs a traveling salesmen problem in a graph of documents. The edges of two documents define whether the two documents are in the top k nearest neighbors.