ACL 2022: SimpleKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language ModelsPaper Documentation:

1. Abstract
   1. Task: Text-based methods still lag behind graph embedding-based methods like TransE, RotatE. They identify the key issue is efficient contrastive learning.
   2. Contribution 1: Introduce three types of negatives (more negatives than before): in-batch negatives, pre-batch negatives, and self negatives
   3. Contribution 2: Combined with InfoNCE loss, our model SimKGC outperforms other embedding-based methods on several benchmark datasets
2. Intro
   1. Limitation of the current models:
      1. Results on popular benchmarks tell a different story that text-based methods still lag behind even with pre-trained language models
   2. Motivation: inspired by the recent progress on contrastive learning
3. Pipeline
   1. Bi-encoder architecture. Two encoders are initialized with the same pre-trained language model but do not share parameters.
   2. Negative sampling: for knowledge graph completion the training data only consists of positive triples. Therefore, negative sampling needs to sample one or more negative triples to train discriminative models.
   3. In-batch negatives; pre-batch negatives; self-negatives
   4. loss function: InfoNCE loss with additive margin
4. Experiments:
   1. Comparison between embedding-based methods, text-based methods, and their models with ablation studies.
   2. Results show that self-negatives make the model much more effective
5. Limitations:
   1. SimKGC does not enable easy-to-understand interpretations
   2. dealing with false negatives
   3. Are there any loss functions that perform better than the infoNCE loss?
6. Takeaways
   1. Analysis to get further insights is important to guide future research.
   2. Ideas may be simple, but strong and solid.