Tag: roberta

• Florence A New Foundation Model for Computer Vision (24 Oct 2023)
  

  This is my reading note for Florence: A New Foundation Model for Computer Vision. This paper proposes a foundation model for vision (image/video) and text based on UniCL loss. It uses Swin-transformer and Roberta for the encoder.

• RoBERTa A Robustly Optimized BERT Pretraining Approach (07 Oct 2023)
  

  This is my reading note for RoBERTa: A Robustly Optimized BERT Pretraining Approach. This paper revisits the design choice of BERT. It provides that 1) adding more data; 2) using larger batch size; 3) training for more iterations could significantly improves the performance. In addition, using longer sentence/context could also improve performance and next sentence prediction is no longer useful.

• Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (22 Sep 2023)
  

  This is my reading note for Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone. This papers propose a two-stage pre-training strategy: (i) coarse-grained pre-training based on image-text data; followed by (ii) fine-grained pre-training based on image-text-box data.

• An Empirical Study of Training End-to-End Vision-and-Language Transformers (21 Sep 2023)
  

  This is my reading note for An Empirical Study of Training End-to-End Vision-and-Language Transformers. This paper provides a good review and comparison of multi modality (video and text) model’s design choice.