Efficient Open-Domain Question Answering

The official website for the open domain question answering challenge at NeurIPS 2020.

Home
Sign up for notifications
Task definition, data, and evaluation
Get the data
Baselines
Making a submission

This leaderboard displays all results for submissions to the EfficientQA unrestricted track. It includes all systems that were submitted to the official leaderboard, used for each of the EfficientQA restricted tracks, as well as systems that were run outside of the leaderboard system.

Model Participant Affiliation System size
* = self-reported
Accuracy
UnitedQA Microsoft Research & Dynamics 365 AI Microsoft Research & Dynamics 365 AI -- 54.00
Hybrid Facebook AI QA@FBAI -- 53.89
Model 2 FAIR-Paris&London Facebook 5.85 GiB 53.33
Model 1 FAIR-Paris&London Facebook 5.66 GiB 52.78
soseki ousia_tohoku Studio Ousia & Tohoku University -- 52.44
R2D2_v2_full KNOT@FIT Brno University of Technology *66 GiB 50.33
Soseki ousia_tohoku Studio Ousia & Tohoku University 5.63 GiB 50.17
Soseki ousia_tohoku Studio Ousia & Tohoku University 4.86 GiB 50.11
Soseki ousia_tohoku Studio Ousia & Tohoku University 5.37 GiB 49.39
  Noah_NLP Huawei Noah’s Ark Lab -- 48.06
  Noah_NLP Huawei Noah’s Ark Lab -- 47.61
R2D2_v2 KNOT@FIT Brno University of Technology 5.95 GiB 47.28
    Salesforce -- 46.83
R2D2 KNOT@FIT Brno University of Technology 5.38 GiB 46.06
    Salesforce *5.7 GiB 45.83
CDPR-FID BearQA UC Berkley -- 40.83
R1D1 KNOT@FIT Brno University of Technology 5.29 GiB 40.56
uclnlp-fair-efficientqa-xlg uclnlp-efficientqa University College London and Facebook AI Research *1.16GiB 39.11
DPR model serving using Vespa.ai Vespa Team Vespa.ai -- 35.00
REALM Orgainizers EfficientQA 27.19 GiB 34.89
uclnlp-fair-efficientqa-lg uclnlp-efficientqa University College London and Facebook AI Research 336.23 MiB 33.44
RDR-500MB naver-clova-parsing Naver Clova 484.69 MiB 32.06
RDR-6GB naver-clova-parsing Naver Clova 5.78 GiB 32.06
EFF-Large Beauty-P PA Tech -- 31.11
ba45-micro BA45 BA45 5.57 GiB 31.11
ba45-micro BA45 BA45 5.53 GiB 29.97
uclnlp-fair-efficientqa-lg uclnlp-efficientqa University College London and Facebook AI Research 226.09 MiB 29.44
ba45-micro BA45 BA45 5.33 GiB 28.83
RDR-500MB naver-clova-parsing Naver Clova 496.99 MiB 28.22
T5-1.1-XL + SSM Organizers EfficientQA 5.65 GiB 27.89
EFF-Small Beauty-P PA Tech -- 27.89
NTUSTTTTT Jia-Zhi_Guo National Taiwan University of Science and Technology 5.65 GiB 27.17
uclnlp-fair-efficientqa uclnlp-efficientqa University College London and Facebook AI Research 29.43 MiB 26.78
uclnlp-fair-efficientqa uclnlp-efficientqa University College London and Facebook AI Research 157.26 MiB 25.78
uclnlp-fair-efficientqa uclnlp-efficientqa University College London and Facebook AI Research 5.87 GiB 25.78
hsjdpr LSLL LSLL 13.97 GiB 25.61
aqa-udm AQA-UDM University of Montreal 14.17 GiB 25.22
uclnlp-fair-efficientqa uclnlp-efficientqa University College London and Facebook AI Research 28.61 MiB 24.83
ssm_nq_torch machnev Higher School of Economics 3.02 GiB 20.11
FRY-T FRY FRY 423.29 MiB 19.39
NTUSttttt Chia-Chih_Kuo National Taiwan University of Science and Technology 488.96 MiB 18.39
T5-1.1-small + SSM Organizers EfficientQA 486.61 MiB 17.61