Is the LMSYS chatbot arena leaderboard trustworthy?
Premium
18
Ṁ69702027
57%
chance
1D
1W
1M
ALL
LLMs can distinguish their own output from the output of different LLMs and they have a preference for their own output, so it's technically feasible to manipulate the leaderboard by throwing an LLM at the chatbot arena to upvote its own completions.
Has this happened yet? Will it happen soon?
Resolves NO iff, before 2027/7/1, credible media reports state that the lmsys leaderboard has been manipulated with sockpuppet accounts / fraudulent voting. A statement coming directly from lmsys would also count.
Resolves YES otherwise.
This question is managed and resolved by Manifold.
Get
1,000
and3.00
Sort by:
@MalachiteEagle yes (though note that the question would resolve NO in that case). I'm only counting 155 models now.
Related questions
Related questions
Which LLM will be rank 1 at the end of 2025 on ChatBot Arena?
What organization will have the highest ELO score in the LMSYS Org Chatbot Arena Leaderboard at the end of Dec, 2024?
What organization(s) will be ranked #1 in the LMSYS Org Chatbot Arena Leaderboard at the end of December 2024?
Who will ever rank #1 in LMSYS Chatbot Arena Leaderboard in 2025?
Chatbot Arena - top 3 labs EOY 2024
Will GPT-4.5 top the LLMSys Chatbot Arena leaderboard within a month of its release?
78% chance
Who will ever rank Top 10 in LMSYS Chatbot Arena Leaderboard in 2025?
Will any LLM outrank GPT-4 by 150 Elo in LMSYS chatbot arena before 2025?
20% chance
Which organizations are responsible for the new anonymous models on lmsys chatbot arena? (August)
Will GPT-5 top the LLMSys Chatbot Arena leaderboard within a month of its release?
85% chance