Evals/BIG-Bench Hard

EVL

BIG-Bench Hard

Hard reasoning tasks

benchmarkReasoningAverage accuracy across the 23 tasks6.5K items

What this measures

BIG-Bench Hard — 23 tricky tasks (logical deduction, date arithmetic, sarcasm detection, multi-step puzzles) where earlier models struggled. A grab-bag stress test of reasoning.

Sample

'If today is Wednesday and it was 3 days before yesterday, what day was it?' — plus 22 other puzzle types.

Leaderboard · 100 models

Average accuracy across the 23 tasks — higher is better

1T3Q-qwen2.5-14b-v1.0-e3Open LLM LB · 2026-05

65.5

2T3Q-Qwen2.5-14B-Instruct-1M-e3Open LLM LB · 2026-05

65.5

3shuttle-3Open LLM LB · 2026-05

64.1

4internlm2_5-20b-llamafiedOpen LLM LB · 2026-05

63.5

5ultiima-72B-v1.5Open LLM LB · 2026-05

63.4

6internlm2_5-20b-chatOpen LLM LB · 2026-05

62.8

7calme-3.2-instruct-78bOpen LLM LB · 2026-05

62.6

8calme-3.1-instruct-78bOpen LLM LB · 2026-05

62.4

9Homer-v1.0-Qwen2.5-72BOpen LLM LB · 2026-05

62.3

10calme-2.4-rys-78bOpen LLM LB · 2026-05

62.2

11test-2.5-72BOpen LLM LB · 2026-05

62.2

12CalmeRys-78B-Orpo-v0.1Open LLM LB · 2026-05

61.9

13Qwen2.5-72B-2x-Instruct-TIES-v1.0Open LLM LB · 2026-05

61.9

14Qwen2.5-72B-InstructOpen LLM LB · 2026-05

61.9

15Gilgamesh-72BOpen LLM LB · 2026-05

61.8

16calme-2.2-qwen2.5-72bOpen LLM LB · 2026-05

61.8

17calme-2.1-qwen2.5-72bOpen LLM LB · 2026-05

61.7

18LeTriomphant2.2_ECE_iLABOpen LLM LB · 2026-05

61.6

19Rombos-LLM-V2.5-Qwen-72bOpen LLM LB · 2026-05

61.3

20ultiima-72BOpen LLM LB · 2026-05

61.1

21Qwen2.5-72B-Instruct-abliteratedOpen LLM LB · 2026-05

60.5

22Galactic-Qwen-14B-Exp2Open LLM LB · 2026-05

59.9

23Qwen2.5-72B-Instruct-abliteratedOpen LLM LB · 2026-05

59.9

24BigQwen2.5-52B-InstructOpen LLM LB · 2026-05

59.8

25calme-2.3-rys-78bOpen LLM LB · 2026-05

59.6

26FluentlyLM-PrinumOpen LLM LB · 2026-05

59.5

27calme-2.1-rys-78bOpen LLM LB · 2026-05

59.5

28calme-2.2-rys-78bOpen LLM LB · 2026-05

59.3

29EVA-Qwen2.5-72B-v0.2Open LLM LB · 2026-05

59.1

30RYS-XLargeOpen LLM LB · 2026-05

58.8

31RYS-XLarge-baseOpen LLM LB · 2026-05

58.7

32Qwen2.5-95B-InstructOpen LLM LB · 2026-05

58.5

33qwen2.5-test-32b-itOpen LLM LB · 2026-05

58.3

34Apollo_v2-32BOpen LLM LB · 2026-05

58.3

35Rombos-LLM-V2.5-Qwen-32bOpen LLM LB · 2026-05

58.3

36novablast-previewOpen LLM LB · 2026-05

58.2

37RombosBeagle-v2beta-MGS-32BOpen LLM LB · 2026-05

58.1

38ultiima-32BOpen LLM LB · 2026-05

58.1

39Linkbricks-Horizon-AI-Avengers-V2-32BOpen LLM LB · 2026-05

58.1

40TheBeagle-v2beta-32B-MGSOpen LLM LB · 2026-05

58.1

41Oxyge1-33BOpen LLM LB · 2026-05

58.0

42magnum-v2-72bOpen LLM LB · 2026-05

57.9

43QwentileSwapOpen LLM LB · 2026-05

57.7

44magnum-72b-v1Open LLM LB · 2026-05

57.7

45magnum-v1-72bOpen LLM LB · 2026-05

57.7

46Linkbricks-Horizon-AI-Avengers-V1-32BOpen LLM LB · 2026-05

57.6

47MG-FinalMix-72BOpen LLM LB · 2026-05

57.5

48Qwen2-72B-InstructOpen LLM LB · 2026-05

57.5

49Qwen2-72B-Orpo-v0.1Open LLM LB · 2026-05

57.4

50calme-2.1-qwen2-72bOpen LLM LB · 2026-05

57.3

51Llama-3.1-SauerkrautLM-70b-InstructOpen LLM LB · 2026-05

57.2

52Awqward2.5-32B-InstructOpen LLM LB · 2026-05

57.2

53Qwentile2.5-32B-InstructOpen LLM LB · 2026-05

57.2

5470B-L3.3-Cirrus-x1Open LLM LB · 2026-05

57.1

55internlm2_5-7b-chatOpen LLM LB · 2026-05

57.0

56Dracarys-72B-InstructOpen LLM LB · 2026-05

56.9

57Set-70bOpen LLM LB · 2026-05

56.9

58zetasepic-abliteratedV2-Qwen2.5-32B-Inst-BaseMerge-TIESOpen LLM LB · 2026-05

56.8

59calme-2.2-qwen2-72bOpen LLM LB · 2026-05

56.8

60Arcee-NovaOpen LLM LB · 2026-05

56.7

61Linkbricks-Horizon-AI-Avengers-V5-32BOpen LLM LB · 2026-05

56.7

62RomboUltima-32BOpen LLM LB · 2026-05

56.7

63L3.3-MS-Nevoria-70bOpen LLM LB · 2026-05

56.6

64Llama-3.3-70B-InstructOpen LLM LB · 2026-05

56.6

65Linkbricks-Horizon-AI-Avengers-V4-32BOpen LLM LB · 2026-05

56.6

66Qwen2.5-32B-Instruct-abliterated-v2Open LLM LB · 2026-05

56.5

67Linkbricks-Horizon-AI-Avengers-V3-32BOpen LLM LB · 2026-05

56.5

68Qwen2.5-32B-InstructOpen LLM LB · 2026-05

56.5

69Geedorah-14BOpen LLM LB · 2026-05

56.5

70Rosemary-14bOpen LLM LB · 2026-05

56.4

71Qwen2-VL-72B-InstructOpen LLM LB · 2026-05

56.3

72Smaug-Qwen2-72B-InstructOpen LLM LB · 2026-05

56.3

73Progenitor-V1.1-LLaMa-70BOpen LLM LB · 2026-05

56.2

74Ponder-14B-linearOpen LLM LB · 2026-05

56.2

75Linkbricks-Horizon-AI-Avengers-V6-32BOpen LLM LB · 2026-05

56.2

76Wendy-14BOpen LLM LB · 2026-05

56.2

77L3.3-Nevoria-R1-70bOpen LLM LB · 2026-05

56.2

78Phi-4-RR-ShoupOpen LLM LB · 2026-05

56.1

79SuperNova14bOpen LLM LB · 2026-05

56.1

80Fugazi14bOpen LLM LB · 2026-05

56.1

81Phi-4-Sce-exp-v0.1Open LLM LB · 2026-05

56.1

82Rune-14bOpen LLM LB · 2026-05

56.1

83huihui-ai-abliterated-Qwen2.5-32B-Inst-BaseMerge-TIESOpen LLM LB · 2026-05

56.0

84Nova-14b-sceOpen LLM LB · 2026-05

56.0

85NovaScotia-14b-stockOpen LLM LB · 2026-05

56.0

86Lineage-14BOpen LLM LB · 2026-05

56.0

87Alice-14BOpen LLM LB · 2026-05

56.0

88Phi-4-RStock-v0.1Open LLM LB · 2026-05

56.0

89Thalassic-Alpha-LLaMa-70BOpen LLM LB · 2026-05

56.0

90Llama-3.1-70B-InstructOpen LLM LB · 2026-05

55.9

91Mithril-14B-sceOpen LLM LB · 2026-05

55.9

92Phi-4-Model-Stock-v4Open LLM LB · 2026-05

55.9

93Phi-4-ReasoningRPOpen LLM LB · 2026-05

55.9

941up-14bOpen LLM LB · 2026-05

55.8

95Loke-14B-sceOpen LLM LB · 2026-05

55.8

96Rombos-LLM-V2.6-Nemotron-70bOpen LLM LB · 2026-05

55.8

97Luminis-phi-4Open LLM LB · 2026-05

55.8

98Venti-20bOpen LLM LB · 2026-05

55.8

99caramel-14BOpen LLM LB · 2026-05

55.8

100Oasis-14B-tiesOpen LLM LB · 2026-05

55.8

Benchmark health

headroom

Plenty of room at the top — this benchmark still separates frontier models clearly.

63.4

top-10 mean

65.5

best

100

models

What improves this score

Datasets & environments shown to raise it.

—+7.0

FineWeb-Edu+3.0

ALFWorld+3.0

UltraFeedback+2.5

BabyAI+2.0

Research using this benchmark

all →

Orca: Progressive Learning from Complex Explanation Traces of GPT-4 How Hard is it to Rig a Benchmark? A Social Choice Analysis of Leaderboard Robustness The Coupling Tax: How Shared Token Budgets Undermine Visible Chain-of-Thought Under Fixed Output Limits Post Reasoning: Improving the Performance of Non-Thinking Models at No Cost

Matched by name in title/abstract.