Official Code for our paper:
LLMs are Brittle to Simple Code Transformations: Introducing CETBench – A Benchmark for Code-Equivalence Checking
CETBench is accepted in in Findings of ACL 2026.
Paper Link: https://aclanthology.org/2026.findings-acl.2070/
We use python 3.10.
git clone https://github.com/dair-iitd/CETBench.git
cd CETBench
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtThe input dataset must be provided as a JSON file. Each JSON object represents a single programming problem.
A unique identifier for the programming problem.
"name": "problem_001"This field should uniquely identify every problem in the dataset.
Contains all correct reference solutions for the programming problem.
"solutions": {
"language": [3, 3, 4],
"solution": [
"...source code...",
"...source code...",
"...source code..."
]
}A list of integer language identifiers.
Each element corresponds to the solution at the same index in solution.
| ID | Programming Language |
|---|---|
| 1 | Python |
| 2 | C++ |
| 3 | Python 3 |
| 4 | Java |
For example,
"language": [3, 4, 3]means
solution[0]is written in Python 3solution[1]is written in Javasolution[2]is written in Python 3
A list of strings, where each string contains the complete source code for one correct solution.
The lengths of language and solution must be identical, and entries are matched by index.
Contains incorrect or buggy solutions in the same format as solutions.
"incorrect_solutions": {
"language": [3, 4],
"solution": [
"...incorrect solution...",
"...incorrect solution..."
]
}The same language identifiers and indexing rules apply.
Public test cases may optionally be included.
"public_tests": {
"input": [
"ab\naa\n",
"ab\nba\n",
"nzwzl\nniwel\n"
],
"output": [
"aa\n",
"-1\n",
"niwel\n"
]
}A list of input strings.
Each string represents the complete standard input (stdin) for a single test case.
A list of expected output strings.
Each string represents the complete expected standard output (stdout) corresponding to the input at the same index.
The lengths of input and output must be identical.
Private (hidden) evaluation test cases follow exactly the same format.
"private_tests": {
"input": [
"...",
"..."
],
"output": [
"...",
"..."
]
}{
"name": "problem_001",
"solutions": {
"language": [3, 4],
"solution": [
"print('Hello')",
"public class Main { ... }"
]
},
"incorrect_solutions": {
"language": [3],
"solution": [
"print('Wrong Answer')"
]
},
"public_tests": {
"input": [
"ab\naa\n",
"ab\nba\n"
],
"output": [
"aa\n",
"-1\n"
]
},
"private_tests": {
"input": [
"..."
],
"output": [
"..."
]
}
}- Each JSON object corresponds to one programming problem.
- Every entry in
languagemust correspond to the solution at the same index insolution. - Every test case input must have a corresponding expected output at the same index.
- Solution strings should contain the complete source code exactly as intended to be compiled or executed.