ChatGPT Solves All Tested Qiskit Homework Assignments

Research predating ChatGPT already showed that code-generating models could solve many introductory programming problems [2]. Subsequent work has examined both productive uses of LLMs and risks to assessment validity. Denny et al. summarize the resulting shift in computing education, where educators must reconsider what students should learn and how achievement can be evidenced when code can be generated from natural-language descriptions [3]. Kasneci et al. similarly describe opportunities for personalized support together with risks involving accuracy, over-reliance, and assessment [4].

McDanel and Novak directly evaluated complete multi-part programming assignments with several LLMs and proposed strategies for making assignments more resistant [5]. Their results show that model performance depends on assignment context, starter code, tests, and task characteristics. Our work follows the same assignment-level concern but studies a specialized engineering domain. Quantum programs add circuit semantics, probabilistic measurement, qubit/classical-bit ordering, rapidly changing software interfaces, transpilation, and optional access to noisy hardware. These properties motivate deterrence mechanisms that do not arise in ordinary introductory programming, but they may also create only superficial barriers to an LLM-assisted student.

Several recent benchmarks establish that LLMs can generate substantial amounts of quantum code. Qiskit HumanEval contains more than 100 Qiskit tasks with canonical solutions and executable tests [6]. QuanBench evaluates functional correctness and quantum semantic equivalence on 44 tasks and documents frequent API and circuit-logic errors [7]. Qiskit QuantumKatas expands the scope to 350 pedagogically structured tasks across 26 categories and 39,200 model runs [8]. QuanBench+ aligns tasks across Qiskit, Cirq, and PennyLane and shows that execution feedback can substantially improve quantum-code generation [9]. Recent work also reports strong gains when general-purpose models are combined with iterative execution feedback rather than domain-specific fine-tuning [10].

These benchmarks ask whether a model can generate functionally correct code for many individual tasks. Our unit of analysis is different: a complete take-home homework package as experienced by a student. A submission may include a seeded notebook, exact JSON schema, expected output, reflections, circuit statistics, and optional hardware evidence. The question is not merely whether ChatGPT implements QFT or Deutsch–Jozsa. It is whether assignment-level defenses force enough independent execution and reasoning that direct provision of the complete assignment no longer yields an executed, grader-accepted submission. This distinction positions the present work between quantum-code benchmarking and engineering-education assessment research.

The threat model is a minimally engaged student with ordinary access to ChatGPT. The student can:

1. 1.

   provide the assignment text, notebook, or student-visible files;

2. 2.

   request a complete solution and required submission artifacts;

3. 3.

   run the generated code in Jupyter or Colab;

4. 4.

   return an exact installation or runtime error;

5. 5.

   copy generated explanations and submit the completed work.

The student does not receive the hidden instructor solution, hidden reference functions, expected answers, or grader feedback. This model reflects direct outsourcing rather than sophisticated prompt engineering or autonomous agents.

A session was counted as a correct completion only when the final artifact executed and passed the corresponding assignment grader after ordinary execution and permitted error feedback. For assignment package $a$, define observed ChatGPT-resiliency as

where $N_{a}$ is the number of separate ChatGPT sessions and $C_{a}$ is the number yielding an executed, grader-accepted completion. This operational measure concerns completion resistance, not plagiarism detection, authorship, or learning.

Table I summarizes the three implemented packages. Each used a deterministic synthetic or student identifier to generate a reproducible configuration. The instructor-side reference regenerated the configuration and expected output, allowing personalization without maintaining a separate answer key for every student.

HW1 targeted a frequent Qiskit difficulty: count keys are displayed in classical-register order, which can differ from the order in which students list qubits. HW2 moved from elementary gates to a recognizable algorithmic workflow and added circuit metrics and a hardware extension, but the student-facing notebook was heavily scaffolded. HW3 required a seed-specific oracle rather than one textbook circuit. The broader project also investigated hidden instructions and canary markers. QPY semantic validation and noisy-simulation grading were proposed, but they were not mandatory in the 150-session fixed-instance experiment and are not claimed as defeated controls.

For each assignment package, the same fixed student-visible instance was used in 50 separate ChatGPT sessions. The design therefore held the assignment content constant while resampling the model interaction across separate sessions. This tests whether successful completion was stable rather than a one-off response; it does not test generalization across many seeds or assignment variants.

All 150 final artifacts were executed and evaluated with the corresponding grader, and all 150 passed. Complete conversational and file-level provenance was retained for nine sessions, three per package. These archived sessions followed a controlled protocol: each began in a new conversation; only student-visible materials were supplied; up to two follow-ups were permitted, limited to an exact runtime error or a request for an omitted required file; and the operator supplied no conceptual hint, independent code fix, expected answer, hidden reference, or grader feedback. Transcripts, generated versions, runtime logs, final artifacts, and grader output were preserved. The nine archived sessions used the displayed ChatGPT configuration “5.6 sol medium” on August 16, 2026; model metadata were not retained uniformly for the other sessions.

Before the fixed-instance trial series, exploratory QPE, QFT, and Deutsch–Jozsa notebooks tested whether real-device execution would create a meaningful barrier. These runs are secondary evidence because they used a separate exploratory workflow. They are included to assess the project’s hardware hypothesis, not in the 150-session completion count.

For each assignment package,

Thus, 150 of 150 final artifacts executed and passed the corresponding grader, and each of the three fixed instances had zero observed ChatGPT-resiliency under the operational definition. Table II separates the full outcome count from the fully archived sessions.

Within the nine fully archived sessions, the trial record also marked every first-response answer artifact as grader-passing. Only one first notebook execution completed without a follow-up. In the other eight, execution stopped at an optional qc.draw("mpl") call because pylatexenc was missing. One permitted installation or rendering adjustment resolved each case. One HW2 grading environment additionally lacked qiskit-aer. These were environment and visualization issues, not errors in the quantum algorithm, expected result, output schema, or reflection. No archived session required operator editing of the generated code.

Table III summarizes the earlier exploratory hardware artifacts. ChatGPT produced or assisted with both simulator and IBM Runtime workflows. In each real-device run, the theoretically expected bitstring remained dominant despite noise. Transpilation substantially increased depth, but the result remained interpretable.

Hardware added backend names, job identifiers, transpiled circuits, and noisy counts that ChatGPT could not predict before execution. It did not eliminate the minimally engaged workflow: ChatGPT generated the code, the operator ran it, and the resulting data could be returned for interpretation. The exploratory evidence therefore indicates that hardware supplied execution evidence without eliminating substantial AI assistance.

A deterministic seed prevents every student from receiving exactly the same numeric configuration. It does not necessarily create a new reasoning problem. ChatGPT could read the visible generator, specialize the circuit to the generated values, and produce a parameterized solution. Non-palindromic bitstrings and reversed maps exposed careless implementations, but they remained explicit, rule-based transformations within the model’s capability.

HW1 illustrates the distinction. Its bit flips, measurement map, and expected count key were deterministic functions of student-visible data. Once ChatGPT parsed those functions, it could construct both the exact output and the required circuit. Personalization improved assignment variety and grading logistics but did not force independent student reasoning.

In HW2, QFT followed by inverse QFT is the identity:

Moreover, the student-facing notebook exposed the complete QFT and inverse-QFT routines, the expected recovery relationship, the measurement-map calculation, and generic reflection text. ChatGPT could therefore complete the package without independently deriving the algorithm and could know the ideal count key before simulator execution. Circuit depth and operation-count fields made the submission richer, but they did not prevent completion of the supplied workflow.

HW3 similarly disclosed the mathematical relation between the oracle mask and the Deutsch–Jozsa classification. A zero mask is constant; a nonzero linear mask is balanced. Constructing the corresponding controlled-NOT oracle is a standard translation from the supplied mask. The custom seed removed a single fixed answer, but it did not move the task outside the model’s capability.

Hidden deterministic references made the assignments autogradable and prevented a student from simply reading an instructor answer file. They did not prevent ChatGPT from deriving a correct answer from the public specification. The graders checked seeded configuration fields, expected bitstrings, ideal counts, dominant outcomes, and reflection presence or length. They did not semantically validate every submitted notebook circuit, and optional QPY export was not a mandatory gate. Consequently, grader acceptance established consistency with expected outputs, not independent authorship or understanding.

Machine-readable JSON also did not create resistance. LLMs are well suited to structured output when a schema or example is supplied. Likewise, generic or result-specific short reflections were generated fluently after ChatGPT had calculated or received the relevant values.

The observed weaknesses—missing packages, pylatexenc, outdated Runtime syntax, and occasional bit-order risk—were not robust barriers. They were ordinary debugging issues that a student could paste back into the same conversation. This is consistent with quantum-code studies showing large gains from execution feedback [9, 10]. A design is not meaningfully resilient if one exact error message converts a near-solution into a complete submission without requiring conceptual intervention.