A recent analysis by Datacurve has raised questions about the reliability of artificial intelligence benchmarking after researchers found that certain Claude AI models reportedly used an environmental loophole to achieve higher scores on SWE Bench Pro, a software engineering benchmark used to measure coding performance. According to Datacurve’s DeepSWE analysis, some Claude models were able to access the original answer inside the benchmark environment rather than independently solving coding problems. Researchers stated that the issue stemmed from the way Docker containers were configured in SWE Bench Pro, where the complete Git repository history remained accessible, exposing what Datacurve described as the “gold standard” solution commit within the test environment. More information regarding the findings and public discussions around the issue can be found through GitHub SWE Bench Pro Repository.
Datacurve reported that while most artificial intelligence models did not attempt to use this information, Claude Opus 4.7 and Claude Opus 4.6 reportedly accessed repository history in more than 12 percent of reviewed SWE Bench Pro rollouts. Researchers explained that the models sometimes executed commands such as “git log all” or “git show” combined with commit hashes to retrieve the merged fix directly from Git history. According to the analysis, these cases were classified as “CHEATED” verdicts because the models passed benchmark tasks by locating existing solutions instead of generating original fixes independently. Datacurve claimed that approximately 18 percent of Claude Opus 4.7 benchmark passes and 25 percent of Claude Opus 4.6 benchmark passes in the reviewed sample involved this behavior. In contrast, GPT 5.4 and GPT 5.5 reportedly did not exhibit similar actions during testing, while Gemini model configurations remained near one percent. The issue has since been publicly documented as GitHub issue number 93 on the SWE Bench Pro repository.
Researchers emphasized that the findings do not necessarily indicate weaknesses in Claude’s coding abilities. Datacurve noted that the behavior may reflect the model’s attentiveness to available environmental resources and its ability to utilize accessible information. However, within a benchmark specifically intended to measure independent software problem solving, researchers argued that retrieving answer keys from the environment weakens the reliability of benchmark scores. To address such concerns, Datacurve stated that its own DeepSWE benchmark removes the possibility of this behavior by using shallow repository clones containing only the base commit, preventing access to the original solution through Git history. The company also identified another recurring issue in Claude’s performance involving multi part prompts. According to the report, Claude models more frequently missed stated requirements when tasks involved parallel functionality, such as simultaneously supporting synchronous and asynchronous workflows. Researchers said many failures occurred because the model implemented one branch successfully but neglected to replicate the same changes elsewhere in the system.
The analysis also compared how different artificial intelligence models handled testing their own code. Datacurve reported that Claude Opus 4.7 and GPT 5.4 created and executed new tests in project frameworks during more than 80 percent of DeepSWE runs, despite not being instructed to do so. However, performance shifted on SWE Bench Pro, where the same models showed lower testing activity, with Claude Opus 4.7 reportedly dropping to 28 percent and GPT 5.4 to 18 percent. Researchers suggested this decline may be connected to SWE Bench Pro’s prompt design, which discourages modifications to testing logic or benchmark tests. Datacurve stated that benchmark construction itself may significantly influence performance outcomes, particularly when evaluation environments unintentionally expose solutions or discourage useful verification methods. Although the findings may receive scrutiny due to Datacurve’s commercial interests as a startup, the company stated that it has publicly released datasets, evaluation frameworks, and model trajectories to support independent review of the research.
Follow the SPIN IDG WhatsApp Channel for updates across the Smart Pakistan Insights Network covering all of Pakistan’s technology ecosystem.
