Separating signal from noise in coding evaluations

OpenAI no longer recommends SWE-Bench Pro

A new analysis from OpenAI reveals issues in SWE-Bench Pro, a popular coding benchmark, raising concerns about reliability and accuracy in evaluating AI models.
238 points by sk4rekr0w 105 days ago openai.com 93 comments
Summary not yet generated for this article.

Comments (93 of ) ⌛

Loading more comments... ⌛
Scroll down to load more comments
No comments available.
Type to search HN and engineering blog articles
Press Escape to close  |  ↑↓ to navigate  |  Enter to open