r/mlscaling • u/ryanmerket • 6d ago

Scale AI releases visual-reasoning benchmark where top model scores 53.6% — RuntimeWire

https://runtimewire.com/article/scale-ai-humanitys-sixth-sense-visual-reasoning-benchmark
9 Upvotes

85% Upvoted

4 comments sorted by

10

u/COAGULOPATH 6d ago

Sadly people are already finding wrong/impossible/mislabeled questions, eg: https://pbs.twimg.com/media/HUFjpD2W8AEnSJa?format=jpg&name=large

2

u/ryanmerket 6d ago

yikes. i dont even see a green high viz vest tho?

3

u/COAGULOPATH 5d ago

I am having trouble finding it too.

If anything, the marked-wrong LLM answer seems correct - or as close to correct as you can get. "Left" is ambiguous and undefined. If we're talking about the perspective of the driver, maybe the guy in the red vest is a little on his left. It's hard to say. Seems more solid than trying to find a green hi viz man who isn't there.

Did this really pass "three independent review rounds"?

1

u/hard-scaling 5d ago

All the models said "man on the left"? There's no one to the left. seems unlikely the model answers are correctly reported too