This happened circa 2021...
A large company that holds an annual event in San Francisco gave me a test to evaluate the questions, whether they had correct answers, and whether they adequately tested candidates' knowledge.
Those who pass this test receive a certification (I took it and passed, and was later invited to help update the test due to a relationship with my employer).
A few months later I received the initial results, and for 4 questions, the results were inconsistent. Of the candidates who passed the test, many got those questions wrong.
So I suggested some changes to the phrasing to make the context and meaning clearer. I thought this was the root of the problem.
But the last question intrigued me because there was one clearly correct answer and the others were clearly false, and there was no apparent reason to get it wrong. It was so obvious!
If people were good enough to get the other questions right, they should get this one right too. What would explain that inconsistency?
Then an idea came to me: I took the question statement, googled it, and ended up finding an Indian website where that question was published.
The test question had been leaked. But that website indicated the wrong answer. 🤦♂️
So, those who followed that website and memorized the answers they suggested ended up being disadvantaged compared to those who studied or had the experience to identify the correct answer.
What can be done to prevent leaks?
Recreate the entire test from scratch? It's a lot of work and doesn't guarantee that the new test won't leak again.
Or keep the test and let the fools who blindly trust that website get screwed? 😹
