Last week, as part of our Outdated Answers project, we ran a test on Stack Overflow to see what would happen if we stopped pinning the accepted answer to the top of the list of answers. As expected, there were no negative repercussions when we made this change. So, as promised, today we are making this change permanent.
By default, answers are now sorted strictly by Votes (descending order by highest score), and the accepted answer’s order in the list is based on its score. If you prefer to sort by Active (descending order by answer’s created or edited timestamp) or Oldest (ascending order by answer’s created timestamp), the accepted answer is also unpinned.
Read on if you’re interested in details on what we learned from the experiment.
How we conducted the test
For questions where the highest scored answer was not the accepted answer and the user’s preference was the default Votes sort, 50 percent of users saw the accepted answer pinned to the top and the other 50% saw answers sorted by the highest score with the accepted answer unpinned. For both sorts, our success metric was the rate at which users either upvoted or copied.
We analyzed the data in a number of ways.
Users copied or voted on any answer. We looked at users who copied all or part of any answer, or users who took any voting action (upvote, downvote, etc.) on any answer. Our hypothesis was that there would be no statistically significant difference between the two sorts, but we found that when the accepted answer was unpinned there was a 4% increase in copying and voting. Success rate was 20.3% when the accepted answer was pinned, and 21.1% when it was unpinned. (In case you are wondering, upvotes far outnumber downvotes, so we weren’t terribly concerned that the two types of votes were mixed together in this analysis.)
Users copied the first answer. There was a whopping 61.6% increase in users copying from the top answer when the accepted answer was unpinned and the highest-scoring answer was first in the list of answers. Success rate was 6.9% when the accepted answer was pinned, and 11.2% when it was unpinned.
Users upvoted the first answer. We did not have enough sample size to draw statistically significant conclusions, but there was a 90.5% increase i