I presented people with a series of subtle variations of the basic colors (shown below; ignore the black and white sets), and then simply asked them to pick one of them as the one they most identify as that color. In other words, which variant is the "most blue" or "closest to what you would call red."
Here are the results:
As noted, this is after 220 responses, which I felt was enough to get a representative sample. Clearly, some colors are more easy for people to agree on, like orange and green and purple and yellow. Meanwhile, others were more "controversial," such as gray and to a lesser extent blue and pink and, the one that was the most fascinating to me: brown, with no clear majority consensus at all.