The difference between objective and subjective perceptions

While at our Weatherhead Center workshop, we presented the results of our expert perceptions pilot study. Dawn Brancati kindly served as discussant, and she made a number of very useful and thought-provoking comments. In this post, we are going to focus on one of them.

To provide a bit of background, the core of our survey is a group of forty-nine questions encompassing all stages of the electoral process—as defined by the UN here and explained well by Jørgen Elkit and Andy Reynolds in a 2005 Democratization article.

The comment we are focusing on today is Dawn’s suggestion that we distinguish between objective (factual) measures and subjective (judgmental) measures within our survey. Understanding the difference between objective and subjective measures is potentially important to our project for several reasons, but for today one stands out.

Specifically, we believe that distinguishing objective from subjective measures could provide us a good way of controlling for differences between experts. This ability might be particularly important for electoral contests where no consensus exists about whether they meet standards of electoral integrity.

Consider the following table and figure. Table 1 shows the distribution of responses in the Czech Republic to the question “Did the election trigger violent protests?” Our Czech experts’ responses are very clear. There were no violent post-election protests and, as such, all the experts say that voters were not threatened with violence. This judgment reflects an objective fact: either there were violence protests or there were not.

Table 1. Did the election trigger violent protests

in the 2012 Czech Republic national election?

	Frequency
Strongly disagree	18
Disagree	0
Neither disagree or agree	0
Agree	0
Don’t know	0
Not applicable	0

Now let’s look at Figure 1, which summarizes the responses to the question: “Do rich people buy elections?” in the Czech Republic. This question is one of four in our survey that is worded exactly as it is in the latest round of the World Values Survey. This overlap will be very useful for comparing elite and mass perceptions—a topic we will be discussing in future posts.

Figure 1. Responses: Do rich people buy elections in the Czech Republic?

As you can see, unlike for the election violence question there is a divergence of responses. This result is unsurprising since this question is more open for interpretation and subjective judgment than a question about whether there was violence. What is “rich”? What does it take to buy an election? Do the rich need to intervene the results directly or just buy the candidates? Clearly, this question is more subjective and open for interpretation.

We find the differences between potentially objective and subjective questions interesting because the former could be used as a proxy for ‘expert knowledge’ to better understand answers to the latter—we could check whether objective events did or did not occur and then see if individual experts’ answers reflect this fact.

We could then weight experts’ answers to subjective questions according to their answers to objective questions. If an expert ‘fails’ (i.e. by being wrong) many of the objective questions, then it may be worth discounting his/her answers. This could also partially address a point raised by Andreas Schedler in his 2012 piece in Perspectives on Politics (available here). Schedler stresses the importance of holding experts accountable for their judgments. Evaluating our experts’ knowledge through an external verification method like the one we propose here could provide a clear means of improving our confidence in the aggregate survey results.

Thanks again to Dawn Brancati for her insights.

Please feel free to leave your own feedback in the comments section below.

-Ferran and Rich