Subjective vs. objective evaluation of ontological statements with crowdsourcing

In this paper we propose and test a methodology for evaluation of statements of a multi-viewpoint ontology by crowdsourcing. The task for the workers was to assess each of the given statement as true statements, controversial viewpoint statement or error. Typically, in crowdsourcing experiments the workers are asked for their personal opinions on the given subject. However, in our case their ability to objectively assess others’ opinions is examined as well. We conducted two large-scale crowdsourcing experiments with about 750 ontological statements originating from diverse single-viewpoint ontologies. Our results show substantially higher accuracy in evaluation for the objective assessment approach compared to the experiment based on personal opinions.