A Critical Assessment of Steenbergen’s Discourse Quality Index
Consider the DQI’s method of measurement. One must ask what the codes are picking out exactly, and in what way they are relevant to one another. It seems absurd to suggest that the figures assigned to the codes represent units of “ideal discourse”. Steenbergen et al state that the codes are scaleable (Steenbergen et al 2003), which suggests that this is the conception intended. The application reveals problems with this approach to understanding discourse quality. For example, one would want to consider successful a discourse which produced genuine consensus through debate, even if for the most part it involved negative but ineffectual speech acts. The DQI does not discriminate between speeches in this way, but understands quality based on an average of the quality it perceives to exist in all speeches.
The notion of effectiveness is a crucial problem for the DQI. It is unclear how one could judge the effect of a speech on the discourse or its outcome, by examining only the content of that speech. Such a judgment would appear to involve considering the subjective dimensions of subsequent speech acts. These considerations appear to be beyond the scope of the objective measurement Steenbergen et al describe; it is explicitly acknowledged that the DQI does not measure “authenticity” because of the limitations of its method. Consequently, the way the DQI understands the quality of discourse is potentially misleading. For example, suppose a debate consisted of speeches that were uninterrupted, respectful and well justified. However, the first speech expressed a persuasive threat to other participants, which dissuaded them from contesting the decisions of that participant. The DQI would appear to give a counter-intuitively favourable assessment of that debate – indeed, the threat might even be coded (2) as a demand with a qualified justification.