[Main page]
<<<<< PREVIOUS page
2. LOOKING INTO A MIRAGE: DO THE PATHS CONVERGE?
2.1 A Fixed Social Reality? The Spectre of Positivism
One of the principal aims of triangulation in the social sciences seems to be to corroborate one set of findings with another; the hope is that two or more sets of findings will 'converge' on a single proposition. This view holds much weight in literature on triangulation:
As Blaikie (1991) points out, such views can make sense only if the researcher works within a 'positivistic frame of reference which assumes a single (undefined) reality and treats accounts as multiple mappings of that reality (Silverman 1985:105).' Of course, triangulation in surveying is based on such a premise.
The question is whether it makes sense to conceive of social reality in this way. According to Blaikie (1991:120), for the strict interpretivist, 'social reality is not some 'thing' that may be interpreted in different ways; it is those interpretations.' Guba and Lincoln (1989) are anxious about a simplistic notion of a fixed social reality that remains basically unchanged regardless of one's investigative stance: '... triangulation itself carries too positivist an implication, to wit, that there exist unchanging phenomena so that triangulation can logically be a check (p.240).' And if one takes the position that there is a reality 'out there' separate from ourselves which, however, cannot be known but only hinted at through our constructions, then it is difficult to see how any amount of triangulation (as conceived in the social sciences) can get us any 'closer' to knowledge of that reality.
Arguments about the nature of social reality are well documented, and do not need to be rehearsed here. However, it is clear that if a researcher were to reject the notion of a fixed 'social reality' (knowable or not), then the idea of there being a method that could help a researcher home in on a social reality would make no sense; 'the image of data converging on a single proposition about a social phenomenon' would have to be regarded as a 'phantom image' (Mathison 1988:17), and the researcher would have to give up all thoughts of using triangulation as a methodological resource.
Let us assume, however, for the moment, that there is a social reality that can be mapped, in order to investigate the ways in which methodological triangulation is conceived and widely employed. The rest of this chapter is a discussion of some significant errors which are often committed by those who attempt t oconduct methodologcal triangulation. Regardless of one's ontological or epistemological position, the discussion raises serious questions about what kind of claims could be generated through such a strategy, or what could count as good practice.
2.2 Pulling yourself up by your own bootstraps: the logical fallacy of mutual confirmation
Commonly 'triangulation' is used as a means of establishing the truth of propositions whilst simultaneously establishing the validity of the methods which are used to reach those propositions:
Underlying both the above quotations is the belief that the results generated from one method can somehow confirm those of another. There are two logical fallacies bound up in such a belief.
First, there is a confusion between the 'truth' function of a technique, and its operational function. Operationally, one can define a priori that one measure can serve in place of another. In such circumstances, the proposition developed through one strategy could then be said to be the same as one that would have emerged had the strategy's 'operational twin' been employed (concurrent validation). A logical error in the confirmatory type of social scientific triangulation is that a second strategy is commonly used to prove the truth of a first, rather than simply define it as true; this I call Error Type A. For a second strategy to be able to prove that propositions generated from the first were 'true' - or to establish the degree of validity, reliability of bias -, it would first have to be defined as a strategy which itself produced 'true' propositions.
The equivalent in land surveying would be to claim that one bearing alone could locate the true position of a point, which is clearly impossible; for this reason alone, the term triangulation would be misleading in social research. In practice, social scientists rarely claim that any one method can by itself produce 'true' propositions because if a particular strategy was deemed to be able to produce 'true' propositions, then no 'triangulation' by using a second data source would be necessary.
The second logical fallacy is that agreement between the results of two methods is taken to prove the validity of the second method as well as the first (the principle of 'mutual confirmation') - the logical equivalent of arguing in a circle: 'Proposition/method A is valid/true. Why? Because it agrees with proposition/method B. How do we know proposition/method B is valid/true? Because it agrees with proposition/method A.' In layman's terms it is like trying to pull yourself up by your own bootstraps. This I call Error Type B.
The difference with triangulation in land surveying is that, barring very minor error margins, the instruments are assumed to be reliable and able to provide valid measurements from the outset. The process of triangulation is to locate a point or true position of an object, not to find out whether the instruments actually work (or how they work): for the final answer to be useful at all, the readings must be assumed to be true. In addition, in the form of (land surveying) triangulation known as intersection, the locations from which the readings are taken are also known. However, in the confirmatory function of triangulation in the social sciences, the researcher is simultaneously trying to establish the validity of a proposition and the validity of the method used to produce it, moreover without knowing all the relevant social dimensions (the 'location') of each investigator. The equivalent in land surveying would be to claim to be able to establish the exact geographical position of an object without knowing beforehand which instruments worked, how to interpret the measurements, or what were the locations of the points from which the bearings were taken. Triangulation in land surveying is logically consistent internally; confirmatory triangulation in social research is not remotely logical and the appropriation of the term triangulation gives such activity a veneer of logic and respectability which it does not deserve.
One option is to adopt an operational definition of truth as the agreement between propositions arrived at through two or more different procedures, a form of consensual truth (e.g. intersubjective agreement). Stated as boldly as this, it is hard to say how many social researchers would want to subscribe to this definition of truth. That does not stop many, however, from implying it in their methods when they adopt a 'mutual confirmation' approach. There is a dilemma whether to adopt a positivist position on truth, or to define truth in Humpty Dumpty fashion as whatever one wants it to mean. Neither position is satisfactory, but advocates of the 'mutual confirmation' theory seem to try to adopt both positions simultaneously.
Methodology textbooks do not always clear up this confusion. For example, Hammersley and Atkinson (1995) seem to reject 'confirmatory' triangulation, for fear that 'confirmation' could simply reinforce prejudice or bias, while 'disconfirmation' might lead one to replace one 'mistaken' proposition by an equally misguided one:
That a conclusion could be 'incorrect', 'a result of error', or 'invalid' or that' links between concepts and indicators' could be 'checked' presupposes that there is a correct conclusion at which one could arrive. Either this hides a positivist stance, which the authors reject, or a view that there is a reality but we just cannot know what it is. If the latter is true, then in it is hard to see how Hammersley and Atkinson's comments could be useful, since one would not be able to know if one was getting any closer to the 'truth' of this reality. This point will be explored further in the next section.
2.3 False signposts: the myths of convergence, divergence, and bias
The key mistaken assumption in much triangulation work in the social sciences is that answers that look the same mean the same thing (e.g. Campbell 1956; Webb et al. 1966; Denzin 1970; Jick 1983; Mitchell 1986; Mathison 1988; Morse 1991; Deacon 1998); this I call Error Type C.
Let us examine a hypothetical example of answers in a survey that appear to conflict in some way, despite the fact that they are apparently about the same phenomenon (e.g. happiness). One question may require responses on a defined scale, while another may ask a qualitative question. The scaled answers may appear to express a generally lower degree of happiness than the qualitative answers suggest. However, this would be a very sloppy way of thinking. The general statements one could derive from the scaled answers would demonstrate where the members of the sample fitted themselves on the scale. In contrast, the general statements derived from the qualitative answers represent where the researcher placed (through personal interpretation) the members of the sample on the scale. This would still be so even if the researcher were to compare the two answers of just one individual from the sample.
As has been argued earlier, if the answers apparently 'disagree' this can in no way disconfirm the validity of one of the answers (Error Type A); ideally, it would simply alert the researcher to the fact that two different phenomena are present (or being socially constructed). Unfortunately, if the answers apparently 'agree', this is taken as confirmation of some truth (Error Type A), or a form of mutual confirmation (Error Type B) - both of which have already been shown to be flawed procedures - and the researcher is then likely to overlook that fact that the questions represent enquiry into (or construction of) two different (albeit perhaps related) phenomena.
To summarise this hypothetical example, there are two mistaken beliefs:
Unfortunately, such errors committed in the name of triangulation are not merely hypothetical; even a cursory glance at literature across the social sciences shows them to be all too real. In fact, even in the triangulation literature the same sorts of mistake occur. For example, in a piece of methodological triangulation, Mathison (1988:16) was concerned about inconsistent findings when 'teachers reported using the activity cards extensively but in over 200 classroom observations only 14 such activities occurred.' It was perhaps a little naive to assume that the number generated for frequency of card use would be the same whether the researcher elicited teachers' perceptions or observed behaviours. The point was that effectively different questions were being asked, although they appeared to be questions of the same type ('How many times were activity cards used?') with answers of the same type (numbers).
One should seriously question Jick's (1983:136) claim that multiple methods can be used 'to examine the same dimension of a research problem.' After all, the meaning of an answer can only really be fully understood by reference to the meaning of its corresponding question, which, in turn, is embedded in a particular research method and ontological/epistemological perspective. Consistencies or inconsistencies therefore only appear to be so because the propositions derived from two sources have been removed from their fuller context of meaning.
Given that propositions derived from two separate methods must mean different things it does not make sense to say that they can conflict or agree: the claims made are not of the same type, and as Blaikie (1991) takes pains to point out, the methods used to produce the claims are usually not even based on the same philosophical premises. The assumption that propositions derived from different methods can converge or diverge, I call Error Type E.
The whole issue of convergence becomes even more of a mystery when one compares methodological triangulation with the original concept of triangulation. Land surveying depends on the knowledge that each new piece of information (bearing/reading) is based on the same framework of measurement (i.e. the same kind of question), so that each answer can be understood in the same way. This is a single method approach - and necessarily so -, unlike the multi-method approach of triangulation as conceived in the social sciences. Moreover, in land surveying triangulation, the issue is not whether the two bearings will converge, but where. Convergence always happens eventually; this is a fundamental geometrical truth.
One kind of misleading rhetoric in social scientific literature is that two or more 'bearings' may not converge. This is not necessarily seen by social scientists as a problem, but as a strength of triangulation (e.g. see Mitchell 1986), the belief being that 'the effectiveness of triangulation rests on the premise that the weaknesses in each single method will be compensated by the counter-balancing strengths of another (Jick 1983:138).' Denzin (1970), one of the early authors of this view, wrote:
This view, which I call Error Type F, is countered by Morse (1991:122):
Clearly, if one does not know a priori which method is closest to the 'truth', or how close it is, then it is not possible to use any method as a yardstick to assess degrees of bias or validity in other methods. As Knafl and Breitmayer (1989) point out:
However, it is hard to imagine what would count as a satisfactory outcome of such a discussion. Deacon et al. (1998:57) set out the problem:
Silverman's (1985:105) view that 'the sociologist's role is not to adjudicate between ... accounts' does not help, and Jick (1983) and Mitchell (1986), while recognising the problem of how to weight different data sources, provide no answers either. Blaikie (1991:123) believes that 'regardless of the methodological perspective adopted, decisions about the relative merits of different sources of data can only be settled in the context of some theory; and the choice and application of the theory is a matter of judgement.'
What is one to do, though, if such 'methodological purism' (Deacon et al. 1988:57) is rejected? Mathison (1988) suggests that 'all the outcomes of triangulation, convergent, inconsistent and contradictory .... need to be explained'. If divergence is no more informative than convergence, then why bother to make the distinction, and do such terms mean anything at all in this context?
In fact, what I have argued is that the weighting problem (Error Type F) and the problem of how to interpret divergence/convergence (Error Type E) do not even exist, since propositions derived from different methods or data sources are not of the same type and therefore not open to comparison. The widespread belief that answers from different methods or data sources can converge, diverge or deviate from a social 'truth', and Mitchell's (1986:24) question of 'how to interpret divergent results between numerical data and linguistic data' (Error Type F), turn out to be more unwitting pieces of misdirection and illusion which nevertheless capture many social researchers in their spell.
2.4 How to be lost without even knowing it: a recent example
Deacon et al. (1998:49ff) ask how one is 'to handle instances in which there is a clear inconsistency between data deriving from quantitative and qualitative research', describing this as a 'failure to triangulate' and insist that one should find a way to 'deal with the clash'. Morse (1991:122) is unequivocal:
The following discussion illustrates just how lost researchers can get when attempting to conduct triangulation in social science research. Deacon et al. (1998) conducted a large-scale investigation into social scientists and their media relations in Britain. Six methods (out of a total of eight used) are discussed in the article:
By their own admission there was 'only one instance in which similar or the same respondents were being researched', and that was where some social scientists were in the samples for both (2) and (4). The declared objective of this was not triangulation but elaboration of quantitative findings through qualitative means (p.51).
Deacon et al. compared the findings from (4) and (5), interviewing social scientists and journalists respectively. They claim that social scientists' answers about the existence of conflict and tension corroborated those of journalists. In doing so, they commit several errors. First, there is an assumption that the perceptions as expressed by one group were the same as those of another group (interaction of Error Types D and F). Second, they are committing Error Type A if they believe that one set of answers can confirm the truth of another. Third, it seems that where in their eyes the two sets of answers agree, a truth has been established, an interaction of Error Types C and A in which the implicit claim is 'this is The Truth, because these two data sets agree'.
Such problems abound throughout the article; careful reading will show that all the comparisons of findings from the different methods are illegitimate according to one or more of the Error Types I have identified. For example, method (3), the second survey, was used to 'appraise the validity of the first survey's [2's] findings' (p.56) (Error Type A). Deacon et al. use the 'clear consistency between the two surveys' (p.56) simultaneously to establish the validity of method (3) (Error Type B). All the concerns about results from the different methods apparently diverging, and the uncritical acceptance of results that apparently converge, are a commission of Error Type E.
Finally, in following up their declared need to investigate various 'clashes' (p.50) between the different studies, Deacon et al. add one more type of error to those already identified. The claims on p. 52 made about what social scientists and journalists feel and think are based on the interviews in method 4 and 5. Such generalisations to the wider populations of social scientists and journalists are wholly unreliable and illegitimate since these were not randomised but purposive, quota and opportunity samples - an elementary methodological mistake.
But this is not a triangulation error, and therefore is not included in my Error Types. Even if all the samples had been randomised, so that they were all generalisable to wider populations, this would still not have validated the comparison of the studies. The point is that, since each sample was constructed on a different basis, one has to assume, unless it is proved otherwise, that the social scientists in samples 1, 2, 3 and 4 represent different populations, as do the journalists in samples 1, 5 and 6. Therefore, even if each of the six studies had used the same method and questions, any apparent 'inconsistency' could be attributed to nothing more remarkable than the fact that the samples did not belong to the same population - indeed the same wider populations of 'all social scientists' or 'all journalists'. The studies were conducted on different methodological grounds, and dissimilar samples. It is therefore a mystery how the researchers - without even recourse to statistical or probability statements - were able to decide whether or not the apparent differences in results were significant. Comparing the results of two samples as though they belong to the same population when there is no methodological or statistical demonstration that they do belong to the same population is a commission of Error Type G.
Such a catalogue of muddle-headed errors would not be so worrying if only neophyte researchers had committed them; perhaps then these researchers could be put back on the straight and narrow path by vigilant supervisors and teachers before too much harm was done. However, the seriousness of the problem becomes apparent when one finds studies such as the above published by such highly respected and influential researchers.
Methodological Triangulation,
Or How To Get Lost Without Being Found Out
by Alexander Massey BA PGCE MA MSc
[page 2 of 3]