Research

The Plural of “Anecdote” is not “Evidence”

John CreswellJohn Creswell is a well-known educational researcher, much published and easy to read (well, easier than many anyway).

He keeps a blog on his web page (http://johnwcreswell.com), but although he may be a much more respected, experienced and published author and researcher than the guy writing this (not much of an achievement really), his blog (http://johnwcreswell.com/category/blog/) is pretty useless; from 2013 to date he’s posted about five paragraphs, and one of those was a reply to a comment. Sadly, the website appears to be little more than a poorly designed publicity tool for Creswell’s books. But I digress, because that really isn’t the point of this post.

I’d been asked to take a look at the following two references from Creswell’s line of texts focused on novice and intermediate researchers, and to synthesize (I’m still a bit fuzzy on what that means exactly), my impressions of the following readings:

Creswell, J. (2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.), pp. 69–110. Thousand Oaks, CA: Sage Publications.

Creswell, J. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.), pp. 3–21. Thousand Oaks, CA: Sage Publications.

creswell_books

So here goes my synthesis of the above…

Creswell’s 2013 work does what it says on the cover. The book provides the author’s take on five different approaches to qualitative research.

One quick note to anybody reading this who wants to re-open the debate on quantitative vs. qualitative research and their comparative merits—don’t. This aint the place, I’ve had that debate with the one or two of you I’m thinking of, and I really can’t be bothered to poke myself in the eye with that blunt stick again.

So, moving on, Creswell focuses his book on the following approaches to examining one or more research questions:

Narrative Research

The analysis of a collection of stories which emerged “through the interaction or dialogue of the researcher and the participant(s).” (Creswell, 2013, p.71)

In my opinion narrative research is an excellent way of gathering opinion and motivational data—why did you do something? What do you think about…? But beware of trying to answer factual questions about phenomena with narrative data—my child developed autism after s/he had her childhood vaccinations, therefore the MMR vaccine causes autism. You can collect hundreds, even thousands of passionate stories advocating the same opinion. This doesn’t make the opinion remotely factual. Just an opinion. The plural of anecdote is not evidence, and whilst Creswell identifies some strengths and weaknesses in the narrative approach, in my opinion, very few qualitative researchers give this fundamental issue the weight it deserves.

Phenomenological Research

Is the inquiry into a given event, concept or experience; examples might include race or gender bias in teaching, or bullying in an educational setting. Creswell described data collection in phenomenological research as typically “interviewing individuals who have experienced the phenomenon.”

In my opinion, this approach shares similar weaknesses to that of narrative research only possibly more so. In the case of an interview, the researcher is phrasing the investigative questions posed to the participant, and thus by the very nature of the questions may unintentionally acknowledge or deny the existence of the issue. In addition, participant statements suffer from the same essential weakness as narrative research—anecdotes are not evidence. Thousands of people may claim to have seen a magician levitate across the Grand Canyon, but it doesn’t mean he really did.

To gain a more accurate perspective on narrative and interview-based accounts, such information must be triangulated with other independent variables in order to confirm or deny the veracity of the statements collected. Eisner (1991), referred to this triangulation of sources as creating “a confluence of evidence that breeds credibility (p. 110).”

Uncorroborated anecdote may be a nice story or even a collection of nice stories—that may even chime with your research questions—but at the end of the day, without corroboration they’re still opinion laced with researcher interpretation and should be approached as such. This doesn’t mean that the data lacks validity or worth. On the contrary, a central research question may be something like: “What is the opinion of a representative sample of Toronto elementary and high school teachers on the current sex education curriculum for grades K–12?” In which case, interview data might be of paramount relevance and an accurate representation of the answer. Once again, it depends upon the nature and the structure of the research question/s.

Grounded Theory Research

In Creswell’s opinion moves beyond the description of narrative and phenomenological research, to the generation or discovery of a theory. Some might disagree, but I would put forward Pinar’s ideas of currere in his 1975 paper The Method of Currere, as an example of where the researcher has generated “a general explanation (a theory) of a process, an action, or an interaction shaped by the views of a large number of participants.” (p. 83)

Let me reiterate a favourite theme here, in this case the word theory is not a scientific theory (testable, falsifiable, replicable, predictable, comprised of laws and experimental evidence like evolution or gravity), but in the scientific sense is more of a hypothesis; however detailed or substantive the idea might be, and regardless of the systematic approach to its generation.

Ethnographic Research

Creswell (2013) defines ethnographic research as focussing on “an entire culture-sharing group” (p. 90), examining the shared experiences of a few or many individuals. For example, Long (2011) examined the shared experiences of undergraduate biology students that declared a strong belief in biblical accounts of human creation, when exposed to scientific theories of evolution as part of their university studies. As Creswell described, Long focussed on developing a “complex, complete description of the culture” (p. 91) of the group, in the manner of a realist ethnography reproducing participants’ views through chosen quotations and his own declared research perspective.

Once again, ethnographic research suffers from similar pitfalls to that of both narrative and phenomenological research. The interpretation of the investigator is key, the investigator is still coding and choosing quotations based on his/her own research perspective and judgment of fit to predefined research categories. This approach is also extremely time-consuming, and requires more than a passing familiarity with the principles of cultural anthropology. Nevertheless, long-term observation in a systematic, replicable manner can provide for what Creswell (2013) refers to as a “holistic cultural portrait” (p. 96) of the subject group.

Case Study

Now we come to my particular favourite. Creswell borrows from Yin (2009) when he defines case study research as “the study of a case within a real-life, contemporary context or setting” (p. 97). Case study research differs from the previous three approaches in that it can encompass multiple sources of information including “observations, interviews, audiovisual material, documents and reports” (p. 97) from a single or multiple cases.

In my own case as a researcher I have previously chosen to analyse the legal records of United States court cases with reference to cases challenging the teaching of biblical creationism in US science classes. Studying this material provided the opportunity to analyse all submitted evidence, expert witness testimony delivered under oath, cross-examination of lay and expert witnesses by legal counsel and judicial opinion. All such material was freely available without copyright or ethical disclosure restrictions, no time constraint existed for obtaining or using the data and a range of historical cases analysed by expert criticism over time, existed in pre-transcribed formats. The major drawback with such an approach is the sheer amount of data you may be forced to select according to research criteria—purposeful sampling.

That of course, and you are probably forced to assume that not everybody in a court of law is lying.

Obviously, legal material is not the only source of data collection open to case study research, but does offer, in my opinion, an excellent, easily available source of data that is both verifiable and replicable by future researchers. Nevertheless, the usual caveat applies as to interpretation of material by the researcher, their research stance and the structure of the research questions.

So, to summarise the chapter, we have a selection of approaches to qualitative inquiry which encompass most of the techniques novice post-grad’s are going to need in the area. There are short sketches of what they are and how & when to use them, and brief discussions of the challenges in each approach. The author closes with a comparative table of the major features of each approach. A brief scan of the rest of the book shows a logical progression through frameworks, design, data collection, analysis, writing up, validation and conclusions. Even a chapter on philosophical assumptions and interpretive frameworks. As a result, it allows a novice researcher such as myself to make a more informed choice for my own work, but from what I have read in detail, the shortcomings of each approach that I have identified above remain largely unaddressed.

I’m going to have to read the rest of the book though, before I can really know that for sure.

Creswell’s 2014 introductory chapter to the fourth edition of Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, proceeds on the pretty inarguable premise that any academic researcher must be able to clearly convey the purpose and findings of their own work, both in research proposals and research papers.

This is a guide to help both graduate students and professional researchers organize their thoughts and produce a half-decent finished product. The entire text outlines and explains each section of a finished research proposal and aids an author in deciding what approach to take and what information to include. Chapter one (pp. 3–21) covered for this post follows nicely from the previous reading, defining and outlining what Creswell sees as the four most common worldviews that researchers may utilize during their research and the three types of research design that may be utilized to fit both the worldview of the researcher and the nature of the research question/s.

Worldview—Postpositivist

Also referred to as the scientific method of empirical evidence gathering. This is epitomised by a deterministic philosophy which states that causes probably determine effects or outcomes. It is also reductionist, in the sense that hypotheses can be reduced to testable, falsifiable and replicable inquiry, resulting in empirical evidence.

Creswell’s idea of the scientific method being simply a process which “begins with a theory, collects data that either refutes or supports the theory, and then makes necessary revisions before additional tests are made.” (p. 7) Is over-simplified and naive at best, misleading at worst, as it makes no discussion of the concepts of testability, falsifiability, replicability, predictability, tentativeness and other aspects of modern scientific investigation, such as control measures and double-blinded or even triple-blinded protocol. Though, to be fair, that may be a little beyond the scope of his chapter, a professional researcher of Creswell’s stature shouldn’t try to provide a definition of a research paradigm in a manner that is over-simplified to the point of being dismissive of critical aspects.

Worldview—Social Constructivist

Creswell (2014) explains that social constructivists “seek understanding of the world in which they live and work (p. 8).” This understanding is formed by an interaction with others and society as a whole—resulting in the term social constructivism. And at this this point I would ask, don’t we all seek such an understanding? Though Creswell goes on to explain that this leads the researcher to look for a “complexity of views rather than narrowing meanings into a few categories or ideas (p. 8).” This seems to me that Creswell is suggesting that this is how the scientific method approaches research, and is differentiating the two worldviews at least partially on this basis. I would suggest that if this is Creswell’s position, he would make a poor scientist, and needs to take a look at some of the latest research on string theory (Gu, Klemm, Marino, & Reuter, 2015), or macroevolution according to a confluence of data (Benton, 2015), and the complexity of views being considered therein, which by no means may be thought of as narrowing meanings to a few ideas. Indeed it is the complexity of views expressed within the realms of peer-reviewed research that form the dynamic of progress via the scientific method.

Creswell states that the goal of social constructivist research is to “rely as much as possible on the participants’ views of the situation being studied (p. 8).” He goes on to argue that, “[m]eanings are constructed by human beings as they engage with the world they are interpreting (p. 8).” A position which—as has previously been discussed—is fraught with the difficulties inherent in participant bias, faulty recollection, inconsistent or faulty researcher and participant interpretation and the very nature of research questions posed.

Worldview—Advocacy and Participatory

The advocacy and participatory worldview stems, according to Creswell (2014), from “individuals who felt that the postpositivist assumptions imposed structural laws and theories that did not fit marginalized individuals in our society or issues of social justice” (p. 9). Related research is therefore closely related with political aims for societal reform and emancipation of marginalized or disenfranchised sectors. A base position with which I can find little to argue. In my opinion, curriculum and thus education is a product of the times and political climate—as has been argued in more detail elsewhere on this site—and as such, certain sectors of society are often marginalized. Social justice is a dynamic which in my opinion any forward-thinking society should consistently strive to improve, and any research paradigm which places these issues in a position of primary consideration is of great importance to any field of endeavour.

Worldview—Pragmatic

Pragmatism explains Creswell (2014), is a worldview which has arisen from “actions, situations, and consequences rather than antecedent conditions (p. 10).” There is a concern with what works and practical solutions to problems rather than method. Research based on the pragmatic approach apparently “is not committed to any one system of philosophy and reality (p.10).”

At which point I begin to snigger and will thoroughly refute any argument which even begins to suggest that there may be more than one measurable, accessible, testable, falsifiable reality. This is just the sort of pseudoscientific language with which I lose patience pretty quickly.

“Truth is what works at the time. It is not based in a duality between reality independent of the mind or within the mind (p. 11).” If Creswell knows what he meant by this, then he hasn’t seen fit to share it with his readership. Such a statement is itself open to interpretation and requires an explanation which is quantum in its duality, it is both there and not there. It’s there if you take a flying leap of imagination and read far more into the statement than is on the paper, and it’s not there if you look at it through the unforgiving lens of the English language. It makes no sense.

Cutting through the pseudoscience, Creswell suggests that mixed method inquiry (see below), lends itself well to the pragmatic stance. Cynically one might suggest that this is because many pragmatists probably can’t make up their mind as to what they’re trying to say or investigate… I couldn’t possibly comment.

Research Design—Quantitative

Creswell defines quantitative research as a “means for testing objective theories by examining the relationship among variables (Creswell, 2014, p. 4).” This can include the standard scientific method of testing and rejecting or accepting a null hypothesis, typically on the basis of accepted statistical analysis resulting from observation and/or controlled experiment. For example, the research question may be phrased along the lines of: “There is no relationship between examination results and hours of tuition and home study combined.” Given a representative sample and a well-designed protocol, it may then be possible to generalize the results of such a study across a larger body than the original sample population.

Research Design—Qualitative

Creswell sees qualitative research as a means for “exploring and understanding the meaning individuals or groups ascribe to a social or human problem (p. 4).” Although I would personally feel more comfortable using the word issue rather than problem on the basis of the negative inference usually assigned to the word problem. An academic question may be more of a problem in the puzzle or quizzical sense rather than a negative one. But maybe that’s just me.

Creswell describes a process which is founded upon interpretation of meanings which are rarely clear-cut or supported by testable, replicable data, and rely upon “individual meaning” (p. 4). This is a process that—by its own definition—cannot be generalized in the same manner as a quantitative study, but can reveal frequently repeated explanations for given phenomena (see the descriptions of narrative and ethnographic research approaches discussed earlier in this post).

Research Design—Mixed Methods

As the name would suggest, combines or mixes both quantitative and qualitative forms in the same investigation. Creswell rightly stresses that this is more than simply using both methods of data collection and analysis. Performed well, a mixed methods study uses both methods in a mutually supportive manner. For example, a quantitative study may reveal a statistically significant trend by biology teachers to avoid teaching certain topics in a given geographical region, but a qualitative study may be required to answer why that might occur.

Crewswell (2014) goes on to suggest that certain worldviews tend to be associated with certain approaches to research design (e.g., postpositivist worldview to a quantitative approach), and whilst this might be the case in a stereotypical view, I can’t help but come to the opinion that in the real-world, a pragmatic point of view would be to use whatever approach fits the research question. If the research question lends itself to a quantitative approach, e.g., is there a relationship between…? Is there a significant difference between performance before and after…? Or a qualitative approach, e.g., why do some high school science teachers express an ambivalence to teaching global warming consensus theories or evolutionary biology? Or mixed method, is there a significant difference between the proportion of high school science teachers which express ambivalence in teaching publicly controversial issues in science, in religious-based school boards compared to public school boards; and if so, why does such a difference exist?

Once again, the approach to research should really be dependent upon a correctly formed research proposal, and central to the proposal would be a correctly formed research question or questions. In my opinion, without such a well-formed research question, the entire premise of the inquiry is fundamentally flawed from the outset.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s