Design Based Research vs. Mixed Methods: The Differences and Commonalities

This post is a summary and some reflection on a discussion paper posted to ITForum. It’s by Goknur Kaplan Akilli and is titled Design Based Research vs. Mixed Methods: The Differences and Commonalities. The author is a PhD candidate with some interesting research runs on the board.

The following contains two main sections

1. Reflections – my, as yet incomplete, meanderings on the paper.
2. Summary – an attempt to understand the paper.

Reflections

This paper seems to indicate that the education discipline, like information systems, appears to be struggling with understanding where design research fits. How is it different? How to do it well? Reading this paper should help me, given that one of my current tasks is to re-write chapter 3 of my thesis and essentially set out what I understand about these questions.

Given my background in information systems much of my thoughts on the following are influenced by that work. One representation of that work is given on the design research in information systems page. However, I don’t agree with a number of the points made there.

Questions and points of disagreement or depature

DBR: a methodology or a paradigm

There appears to be some confusion over whether or not DBR is a methodology or a paradigm. Which may come back to a lack of agreement on the difference amongst the author and her cited sources.

In the last paragraph of the section on DBR the author suggests

Lastly, the immaturity of the methodology is another criticism (Kelly, 2004; Wang & Hannafin, 2005), which consists of methodological challenges that need to be addressed if DBR is to be developed “from a loose set of methods into a rigorous methodology” (Kelly, 2004, p.116).

In the conclusion the author suggests

DBR is more of a generic paradigm rather than a method in the way that mixed methods research is. DBR offers a new worldview of theory development and refinement along with design to construct design sciences of education.

So which is it, paradigm or method?

Dede’s “conditions of success” criticism

In a couple of sentences the author describes Dede’s (2004, 2005) criticisms of DBR related to “conditions of success”.

For instance, Dede (2004) argues that there seems to be hardly any standards to decide whether a design should be dropped or sustained and further explores due to its promising nature by differentiating it from its “conditions of success” (p.109). However this is not possible, since the findings in DBR are strongly bounded with contextual variables shaping the design’s “desirability, practicality and effectiveness” (Dede, 2005, p.7).

At this point in time, I don’t understand (at all) this criticism. Appears I have some more reading to do.

Interventions embody theory

In characterising DBR the author follows (DBRC 2003) that the interventions arising from DBR embody specific theoretical claims about teaching and learning. I’m assuming that one reason for this is that as research, DBR should be purposeful about its interventions and shouldn’t simply be trying any idea that crops up, it needs to be informed by theory. Within the information systems field and its approach to design research the theories which a design embodies are called “kernel theories”, a term coined by Walls et al (1992).

One of the foundations of this work is Simon’s work on sciences of the artifical work that underpins the interest in design research in a number of fields, including information systems.

Simon talks about the idea that it is possible to design a successful artifact that makes a contribution without being fully aware of all of the theories/knowledge which underpin the artifact. The example often used is that the folk who built the first airplanes had little understanding of the aeronautical sciences, the physics that explained why their machines flew.

There is also the problem that simply following learning and teaching theories my lead to inattentional blindness (aka perceptual blindness). A situation where the established theories of learning and teaching limit what you can actually see or envision happening in a given situation.

The possibility that requiring an embodiment of existing learning and teaching theory in all educational DBR is troubling if you agree that the vast majority of learning and teaching theories have been developed by more “traditional research”. An approach to research which, according to the DBR proponents, have some significant flaws. Hence the patterns embodied by current theories of learning and teaching may have some flaws which is limiting possibilities.

A later characteristic of DBR – interactive, collaborative, iterative and flexible processes – address this somewhat in the recognition that DBR is flexible enough to react to situations where the expected outcomes did not occur in reality and consequently led to a re-thinking of these understanding sof the world.

So perhaps the characteristic re: embodying theory should be understood that at the end of the DBR process the work should embody some sort of theory around learning and teaching. But perhaps, when it started, that theory was not well understood or espoused, or perhaps a new one developed as the iterative process was followed.

The role of theory

Throughout the paper, and the literature it quotes, there appears to be some issues with the definition of theory including:

• No clear definition of what the author or the literature thinks theory is (or not).
• Many different terms that are close to theory but seem to indicate a difference e.g. prototheories, design theory, design principles, “usable knowledge”..

All this seems to point to a lack of agreement to a fairly fundamental building block of this argument. Especially if you agree that the ultimate aim of research is to generate knowledge which should typically aim to be expressed as theory.

This is especially troubling given that one of the stated criticisms of DBR is that it often doesn’t make a significant contribution of theory. This might well be expected if the current understanding of theory within the education discipline is more appropriate to traditional research and somewhat under done or limiting due to the nature of design research.

The nature of theory also raises its head in terms of the problems facing DBR in “universality of findings”.

I also wonder if my connection with Shirley and her work on the nature of theory in information systems colours my perspective. I also think that only very limited reading I’ve done of the education based DBR literature also shows, as I believe some of that literature does address this issue. Though not with the same outcomes as Shirley’s work.

Misc questions

Much of the design research thinking in information systems is focused on the IT (or IS) artifact. The aim of design research is to construct or develop theory to guide the construction of an IT artifact. What, if any, is the similar aim in education?

Summary

The basic aim of the paper is to establish that there is a difference between DBR and mixed methods as a research methodology. It has two main sections, one each on the respective approaches. One assumes (given that I haven’t read the paper) that these two sections explain the differences and serves the authors purpose.

In the authors conclusion she argues

• Mixed methods is a research method. A 3rd methodology that arose from the qualitative/quantitative paradigm wars.
• DBR more of a generic paradigm, claims it is a wicked paradigm since it deals with wicked problems (Rittel & Webber, 1973).
• DBR offers a new worldview of theory development and refinement.
• But it also offers a newly-emerging research methodology drawing on different fields of design and education, including mixed methods.
• DBR produces knowledge that is
  • dynamic,
    It is knowledge that changes in relation to context, shaped by time, place, actors and actions.
  • usable,
    Knolwedge that informs theories and real-world practices.
  • glocal.
    Local in that it produces tentative generalizations that are drawn from initial implementations. Suggested it is global because these generalisations can be “globalised” with studies that have similar contexts.

Design-Based Research

First, the author establishes some of the variety in perspectives, or at least terminology, used around DBR

Design Based Research Collective (DBRC) (2003, p.5) characterizes DBR as a research paradigm that “blends empirical educational research with the theory-driven design of learning environments,”

Wang and Hannafin (2005) define it as “a systematic but flexible methodology [italics added] aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings, and leading to contextually-sensitive design principles and theories”

DBR aims to develop and refine theories via closely linked strategies rather than testing intact theories using traditional methodologies (Edelson, 2002).

Suggests that the origins of DBR were to

• develop a design science of education – a connection back to Simon (1969).
• develop a methodology to help develop design theory (Collins, 1992)
• prevent the detachment of educational research (laboratory settings) from problems and issues of everyday practice
• close the credibility gap (Levin & O’Donnell, 1999)
• develop more “usable knowledge”.

The main characteristics of DBR are

• Pragmatic.
  It is based in real-world situations. Attempts to improve those through interventions, but at the same time make a contribution to theory. The value of theory is in its utility to practitioners and other designers.
• Theory-driven and grounded in real-world contexts.
  The interventions embody specific theoretical claims about teaching and learning.
• Uses a process that is interactive, collaborative, iterative and flexible.
  It continues to respond to the findings within the real world setting.
• Is integrative through the richness and variety of theories, methods and procedures utilized to meet research needs.
  Multiple (mixed) methods are used to analyse and refine the intervention.
• Contextualised.
  It cannot be thought of as indepedent from context, must involve authentic settings.

Criticisms of DBR include

• “Conditions of success” – mentioned above as something I don’t get
• Absence of theoretical foundation or contribution
  Dede (2004) suggests this is due to different skills for creative designers and rigorous scholars. Also the problem of “innovation fascination” leading to under-conceptualized research in order to try the new toy.

  Is suggested that DBR is over-methodologized and its tendency towards excessive data collection (Nona, are you reading this? 😉 ) which result in only tiny contributions to theory.

• Generalisation.
  The very contextual nature of DBR is seen as making it difficult to make generalisations to other contexts. Some arguments here about the close interaction between researcher/practitioner necessarily limiting the ability to be rigorous or objective.
• Immaturity of the methodology
  Suggests DBR is more a loose set of methods than a rigorous methodology (Kelly, 2004). He suggests that DBR studies are descrbied as a set of processes rather than describing the essential underlying conceptual structure.

Mixed methods research

Defined as a methodology that uses multiple approaches in all stages of research.

Theoretical assumptions include

• Pragmatist philosophy
  Researchers avoid philosophical arguments about research methods and mix approaches based on the utility the approach will give within a particular problem or context.
• Compatibility thesis
  Assumption that quantitative and qualitative methods are compatible and can be mixed.
• principle of mixed methods research
  Methods are mixed in a way that uses their complementary strengths and non-overlapping weaknesses.

Four additional criteria

1. Sequence of data collection approaches.
   Concurrently/sequentially, intra or inter-method mixing. Connected with “data triangulation” and “method triangulation”.
2. Which method was given priority
3. Stage of integration – where did the mixing or connecting of methods occur
4. Theoretical perspectives – the researchers’ personal stances toward the topcis.

From these a diverse typology of mixed methods research is outlined

A strength of mixed methods is the ability to answer both exploratory and confirmatory questions at the same time – i.e. verify and generate theory.

A strength of mix-methods research is the availability of information about how to do such research well.

Major criticism of it is the “incompatibility thesis” which argues that quantitative and qualitative research paradigms should not be mixed. (Onwuegbuzie and Leech, 2005)

References

Dede, C. (2004). If design-based research is the answer, what is the question? Journal of the Learning Sciences, 13(1), 105-114.

Dede, C. (2005). Why design-based research is both important and difficult. Educational Technology, 45(1), 5-8.

Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8, 35-37.

diSessa, A. A., & Cobb, P. (2004). Ontological innovation and the role of theory in design experiments. Journal of the Learning Sciences, 13(1), 77-103.

Kelly, A. E. (2004). Design research in education: Yes, but is it methodological? Journal of the Learning Sciences, 13(1), 115-128.

Onwuegbuzie, A. & Leech, N. 2005. “Taking the “Q” Out of Research: Teaching Research Methodology Courses Without the Divide Between Quantitative and Qualitative Paradigms.” Quality and Quantity 39, 267-296.

Walls, J., Widmeyer, G., & El Sawy, O. A. (1992). Building an Information System Design Theory for Vigilant EIS. Information Systems Research, 3(1), 36-58.