Assembling the heterogeneous elements for (digital) learning

Category: reflectivealignment Page 1 of 3

Learning Analytics: engaging with and changing learning and teaching

The following is an attempt to build a bit more on an earlier idea around the use of learning analytics. It’s an attempt to frame a different approach to the use of learning analytics and to share these ideas in preparation for a potential project.

In part, the project is based on the assumption that the current predominant applications of learning analytics are

  1. By management as a tool to enable “data-based” decision making.
  2. By providing tools to students that allow them to reflect on their learning.
  3. By researchers.

And that as identified by Dawson, Heathcote & Poole (2010) there is a

lack of research regarding the application of academic analytics to inform the design, delivery and future evaluations of individual teaching practices

i.e. while there existing applications of learning analytics by/for management, researchers and students is important and should continue, there is a need to explore how learning analytics can be used by teaching staff to inform and improve their practice.

The theoretical basis the current project idea are, in summary,

  1. Drawing on Seely-Brown & Duiguid’s (1991) ideas around how “abstractions detached from practice distort or obscure intricacies of that practices” there is value in examining how what learning analytics might do by focusing on an in-depth engagement with actual academic practice to better enable exploration, understanding and innovation around the application of learning analytics to individual teaching practices.
  2. The quality of student learning outcomes is influenced by the conceptions of learning and teaching, and the perceptions of the teaching environment held by teaching staff (Trigwell, 2001; Prosser et al, 2003; Richardson, 2005; Ramsden et al, 2007).
  3. Learning analytics can be useful in revealing different and additional insights about what is going on within a course (and in other courses).
  4. Transforming the insights from learning analytics to informed pedagogical action is, for the majority of academics, complex and labour intensive (Dawson, et al, 2010).
  5. Distributed leadership – built on foundations of distributed cognition and activity theory – seeks to distribute power (the ability to get things done) through a collegial sharing of knowledge, of practice, and reflection within a socio-cultural context. (Spillane et al, 2004; Parrish et al 2008).
  6. Encouraging academics to engage in reflection on their teaching is an effective way to enhance teaching practice and eventually student learning (Kreber and Castleden, 2009).

Consequently, the project seeks to engage groups of academics in cycles of participatory action research where they are encouraged and enabled to explore and reflect on their courses they have taught with various learning analytics tools and other lenses. In preparation for this a range of existing analytics tools and forms of analysis will be applied to the courses. In response to the cycles the tools/analyses may be modified or new ones created. In particular, the tools will be modified/developed to make it easier for academics to transform the information provided by the application of learning analytics into informed pedagogical action.

In particular, the project will explore how the tools can be modified to enable the sharing of knowledge, practice and reflection between the participants and eventually the broader academic community. To break down the course-based silos and make it easier for academics to see what other staff have done and with what impacts.

References

Dawson, S., Heathcote, L., & Poole, G. (2010). Harnessing ICT potential: The adoption and analysis of ICT systems for enhancing the student learning experience. International Journal of Educational Management, 24(2), 116-128. doi:10.1108/09513541011020936

Kreber, C. and H. Castleden (2009). “Reflection on teaching and epistemological structure: reflective and critically reflective processes in ‘pure/soft’ and ‘pure/hard’ fields.” Higher Education 57(4): 509-531.

Parrish, D., Lefoe, G., Smigiel, H., & Albury, R. (2008). The GREEN Resource: The development of leadership capacity in higher education. Wollongong: CEDIR, University of Wollongong.

Prosser, M., P. Ramsden, et al. (2003). “Dissonance in experience of teaching and its relation to the quality of student learning.” Studies in Higher Education 28(1): 37-48.

Spillane, J., Halverson, R., & Diamond, J. (2004). Towards a theory of leadership practice: a distributed perspective. Journal of Curriculum Studies, 36(1), 3-34.

Ramsden, P., M. Prosser, et al. (2007). “University teachers’ experiences of academic leadership and their approaches to teaching.” Learning and Instruction 17(2): 140-155.

Richardson, J. (2005). Students’ approaches to learning and teachers’ approaches to teaching in higher education. Educational Psychology, 25(6), 673-680.

Seely Brown, J., & Duguid, P. (1991). Organizational learning and communities-of-practice: Toward a unified view of working, learning, and innovation. Organization Science, 2(1), 40-57.

Trigwell, K. (2001). “Judging university teaching.” The International Journal for Academic Development 6(1): 65-73.

Schemata and the source of dissonance?

The following is intended to be an illustration of one of the potential origins of the gap between learning technologists and educators. It picks up on the idea of schemata from this week’s study in one course and connects to my point about the dissonance between how educational technology is implemented in universities and what we know about how people learn.

I’m sure folk who have been around the education discipline longer than I will have seen this already. But it is a nice little activity and not one that I’d seen previously.

An experiment

Read the following paragraph and fill in the blanks. If you’re really keen add a comment below with what you got. Actually, gathering a collection of responses from a range of people would be really interesting.

The questions that p________ face as they raise ch________ from in_________ to adult are not easy to an _________. Both f______ and m________ can become concerned when health problems such as co_________ arise anytime after the e____ stage to later life. Experts recommend that young ch____ should have plenty of s________ and nutritious food for healthy growth. B___ and g____ should not share the same b______ or even be in the same r______. They may be afraid of the d_____.

Now, take a look at the original version of this paragraph.

Is there any difference between it and what you got? Certainly was for me.

Schemata

This problem was introduced in a week that was looking at Piaget and other theories about how folk learn. In particular, this example was used as an example of the role schemata play in how people perceive and process the world and what is happening within it.

I am a father of three wonderful kids. So, over the last 10+ years I’ve developed some significant schemata around raising kids. When I read the above paragraph, the words that filled the blanks for me were: parents, children, infant, answer, fathers, mothers,….and it was hear that I first paused. None of my children really suffered from colic, so that didn’t spring to mind, but I started actively searching for ways I could make this paragraph fit the schemata that I had activated. i.e. I was thinking “parent”, so I was trying to make these things fit.

Schemata are mental representations of an associated set of perceptions etc. The influence how you see what is going on.

I’m somewhat interested in seeing what words others have gotten from the above exercise, especially those without (recent) experience of parental responsibilities.

A difference of schemata

Learning technologists (or just plain innovative teachers) have significantly different schemata than your plain, everyday academic. Especially those that haven’t had much experience of online learning, constructivist learning, *insert “good” teaching practice of your choice*. Even within the population of learning technologists there is a vast difference in schemata.

Different schemata means that these folk see the world in very different ways.

A triumph of assimilation of accommodation

The on-going tendency of folk to say things like (as in an article from the Australian newspaper’s higher education section) Online no substitute for face to face teaching says something about their schemata and (to extend the (naive/simplistic) application of Piaget) the triumph of assimilation over accommodation.

For Piaget people are driven to maintain an equilibrium between what the know and what they observe in the outside world. When they perceive something new in the world they enter a state of disequilibrium and are driven to return to balance. For Piaget, there are two ways this is done.

  1. Assimilation – where the new insight is fitted into existing schemata.
  2. Accommodation – where schemata are changed (either old are modified or new are created) to account for the new insights.

I’d suggest that for a majority of academic staff (and senior management) when it comes to new approaches to learning and teaching their primary coping mechanism has been assimilation. Forcing those new approaches into the schemata they already have. i.e. the Moodle course site is a great place to upload all my handouts and have online lectures.

As I’ve argued before I believe this is because the approaches used to introduce new learning approaches in universities have had more in common with behaviourism than constructivism. Consequently the approaches have not been all that successful in changing schemata.

30% of information about task performance

Over on the Remote Learner blog, Jason Cole has posted some information about a keynote by Dr Richard Clark at one of the US MoodleMoots. I want to focus on one key quote from that talk and its implications for Australian higher education and current trends to “improve” learning and teaching and adopt open source LMS (like Moodle).

It’s my argument that this quote, and the research behind it, has implications for the way these projects are conceptualised and run. i.e. they are missing out on a huge amount of potential.

Task analysis and the 30%

The quote from the presentation is

In task analysis, top experts only provide 30% of information about how they perform tasks.

It’s claimed that all the points made by Clark in his presentation are supported by research. It appears likely that the support for this claim comes from Sullivan et al (2008). This paper address the problem of trying to develop procedural skills necessary for professions such as surgery.

The above quote arises due to the problems experts have in describing what they do. Sullivan et al (2008) offer various descriptions and references of this problem in the introduction

This is often difficult because as physicians gain expertise their skills become automated and the steps of the skill blend together [2]. Automated knowledge is achieved by years of practice and experience, wherein the basic elements of the task are performed largely without conscious awareness [3]. This causes experts to omit specific steps when trying to describe a procedure because this information is no longer accessible to conscious processes [2]

Then later, when describing the findings of their research they write

The fact that the experts were not able to articulate all of
the steps and decisions of the task is consistent with the expertise literature that shows that expertise is highly automated [2,3,5] and that experts make errors when trying to describe how they complete a task [3,6,7]. In essence, as the experts developed expertise, their knowledge of the task changed from declarative to procedural knowledge. Declarative knowledge is knowing facts, events, and objects and is found in our conscious working memory [2]. Procedural knowledge is knowing how to perform a task and includes both motor and cognitive skills [2]. Procedural knowledge is automated and operates outside of conscious awareness [2,3]. Once a skill becomes automated, it is fine-tuned to run on autopilot and executes much faster than conscious processes [2,8]. This causes the expert to omit steps and decision points while teaching a procedure because they have literally lost access to the behaviors and cognitive decisions that are made during skill execution [2,5].

The link to analysis and design

A large number of universities within Australia are either:

  1. Changing their LMS to an open source LMS (e.g. Moodle or Sakai), and using this as an opportunity to “renew” their online learning; and/or
  2. Busy on broader interventions to “renew” their online learning due to changes in government policies such as quality assurance, graduate attributes and a move to demand funding for university places.

The common process being adopted by most of these projects is from the planning school of process. i.e. you undertake analysis to identify all relevant, objective information and then design the solution on that basis. You then employ a project team to ensure that the design gets implemented, and finally you put in a skeleton team that maintains the design. This works in terms of information systems (e.g. the selection, implementation and support of a LMS) or broader organisational change (e.g. strategic plans).

The problem is that the “expert problem” Clark refers to above means that it is difficult to gather all the necessary information. It’s difficult to get the people with the knowledge to tell all that they know.

A related example.

The StaffMyCQU Example

Some colleagues and I – over a period of almost 10 years – designed, supported, and evolved an information system call Staff MyCQU. An early part of it’s evolution is described in the “Student Records” section of this paper. It was a fairly simple web application that provided university staff with access to student records and range of related services. Over it’s life cycle, a range of new and different features were added and existing features tweaked, all in response to interactions with the system’s users.

Importantly, the systems developers were also generally the people handling user queries and problems on the “helpdesk”. Quite often, changes to the system would result in tweaks and changes. Rather than being designed up front, the system grew and changed with people using it.

The technology used to implement Staff MyCQU is now deemed ancient and, even more importantly, the system and what it represents is now politically tainted within the organisation. Hence, for the last year or so, the information technology folk at the institution have been working on replacement systems. Just recently, there’s been some concrete outcomes of that work which has resulted in systems being shown to folk, including some of the folk who had used Staff MyCQU. On being shown a particular feature of the new system, it soon became obvious that the system didn’t include a fairly common extension of the feature. An extension that had actually been within StaffMyCQU from the start.

The designers of the new system, with little or no direct connection with actual users doing actual work, don’t have the knowledge about user needs to design a system that is equivalent to what already exists. A perfect example of why the strict separation of analysis, design, implementation and use/maintenance that is explicit in most IT projects and divisions is a significant problem.

The need for growing knowledge

Sullivan et al (2008) suggest cognitive task analysis as a way to better “getting at” the knowledge held by the expert, and there’s a place for that. However, I also think that there is a need, especially in some contexts, for recognition that the engineering/planning method is just not appropriate for some contexts. In some contexts, you need more of a growing/gardening approach. Or, in some cases you need to include more of the growing/gardening approach into your engineering method.

Rather than seeking to gather and analyse all knowledge separated from practice and prior to implementation. Implementation needs to be designed to pay close attention to knowledge that is generated during implementation and the ability to act upon that knowledge.

Especially for wicked problems and complex systems

Trying to improve learning and teaching within a university is a wicked problem. There are many different stakeholders or groups of stakeholders, each with a different frame of reference which leads to different understanding of how to solve the problem. Simple techno-rational solutions to wicked problems rely on the adoption of one of those frames of reference and ignorance of the remainder.

For example, implementation of a new LMS is seen as an information technology problem and treated as such. Consequently, success is measured by uptime and successful project implementation. Not on the quality of learning and teaching that results.

In addition, as you solve wicked problems, you and all of the stakeholders learn more about the problem. The multiple frames of reference change and consequently the appropriate solutions change. This is getting into the area of complex adaptive systems. Dave Snowden has a recent post about why human complex adaptive systems are different.

Prediction

Universities that lean too heavily on engineering/planning approaches to improving learning and teaching will fail. However, they are likely to appear to succeed due to the types of indicators they choose to adopt to as measurements of success and the capability of actors to game those indicators.

Universities that adopt more of a gardening approach, will have greater levels of success, but will have a messier time of it during their projects. These universities will be where the really innovative stuff comes from.

References

Sullivan, M., A. Oretga, et al. (2008). “Assessing the teaching of procedural skills: can cognitive task analysis add to our traditional teaching methods.” The American Journal of Surgery 195: 20-23.

The grammar of school, psychological dissonance and all professors are rather ludditical

Yesterday, via a tweet from @marksmithers I read this post from the author of the DIYU book titled “Vast Majority of Professors Are Rather Ludditical”. This is somewhat typical of the defict model of academics which is fairly prevalent and rather pointless. It’s pointless for a number of reasons, but the main one is that it is not a helpful starting point for bringing a out change as it ignores the broader problem and consequently most solutions that arise from a deficit model won’t work.

One of the major problems this approach tends to ignore is the broader impact of the grammar of school (first from Tyack and Cuban and then Papert). I’m currently reading The nature of technology (more on this later) by W. Brian Arthur. The following is a summary and a little bit of reflection upon a section titled “Lock-in and Adaptive Stretch”, which seems to connect closely with the grammar of school idea.

Psychological dissonance and adaptive stretch

Arthur offers the following quote from the sociologist Diane Vaughan around psychological dissonance

[In the situations we deal with as humans, we use] a frame of reference constructed from integrated sets of assumptions, expectations and experiences. Everything is perceived on the basis of this framework. The framework becomes self-confirming because, whenever we can, we tend to impost it on experiences and events, creating incidents and relationships that conform to it. And we tend to ignore, misperceive, or deny events that do not fit it. As a consequence, it generally leads us to what we are looking for. This frame of references is not easily altered or dismantled, because the way we tend to see the world is intimately linked to how we see and define ourselves in relation to the world. Thus, we have a vested interest in maintaining consistency because our own identity is at risk.

Arthur goes onto to suggest that “the greater the distances between a novel solution and the accepted one, the large is this lock-in to previous tradition”. He then defines the lock-in of the older approach as adaptive stretch. This is the situation where it is easier to reach for the old approaches and adapt it to the new circumstances through stretching.

Hence professors are ludditical

But haven’t I just made the case, this is exactly what happens with the vast majority of academic practice around e-learning. If they are using e-learning at all – and not simply sticking with face-to-face teaching – most teaching academics are still using lectures, printed notes and other relics of the past that they have stretched into the new context.

They don’t have the knowledge to move on, so we have to make them non-ludditical. This is when management and leadership at universities rolls into action and identifies plans and projects that will help generate non-ludditical academics.

The pot calling the kettle black

My argument is that if you step back a bit further the approaches being recommended and adopted by researchers and senior management; the way those approaches are implemented; and they way they are evaluated for success, are themselves suffering from psychological dissonance and adaptive stretch. The approaches almost without exception borrow from a traditional project management approach and go something like:

  • Small group of important people identify the problem and the best solution.
  • Hand it over to a project group to implement.
  • The project group tick the important project boxes:
    • Develop a detailed project plan with specific KPIs and deadlines.
    • Demonstrate importance of project by wheeling out senior managers to say how important the project is.
    • Implement a marketing push involving regular updates, newsletters, posters, coffee mugs and presentations.
    • Develop compulsory training sessions which all must attend.
    • Downplay any negative experiences and explain them away.
    • Ensure correct implementation.
    • Get an evaluation done by people paid for and reporting to the senior managers who have been visibly associated with the project.
    • Explain how successful the project was.
  • Complain about how the ludditical academics have ruined the project through adaptive stretching.

Frames of reference and coffee mugs

One of the fundamental problem with these approaches to projects within higher education is that it effectively ignores the frames of reference that academics bring to problem. Rather than start with the existing frames of reference and build on those, this approach to projects is all about moving people straight into a new frame of reference. In doing this, there is always incredible dissonance between how the project people think an action will be interpreted and how it actually is interpreted.

For example, a few years ago the institution I used to work for (at least as of CoB today) adopted Chickering and Gamson’s (1987) 7 principles for good practice in undergraduate teaching as a foundation for the new learning and teaching management plan. The project around this decision basically followed the above process. As part of the marketing push, all academics (and perhaps all staff) received a coffee mug and a little palm card with the 7 principles in nice text and a link to the project website. The intent of the project was to increase awareness of the academics of the 7 principles and how important they were to the institution.

The problem was, that at around this time the institution was going through yet more restructures and there was grave misgivings from senior management about how much money the institution didn’t have. The institution was having to save money and this was being felt by the academics in terms of limits on conference travel, marking support etc. It is with this frame of reference that the academics saw the institution spending a fair amount of money on coffee mugs and palm cards. Just a touch of dissonance.

What’s worse, a number of academics were able to look at the 7 principles and see principle #4 “gives prompt feedback” and relate that to the difficulty of giving prompt feedback because there’s no money for marking support. Not to mention the push from some senior managers about how important research is to future career progression.

So, the solution is?

I return to a quote from Cavallo (2004) that I’ve used before

As we see it, real change is inherently a kind of learning. For
people to change the way they think about and practice education, rather than merely being told what to do differently, we believe that practitioners must have experiences that enable appropriation of new modes of teaching and learning that enable them to reconsider and restructure their thinking and practice.

Rather than tell academics what to do, you need to create contextualised experiences for academics that enable appropriation of new models of teaching and learning. What most senior managers at universities and many of the commentators don’t see, is that the environment at most universities is preventing academics from having these experiences and then preventing them from appropriating the new models of teaching.

The policies, processes, systems and expectations senior managers create within universities are preventing academics from becoming “non-ludditical”. You can implement all the “projects” you want, but if you don’t work on the policies, processes, systems and expectations in ways that connect with the frames of reference of the academics within the institution, you won’t get growth.

References

Cavallo, D. (2004). Models of growth – Towards fundamental change in learning environments. BT Technology Journal, 22(4), 96-112.

Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 39(7), 3-7.

Functional fixedness, analytics, and the LMS

A blog post on the website of Gilfes Education Group (apparently a “network of independent education experts”) picks up on the Indicators project and its take on academic analytics. The post seems to largely in agreement with what we’re doing and the reasons behind it.

The following seeks to pick up on a point made in the Gilfus post about the problem arising from ownership of the data and some of the other barriers that have been proposed. The argument I develop in the following that functional fixedness is a major barrier to the effective appropriation of academic analytics to help improve learning and teaching.

But first, an experiment

Imagine if you will that we’re in a room together. I’m going to set you a task. Here’s some matches, a box of tacks and a candle (see the image below). Your task is to attach the candle to a cork board on the wall in way that means that wax from the candle does not drip onto the table that is underneath the cork board.

Candle problem set up

How do you do it?

The solution is given an image at the end of this post.

Apparently, if I rephrase the problem solution a little to the following, it might improve your chances of success.

Here’s some matches, a box of tacks and a candle (see the image below). Your task is to attach the candle to a cork board on the wall in way that means that wax from the candle does not drip onto the table that is underneath the cork board.

Functional fixedness

If you’re anything like my brother-in-law on whom I tested this out in person, you did not arrive at the solution quickly, if at all. This experiment is called the candle problem and has been used to demonstrate the problem of functional fixedness.

Functional fixedness suggests that you have fixated on the design function of the object – i.e. the box of tacks is designed to hold the tacks – so much that you cannot see how it might be put to a different use to solve this problem. To put it in the words of German and Barrett (2005)

Problem solving can be inefficient when the solution requires subjects to generate an atypical function for an object and the object’s typical function has been primed

In other words, the problem description above had the box’s typical function primed as holding the tacks, hindering your ability to see another use for the box.

Academic analytics, the LMS and functional fixedness

For most universities there is an existing set of information systems. There’s the learning management system (LMS) in which learning takes place, and there is the data warehouse and associated business intelligence tools for providing reports and information. The people within these organisations, especially those already supporting (the IT folk) and using (management) the data warehouse, have been primed to see a typical use for these systems. They are fixated on using the LMS and data warehouse in a particular way.

Add into this mix the typical under resourcing/inefficient management of IT, and the typical top-down, techno-rational approach to management and it is simply too difficult for organisational members to see the case for moving aspects of academic analytics into the LMS.

It doesn’t help that it’s messy

The matter isn’t helped much by the benefits of moving aspects of academic analytics into the LMS are somewhat uncertain and messy. Being uncertain and messy aren’t characteristics of an approach likely to overcome functional fixedness. Especially in organisational environments where being efficient (defined as doing what we already do or have strategically planned to do) is the main intermediate goal. But then this is why innovation is hard in organisations, innovation is messy.

References

German, T. and H. C. Barrett (2005). “Functional fixedness in a technologically sparse culture.” Psychological Science 16(1): 1-5.

Solution

The solution to the Candle problem is represented in the following image.

Candle problem solution

Course websites and "libertarian paternalism"

Stephen Downes makes a valid point about my recent question about whether or not academics should manually create websites. I agree with his underlying point that academics should not be forced to use the institutional approach. Given any option I would not suggest such an approach.

Incompetent paternalism

However, at least within some Australian institutions academics are being forced to accept an institutional approach. That approach is typically expressed as “minimum service standards” which are specified by management. The academics are than expected to manually fulfill those standards. I have a problem with this approach, but if it is being adopted, then at the very least implement it in an efficient and effective way.

What is happening in these situations could be described as “incompetent” paternalism. Academics are being treated as children (a theory X perspective of academics underpins this approach). Management as the parents have to specify codes of behaviour. But when it comes to implementing this code of behaviour, management are actually making using an inefficient and ineffective approach.

I disagree strongly with this approach, but if management believe it, is it too much to ask them to do it efficiently and effectively? That’s one perspective, my real interest is in a third way that tries to effectively merge features of both incompetent paternalism and the academic free for all.

The “libertarian” paternalist alternative

The model that evolved in the early part of this decade could be described as a “libertarian” paternalist approach. It’s a bit of a stretch but I think the metaphor works.

The theory was/is that an appropriately skilled group, taking an adopter-focused and emergent development approach could develop a default course site that could effectively be used by a group of courses. That default course site could be automatically created. But since the group was using an emergent development approach, the default course site would continue evolving.

The default course site did not remove the academics freedom of choice. As implemented, academics could modify the default course sites in two ways:

  • Use the “LMS” to modify or add to the default course site; or
    Here’s an example default course site from 2006.
  • Create a real course site.
    Here’s a default course site where the academic has created a real course site.

This wasn’t a perfect solution, it still wasn’t flexible enough. We had plans to enable better merging of the default and real course sites. i.e. if a real course site was created, it could replace the default course site and make better use of the services offered by the “LMS”. But they never got off the drawing board.

In the almost 8 years that this approach was used, the “LMS” averaged around 300 course sites a year. The number of real course sites (i.e. academics doing their own thing) was never reach 10% in any given year. It averaged around 4% of course sites a year.

Lack of appropriation

Importantly, where possible, the aim was to observe what everyone was doing, especially the <10% creating the real courses sites, and use those insights to modify the default course sites.

The current management approach of specifying minimum standards is being driven by external desires, not by the experience of academics and students using the minimum standards.

References

Jones, D. and T. Lynch (1999). A Model for the Design of Web-based Systems that supports Adoption, Appropriation and Evolution. First ICSE Workshop on Web Engineering, Los Angeles.

The confusion of small and big changes

Over the last couple of days I’ve enjoyed a small discussion that has arisen out of some comments Kevin has made on my blog. This post is an attempt to partially engage with the most recent comment. I echo Kevin’s conclusion, I’d love to hear anyone else’s take on this.

The unanswered question

The main point I’d like to discuss is the question of small versus big changes. I have an opinion on this, but there’s not enough evidence to suggest that it’s an answer. The basic question might be phrased as: How do you improve the quality significant improvement in the quality of L&T in universities? You could make this much more general, along the lines of “How do you change organisational practices?”, but I’m going to stick with the specific.

I’m familiar with two broad responses:

  • Revolutionary (usually top-down) change; and
    This is where the necessary change is identified by someone, eventually they get agreement/the ability to implement the change through some sort of change management process. This usually involves some big change. e.g. the adoption of a new LMS for a university, trashing the LMS and adopting WPMU for L&T, adopting university wide graduate attributes, requiring every academic to have a formal teaching qualification etc. Or even more radical, the death of universities and their replacement by something else.
  • Evolutionary (usually bottom-up) change.
    Small-scale changes in practice, usually at the local level.

Kevin’s comment gives a good summary of the common problem with the evolutionary change approach

In my experience, especially at a large institution, taking the “small changes” route is the road to perdition. For me, this means I have to engage in a million little negotiations to get the small to accumulate to something significant. At the rate I’m going it will take me two lifetimes to bring about real change in the English Department.

As I mentioned above and indicate by the heading for this section, I don’t have what I would call an answer. I have an argument for the approach I would take and some evidence to support it, but I don’t think I can call it “the answer” (yet).

What I think is the answer

Last I year I gave a presentation called Herding cats, losing weight and how to improve learning and teaching (slides and video are available). In that presentation, the analogy used is that revolutionary change is like herding cats and that evolutionary change is like losing weight. Using this analogy I argue that the herding cats approach to improve the quality of teaching at a University has not worked empirically and that there is significant theory to explain why it will never work. That same theory suggests that an evolutionary approach informed by lessons learned from weight loss, is much more promising.

The general solution I suggest is one slide 200 or so (it was only a 60 minute presentation) and goes under the title “reflective alignment” and can be summarised as

All aspects of the learning and teaching environment are aligned to enable and encourage academic staff to reflect on their teaching with the aim of achieving 3rd order change.

Framed another way, the teaching environment at a university encourages and enables academics to be changing their thinking and practice of teaching. That is essentially do what they do now, make small changes each time they teach a course, but rather than changes that are not constrained by the same ways of thinking about teaching.

Having academics continually making these sorts of 3rd order changes (within an institution that encourages and enables them to make those 3rd order changes) will result (I think) in radically different and significantly improved learning and teaching.

When small changes won’t work

Like Kevin, I think that universities relying on small changes to improve learning and teaching will not work. Mostly because the university environment does not encourage nor enable the type of small scale changes that are required.

In the herding cats presentation a large part of the time was listing the parts of the university teaching environment that actively prevents the type of 3rd order change that is necessary. In fact, much of the bleating in posts on this blog are complaining about these limitations. Some examples include:

  • Rewards that favour research, not teaching.
    No matter how many formal teaching qualifications an academic is forced to acquire, if they get promoted (both at their current and other universities) through the quality of their promotion, then they will focus on research, not teaching.
  • Pressures arising from quality assurance and simplistic KPIs.
    Since the mid-1990s I’ve observed that it is only the courses with large failure rates or student complaints that get any attention from university management. Students, like most people get scared when their expectations aren’t meant. That means if you try something innovative students will complain. In addition, if you try something innovative you might have problems, which management hate. If you try something different, you are more likely to have to waste time responding to “management concerns”. The presentation references research showing that this is preventing academics from trying innovative work.

    With the rise of quality assurance and corporate aproaches to management, this trend is getting worse.

  • Removal of autonomy;
    As I’ve argued in a couple of posts top-down management is removing academic autonomy and perhaps purpose and subsequently reducing academic motivation.
  • Constraining systems;
    Increasingly universities are using information systems to perform learning and teaching. Those systems are designed on particular assumptions that limit the ability to change. The most obvious example is the LMS (be it open or closed source). This recent post includes discussion of this point around the LMS.

    The people, processes and policies within universities are being set up to use these systems. If you use something different, you are being inefficient.

  • Simplistic understandings of innovation.
    When it comes to understanding innovations (e.g. something as simple as a new LMS), universities have naive perspectives of the adoption process. As recognised by Bigum and Rowan (2004) this naive perspective assumes that the innovation passes through the adoption process largely unchanged, which means that the social must conform with the innovation.

    i.e. As the institution starts to adopt Moodle across all its courses, Moodle can and should stay exactly the same. You only need to show people how to use Moodle, nothing more. If what they want to do is not supported by Moodle, then they need to conform to what Moodle does, regardless of the ramifications.

My argument is that all of this and other factors within a university environment actively prevent small changes having broad outcomes. The university environment is actively discouraging 3rd order change and isn’t even very good at achieving 2nd order change.

But small change can’t make a big difference

Ignoring all that, people still get stuck on the idea of lots of small change creating really big change. They are wrong.

To justify that, first let me draw on people recognised as being much smarter and more important than I (Weick and Quinn, 1999)

The distinctive quality of continuous change is the idea that small continuous adjustments, created simultaneously across units, can cumulate and create substantial change.

The main reason people have trouble with this idea (I think) is that they think that the world is ordered and predictable. That the world is an ordered system. If you make a small change, you get a small effect. However, when you’re talking about a complex system, small changes can create radical outcomes.

I don’t have time to expand on this here, it’s talked about in the presentation I mentioned above. Anyway, Dave Snowden and any number of other people make this point better than I.

Big and small change in the wrong place

Here’s a new idea. One of the reasons why I think most universities are failing to improve the quality of their teaching is that they are focusing on big and small change in the wrong places.

In my experience, most universities are trying to make big improvement in teaching by introducing big changes in what academics do. Use a different system, use a different pedagogy, radically change your teaching so you are constructively aligned, get a teaching qualification etc. But at the same time, there is no radical change in the how the teaching environment works. There are no solutions to the above problems with the environment.

What I am suggesting is that there should be big changes in the environment to enable small changes on the part of the academic. In fact, in the presentation I argue that the aim is to help the academics do what good teaching academics have always done (Common, 1989)

Master teachers are not born; they become. They become primarily by developing a habit of mind, a way of looking critically at the work they do by developing the courage to recognise faults, and struggling to improve.

References

Bigum, C. and L. Rowan (2004). “Flexible learning in teacher education: myths, muddles and models.” Asia-Pacific Journal of Teacher Education 32(3): 213-226.

Common, D. (1989). “Master teachers in higher education: A matter of settings.” The Review of Higher Education 12(4): 375-387.

Weick, K. and R. Quinn (1999). “Organizational change and development.” Annual Review of Psychology 50: 361-386.

The road not taken

A recent post of mine continued the trend of reflecting on the impacts – in my mind negative impacts – of a top-down, compliance driven culture in higher education. This bit has been encouraged by
a comment on that post which makes a number of interesting points, at least in terms of encouraging some additional thinking on my part. It’s also serindipitously coincided with some recent local events.

My interpretation

I’ve interpreted the post as suggesting that no oversight can lead to a proliferation of chaos or bad practice. In terms of talking about teaching and learning within a university I tend to agree – more on this below. There’s also a point about moving academics beyond some of their existing practices and the suggestion that the top-down chain of command isn’t really a solution. It closes with something sparked this post

Yes, this means we have to sell, not try to dictate. Long road.

Free-for-all, top-down compliance and chaos

“Chaos” or complexity is not necessarily a bad thing. However, I do except that an organisation – like a university – does generally have to do something to ensure that the quality of its teaching and learning is improving. (Note: one of my principles is “It’s not how bad you start, but how quickly you get better”. I don’t believe in “being good” as a goal, it’s an on-going process.) At the very least I think a university taking public funds has to demonstrate that it is using those funds somewhat effectively.

This is why in the post that started this thread I proposed that the first stage of improving learning and teaching (i.e. what the teacher is) is not way to achieve this. In that stage, each academic is left to their own devices. What they do is up to them and their preferences and capabilities. There is little or no support. In my experience with this stage, there are some examples of very good teaching, but the vast majority is somewhat lacking.

This is where the process/quality/teaching nazis appear. These include consultants, government, educational researchers, senior management, IT folk etc. Each of these folk have the solution. The quality of teaching would be wonderful within the organisation if only every academic used process Y, product X. If every course had mapped its graduate attributes and had a course site that met a minimum service standard, then the quality of teaching would be wonderful. So, let’s set up a project team, specify the outcomes, implement them and then report success. Typically the aim is something along the lines of “Develop a systemic University-wide approach to learning and teaching” or perhaps even worse prove efficiency and control by aiming to “Centralise the strategic planning and managing of funds for learning and teaching support, activities and initiatives”.

In my experience these approaches never work. Mainly because the decisions made by the centralised, systemic University approach to learning and teaching are informed by experiences far removed from the realities of the teaching academics. The people making the decisions are generally senior managers who have either no recent teaching experience or only very narrow teaching experiences. Instead, the experience of these folk quickly becomes limited to the “systemic university-wide approach to learning and teaching”. That is, the initiatives they identify as important (e.g. mapping graduate attributes) become their main experience. Everything they think and do arises from that project. Their experience limits what decisions they can make.

What’s worse, the current management environment in Australian university encourages short-term (5 year) contracts for senior managers. In order to keep their job or move onto a new one, these managers have to prove their “ability to lead”. This means that they have to have successfully “led” completed projects which they can put on their CV. What’s worse, those projects have to fit within the current fads within higher education. The priority of these managers is not improving the experience of coal-face teaching academics, it’s about achieving the successful implementation of “systemic University-wide” projects.

This is why senior management can be so confident saying that Project X is a great success, when the coal-face teaching academics will be telling a very different story. This is what Chris Argyris (1990) termed organisational defensive routines and model 1 behaviour in organisations – discussed in this post.

So, in the second way, which I describe as “what management does”, the decisions about how to improve learning and teaching are being made by people who have limited experience of coal-face teaching and who also have significant motivating factors to have successful projects. Is it any surprise that this approach doesn’t create long-term sustainable change?

Rather than create the “proof” of effectiveness required by those providing the funds, this approach creates compliance and task corruption. i.e. the KPIs are met, but by ticking the boxes, not in outcome. For example, I know of an institution that has developed a “checklist” for course websites. It’s a long list of requirements that a minimum course site is expected to fulfill. The idea is that academics that build these sites, and their colleagues of moderate the course and course site, will work their way through the checklist ensuring that each requirement is fulfilled. In reality, a significant number of academics are asking “Is you’re site ready?” and then ticking all the boxes.

The road not taken

My argument is that there is a third way that promises better outcomes, but it’s continuing to the road not taken. Which is somewhat surprising for my current institution given that it’s strategic plan includes the following in its vision

We strive to understand their environment and situation, their
circumstances and goals, so we can help them achieve what they want to achieve and be who and what they want to be, one person at a time.

This is a brilliant summation of what I’m trying to get at with the “third way”. My post from yesterday gives some background into the origins of this perspective (more to come).

In terms of the third way, I should have mentioned Dave Snowden’s “how to manage a birthday party story” (the video is below) which also fits nicely with the three ways I’ve expressed. Here’s the connection I make:

  • chaotic system == what the teacher is.
  • ordered system == what management does.
  • complex system == what the teacher does.

References

Argyris, C. (1990). Overcoming Organisational Defenses: Facilitating Organisational Learning, Prentice Hall.

The role of experience

Peter Albion picked up on an earlier post of mine and offers a brief description of his own experience within Australian universities. In particular, the increasing focus on compliance with bureaucratic systems as a means of assuring quality, a move back to hierarchies of command and control and apparent adoption of a Theory X view. A view that resonates with what I see within my current institution and one others talk about.

This morning I was listening to this talk by Baroness Susan Greenfield. In the end she suggests that online network is potentially harmful, but I’m going to ignore that. One of the fundamental planks for her argument is brain plasticity. i.e. that the brain is shaped by what we do with it. What we experience, what we think shapes our brain.

What is the current environment of compliance, command and control, and Theory X doing to the thoughts and brains of the academics that work within them?

Dan Pink talks about motivation and suggests that it requires workers to have feelings of autonomy, mastery and purpose. When it comes to learning and teaching within universities, I’ve argued previously that for some the current environment provides anything but that combination.

As it happens, I’m also reading at the moment a book by James Zull called The Art of Changing the Brain: Enriching the practice of teaching by exploring the biology of learning. I think this quote is interesting (emphasis in original)

..no outside influence or force can cause a brain to learn. It will decide on its own. Thus, one important rule for helping people to learn is to help the learner feel she is in control.

For me, the lesson here is that if you want to improve learning and teaching at Universities, the academics have to feel that they are in control. This does not mean they do their own thing. As Peter wrote

There is some benefit in ensuring that certain basics are in place but there is also room for some variation that provides scope for the next improvement to emerge.

The academic has to feel like they are in charge of that next improvement, to have the room for some variation. The compliance, top-down culture infecting universities (in Australia at least) is removing that control and is often ineffective in ensuring that the basics are in place because it has removed the motivation (in the form of autonomy, mastery and purpose) from the academics.

The need for a third way

One of the themes for this blog is that the majority of current approaches to improving learning and teaching within universities simply don’t work. At least not in terms of enabling improvement in a majority of the learning and teaching at an institution. Recently I finally completed reading the last bits of the book Nudge by Thaler and Sunstein. Chapter 18 is titled “The Real Third Way”. This post explores how that metaphor connects with some of the thinking expressed here.

The real third way

Thaler and Sunstein mention that the “20th century was pervaded by a great deal of artificial talk about the possibility of a ‘Third Way'” in politics. Their proposal is that libertarian paternalism, the topic of the book, represents a real third way. I’m not talking politics but there appears to be the same need to break out of a pointless dichotomy and move onto something more useful.

The characterisations of the two existing ways provided by Thaler and Sunstein are fairly traditional (stereotypical?) extremes of the political spectrum. i.e.:

  1. Liberal/Democrat – “enthusiasm for rigid national requirements and for command-and-control regulation. Having identified serious problems in the private market, Democrats have often insisted on firm mandates, typically eliminating or at least reducing freedom of choice.”.
  2. Conservative/Republican – have argued against government intervention and on behalf of a laissez-fair approach with freedom of choice being a defining principle. They argue that “in light of the sheer diversity of Americans one size cannot possibly fit all”.

Thaler and Sunstein’s third way – libertarian paternalism – is based on two claims:

  1. Choice architecture is pervasive and unavoidable.
    Small features of social situations have a significant impact on the decisions people make. The set of these features – the choice architecture – in any given social situation already exists and is already influencing people toward making good or bad decisions.
  2. Choice architecture can be manipulated while retaining freedom of choice.
    It is possible to make minor changes to the set of features in a social situation such that it encourages people to make “better” decisions, whilst still allowing them to make the “bad” decision, if that’s what they want.

Connections with improving learning and teaching

Early last year I borrowed and slightly modified Bigg’s 3 levels of teaching to identify 3 levels of improving learning and teaching. Obviously there is a numerical connection between these 3 levels and the “3 ways” outlined above. The more I’ve thought about it, the more I realise that the connections are more significant than that, and that the “3rd way” seems to be a useful way to position my beliefs about how to improve learning and teaching within a university. Here goes my first attempt at explicating it.

Expanding upon the 3 levels of improving L&T

The 3 levels I initially introduced can be expanded/morphed into ways or into stages. In terms of stages, I could probably argue that the levels/stages represent a historical evolution of how learning and teaching has been dealt within in Universities. Those three stages are:

  1. What the teacher is (i.e. ignore L&T).
    This is the traditional/historical stage that some long term academics look back on with fond memories. Where university management didn’t really get involved with teaching and learning. Individual academics were left to teach the course they way they felt it should be taught. There was little over sight and little need for outside support.

    The quality of the teaching was solely down to the nature of the teacher. If they were a good teacher, good things happened. If bad….. This was the era of selective higher education where, theoretically, only the best and the brightest went to university and most were seen to have the intellectual capability and drive to succeed regardless.

    For a surprising number of universities, especially those in the top rank of universities, this is still primarily how they operate. However, those of us working in “lesser” institutions are now seeing a different situation.

  2. What management does (i.e. blame the teacher).
    Due to the broadly publicised characteristics of globalisation, the knowledge economy, accountability etc. there is now significant pressure upon universities to demonstrate that the teaching at their institutions is of high quality. Actually, this has morphed into proxy measures where the quality of teaching is being measured by ad hoc student memories of their experience (CEQ surveys), how many of the academics have been forced to complete graduate certificates in higher education, what percentage of courses have course websites and how well the institution has filled out forms mapping graduate attributes.

    All of these changes to the practice of teaching and learning are projects that are initiated and “led” by senior university management. The success of the institution is based on how well senior university management have been in completing those projects.

    As each new fad arises within government of the university sector, there is a new set of projects to be completed. Similarly, when a new set of senior management start within an institution, there is a new set of projects to be completed. In this case, however, the projects aren’t typically all that new. Instead they are simply the opposite of what the last management did. i.e. if L&T support was centralised by the last lot of management, it must now be de-centralised.

    Most academics suffering through this stage would like to move back to the first stage, I think they and their institutions need to move onto the next one.

  3. What the teacher does.
    For me this is where the institution its systems, processes etc are continually being aligned to encourage and enable academics to improve what they are doing. The focus is on what the teacher does. This has strong connections with ideas of distributive leadership, the work of Fullan (2008) and Biggs (2001).

    For me implementing this stage means taking an approach more informed by complex adaptive systems, distributive leadership, libertarian paternalism, emergent/ateleological design and much more. This stage recognises that in many universities stage 1 doesn’t work any longer. There are too many people and skills that need to be drawn upon for successful teaching that academics can’t do it by themselves (if they ever did). However, that doesn’t mean that the freedom of academics to apply their insights and knowledge should be removed.

So, now I’ve expanded on those, time to connect these three ways with some other triads.

Connections with politics

The following table summarises what I see as the connections with the 3 stages of improving learning and teaching and the work of Thaler and Sunstein (2008).

  1. Conservative/republican == What the teacher is.
    i.e. the laissez-faire approach to teaching and learning. Academics are all too different, no one system or approach to teaching can work for us.
  2. Liberal/democrat == What management does.
    There are big problems with learning and teaching at universities that can only be solved by major projects led by management. Academics can’t be trusted to teach properly we need to put in place systems that mandate how they will teach and force them to comply.
  3. Libertarian paternalism == What the teacher does.
    The teaching environment (including the people, systems, processes, policies and everything else) within a university has all sorts of characteristics that influence academics to make good and bad decisions about how they teach. To improve teaching you need to make small and on-going changes to the characteristics of that environment so that the decisions academics are mostly likely will improve the quality of their teaching and learning. A particular focus should be on encouraging and enabling academics to reflect on their practice and take appropriate action.

Approaches to planning

This morning George Siemens pointed to this report (Baser and Morgan, 2008) and made particular mention of the following chart that compares assumptions between two different approaches to planning.

Comparison of assumptions in different approaches to planning (adapted from )
Aspect Traditional planning Complex adaptive systems
Source of direction Often top down with inputs from partners Depends on connections between the system agents
Objectives Clear goals and structures Emerging goals, plans and structures
Diversity Values consensus Expects tension and conflict
Role of variables Few variables determine the outcome Innumerable variables determine outcomes
Focus of attention The whole is equal to the sum of the parts The whole is different than the sum of the parts
Sense of the structure Hierarchical Interconnected web
Relationships Important and directive Determinant and empowering
Shadow system Try to ignore and weaken Accept most mental models, legitimacy and motivation for action is coming out of this source
Measures of success Efficiency and reliability are measures of value Responsiveness to the environment is the measure of value
Paradox Ignore or choose Accept and work with paradox, counter-forces and tension
View on planning Individual or system behaviour is knowable, predictable and controllable Individual and system behaviour is unknowable, unpredictable and uncontrollable
Attitude to diversity and conflict Drive for shared understanding and consensus Diverse knowledge and particular viewpoints
Leadership Strategy formulator and heroic leader Facilitative and catalytic
Nature of direction Control and direction from the top Self-organisation emerging from the bottom
Control Designed up front and then imposed from the centre Gained through adaptation and self-organisation
History Can be engineered in the present Path dependent
External interventions Direct Indirect and helps create the conditions for emergence
Vision and planning Detailed design and prediction. Needs to be explicit, clear and measurable. A few simple explicit rules and some minimum specifications. But leading to a strategy that is complex but implicit
Point of intervention Design for large, integrated interventions Where opportunities for change present themselves
Reaction to uncertainty Try to control Work with chaos
Effectiveness Defines success as closing the gap with preferred future Defines success as fit with the environment

I was always going to like this table as it encapsulates, extends and improves my long term thinking about how best to improve learning and teaching within universities. I’ve long ago accepted (Jones, 2000; Jones et al, 2005)) that universities are complex adaptive systems and that any attempt to treat them as ordered systems is doomed to failure.

I particularly liked the row on shadow systems as it corresponds with what some colleagues and I (Jones et al, 2004) suggested sometime ago.

In terms of connections with the stages of improving learning and teaching,

  1. No planning == What the teacher is.
    i.e. there is no real organisational approach to planning how to improve learning and teaching. It’s all left up to the academic.

    Often “traditional planning” proponents will refer to the complex adaptive systems approach to planning as “no planning”. Or worse they’ll raise the spectre of no control, no discipline or no governance over the compelx adaptive systems planning approach. What they are referring is actually the no planning stage. A CAS planning approach, done well, needs as much if not more discipline and “governance” as a planning approach, done well.

  2. Traditional planning == What management does.
    University management (at least in Australia) is caught in this trap of trying to manage universities as if they were ordered systems. They are creating strategic plans, management plans, embarking on analysis and then design of large scale projects and measuring success by the completion of those projects, not on what they actually do to the organisation or the quality of learning and teaching.
  3. Complex adaptive systems == What the teacher does.
    The aim is to increase the quantity and quality of the connections between agents within the university. To harness the diversity inherent in a large group of academics to develop truly innovative and appropriate improvements. To be informed by everything in the complex adaptive systems column.

Orders of change

There also seems to be connections to yet another triad described by Bartunek and Moch (1987) when they take the concept of schemata from cognitive science and apply it to organisational development. Schemata are organising frameworks or frames that are used (without thinking) to make decisions. i.e. you don’t make decisions about events alone, how you interpret them is guided by the schemata you are using. Schemata (Bartunek and Moch, 1987):

  • Help identify entities and specify relationships amongst them.
  • Act as data reduction devices as situations/entities are represented as belonging to a specific type of situation.
  • Guide people to pay attention to some aspects of the situation and to ignore others.
  • Guide how people understand or draw implications from actions or situations.

In moving from the cognition of individuals to organisations, the idea is that different organisations (and sub-parts thereof) develop organisational schemata that a sustained through myths, stories and metaphors. These organisational schemata guide how the organisation understands and responds to situations in much the same way as individual schemata. e.g. they influence what is important and what is not.

Bartunek and Moch (1987) then suggest that planned organisational change is aimed at trying to change organisational schemata. They propose that successful organisational change achieves one or more of three different orders of schematic change (Bartunek and Moch, 1987, p486):

  1. First-order change – the tacit reinforcement of present understandings.
  2. Second-order change – the conscious modification of present schemata in a particular direction.
  3. Third-order change – the training of organisational members to be aware of their present schemata and thereby more able to change these schemata as they see fit.

Hopefully, by now, you can see where the connection with the three stages of improving teaching and learning are going, i.e.

  1. First-order change == What the teacher is.
    Generally speaking how teaching is understood by the academics doesn’t change. Their existing schemata are reinforced.
  2. Second-order change == What management does.
    Management choose a new direction and then lead a project that encourages/requires teaching academics to accept the new schemata. When the next fad or the next set of management arrives, a new project is implemented and teaching academics once again have to accept a new schemata. If you’re like me, then you question whether or not the academics are actually accepting this new schemata or they are being seen to comply.

    The most obvious current example of this approach is the current growing requirements for teaching academics to have formal teaching qualifications. i.e. by completing the formal teaching qualification they will change their schemata around teaching. Again, I question (along with some significant literature) the effectiveness of this.

  3. Third-order change == What the teacher does.
    The aim here is to have an organisational environment that encourages and enables individual academics to reflect on their current schemata around teaching and be able to change it as they see problems.

    From this perspective, I see the major problem within universities not being that academics don’t have appropriate schemata to improve teaching, but that the environment within which they operate doesn’t encourage nor enable them to implement, reflect or change their schemata.

Conclusions

I think there is a need for a 3rd way to improving learning and teaching within universities. It is not something that is easy to implement. The 2nd way of improving learning and teaching is so embedded into the assumptions of government and senior management that they are not even aware of (or at best not going to mention) the limitations of their current approach or that there exists a 3rd way.

Look down the “Traditional planning” column in the table above and you can see numerous examples of entrenched, “common-sense” perspectives that have to be overcome if the 3rd way is to become possible. For example, in terms of diversity and conflict, most organisational approaches place emphasis on consensus. Everyone has to be happy and reading from the same hymn sheet, “why can’t everyone just get along?”. The requirement to have a hero leader and hierarchical organisational structures are other “common-sense” perspectives.

Perhaps the most difficult aspect of implementing a 3rd way is that there is no “template” or set process to follow. There is no existing university that has publicly stated it is following the 3rd way. Hence, there’s no-one to copy. An institution would have to be first. Something that would require courage and insight. Not to mention that any attempt to implement a 3rd way should (for me) adopt an approach to planning based on the complex adaptive systems assumptions from the above table.

References

Baser, H. and P. Morgan (2008). Capacity, Change and Performance Study Report, European Centre for Development Policy Management: 166.

Bartunek, J. and M. Moch (1987). “First-order, second-order and third-order change and organization development interventions: A cognitive approach.” The Journal of Applied Behavoral Science 23(4): 483-500.

Biggs, J. (2001). “The Reflective Institution: Assuring and Enhancing the Quality of Teaching and Learning.” Higher Education 41(3): 221-238.

Fullan, M. (2008). The six secrets of change. San Francisco, CA, John Wiley and Sons.

Jones, D. (2000). Emergent development and the virtual university. Learning’2000. Roanoke, Virginia.

Jones, D., J. Luck, et al. (2005). The teleological brake on ICTs in open and distance learning. Conference of the Open and Distance Learning Association of Australia’2005, Adelaide.

Thaler, R. and C. Sunstein (2008). Nudge: Improving decisions about health, wealth and happiness. New York, Penguin.

Understanding what teachers do: First step in improving L&T

The following is an attempt to explain the initial description and rationale of an exploratory research project (perhaps ethnographic, narrative inquiry or some similar qualitative methodology) aimed at understanding what teachers/academics actually experience within a particular environment during a single term. The assumption is that by better understanding the lived experience of the teaching staff you can better understand why (or why not) teaching is likely to improve.

In terms of suggestions and advice, I’m really keen to hear from people who might have some insights to share around:

  • methodology;
    What good methods are there to gain the type of insight I’m interested in without being to onerous for the academics involved?
  • related literature;
    Where is the literature that talks about this sort of approach within university teaching, or perhaps more broadly in education?

Why? – The personal aspect

I’m interested in this because I coming to the opinion that it is the quality, quantity and diversity of the connections within the network of people, policies, technologies and other system objects that enable or constrain the ability for a university to improve it’s teaching and learning. In particular, the connections which surround the teaching staff and students define what they experience and that experience impacts what they are likely to do (or not). My bias is that I think the network/environment surrounding most staff/students is actively preventing improvement in learning and teaching.

In my current job I am expected to help improve the quality of teaching and learning. Much of what I do (e.g. Moodle curriculum mapping, the broader alignment project, and the indicators Moodle block) is aimed at modifying the environment/network around teaching staff to enable and encourage them to improve their teaching. But this is only half the equation.

Aside: My focus is on teaching staff. It is not on students. While I agree 100% that student-centered approaches (there’s a lot of “buzz word” around that phrase, I hesitate to use it) to learning are the most effective, I don’t teach students. My job is to help other academic staff improve what they are doing, to create an environment in which they start to think that student-centered learning is not only a good thing (which many of them do) but that the environment actually helps them implement such approaches, rather than actively hinders it. I’ve seen too many attempts to encourage student-centered approaches that ignore the teaching staff and consequently get hamstrung from reactance.

The other half of the equation is getting a good understanding of the environment/network as experienced by the teaching staff. Up until now I’ve been relying on my recent experience of the same environment (which is now 3+ years old) and ad hoc discussions with colleagues (which is limited by all sorts of bias). This understanding is necessary because of the need to:

  • Be more aware of what some of the potential problems or needs are that need addressing.
  • Design the interventions to address those problems.
  • Understand the impact and ramifications of those interventions.
  • Provide evidence to others of the problems within the environment and the value of the interventions.

Why? – the research perspective

So that’s the personal perspective, what about the research perspective?

First off, one of the “buzzwords” within education fields at the moment is distributive leadership. Here’s something I wrote describing distributive leadership in the alignment project blurb

Parrish et al (2008) define distributive leadership as the distribution of power through a collegial sharing of knowledge, of practice, and reflection within a socio-cultural context. Zepke (2007) argues that this is more than the delegation of tasks and responsibilities, and more than collaborative practice. Spillane et al (2004, p. 9) argue that, based on its foundations in distributed cognition and activity theory, distributive leadership is not limited to people, but can also be attributed to artefacts such as language, notational systems, tools and buildings. Leadership activity is distributed through an interactive web of actors, artefacts and situation (Spillane et al., 2004, p. 20).

Spillane et al (2004) go onto define leadership as

the identification, acquisition, allocation, co-ordination, and use of the social, material, and cultural resources necessary to establish the conditions for the possibility of teaching and learning.

i.e. the identification of the social, material and cultural resources within an organisation is an important part of creating the conditions for teaching and learning.

Support for the importance of the environment in terms of its impact on learning outcomes comes from 30 years of empirical research by Prosser et al (2003) and Ramsden et al (2007) which has produced abundant empirical inquiry and theory that links the quality of student learning outcomes with: (1) the approaches to learning taken by students; (2) the students’ perceptions of the learning context; and (3) the approaches to teaching practiced by teaching staff. In turn, this research confirms the findings of other leadership studies by illustrating that variation in teaching approaches is associated with perceptions of the academic environment (Ramsden et al., 2007).

In terms of models of what we know about teaching and learning, the importance of the environment and context is illustrated by Trigwell’s (2001) model of teaching (click on the images to see larger versions)

Trigwell's model of teaching

And by Richardson’s (2005) model of teachers’ approaches to teaching

Integrated model of teachers' approaches to teaching

While pedagogue’s are likely to adopt teaching approaches that are consistent with their conceptions of teaching there may be differences between espoused theories and theories in use (Leveson 2004). While pedagogues may hold higher-level view of teaching other contextual factors may prevent use of those conceptions (Leveson 2004). Environmental, institutional, or other issues may impel pedagogues to teach in a way that is against their preferred approach (Samuelowicz and Bain 2001). While conceptions of teaching influence approaches to teaching, other factors such as institutional influence and the nature of students, curriculum and discipline may also influence teaching approaches (Kember and Kwan 2000). Prosser and Trigwell (1997) found that pedagogue’s with a student-focused approach were more likely to report that their departments valued teaching, that their class sizes were not too large, and that they had control over what was taught and how it was taught. Other contextual factors that frustrate pedagogues’ intended approaches to teaching may include senior staff with traditional teacher-focused conceptions raising issues about standards and curriculum coverage and students who induce teachers to adopt a more didactic approach (Richardson 2005). In addition, teachers who experience different contexts may adopt different approaches to teaching in those different contexts (Lindblom-Ylanne, Trigwell et al. 2006).

i.e. the perceptions of the environment in which they teach held by teaching staff have a direct effect on how they teach. If you want to improve the quality of teaching within a university you have to understand how the academics perceive/experience the environment.

Most of the literature I’ve seen to date by Prosser and his colleagues has been mostly survey based. I’m interested in a more detailed insight into the actual lived experience, rather than ad hoc recollections filtered through survey questions.

What

To my way of thinking this has to be a exploratory, qualitative and ethnographic investigation. I’m looking to gain insight into the day to day lived experience of academics and how they react to that experience, what it does to them. I need to read up some more. What follows are some initial thoughts.

Murthy (2008) describes how good ethnography “effectively communicates a social story, drawing the audience into the daily lives of the respondents”. This is what I’m trying to get to, I want the stories of the daily lives of the academics around learning and teaching. Murthy (2008) goes on to give an overview of digital ethnography, but nothing immediately helpful…but it seems connected to what I was thinking of doing. Hookway (2008) also looks promising but the site is down for scheduled maintenance.

How

So, without much reading, I’ve been thinking about starting this with a small exploratory study along the following lines:

  • Approach half a dozen academics from my current institution.
    Selected to be somewhat diverse in terms of likely experience in terms of location, subject etc.
  • Invite them to be co-researchers.
    I’d rather they were collaborators than research subjects. I want them to have greater ownership and motivation to be involved. I want the value of their insight not just into the everyday experience of teaching, but also research.
  • For a single teaching term, ask them to contribute to a blog stories about their experience with teaching.
    Whenever they do something around teaching, something different, something frustrating etc. write a story on the blog. As short of as long as they like. It might be a personal blog, or it might be a group blog. It might well have to be a private blog.
  • At the end of term, employ various methods to analyse the data.
  • Present it locally and publish it.

References

Biggs, J. (1996). “Enhancing teaching through constructive alignment.” Higher Education 32(3): 347-364.

Kember, D. and K.-P. Kwan (2000). “Lecturers’ approaches to teaching and their relationship to conceptions of good teaching.” Instructional Science 28(5): 469-490.

Leveson, L. (2004). “Encouraging better learning through better teaching: a study of approaches to teaching in accounting.” Accounting Education 13(4): 529-549.

Lindblom-Ylanne, S., K. Trigwell, et al. (2006). “How approaches to teaching are affected by discipline and teaching context.” Studies in Higher Education 31(3): 285-298.

Hookway, N. (2008). “‘Entering the blogosphere’: some strategies for using blogs in social research.” Qualitative Research 8(1): 91-113.

Murthy, D. (2008). “Digital Ethnography.” Sociology 32(5): 837-855.

Parrish, D., G. Lefoe, et al. (2008). The GREEN Resource: The development of leadership capacity in higher education. Wollongong, CEDIR, University of Wollongong: 64.

Prosser, M. and K. Trigwell (1997). “Relations between perceptions of the teaching environment and approaches to teaching.” British Journal of Educational Psychology 67(1): 25-35.

Prosser, M., P. Ramsden, et al. (2003). “Dissonance in experience of teaching and its relation to the quality of student learning.” Studies in Higher Education 28(1): 37-48.

Ramsden, P., M. Prosser, et al. (2007). “University teachers’ experiences of academic leadership and their approaches to teaching.” Learning and Instruction 17(2): 140-155.

Samuelowicz, K. and J. Bain (2001). “Revisiting academics’ beliefs about teaching and learning.” Higher Education 41(3): 299-325.

Spillane, J., R. Halverson, et al. (2004). “Towards a theory of leadership practice: a distributed perspective.” Journal of Curriculum Studies 36(1): 3-34.

Trigwell, K. (2001). “Judging university teaching.” The International Journal for Academic Development 6(1): 65-73.

Zepke, N. (2007). “Leadership, power and activity systems in a higher education context: will distributive leadership server in an accountability driven world?” International Journal of Leadership in Education 10(3): 301-314.

Framing a body of research and innovation

Markus has finally posted the set of questions for framing a research proposal idea that he showed me weeks ago. This post is an attempt to use those questions to frame what I’m doing in my current position.

A simple test for BIM purposes, ignore

Write a research question

How can you improve the quality of learning and teaching within a university?

What is the important theoretical or conceptual setting?

Too much work in this area has not focused on the academic staff and factors or approaches that would encourage and enable them to change their teaching behaviour. The behaviour of the teacher is a key component of the quality of learning and teaching. Teaching and learning is getting more difficult.

Write 2-3 key points about why it is important (Significance? Innovation?)

  • Governments and other external stakeholders are increasingly demanding proof that university learning and teaching is good and getting better.
  • Most university learning and teaching is of less than good quality.
  • Most, if not all, interventions to improve L&T are not succeeding in widespread, long-term behaviour change (i.e. improving quality).

Write 1-2 brief objective points (formulate full aims later)

  • Evaluation of existing interventions, informed by behaviour change research, will reveal significant short-comings.
  • Interventions, informed by behaviour change research, will result in significantly greater levels of improvement of L&T.

Write a couple of points about the approach you might take

This is not necessarily just about a single research project and it is unlikely to have, at least initially, significant institutional support or recognition. A collaborative approach involving many different projects and people will be done. However, it seems likely that an on-going process of:

  1. Learning more about how behaviour change research may inform this project.
  2. Combining these insights with other knowledge to design interventions.
  3. Use the insights to evaluate both existing and new interventions. Also to further understanding of the current context.
  4. Publish.

Will I have all the expertise/resources? Should I be talking to collaborators? Might the proposal be strengthened by having a team?

A team is absolutely necessary. Knowledge of behaviour change research is essential. As is good knowledge of the local context and educational and technical skills to implement the interventions.

Do I or will I need some ‘proof of concept’, preliminary data or demonstration of competency to undertake the project?

Yes, the on-going process above will have to start small and learn from there.

Will the team/partners be competitive in this field?

At the moment, yes. There doesn’t appear to be too many people taking this approach.

What would the outcome(s) be and who benefits?

Hopefully, there will be improvements in L&T in the local context and research publications and grants around that work.

What is the ‘WOW’, ‘HOOK’, or ‘EXCITEMENT’ factor?

It appears that this is a new way of approaching an intractable problem.

Further thinking – behaviour change and improving L&T

This post is an attempt to synthesize and reflect more upon two posts from yesterday. One from me trying to explain an early form of a framework for improving L&T based on findings from psychology behavior change research. The second from Markus that expands on the potential psychological foundations for this thinking by mentioning:

  • The recommendations section of a UK National Health Service (NHS) document outlining some generic principles for initiatives to support attitude and behaviour change.
  • The concept of salutogensis, at least as a cliff hanger for a subsequent post.

The following is my attempt to summarise and make some initial connections between this work and ideas around improving learning and teaching.

What behaviour change programs need to do

The NHS document is said to highlight the need for “interventions and programmes aimed at changing behaviour” to:

  • Be based on a sound knowledge of community needs and should build upon the existing skills and resources within a community.
    To me, for improving and learning and teaching this speaks of a need for empathy-driven innovation and for a focus on what the teacher does.
  • Equip practitioners with the necessary competencies and skills to support behaviour change, using evidence-based tools.
    My understanding is that “practitioners” refer to the people helping the change, not the actual teachers. Interesting.
  • Evaluate all interventions.
    Something not really being done around L&T within universities.

These are derived somewhat from the principles for interventions presented. It is again the importance placed on understanding the community as the foundation for change that resonates for me. You have to start with knowledge of the community and their strengths, perceptions etc.

Salutogensis

The following is based on a quick reading of this paper.

Focus on origins of health, rather than causes of disease

Interesting thought, salutogensis arose from an attempt to understand why women who survived concentration camps in the Second World War, had the capability for good health.

Is there a connection hear with L&T?

I would perhaps characterise the current focus in Australian universities on changes driven by the need to respond to government as attempts to prevent disease, rather than encourage health. …tentative and tenuous.

The human is at the heart

Discussions this morning lead to some reflection on metaphor – an area in which my wife has done work. The idea is that the metaphors we use to describe complex concepts reveal how we think about those concepts, it informs our decisions.

Increasingly, some colleagues and I are seeing evidence of senior management holding a machine metaphor when it comes to learning and teaching. The academic staff involved in learning and teaching are seen as machines, as inter-changeable machines. You can take that machine from there, and place it over there and it will do the same job. It doesn’t matter which machine, they all do the same thing.

It is this metaphor which enables the adoption of techo-rational approaches to improving learning and teaching such as the idea that every academic should have a graduate certificate. The metaphor here is that if you add a new widget to the machine it will perform the task with greater efficiency or with greater effectiveness.

The trouble is that people aren’t machines. They don’t passively accept the addition of a widget. There’s no direct causal link between addition of a widget and changes in behaviour.

Lundstrom and Eriksson (2006) see health promotion as

three phases, the first one that recognizes the background (the determinants), the second one that sets an objective (to lead an active productive life), and the third one is the activity (the enabling process) where the determinants are used to reach the objective in a dialectic relationship between people, the setting and the enablers.

As a “dialectic relationship” it is an on-going dialog, the person doesn’t take it unquestioningly. They go on to say

At the heart there is the human being seen as an active participating subject, respected in her full human rights.

Something that is increasingly missing in L&T within universities.

The salutogenesis framework

A framework might be possible by drawing on the salutogenic focus on three aspects

  1. Focus on problem solving/finding solutions.
  2. Identifying GRRs that help people move in the right direction.
    GRR = General Resistance Resources – biological, material and pyschosocial factors that help people see their lives as consistent, structured and understandable. A person with these resources has a better chance of dealing with challenges.
  3. Identify the SOC – the pervasive sense that enables this move.
    The SOC is the ability to use the GRRs. The GRRs create life experiences that promote a strong SOC. (This is somewhat confusing but the message I’m getting from the paper.) The SOC is the capabaility to perceived you can manage in any situation. It is flexible and not focused on a specific set of strategies.

Will have to wait for Markus’ explanation to make this understandable to me.

There are, however, some positive resonances with this ideas and my current thoughts, will have to see how that plays out.

Identifying and designing interventions to improve L&T – a behaviour change framework?

The following is an attempt to describe early attempts at developing a framework for evaluating existing and developing new interventions aimed at improving learning and teaching within universities. It builds on some ideas from an earlier post.

Sorry, but the first couple of sections seem necessary to convince myself that this is connected with some of my earlier thinking.

Rationale and assumptions

As mentioned many times on this blog, I think most attempts to improve learning and teaching at Universities really don’t work as they don’t change what the majority of academics do. At best they get a bit of compliance. The aim here is to come up with a framework/guidelines that help evaluate and design such interventions.

Empathy driven innovation

While the following uses a framework from psychology, the basic message is the same as a range of other work. That message is, focus on the academics, understand what they are currently doing and experiencing, and design interventions that connect with that experience. Most recently this was discussed in a a post on empathy-driven innovation.

Much of my thinking around improving learning and teaching is based on this sort of approach, the following tries and provide some additional guidance into how to do this. But underpinning it all, is a focus on the actual teaching experience of the academic staff (and the learning experience of the students) as the primary focus.

The aim is to understand what they are experiencing and develop ideas about how that experience can be improved, as perceived by the staff and students and not as perceived by management, the government or some research ideal. The focus is on what the teacher does, not what management does.

Improving L&T is an attempt to change behaviour

An assumption underpinning this thinking is that when you are trying to improve learning and teaching, what you are actually trying to do is to change the behaviour of the teaching academics. If you don’t change the behaviour of the academics, then there is no way you can expect the quality of learning and teaching to change.

This is perhaps the main problem that I have with existing methods being employed to improve learning and teaching at Universities. Existing methods don’t actually consistently result in behaviour change amongst a significant percentage of university teaching staff. I’ve argued before that they don’t work and that there is evidence to support this.

The question is, what’s the solution?

Conceptions and teaching

An example of this mismatch can be seen with the on-going focus on teacher qualifications. For example, the Australian government has introduced KPIs for university learning and teaching that include the use of the % of university teaching staff with teaching qualifications as a KPI. I’ve argued this this attempt will encourage compliance behaviour on the part of institutions and academics.

The institutions are already at it. My current institution is currently introducing a new “desirable criteria” for people in charge of courses – having a formal teaching qualification. The assumption is that this will increase the quality of teaching. It won’t.

At best, a formal teaching qualification will change the conceptions of teaching held by an academic. The “teacher’s thinking” layer in the following diagram. There is significant research (e.g. Trigwell, 2001; Richardson, 2005) that suggests that unless all of the layers in the diagram are in alignment, you won’t get improvement in teaching.

Trigwell's model of teaching

That is, even with a formal teaching qualification, if the context, disciplinary expectations and many other factors aren’t right, then there will be no behaviour change. The teacher will continue to employ the same strategies as always.

How to encourage and enable behaviour change

How to change behaviour is something that the psychologists study. This current line of work is a collaboration between myself and a psychologist colleague. The current aim is to try and distill what is known about behaviour change in psychology into a form that helps explain how someone – with a deep knowledge of the current experience of academics – can encourage and enable behaviour change that results in improved learning and teaching.

In earlier work I’ve drawn on insights from Roger’s (1995) diffusion theory as a guide. There are definite connections between diffusion theory and the following.

The initial spark for this thinking is a paper by Michie et al (2008) which suggests that:

  • There are determinants of behaviour change, (see the table below) someone is unlikely to change behaviour when these determinants have the wrong value.
  • There are known techniques for encouraging behaviour change.
  • These techniques are each seen as likely to influence a certain subset of the determinants.
Key Determinants of Behaviour Change from Fishbein et al., 2001; Michie et al., 2008
Fishbein et al Michie et al
Self-standards Social/professional role and identity
Knowledge
Skills Skills
Self-efficacy Beliefs about capabilities
Anticipated outcomes/attitude Beliefs about consequences
Intention Motivation and goals
Memory, attention and decision processes
Environmental constraints Environmental context and resources
Norms Social influences
Emotion
Action planning

From a L&T perspective, the application of this knowledge seems to be something like:

  • Identify something that is known to be good practice, that improves L&T, but which isn’t being done in the current context.
  • Use the determinants to identify areas of weakness, areas for improvement, around the current context. Especially in terms of the “good” practice.
  • Identify generic behaviour change techniques that address those areas of weakness.
  • Draw on empathy driven innovation to develop specific interventions, which are based on the information gathered in the previous two steps.

An example

In a previous post that seeks to summarise the Michie et al (2008) paper, I use the following example of the level of interaction or participation of staff in a course site. In the following, I compare/contrast the more traditional approach and the type of approach that I think might work better.

The traditional approach

The more traditional approach is what I suggests is “what management does”. Management will see that there is a problem, i.e. staff aren’t participating enough. They then take it upon themselves to perform some action, which might include:

  • Require staff to get a teaching qualification.
    The assumption being that with a teaching qualification they know more about the importance of staff interaction and hence will increase their interaction.
  • Create a policy that requires a certain level of participation.
    At the very least they might say that every course site must have a discussion forum. Over time they might add requirements like: every student query must be answered within two days, or every staff member must make 5 posts a week to the course discussion forum.
  • Run a staff development session.
    Organise an external expert on the benefits of staff participation to run a session at the institution and invite teaching staff to come along to listen.
  • Reward staff who interact at appropriate levels.
    Give them extra money or recognition as good teachers. etc.

Most of the above interventions would require a fair bit of work. Most, if not all, of the senior management at the institution would need to be involved, a working party might need to be formed, appropriate consultation, and if they’re really smart they’ll make sure that someone important within the institution is publicly seen encouraging this goal.

I’ve seen the above approach used again and again. I don’t think I’ve ever seen it work to any great extent.

The behavioural change/empathy-driven approach

Using the process above:

  • Identify the good practice.
    Done, we wish to increase the level of staff participation in a course site/forum.
  • Use the determinants to identify areas of weakness in the current context.
    As an example, let’s assume that after an appropriate process we have identified the following:
    • Skills: many staff don’t have the skills or insight necessary to increase their interaction with their course in effective ways.
    • Environmental constraints: there’s nothing in the existing context that helps them perform this task. In fact, the rewards system tends to suggest that research is more important than teaching.
    • Norms: there is little visible evidence of what the norms are in terms of appropriate levels of staff participation. Staff don’t know how much other staff are participating, they don’t know what is acceptable and what is not.
    • Anticipated outcomes: many staff don’t see the connection between increasing their participation in a course site and benefits to students. They do see how increasing their level of participation increases the time they spend on teaching and decreases the time they spend on research.
  • Identify generic behaviour change techniques that target these determinants.
    Techniques that target skills include: monitoring; self-monitoring; graded task, starting with easy tasks; modelling/demonstration of behaviour by others; goal/target specified: behaviour or outcome. Norms can be addressed by: modelling/demonstration of behaviour by others; social processes of encouragement, pressure, support; and prompts, triggers, cues. etc.
  • Use empathy-driven innovation to design specific interventions.
    Embed into the LMS a graph that shows the academic staff members level of participation in their course site (self-monitoring) and compare it against the levels of participation in a group of related courses (social processes of encouragement). Have that graph show up whenever a staff member logs into the LMS. Include in the graph some easily visible representation of the connection between student failure rate and staff participation. Include a link “how to increase participation” to a page that outlines various techniques for increasing participation (prompts, triggers, cues) and includes comments or suggestions from other staff about how they did it (modeling/demonstration of behaviour by others).

    Once this intervention is implemented, spend a lot of time observing what happens with the use of this intervention and make on-going change.s

Differences

Apart from the obvious, there is one major difference between these two approaches which I must make explicit. The traditional method spends a lot more time on the design of idealistic solutions that are disconnected from the everyday experience (e.g. the setting up and encouraging of people to enrol in a graduate certificate in higher education). Often, such approaches all but ignore the reality of the lived experience (e.g. policies that state a requirement but for which there’s been no work to support implementation). The traditional approach tends to be design heavy.

The empathy-driven approach, focuses on the lived experience, on what happens day-to-day. There is some design work, but much of the time is spent understanding what is going on everyday and trying to respond in an informed way.

The questions

It is still very early days for this work. There are lots of questions, here are just some.

How do you evaluate the level of determinants?

In the above example, I outlined some of my ideas of what values the behavioural change determinants around increasing staff participation on a course site might be. Basing the design on my thoughts is not a good start. It has to be based on a deep understanding of the experience/beliefs of the teaching staff.

How can you develop this understanding?

Are there survey instruments or other approaches from psychology that allow you to get some understanding of the current “level” of these determinants across an institution?

Which are the most important determinants?

Michie et al (2008) identify 7 sets of determinants, there are certain to be complexities within each one. In any large population you are likely to get widely differing results.

How do you figure out which of the determinants is most important or difficult?

Answers to this could help you guide the selection of which determinants to address and/or which of the techniques to adopt.

What’s the relationship between the determinants?

It’s likely that there would be relationships between the determinants. For example, if you increase skills in an appropriate way, I would imagine that this would change beliefs about self-efficacy, at least to some level.

What are the known relationships between the determinants?

What are the factors within determinants?

Fishbein et al (2001) talk about norms as being one of the determinants. Michie (et al) 2008 expand this out to include social influences, emotion and action planning. This is just one example, it is likely that each of the determinants embody a collection of factors. For example, self-efficacy might be a combination of characteristics of the person, the type of task and the previous experience of those involved.

What are the factors or sub-components of each of the determinants?

What types of behaviour are there?

Are all behaviours the same? Is a particular teaching behaviour the same as diet or how you control your children?

Are there different categorisations of behaviours with different characteristics?

What are the specifics of change techniques?

Michie et al (2008) just give a short title to these techniques. I imagine that each one is worthy of a literature that seeks to define, describe and evaluate interventions that have been designed on the basis of a specific technique.

What are the specifics of each change technique? What are the success factors? How do you choose which technique might be most appropriate?

References

Michie, S., Johnston, M., Francis, J., Hardeman, W., & Eccles, M. (2008). From theory to intervention: Mapping theoretically derived behavioural determinants to behaviour change techniques. Applied Psychology: An International Review, 57(4), 660-680.

Rogers, E. (1995). Diffusion of Innovations (4th ed.). New York: The Free Press.

From theory to intervention: Mapping theoretically derived behavioural determinants to behaviour change techniques

The following draws on principles/theory from psychology to guide thinking about how to incorporate “data” from “academic analytics” into an LMS in a way that encourages and enables academic staff to improve their learning and teaching. It’s based on some of the ideas that underpin similar approaches that have been used for students such as this Moodle dashboard and the signals work at University of Purdue.

The following gives some background to this approach, summarises a paper from the psychology literature around behaviour modification and then explains one idea for a “signals” like application for academic staff. Some of this thinking is also informing the “Moodle curriculum mapping” project.

Very interested in pointers to similar work, suggestions for improvement or expressions of interest from folk.

Background

I have a growing interest in how insights from psychology, especially around behaviour change can inform the design of e-learning and other aspects of the teaching environment at universities in a way to encourage and enable improvement.
Important: I did not say “require”, I said “encourage”. Too much of what passes in universities at the moment takes the “require” approach with obvious negative consequences.

This is where my current interest in “nudging” – the design of good choice architecture and behaviour modification is coming from. The basic aim is to redesign the environment within which teaching occurs in a way the encourages and enables improvement in teaching practice, rather than discourages it.

To aid in this work, I’ve been lucky enough to become friends with a pyschologist who has some similar interests. We’re currently talking about different possibilities, informed by our different backgrounds. As part of that he’s pointing me to bits in the psychological literature that offers some insight. This is an attempt to summarise/reflect on one such paper (Michie et al, 2008)

Theory to intervention

It appears that the basic aim of the paper is to

  • Develop methods to clarify the list of behaviour change techniques.
  • Identify links between the behaviour change techniques and behavioural determinants.

First a comparison of two attempts at simplifying the key behavioural determinants for change – the following table. My understanding is that there are some values of these determinants that would encourage behaviour change, and others that would not.

Key Determinants of Behaviour Change from Fishbein et al., 2001; Michie et al., 2004
Fishbein et al Michie et al
Self-standards Social/professional role and identity
Knowledge
Skills Skills
Self-efficacy Beliefs about capabilities
Anticipated outcomes/attitude Beliefs about consequences
Intention Motivation and goals
Memory, attention and decision processes
Environmental constraints Environmental context and resources
Norms Social influences
Emotion
Action planning

It is interesting to see how well the categories listed in this table resonate with the limits I was planning to talk about in this presentation. i.e. it really seems to me, at the moment, that much of the environment within universities around teaching and learning is designed as to reduce the chance of these determinants to be leaning towards behaviour change.

Mapping techniques to determinants

They use a group of experts in a consensus process for linking behaviour change techniques with determinants of behaviour. The “Their mapping” section below gives a summary of the consensus links. The smaller headings are the determinants of behaviour from the above table, the bullet points are the behaviour change techniques.

Now, I haven’t gone looking for more detail on the techniques. The following is going to be based solely on my assumptions about what those techniques might entail – and hence it will be limited. However, this should be sufficient for the goal of identifying changes in the LMS environment that might encourage change in behaviour around teaching.

First, let’s identify some of the prevalent techniques, i.e. those that are mentioned a more than once and which might be useful/easy within teaching.

Prevalent techniques

Social encouragement, pressure and support

The technique “Social processes of encouragement, pressure, support ” is linked to 4 of the 11 determinants: Social/professional role and identity, beliefs about capabilities, motivation and goals and social influences. I find this interesting as it can be suggested that most teaching is a lone and invisible act. Especially in a LMS where what’s going on in other courses. Making what happens more visible might enable this sort of social process.

There’s also some potential connection with “Information regarding behaviour of others” which is mentioned in 3 of 11.

Monitoring and self-monitoring

Get mentioned as linked to 4 of 11 determinants. Again, most LMS don’t appear to give good overall information about what a teacher is doing in a way that would enable monitoring/self-monitoring.

Related to this is “goal/target specified”, part of monitoring.

There’s more to do here, let’s get onto a suggestion

One suggestion

There’s a basic model process embedded here, something along the lines of:

  • Take a knowledge of what is “good” teaching and learning
    For example, Fresen (2007) argues that the level of interaction, facilitation or simply participation by academic staff is a critical success factor for e-learning. There’s a bunch more literature that backs this up. And our own research/analysis has backed this up. Courses with high staff participation show much higher student participation and a clearer correlation between student participation and grade (i.e. more student participation, the higher the grade).
  • Identify a negative/insight into the behavioural determinants that affect academic staff around this issue.
    There are a couple. First, it’s not uncommon for staff to have an information distribution conception of teaching. i.e. they see their job as to disseminate information. Not to talk, to communicate, or participate. Associated with this is that most staff have no idea what other staff are doing within their course sites. They don’t know how often other staff are contribution to the discussion forum or visiting the course site.
  • Draw on a behavioural technique or two to design an intervention in the LMS that can encourage a behaviour change. i.e. that addresses the negative in the determinants.
    In terms of increasing staff participation you might embed into the LMS a graph like the following. Embed it in such a way as the first thing an academic sees when they login, is the graph – perhaps on part of the screen.

    Example staff posts feedback

    What this graph shows is for a single (hypothetical) staff member the number of replies they have made in course discussion forums for the three courses the staff member has taught. The number of replies is shown per term, in reality it might be shown by week of term – as the term progresses.

    This part can hit the “monitoring”, “self-monitoring” and “feedback” techniques.

    The extra, straight line represents the average number of replies made by staff in all courses in the LMS. Or alternatively, all courses in a program/degree into which the staff member teaches. (Realistically, the average would probably change from term to term).

    This aspect hits the “social processes of encouragement, pressure, support”, “modelling/demonstration behaviour of others”. By showing what other people are doing it is starting to create a social norm. One that might perhaps encourage the academic, if they are below the average, to increase their level of replies.

    But the point is not to stop here. Showing a graph like this is simple using business intelligence tools and is only a small part of the change necessary.

    It’s now necessary to hit techniques such as “graded task, starting with easy tasks”, “Increasing skills: problem-solving, decision-making, goal-setting”, “Planning, implementation”, “Prompts, triggers, cues”. It’s not enough to show that there is a problem, you have to help the academic with how to address the problem.

    In this case, there might be links associated with this graph that show advice on how to increase replies or staff participation (e.g. advice to post a summary of the week’s happenings in a course each week, or some other specific, context appropriate advice). Or it might also provide links to further, more detailed information to shed more light on this problem. For example, it might link to SNAPP to show disconnections.

    But it’s even more than this. If you wanted to hit the “Environmental changes (e.g. objects to facilitate behaviour)” technique you may want to go further with than simply showing techniques. You may want to enable this “showing of techniques” to be within a broader community where people could comment on whether or not a technique worked. It would be useful if the tool help automate/scaffold the performance of the task, i.e. moved up the abstraction layer from the basic LMS functionality. Or perhaps the tool and associated process could track and create “before and afters”. i.e. when someone tries a technique, store the graph before it is applied and then capture it at sometime after.

It’s fairly easy to see how the waterfall visualisation (shown below) and developed by David Wiley and his group could be used this way.

education,data,visualization

Their mapping

Social/professional role and identity

  • Social processes of encouragement, pressure, support

Knowledge

  • Information regarding behaviour by others

Skills

  • Goal/target specified: behaviour or outcome
  • Monitoring
  • Self-monitoring
  • Rewards; incentives (inc. self-evaluation)
  • Graded task, starting with easy tasks
  • Increasing skills: problem-solving, decision-making, goal-setting
  • Rehearsal of relevant skills
  • Modelling/demonstration of behaviour by others
  • Homework
  • Perform behaviour in different settings

Beliefs about capabilities

  • Self-monitoring
  • Graded task, starting with easy tasks
  • Increasing skills: problem-solving, decision-making, goal-setting
  • Coping skills
  • Rehearsal of relevant skills
  • Social processes of encouragement, pressure and support
  • Feedback
  • Self talk
  • Motivational interviewing

Beliefs about consequences

  • Self-monitoring
  • Persuasive communication
  • Information regarding behaviour, outcome
  • Feedback

Motivation and goals

  • Goal/target specified: behaviour or outcome
  • Contract
  • Rewards; incentives (inc. self-evaluation )
  • Graded task, starting with easy tasks.
  • Increasing skills: problem-solving, decision-making, goal-setting
  • Social processes of encouragement, pressure, support
  • Persuasive communication
  • Information regarding behaviour, outcome
  • Motivational interviewing

Memory, attention, decision processes

  • Self-monitoring
  • Planning, implementation
  • Prompts, triggers, cues

Environmental context and resources

  • Environmental changes (e.g. objects to facilitate behaviour)

Social influences

  • Social processes of encouragement, pressure, support
  • Modelling/demonstration of behaviour by others

Emotion

  • Stress management
  • Coping skills

Action planning

  • Goal/target specified: behaviour or outcome
  • Contract
  • Planning, implementation
  • Prompts, triggers, cues
  • Use of imagery

References

Fresen, J. (2007). “A taxonomy of factors to promote quality web-supported learning.” International Journal on E-Learning 6(3): 351-362.

Michie, S., M. Johnston, et al. (2008). “From theory to intervention: Mapping theoretically derived behavioural determinants to behaviour change techniques.” Applied Psychology: An International Review 57(4): 660-680.

Implications of cognitive theory for instructional design

The following is a summary/reflection of Winn (1990), the abstract follows

This article examines some of the implications of recent developments in cognitive theory for instuctional design. It is argued that behavioral theory is inadequate to prescribe instructional strategies that teach for understanding. Examples of how instructional designers have adopted relevant aspects of cognitive theory are described. However, it is argued that such adoption is only a first step. The growing body of evidence for the indeterminism of human cognition requires even further changes in how instructional designers think and act. A number of bodies of scholarly research and opinion are cited in support of this claim. Three implications of cognitive theory for design are offered: instructional strategies need to be developed to counter the reductionism implicit in task analysis; design needs to be integrated into the implementation of instruction; designers should work from a thorough knowledge of theory not just from design procedures.

Summary

Suggests problems arise when decisions within instructional design are driven by cognitive theory, not behavioural. Mostly around the assumptions of rationality and predictability and the subsequent appropriateness of the traditional teleological design process used by instructional design. Suggests some approaches/implications that might help address these somewhat.

Reflection

The ideas expressed here offer support for the ideas I’ve been formulating about how to improve learning and teaching at Universities. Which obviously means I think it is an important bit of work by an intelligent person. It probably does have flaws. Will need to read and reflect more.

Still not sure that these principles have been applied broadly enough (though the conclusion seems to indicate yes). Winn has focused on changes to the practice of instructional designers in how they approach design without talking about how they may have to change how they work with the academics. Instructional design, for me, is as much about staff development as it is about design, at least within the current university context. Instructional design within universities can’t scale unless it builds capacity amongst the academic staff and the system to help in design.

Many of these limitations of instructional design are similar to those I’ve been trying to push around the institutional implementation of e-learning and more generally about approaches to improve learning and teaching e.g. graduate attributes.

Introduction

Starts with a definition of instructional design from Reigeluth (1993) – essentially it is a set of decision-marking procedures which, given the outcomes to be achieved and the conditions under which they are to achieve them, develops the most effective instructional strategies.

Generally done with analysis of outcomes/conditions, selection of strategies, iterative testing until some level of success achieved. The decisions are guided by instructional theory.

Gives examples of instructional design processes informed by cognitive theory.

Suggests evidence that cognitive theory is impacting thinking/actions of instructional designers, however, suggests that cognitive theory requires further changes in the way they think/act. Has problems with the analysis and selection/testing stages. Current approaches are not sufficient.

Suggests that instructional design should be driven by an understanding of theories of learning and instruction, rather than mastery of design techniques.

I’m assuming here he means that the type and nature of the steps within design process itself should be informed by these, not what he also recognises is that the decisions made within these steps are already driven by this. I’m a bit slow this morning.

Instructional design and behavioural theory

Supports/explains the notion that instructional design originated in behavioural theory, the dominant learning theory of the time when ID originated. Shows how instructional design processes evolved to fit the needs of behavioural theory. Examples include the reductionist nature of task analysis and pilot testing being sufficient to debug instruction that consisted of stimulus-response prescriptions. i.e. behavourists did not consider that there were “mental operations” within the learner that might mediate between stimulus and response. This resulted in design being separated from implementation.

If instruction can be developed to the point where acceptable student performance is likely to occur, then it matters little whether instruction is implemented immediately after the designer has attained this standard, or at some later time and in some other place.

Connects with literature that acknowledges the separation (Richey, 1986), thinks it creates problems (Nunan, 1983, Streibel, 1989, Winn, 1989) and others which think it desirable (Heinich, 1970 and 1984). Desirable because “it allows instruction to be brought up to a high standard, and then to be distributed far and wide so that all students can enjoy the advantages of top-rate teaching”.

Lastly, suggests the idea that instructional design can be “done by the numbers” also arises from the behavioural tradition. The idea is that any novice designer can be successful if they just follow the process – do it by the numbers.

In summary, 3 important areas where behaviourism still exerts power over instructional design:

  1. Reductionist premise that you can identify the parts, then you can teach the whole.
  2. Separate design from implementation.
  3. Assumption that following good procedures, applied correctly results in good instruction.

Sticking with the behavioural traditions, suggests that these 3 are not a problem, if you’re limiting yourself to low-level skills. The problems arise when you go to high levels of cognitive processing.

Cognitive theory

The aim here is to explain why the three assumptions are problematic as informed by cognitive theory – the obvious though here is what would constructivism or connectivism suggest.

The description of cognitive theory is

Changes in behavior are seen as indirect rather than direct outcomes of learning. Observable behavior is mediated and controlled by such mental activities as the acquisition, organization and application of knowledge about the world (Neisser, 1976; Rumelhart and Norman, 1981); by the development of skills that allow the encoding, storing and relrieval of information (E. Gagne, 1985; Shuell, 1986); by people’s motivation (Keller, 1983); their perception of what a task requires of them (Salomon, 1983a); and their perception of their likelihood of success (Salomon, 1983b; Schunk, 1984). Consequently, students are seen as active in the construction of knowledge and the development of skills, leading to the conclusion that learning is a generative process under the control of the learner (Wittrock, 1979, 1982).

To my somewhat untrained ear, this sounds like it has aspects of constructivism.

References Bonner (1988) as identifying a number of the differences between traditional designers and those informed by cognitive theory including:

  • task analysis;
    Traditionally aims to identify directly observed behaviours. Cognitive theory requires that “unobservable” tasks be analysed. i.e. the mental tasks to be mastered prior to observable performance being possible. examples including identifying declarative and procedural knowledge or schemata required to perform. Also recognition that novice to expert involves many steps that need to mastered.
  • objectives;
    Statements of what the student is to accomplish under what conditions and to what criterion is a behaviourist approach. Cognitive objectives are schematic representations of the knowledge to be acquired and procedures to apply.
  • learner characteristics;
    Focus on the schemata/mental models students bring to instruction, not their behaviours. May not be a clear line between what they need to know and what they know – learner as dirty slate.

    This acknowledges the importance of current knowledge of the world, represented in mental models, for the acquisition of new knowledge and skills. Research (De Kleer and Brown, 1981; Larkin, 1985; and authors contributing to Gentner and Stevens, 1983) has shown that learning occurs as students’ mental models acquire refinement and accuracy.

  • instructional strategies;
    Behaviourism selected instructional strategies based on the type of learning to take place, the type of learning outcome.

    But because the cognitive conception of learning places so much importance on the student’s development of adequate knowledge structures, cognitive procedures and mental models, the designer should create opportunities for what Bonner calls a “cognitive apprenticeship” centered around problem-solving rather than prescribe strategies apriori.

    Some general principle may determine aspects of the strategy, however, it evolves like a conversation.

Rieber (1987) point out, is that instruction that is designed from cognitive principles can lead to understanding rather than just to memorization and skill performance.

This speaks to me because too much of what passes for improving learning and teaching strikes me as most likely to create memorisation and skill performance, not long term change.

The need for further change

While instructional designers are adopting principles from cognitive theory, the idea is that recent thinking in cognitive psychology and related fields brings into question some the assumptions of cognitive theory as currently accepted. Moving onto the reasons:

  • metacognition;
    Metacognition research shows that students have or can be trained to acquire the ability to reflect on their performance and adopt/adapt different learning strategies. This means that the intent of a instructional design can be circumvented if the student finds the chosen strategy problematic. If the instruction is not adaptable or the student doesn’t choose a good strategy, then the instructional design is compromised.

    I wonder what implications this has for constructive alignment and its idea of forcing the student to do the right thing?

  • dynamic nature of learning;
    Very interesting. As the learner learns, they develop knowledge and skill that is different from the start. The analysis performed at the start to select the instructional strategy no longer holds. If the analysis was done now, a different strategy would be required.

    Nunan (1983) develops this line of reasoning in his argument against the value of instructional design, drawing on arguments against the separation of thought from action put forward by Oakshotte (1962) and Polanyi (1958).

  • emergent properties of cognition;
    This is the argument against reductionism. Emergence is defined as the idea where the properties of the whole, cannot be explained solely be examining the individual parts of the whole. The nature of the whole affects the way elements within them behave. The suggestion is that a number of people have claimed that the actions of the human mind exhibit emergent properties (Churchland, 1988; Gardner, 1985).

    The reductionism that underpins task and learner analysis “acts counter to, or at best ignores, a significant aspect of human cognition, which is the creation of something entirely new and unexpected from the “raw material” that has to be learned.

  • plausible reasoning;
    A designer informed by cognitive theory assumes that the thought processes of a student will be as logical as the instruction itself. In order to learn from a machine, the student has to think like a machine (Streibel, 1986). There is lots of evidence to suggest people are not logical. “Plausible reasoning” is Collins (1978) idea that people proceed on hunches and incomplete information. Hunt (1982) suggests plausible reasoning has allowed the human species to survive.

    If we waited for complete sets of data before making decisions, we would never make any and would not have evolved into intelligent beings.

  • situated cognition; and
    Somewhat related to previous. Streibel (1989) argues that “cognitive science can never form the basis for instructional design because it proceeds from the assumption that human reasoning is planful and logical when in fact it is not”. References Brown, Collins and Duguid (1989); Lave (1988) and Suchman (1987) – i.e. situated cognition folk – to argue that the way we solve problems is dependent on the situation in which the problem occurs. We do not use formal or mathematical reasoning.
  • unpredictability of human behaviour.
    The 5 previous points suggest that human behaviour is indeterminate. Csiko (1989) gives 5 types of evidence to argue that the unpredictability and indeterminism of human behaviour is central to the debate concerning epistemology of educational research. Winn (1990) suggests it applies equally well to instructional design:
    1. Individual learner differences interact in complex ways with treatments which make prediction of performance difficult.
    2. Chaos theory suggests the smallest changes in initial states lead to wild and totally unpredictable fluctuations in a systems behaviour. Something that is more pronounced in complex cognitive systems.
    3. Much learning is “evolutionary” in that it arises from chance responses to novel stimuli
    4. Humans have free will which can be exercised and subsequently invalidate any predictions about behaviours made deterministically from data.
    5. Quantum mechanics shows that observing a phenomenon, changes that phenomenon so that the results of observations are probabilities, not certainties.

Though eclectic, this body of argument leads one to question seriously both the assumption of the validity of instructional prescriptions and the assumption that what works for some students will work for others.

While prediction may not be part of instructional design, it is of the theories it depends upon and Reigelluth (183) points out that any theory of instruction, while not deterministic, does rely on the probability that prescriptions made form it for its validity. Without such validity, you may as well rely on trial and error.

Conclusions

Cognitive theory has been incorporated into instructional design, but behaviourism influence remains and that causes problems.

Cognitive task analysis to develop objectives is just as reductionist as behaviourist approaches.. The whole approach designers take needs to be re-examined. Three directions might include:

  1. Analysis and synthesis;
    Addressing reductionist analysis – instructional strategies need to ensure knowledge/skill components are put back together in meaningful ways….e.g. Reigeluth and Stein’s (1983) use of “summarisers” and “synthesizes” in elaboration theory.

    Balance analysis as a design procedure with synthesis as an instructional strategy. Such prescriptions should exist in instructional theories.

  2. Design and implementation;

    For instruction to be successful, it must therefore constantly monitor and adapt to unpredicted changes in student behavior and thinking as instruction proceeds……To succeed, then, instructional decisions need to be made while instruction is under way and need to be based on complete theories that allow the generation of prescriptions rather than on predetermined sets of prescriptions chosen ahead of time by a designer. (p64)

    Requires the teacher to monitor and modify strategies as they’ve been prescribed. Requires teachers to be well schooled in instructional design and a solid knowledge of theories of learning and instructions – so that they can respond in some sort of informed way. e.g. need methods that will allow them to invent prescriptive principles when the need arises.

    Second recommendation is that the designer needs to monitor the actual use of the instructional system during implementation, or for the designer to make provision for the use to change instruction strategies.

  3. Theory and procedure.
    Decisions about instructional strategies need to be based on more than just the application of design procedures. Rather than techniques being taught, the principles should be.

    This problem is made worse by researchers who are content to identify strategies that work on single occasions rather than determine the necessary conditions for their success (Clark, 1983).

Page 1 of 3

Powered by WordPress & Theme by Anders Norén

css.php