Assembling the heterogeneous elements for (digital) learning

Category: missingPs Page 1 of 3

The life and death of Webfuse: What's wrong with industrial e-learning and how to fix it

The following is a collection of presentation resources (i.e. the slides) for an ASCILITE’2012 of this paper. The paper and presentation are a summary of the outcomes my PhD work. The thesis goes into much more detail.

Abstract

Drawing on the 14-year life and death of an integrated online learning environment used by tens of thousands of people, this paper argues that many of the principles and practices underpinning industrial e-learning – the current dominant institutional model – are inappropriate. The paper illustrates how industrial e-learning can limit outcomes of tertiary e-learning and limits the abilities of universities to respond to uncertainty and effectively explore the future of learning. It limits their ability to learn. The paper proposes one alternate set of successfully implemented principles and practices as being more appropriate for institutions seeking to learn for the future and lead in a climate of change.

Slides

The slides are available on Slideshare and should show up below. These slides are the extended version, prior to the cutting required to fit within the 20 minute time limit.

References

Arnott, D. (2006). Cognitive biases and decision support systems development: a design science approach. Information Systems Journal, 16, 55–78.

Behrens, S., Jamieson, K., Jones, D., & Cranston, M. (2005). Predicting system success using the Technology Acceptance Model: A case study. 16th Australasian Conference on Information Systems. Sydney.

Brews, P., & Hunt, M. (1999). Learning to plan and planning to learn: Resolving the planning school/learning school debate. Strategic Management, 20(10), 889–913.

Cecez-Kecmanovic, D., Janson, M., & Brown, A. (2002). The rationality framework for a critical study of information systems. Journal of Information Technology, 17, 215–227.

Central Queensland University. (2004). Faculty teaching and learning report. Rockhampton, Australia.

Davenport, T. (1998). Putting the Enterprise into the Enterprise System. Harvard Business Review, 76(4), 121–131.

Dede, C. (2008). Theoretical perspectives influencing the use of information technology in teaching and learning. In J. Voogt & G. Knezek (Eds.), (pp. 43–59). New York: Springer.

Dillard, J., & Yuthas, K. (2006). Enterprise resource planning systems and communicative action. Critical Perspectives on Accounting, 17(2-3), 202–223.

Fleming, P., & Spicer, A. (2003). Working at a cynical distance: Implications for power, subjectivity and resistance. Organization, 10(1), 157–179.

Haywood, T. (2002). Defining moments: Tension between richness and reach. In W. Dutton & B. Loader (Eds.), (pp. 39–49). London: Routledge.

Hutchins, E. (1991). Organizing work by adaptation. Organization Science, 2(1), 14–39.

Introna, L. (1996). Notes on ateleological information systems development. Information Technology & People, 9(4), 20–39.

Jamieson, K., & Hyland, P. (2006). Factors that influence Information Systems decisions and outcomes: A summary of key themes from four case studies. Adelaide, Australia.

Jones, D. (1996). Solving Some Problems of University Education: A Case Study. In R. Debreceny & A. Ellis (Eds.), Proceedings of AusWebÕ96 (pp. 243–252). Gold Coast, QLD: Southern Cross University Press.

Jones, D. (2002). Student Feedback, Anonymity, Observable Change and Course Barometers. In P. Barker & S. Rebelsky (Eds.), World Conference on Educational Multimedia, Hypermedia and Telecommunications 2002 (pp. 884–889). Denver, Colorado: AACE.

Jones, D. (2003). Course Barometers: Lessons gained from the widespread use of anonymous online formative evaluation. QUT, Brisbane.

Jones, D., & Buchanan, R. (1996). The design of an integrated online learning environment. In A. Christie, B. Vaughan, & P. James (Eds.), Making New Connections, asciliteÕ1996 (pp. 331–345). Adelaide.

Jones, D., & Luck, J. (2009). Blog Aggregation Management: Reducing the Aggravation of Managing Student Blogging. In G. Siemns & C. Fulford (Eds.), World Conference on Educational Multimedia, Hypermedia and Telecommunications 2009 (pp. 398–406). Chesapeake, VA: AACE.

Jones, N., & OÕShea, J. (2004). Challenging hierarchies: The impact of e-learning. Higher Education, 48, 379–395.

Katz, R. (2003). Balancing Technology and Tradition: The Example of Course Management Systems. EDUCAUSE Review, 38(4), 48–59.

Kurtz, C., & Snowden, D. (2007). Bramble Bushes in a Thicket: Narrative and the intangiables of learning networks. In M. Gibbert & T. Durand (Eds.), . Blackwell.

Laurillard, D. (2002). Rethinking University Teaching: A Conversational Framework for the Effective Use of Learning Technologies. London: Routledge.

Light, B., Holland, C. P., & Wills, K. (2001). ERP and best of breed: a comparative analysis. Business Process Management Journal, 7(3), 216–224.

March, J. (1991). Exploration and exploitation in organizational learning. Organization Science, 2(1), 71–87.

Mintzberg, H. (1989). Mintzberg on Management, Inside our Strange World of Organisations. New York: Free Press.

Morgan, Glenda. (2003). Faculty use of course management systems. Educause Centre for Applied Research.

Morgan, Glenn. (1992). Marketing discourse and practice: Towards a critical analysis. In M. Alvesson & H. Willmott (Eds.), (pp. 136–158). London: SAGE.

Pozzebon, M., Titah, R., & Pinsonneault, A. (2006). Combining social shaping of technology and communicative action theory for understanding rhetorical closuer in IT. Information Technology & People, 19(3), 244–271.

Robey, D., Ross, W., & Boudreau, M.-C. (2002). Learning to implement enterprise systems: An exploratory study of the dialectics of change. Journal of Management Information Systems, 19(1), 17–46.

Rossi, D., & Luck, J. (2011). Wrestling, wrangling and reaping: An exploration of educational practice and the transference of academic knowledge and skill in online learning contexts. Studies in Learning, Evaluation, Innovation and Development, 8(1), 60–75.

Seely Brown, J., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42.

Seely-Brown, J., & Hagel, J. (2005). From push to pull: The next frontier of innovation. The McKinsey Quarterly. McKinsey & Company.

Simon, H. (1991). Bounded rationality and organizational learning. Organization Science, 2(1), 125–134.

Sturgess, P., & Nouwens, F. (2004). Evaluation of online learning management systems. Turkish Online Journal of Distance Education, 5(3).

Thomas, J. (2012). Universities canÕt all be the same – it’s time we embraced diversity. The Conversation. Retrieved June 28, 2012, from http://theconversation.edu.au/universities-cant-all-be-the-same-its-time-we-embraced-diversity-7379

Truex, Duane, Baskerville, R., & Travis, J. (2000). Amethodical systems development: the deferred meaning of systems development methods. Accounting Management and Information Technologies, 10, 53–79.

Truex, Duanne, Baskerville, R., & Klein, H. (1999). Growing systems in emergent organizations. Communications of the ACM, 42(8), 117–123.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.

Underwood, J., & Dillon, G. (2011). Chasing dreams and recognising realities: teachersÕ responses to ICT. Technology, Pedagogy and Education, 20(3), 317–330. doi:10.1080/1475939X.2011.610932

Wagner, E., Scott, S., & Galliers, R. (2006). The creation of Òbest practiceÓ software: Myth, reality and ethics. Information and Organization, 16(3), 251–275.

Weick, K., & Quinn, R. (1999). Organizational change and development. Annual Review of Psychology, 50, 361–386.

Ateleological travels in a teleological world: Past and future journeys around ICTs in education

In my previous academic life, I never really saw the point of book chapters as a publication form. For a variety of reasons, however, my next phase in academia appears likely to involve an increasing number of book chapters. The need for the first such chapter has arisen this week and the first draft is due by February next year, which is a timeline to give me just a little pause for thought. (There is a chance that this book might end up as a special edition of a journal)

What’s you perception of book chapters as a form of academic publication? Am particularly interested in the view from the education field.

What follows is a first stab at an abstract for the book chapter. The title for the book/special edition is “Meanings for in and of education research”. The current working title for my contribution is the title to this post: “Ateleological travels in a teleological world: Past and future journeys around ICTs in education”.

Abstract

The Australian Federal Government are just one of a gaggle of global stakeholders suggesting that Information and Communication Technologies are contributing to the creation a brave, new, digital world. Such a digital world is seen as being radically different to what has gone before and consequently demanding a radically different education system to prepare the next generation of learners. A task that is easier said than done. This chapter argues that the difficulties associated with this task arise because the meanings underpinning the design of education systems for the digital world are decidedly inappropriate and ill-suited for the nature of the digital world. The chapter draws upon 15+ years of research formulating an Information Systems Design Theory for emergent e-learning systems for universities to critically examine these commonly accepted meanings, suggest alternate and more appropriate meanings, and discuss the potential implications that these alternate meanings hold for the practice of education and education research.

The plan

The plan is that this chapter/paper will reflect on the primary focus of my research over recent years and encourage me to think of future research directions and approaches. Obviously it will draw on the PhD research and in particular the Ps Framework and the presentation I gave at EdMedia a couple of years ago. It will also draw on the presentation I gave analysing the Digital Education Revolution as part of my GDLT studies this year.

The insanity of changing LMSes/VLEs

There is a definition of insanity that I’ve seen seen attributed to either Einstein or Benjamin Franklin,

“The definition of insanity is doing the same thing over and over again and expecting different results”

That quote, at least for me, has connections with one of more certain origins.

Those who cannot remember the past are condemned to repeat it.

which comes from George Santayana

The connection with LMSes and e-learning

There is an orthodoxy in e-learning at universities. Implement a learning management system like Blackboard, Moodle, Sakai…. Different names, slightly different features but essentially the same type of tool. A big integrated “ring to rule them all”.

At least going by the literature I read and the experience I have the success of LMSes has been far from good. Either the LMS is rarely used or what use it is put to is at a very low level in terms of quality learning and teaching.

Given this is known, then why are many universities up to their second, third and even fourth learning management system? Why are they doing the same thing over and over again and expecting different results?

My answer

In the following presentation I give my answer, which is essentially

  • Implementation of e-learning is really complex and requires a mix of skills and knowledge.
  • It’s easier to adopt a fad – the LMS – than engage with the complexity.

To some extent, this might have some connection with the idea of task corruption.

Those who disagree with the definition of insanity

There is not universal agreement on the validity or source of the “Einstein” quote. George Sanger has a post titled “The definition of Insanity is, perhaps, using that quote”. Of course, I and a number of the folk commenting on the post disagree.

One of the most credible seeming points made against this quote is

It contradicts the notions of experimentation and practice.

Which, on reflection, doesn’t apply. For me at least, experimentation and practice, means that you will not be doing the same thing again and again. You will be trying slightly different things. Each time you practice you will be working to improve what you are doing, to learn from your mistakes.

The source of the insanity quote

Wikiquote attributes the insanity quote to Rita Mae Brown and her book “Sudden Death”

Task corruption in teaching @ university – negative impact of Place?

Busy being a good boy working on the thesis, currently reading a collection of literature to flesh out Chapter 2 which is drawing on the Ps Framework to illustrate the current state of e-learning within Universities. As the last post illustrates, the most recent paper I’m reading is White (2006).

The Ps Framework: a messy version

In her concluding remarks, White draws on the idea of task corruption suggested by Jan Chapman (1996) to describe some of the negative impacts of broader societal issues on learning and teaching at Universities. I’m attracted to this idea for two reasons:

  1. Increasingly I’ve thought most learning and teaching at universities is increasingly of less than stellar quality and “task corruption” provides an interesting (and at current glance, appropriate) perspective on why.
  2. It reinforces the potentially negative impacts that “Place” (one of the components of the Ps Framework) can have on the practice of learning and teaching (again one of many).

In the following, I’m trying to explain what task corruption is and explore what impact it might have on learning and teaching, and particularly e-learning (topic of the thesis), within universities.

What is “task corruption”

Task corruption is where either an institution or individual, conciously or unconsciously, adopts a process or an approach to a primary task that either avoids or destroys the task. Yesterday’s Dilbert cartoon – see below – is a great example.

Dilbert.com

White (2006) identifies two types of task corruption:

  1. amputation; and
    Where parts of the task are no longer performed or are ‘starved’ of attention at the expense of other parts of the task. White (2006) uses the following quote from one of the students she talked with as an example

    I personally believe that the way universities are run today is not necessarily in the best interests of students, but rather in securing numbers to generate a wealthy university and to establish research programs and post graduate programs rather than focusing on the majority of student who come to study in undergraduate degrees.

    I don’t think it would be too hard at some institutions to find a similar student quote in relation to full-fee paying overseas students at commercial campuses.

  2. simulation.
    Where the system or the individual is seen to comply with the task. i.e. they adopt the appearance of task engagement with the aim of avoiding real engagement.

    Perhaps an example of this is what happens in response to a rule at one organisation that states a course shall have no more than 2 assignments, if there is an exam worth more than a certain percentage. If you check course profiles this rule has essentially been followed. But scratch the surface and you find multi-part assignments, including sub-parts that have different due dates than other sub-parts of the same assignment.

Drawing further on Chapman, there is the observation that task corruption occurs most frequently with tasks where it is difficult to define or measure the quality of service (learning and teaching anyone?). Consequently, incentives (or punishiments) are based on quantity rather than quality. This certainly has resonances with personal experience and the almost exclusionary concern about end of term on failure rates, rather than on actual quality of learning.

Sadly, the most recent discussion of this work (Chapman, 2003) is in a journal to which I don’t currently have access. But it is interesting enough to follow up.

Implications for universities

My interest is in how you improve learning and teaching at universities, and one in particular. What implications do these ideas have for that?

Perhaps the most important one I can think of at the moment is to increase awareness of task corruption.

My feel is that task corruption is the dirty little secret of learning and teaching. Most people are aware it goes on, can probably point to some examples but professional pride (and perhaps other reasons) will prevent them from admitting this in a broader sense. In my experience management, especially those at a senior level, have developed the ability to ignore task corruption.

A certain sense of abstraction at the senior management level is good, otherwise you’d never get anything done. But perhaps, it’s been taken too far. Looking for and talking about the forms of task corruption within a university around learning and teaching could be a good first step in identifying those factors within the organisational and the social setting that are contributing to the task corruption. Hopefully as a first step in addressing these problems (and who says I can’t be wildly optimistic).

The problem isn’t limited to senior management. In an organisation that places emphasis on top-down, teleological design procesess the problem is (I believe) likely to occur within instructional design groups, information technology “support” groups etc.

References

Chapman, J. (1996). Hatred and corruption of task (Australian Institute of Social Analysis Working Paper No. 3). Carlton: AISA.

Chapman, J. (2003). Hatred and corruption of task. Organisational and Social Dynamics, 3(1):40-60

White, N. (2006). “Tertiary education in the Noughties: the student perspective.” Higher Education Research & Development 25(3): 231-246.

An information systems design theory for e-learning

Yesterday I gave a presentation at the Australian National University on my PhD. I’m doing it through ANU and this 30 minute presentation is a standard requirement of study. The slides are up on slideshare (embedded below). I recorded the audio and will be trying to put that online later on today and make the slides into a slidecast.

The presentation

It appears embedding the presentation in this post isn’t working at the moment. The slides can be found here on slideshare. — seems the embedding is working now.

The description

It’s been a while since I worked directly on the PhD and creating this presentation was a way to become deeply familiar with the thesis again, in preparation for writing it up. So the presentation is structured in line with the thesis and provides a high level overview of the whole thing.

While the information systems design theory (ISDT) that is the main product of the thesis gets a mention, explaining the design theory is not the primary goal of the presentation. Such descriptions have been given in other papers (Jones and Gregor, 2002; Jones and Gregor, 2004). Instead the emphasis of the presentation is on the other components of the thesis that are in need of some extra work.

Most of the content of the presentation is focused on chapter 2 and the Ps Framework. In fact, must of it is related around the content of a paper I’ve proposed for later in the year.

Essentially the idea is that the practice of e-learning within universities has a definite orthodoxy (which LMS will we adopt). I suggest that for a number of reasons the understandings that underpin that orthodoxy are entirely inappropriate and this is why most university e-learning implementations are plagued by less than widespread use by academics, low quality learning by those that do use it and some concerns around return on investment.

There’s also some early work on the structure of chapter 3 – the research method. But still early days there.

References

Jones, D. and S. Gregor (2004). An information systems design theory for e-learning. Managing New Wave Information Systems: Enterprise, Government and Society, Proceedings of the 15th Australasian Conference on Information Systems, Hobart, Tasmania.

Jones, D. and S. Gregor (2006). The formulation of an Information Systems Design Theory for E-Learning. First International Conference on Design Science Research in Information Systems and Technology, Claremont, CA.

Modified TPACK diagram to illustrate the reality of the entanglement/mess

Frameworks and representation – tidy versus messy

I’m a fan of frameworks and taxonomies. Also known as theories for understanding (Gregor, 2006). It’s the understanding part that I like. They provide, or at least good ones do, a leg up in understanding difficult concepts. As Mischra and Koehler (2006, p 1019) say

Having a framework goes beyond merely identifying problems with current approaches; it offers new ways of looking at and perceiving phenomena and offers information on which to base sound, pragmatic decision making.

As it happens, I’m currently doing a lot of work around one framework and its application and the following arises out of that work.

Two of my current most used frameworks include Dave Snowden’s cynefin framework (Snowden and Boone, 2007) and Mischra and Koehler’s TPACK (2006). Representation is important to frameworks. The cynefin framework, in particular, has a very specific representation that has very specific meaning and purpose.

TPACK framework

The TPACK crew have just released an updated representation of their framework (see the image to the left). I particularly like the addition of ‘contexts’ around the outside. The use of ven diagrams is important, one of the contributions of TPACK is the overlaps.

Tidy versus messy

One of the things I don’t like about frameworks is that they have (for very good reasons) to be tidy. This certainly helps understanding, a key purpose of frameworks, but it also can give the false impression of tidiness, of simplicity of a tame problem. My interest is currently in e-learning within universities, which I consider to be extremely messy. To me it is an example of a wicked problem.

A message version of TPACK

Last week I ran a session on course analysis and design for some CQUniversity academic staff. I used TPACK as one of the major themes. However, at one point I really wanted to emphasise to the participants that none of our discussions should be taken to assume that this is a neat and simple problem. The image to the left is the one I used to reinforce this (they’d already seen the tidy version of TPACK).

In doing this, I sacrificed much of the representational value of TPACK to highlight the messiness involved.

The Ps Framework – Tidy versus messy

For about 3 years (this presentation is the first public evidence) I’ve been working on what is now known as the Ps Framework as part of my PhD.

The first representation of the Ps Framework, taken from the first presentation is included below. A photo of some frozen peas used as a “pun”. The arrows are included, but don’t really mean anything. Still very early days.

Version 1 of the Ps Framework

The next public iteration of a graphical representation of the Ps Framework was the following one for a more recent presentation (you can even watch the video of this one). In this “Place” becomes the underlying context for all the other Ps. Much like the addition of context in TPACK. The frozen peas disappear for nice tidy circles (to some extent each one is meant to be a pea) and the arrows are still there. The arrows are meant to indicate that each of the Ps impacts upon the other in some unspecified way.

Version 2 of the Ps Framework

I had to prepare the above images to deadlines for presentations. I never liked them. Too tidy and they appeared to indicate linear or simple connections between the individual Ps. I don’t believe that. The relationships between these Ps when talking about e-learning implementation within universities is messy, complex and unpredictable – at least beforehand.

So I had to come up with something else. For the last few months of last year Jocene, Nathaniel and I spent a lot of time discussing and arguing about how to represent the Ps Framework. The following is my current best effort – it’s the effort I’ll be using this week at ANU.

a messy version

I have a range of problems with this representation including:

  • It’s still a little too structured.
    i.e. People only overlaps with Past Experience, Purpose and Process. Those overlaps aren’t intentional. They aren’t meant to represent some specific connection. I’m not sure what the connections are, I have an inkling that each and everyone is connected/overlaps with the other but I am stuck with this current conceptualisation.
  • It’s too static.
    The relationships between these components is forever changing. Universities and the place they inhabit are continually changing, each of the other components are changing and each change has some, unpredictable impact on the other components. In my mind I see this dynamic representation of this image where each component is seething and roiling and impacting upon each other.
  • It doesn’t capture perspective.
    Still not certain if this should be another P added to the framework or whether different instantiations of the Ps Framework represent different perspectives. I tend to prefer the latter, but then that leaves unsaid the important point about the perspectives of different groups being very diverse and that this is one of the fundamental problems with e-learning within universities.

Any suggestions?

References

Gregor, S. (2006). “The nature of theory in information systems.” MIS Quarterly 30(3): 611-642.

Mishra, P. and M. Koehler (2006). “Technological pedagogical content knowledge: A framework for teacher knowledge.” Teachers College Record 108(6): 1017-1054.

Snowden, D.J. Boone, M. “A Leader’s Framework for Decision Making”. Harvard Business Review, November 2007, pp. 69-76.

Barriers to innovation in organisations: teleological processes, organisational structures and stepwise refinement

This video speaks to me on so many levels. It summarises many of the problems I have faced and encountered trying to implement innovative approaches to e-learning at universities over the last 15 plus year. I’m sure I am not alone.

Today, I’ve spent a lot of time not directly related to what I wanted to achieve. Consequently, I had planned not to do or look at anything else until I’d finished. But this video resonates so strongly that I couldn’t resist watching, downloading it and blogging it.

I came across the video from a post by Punya Mishra. Some more on this after the video. I should also link to the blog post on the OpenNASA site. Would your University/organisation produce something similar?

If Nona ever gets around to watching this video, I am sure she will see me in a slightly different role in the video. Until recently I had the misfortune to be in the naysayer role. That’s no longer the case. Who said no good could come of organisational restructures?

Barriers to innovation and inclusion

The benefits of being open

Coming across this video, provides further evidence to support an earlier post I made today on the values of being open. I became aware of Punya’s post because of the following process:

  • Almost a year ago Punya published this post on his blog that openly shares the video of a keynote he and Mat Koehler gave.
  • I came across it not long afterwards through my interest in TPACK (formerly known as TPCK).
  • About two weeks ago I decided to use part of the video in some sessions I was running on course analysis and design.
  • A couple of days ago I blogged on an important part of the presentation (not used in the sessions I ran) that resonated with my PhD work.
  • My blog software told Punya’s blog software about my post and added it as a comment to his blog.
  • This afternoon Google Alerts sent me an email that this page on Punya’s blog was linking to my blog (because of the comment – see the comments section in the right hand menu).
  • Out of interest (some might say in the interest of procrastination) I followed the link and saw the video.

I plan to use parts of this video in future presentations around my PhD research. I believe it will resonate with people so much better than me simply describing the abstract principles.

So while not directly contributing to what I wanted to do today. It’s provided with a great advantage in the future.

of a Google Alert I have set on my site. Google emailed me to say that Punya had made this post because his blog software includes a list of the

I’ve spent a lot of time today doing stuff not necessarily directly related to what I wanted to achieve today. To such an extent I’d decided not to blog anymore.

One reason people don't take to new e-learning technology

In a recent post I started my collection of quotes on this blog. I also talked about the “mere exposure effect” and suggested it’s one reason behind the horseless carriage approach to using new technology. It’s also one reason why people resist new technology – especially e-learning/computer technology.

In working on another post, one directly related to the PhD, I came across this article from EDUCAUSE Quarterly titled “The Three-E Strategy for Overcoming Resistance to Technological Change “.

One of the quotes it uses to as evidence of why adoption of new technology is hard is from a book by Carolyn Marvin

For if it is the case, as it is fashionable to assert, that media give shape to the imaginative boundaries of modern communities, then the introduction of new media is a special historical occasion when patterns anchored in older media that have provided the stable currency for social exchange are reexamined, challenged, and defended.

The EDUCAUSE Quarterly article also says the following

As technology professionals, we often fail to see how intimidating technology can be to the user community.

I’d expand this out to include instructional designers and management. Instructional designers often don’t see how intimidating many of their pedagogical innovations (forget the use of technology) are to many academic staff. Many management folk I’ve seen make similar mistakes, though generally worse. Management generally don’t see how new pedagogy and technology, if used effectively, needs a radically different approach to teaching and learning practice. More importantly they don’t see or engage in the fact that this type of radical change often brings into question many of the accepts administrative processes, policies and organisational structures within institutions.

The article also quotes an EDCAUSE review article titled “My Computer Romance”. An expanded quote from this review article

What kept me from seeing and acting on those benefits? The question interests me, and not only out of self-regard. The question is at the heart of “faculty development,” a crude, even misleading phrase that cannot suggest the trick of imagination needed to bring substantial, important knowledge into plain sight and to develop in faculty the resolve and courage to risk failure. For an academic, “failure” is often synonymous with “looking stupid in front of someone.” For many faculty, and maybe for me back in the 1980s, computers mean the possibility of “pulling a Charlie Gordon,” as the narrator poignantly terms it in Daniel Keyes’s Flowers for Algernon.

This has significant implications for personal learning environments that surely represents a significant shift in practice created by the capabilities of a new medium and offers an even greater opportunity for academics to “pull a Charlie Gordon”. The Quarterly article

finishes it’s introduction with the following paragraph

Consider for a moment the impact of Web 2.0 on a professor working in academia for 20 or 30 years. The flattening of knowledge production and the ease of access to information represented by Web 2.0 technologies in many ways negates the concept of the “sage on the stage” or even traditional notions of scholarship. This world is not what most professors are used to, and many are threatened by and therefore resist this kind of change.

The solution

The Quarterly article

suggests that the solution is a strategy for gaining acceptance of technology that embodies “Three Es”

  1. Evident – as potentially useful in making life easier.
  2. Easy to use – to avoid feelings of adequacy.
  3. Essential – as part of going about their business.

Sort of sounds a bit like the insights from TAM and Diffusion of Innovations.

The wrong view

The Quarterly article finishes with this sentence

Only then will faculty effectively use the complex technical infrastructure that we technologists labor so hard to put into place.

God I hate the mindset that underpins that sentence. Or at least the common mindset amongst “support” folk in higher education. This isn’t limited to just information technology people. Instructional designers, quality folk and management all suffer from this view from time to time.

How do we get these poor ill-informed and/or obstinate academics to use the great technology/idea. If only we could do this we would solve all the problems of learning/teaching/research in one fell swoop.

This has been a problem with most people peddling innovation. Indeed, diffusion theory (Rogers, 1995) one of the best known innovation theories, has been criticised for having a pro-innovation bias that, amongst other effects, can separate members of a social system into the superior innovators group and the inferior recalcitrants group (McMaster and Wastell, 2005).

In this paper (Jones and Lynch, 1999) we talk about

  • developer based; and
    A developer-based focus assumes that the new product will automatically replace the old and that adopters will see the benefits of the new product automatically and in the same way as the developers.
  • adopter-based approaches to software development.
    These approaches focus on the adopters and their setting in order to understand the social context and the social function the innovation will serve.

The Es approach strikes me as someone who comes from a developer-based culture taking the first steps towards a more adopted-based approach. But someone who still has the same underlying belief that we build it and they use it.

References

Jones, D. and T. Lynch (1999). A Model for the Design of Web-based Systems that supports Adoption, Appropriation and Evolution. First ICSE Workshop on Web Engineering, Los Angeles.

Rogers, E. (1995). Diffusion of Innovations. New York, The Free Press.

Carolyn Marvin, When Old Technologies Were New: Thinking About Electric Communication in the Late Nineteenth Century (New York: Oxford University Press, 1988), p. 4.

BAM – making e-learning technology more protean

In a post yesterday I talked about how most applications of e-learning within universities seems to actively prevent students and staff leveraging the protean nature of information technology. That is the nature of computer software to be flexible, malleable and customisable.

The rise of “web 2.0” and related concepts has made it easier to put in place elearning technology that is designed to be more protean. In this post I talk about the Blog Aggregation Management (BAM) project and reflect on some of its ideas and implications for making e-learning technology more protean.

What is BAM?

It’s a research project aimed at extending ideas around how to implement e-learning technology at universities, particularly with an emphasis on what the rise of “web 2.0”, software as a service and other related concepts might mean for this practice.

BAM is a set of Perl scripts that aggregates RSS feeds, it matters little from where those RSS feeds originate, registered to individual students and then provides a range of additional services required by university educators. For example, (click on the screenshots to see larger images)

  • Link with the institutional teaching responsibilities database so that staff can see which of their students have (or haven’t) registered their RSS feed.
    BAM show student blog details
  • Show how many posts each staff member’s students have made.
    BAM show all student posts page
  • If required, award a mark and make comments on a student’s posts.
    BAM mark post page

Because of the original context in which BAM was designed (explained in some of the publications and presentations listed in the next section) there are also some scripts to detect plagiarism between student posts.

Origins of BAM

BAM started life as a more flexible way of implementing student journals in a particular course with the intent of encouraging reflection, increasing interactions between students and staff and hopefully increasing student performance. The initial use of BAM for this purpose is talked about in a number of places including:

  • Two presentations given at CQUniversity in 2006 that are available on Google Video. The first talks about the initial design ideas while the second reflects on the experience about half way through the course.
  • A paper describing the use in the initial course.
  • This initial use was also covered in the ELI Guide To Blogging as one of the three case studies.

System usage – examples of the protean nature

The initial application of BAM was intended to encourage student reflection and interaction between staff and students. It succeeded to varying levels of success depending on the staff involved. BAM has been used in all 8 offerings of that course from 2006 to 2008.

It has also been used in 8 other course offerings for a variety of different purposes. The most different was in the course EDED11448, Creative Futuring. EDED11448 was CQUni’s first “Web 2.0 course site” where all of the services used by students and staff in the courses were hosted on external services including del.icio.us, WordPress.com, Wetpaint, and RedBubble.

For EDED11448, BAM was used, in conjunction with Yahoo Pipes to create the Portfolio and Weblog pages. This was done by

  • Students create RedBubble accounts and using this to create their portfolio and their blog.
  • Students register their RedBubble account with BAM.
  • BAM aggregates both the portfolio and blog and produces aggregated RSS feeds.
  • Pipes is used to turn those RSS feeds into a bit of JSON data that can be used by the Javascript on the course website to present the data.

The same idea has been used to create RSS feeds from BAM that aggregate all a staff members students’ posts into one feed. A number of the courses that use BAM can have hundreds of students and tens of staff.

These application of BAM have moved beyond the original design. The protean nature of BAM includes the following:

  • There is choice in what application they use to generate the RSS feeds.
    In some courses that choice is left to the student. In EDED11448 the course designer made a specific choice – RedBubble – for her purpose.
  • There is choice in what BAM is used for.
    The original use was aimed specifically at individual student reflective journals reviewed and marked by staff. EDED11448 aggregated and made public to all students the work of individual students. In some cases the blog posts haven’t been marked.

Implications

I think BAM and the way it operates has the following implications for the practice of e-learning within universities.

  • Increase efficacy and agility while liberating institutional resources.
    This isn’t my view. It’s the one expressed by the authors of the ELI Guide to Blogging. When talking about BAM they say

    One of the most compelling aspects of the project was the simple way it married Web 2.0 applications with institutional systems. This approach has the potential to give institutional teaching and learning systems greater efficacy and agility by making use of the many free or inexpensive—but useful—tools like blogs proliferating on the Internet and to liberate institutional computing staff and resources for other efforts.

  • There is no need to pre-determine and specify all of the technology that staff and students must use.
    Most of the students who have used BAM haven’t really known what a blog is and very few have already had a blog. This lack of knowledge is not a reason to say we must use the blog provided by our LMS in order to minimise confusion. With BAM we recommend that students, who aren’t sure what to do, should make use of WordPress.com to create a blog. But we enable those with more knowledge to be able to use their own.
  • Small pieces loosely joined works.
    To me this is a fundamental characteristic of Web 2.0, the ability to create something larger out of a bunch of small pieces that are all loosely joined. Where each small piece can be replaced or re-tasked depending on the contextual needs. This is simply not possible with traditional enterprise software such as a course management system.
  • There are potential problems but they can be solved, and generally cheaper and easier with this approach.
    The most common question that is asked about BAM is “What happens if a student’s blog provider goes belly up and we can’t access the student’s work?”. This is the “can we depend on external providers” question. The assumption is that organisationally provided systems are more reliable. While that is somewhat questionable, the concern can be mitigated quite easily.

    In BAM’s case, this is done by mirroring. Every hour BAM

    • Visits each student’s RSS feed.
    • If there have been any changes it creates/updates a local copy of the RSS feed.</li

    If the external blog provider ever disappears, we have a copy. These types of problems can be solved.

  • Making existing systems more protean is a good thing.
    A number of benefits arise from systems being more protean. For example, the tools are able to be used for a number of unintended applications and the users are able to use tools that they are familiar with and have a sense of ownership over. For me, this means that making existing systems more protean is a good and worthy thing to do.

Future work

Future work might include:

  • Making BAM more self-serve.
    Currently setting up BAM requires some additional input from technical folk. Wouldn’t be too hard to make this self-serve.
  • Extending the RSS generation capabilities in BAM.
    These are still fairly limited in terms of capabilities. The need some extension in capabilities, especially in increasing the protean nature of such capabilities.
  • Improvements to the BAM interface.
    It was designed by me. Enough said.
  • Enabling more complex group-based manipulation, tagging and commenting within BAM.
    Beyond simple aggregation there is little that can be done. Even marking is not performed with RSS but with databases. One extension might be to create RSS feeds that include comments/marks from markers. Enabling peer marking, commenting and tagging and a range of more complex approaches might also be useful.
  • Looking at supporting privacy capabilities in BAM.
    At the simplest form adding the ability for the student’s RSS feed to be password protected might be useful. At the moment the RSS feed fed into BAM must be freely available. Supporting broader privacy settings makes the tool more flexible.
  • Making existing systems more protean.
    Add RSS feeds to the discussion forums and other features of a learning management system to enable staff and students to start mashing up.
  • Integrating BAM into an existing LMS.
    BAM’s current use is limited to CQUniversity. BAM, at the moment, is essentially a set of scripts that integrate RSS feeds with several CQUniversity systems (online assignment marking, results processing, staff teaching responsibilities, student enrollment etc). This means it doesn’t make sense to sell or release BAM’s code (beyond having people look at it). Another institution would have to rewrite all of BAM to fit with its systems and practices.

    One solution to this might be to integrate BAM with a system like Moodle. These systems already should have data about which staff are responsible for which students, which students are in which course etc.

  • Working closely with a range of different staff to explore and enable different applications of BAM, to extend its protean capabilities and leverage them to improve the learning and teaching experience.
    This is where the real benefit is. Working with staff with different purposes and problems to collaboratively identify approaches and necessary changes to BAM.

The protean nature of modern technology – another limitation of most views of e-learning

A part of my thinking around the Ps Framework I suggest that there are a number of dominant assumptions that underpin the current implementation of e-learning within institutions of higher education. I believe these dominant assumptions limit the quality, efficiency, effectiveness and innovativness of e-learning at Universities. In this post I am trying to identify one of the dominant assumptions associated with the “Product” component of the Ps Framework and its implications.

The dominant assumption I’d like to explore here is that people forget that modern technology is protean. Worse than that, how most universities implement e-learning significantly limits the ability to take full advantage of this protean nature and subsequently limits the quality and innovation possible within e-learning.

Origins

This though has arisen through a combination of work I’ve observed at CQUni over the last year or so and some resources I looked at for the course analysis and design sessions I ran last week.

The resource that got it all started was the keynote presentation given by Mat Koehler and Punya Mishra at the SITE’2008 conference. I originally used the presentation for the first part where they talked about teaching as a wicked design problem. The fact that most people treat teaching as a tame design problem is another one of the major assumptions that negatively impact on e-learning, but that’s a story for another post.

Late last night, as I was putting together the resources used in the course analysis and design sessions I did a Google search for the SITE’2008 keynote. Doing so I came across this blog post from Wesley Fryer. The post provides a summary of the keynote and includes the following snippet

Difference between traditional technologies (pencil, microscope, blackboard) are specific – new technologies are PROTEAN

What does protean mean?

Wikipedia offers this description

with the general meaning of “versatile”, “mutable”, “capable of assuming many forms”: “Protean” has positive connotations of flexibility, versatility and adaptability.

The source of this adjective is Proteus, a sea-god from Greek mythology who can tell the future but will change his shape so he can escape doing so.

Mishra and Koehler draw on the work of a number of folk in describing the digital computer as protean in nature – inherently flexible. For example Kay (1984) describing computers as a meta-medium that can dynamically simulate the details of any other medium (including non-physical media) and his suggestion that we have barely begun to investigate this freedom for representation and expression. A computer is a tool to manipulate symbol systems be they visual, acoustic, textual or numeric.

They also point out that digital computer systems can also mean different things to different people. That different people given the same computer will achieve radically different things with it because of its protean nature.

Indications of the current limitation

I can see indications of ignorance of the protean nature in current organisational practice of e-learning in two main forms.

  1. Almost universal use of processes for the selection and support of information systems that treat information systems as “non-protean”.
  2. Unspoken and unquestioned acceptance of information systems serving only a narrow purpose of its original design.

“Non-protean” processes

Look at how most organisations implement computer systems, including those associated with e-learning. Almost without exception (please let me know if you can point out some exceptions) they follow the traditional teleological systems development life cycle

  1. Identify all requirements.
  2. Examine/build a software system that meets those requirements.
  3. Have a long period of system use where it does not change in order to recoup the costs of the first two stages (or simply minimise costs).
  4. Eventually, when the disconnect between the features of the system and the requirements of the organisation become so great, return to step #1.

Truex, Baskerville and Klein (1998) go into more detail about this traditional approach and why it’s inappropriate.

This approach ignores the protean nature of software in that during the long period of use the system is not to be changed and/or changes are kept to a minimum. The only people who are allowed to change them are the central IT folk and anyone who even thinks about changing or working around these systems is sought out and “dealt with”. Hence the negative conatations of shadow systems.

Traditional organisational approaches to the implementation of e-learning within universities do not leverage one of the greatest strengths of modern computers – its protean nature.

In this paper (Danaher, Luck et al, 2004) my colleagues and I make the following points

Software based systems, such as course management systems, represent a set of cultural patterns frozen for now into a reproducible and constraining form (Clear, 2002). Application areas that have low volatility in requirements make it possible for stable, precisely designed systems – the outcome of traditional development methodologies – to operate satisfactorily with minimal changes for long periods (Truex, Baskerville and Klein, 1999). The criterion for this proposition is the capability of the course management system and the supporting organisation to adapt itself and themselves in response to an ongoing process of shared requirements negotiated within the local institution.

That is, we argued that it was important that organisational implementation of e-learning actively enable and encourage the harnessing of the protean nature of information systems in order to

to support and enhance ongoing and shared constructions of innovation, rather than enact a pragmatic set of existing practices or follow a single conceptualization of what is innovative

In fact, when it comes to enterprise systems standard “best practice” from the research and practitioner literature is to implement a vanilla system. i.e. don’t make any adaptations to the enterprise system as it is too expensive to maintain those changes as the enterprise system evolves.

This is important to e-learning because since early this century a university’s course management system forms the academic system equivalent of enterprise resource planning (ERP) systems in terms of pedagogical impact and institutional resource consumption (Morgan, 2003).

I know of one university that is currently in the process of adopting an open source course management system that is actively explaining how they will be implementing a vanilla system. At least that seems to be the narrative emerging from the information technology division.

Limited views on capabilities of technology

As with the monk in the Introducing the book sketch struggling to come to grips with the new fangled technology of the book. Less than experienced designers of e-learning often don’t grasp or see the full potentials of a new technology. Too often falling into the horseless carriage problem.

For example, it continues to be a trend that when universities or academics purchase some space in SecondLife the first building they construct is some form of auditorium in which they can give lectures.

Even if they are particularly innovative in terms of learning and teaching most folk using SecondLife still automatically think that any potential application of Second Life will require students to be able to get in world. Something that can be expensive, if not impossible for them because of bandwidth and hardware requirements.

Few folk make the connection that Second Life can also be a multimedia production platform. Similar to what folk I’ve worked with have done with machinimas (see example below) to support case studies in an Auditing course.

The use of Second Life to produce machinimas provided a cheap and effective way to produce “video” of authentic scenarios without requiring students to access Second Life and without the huge cost of a “real life” production. It’s also an example of the “protean” nature of Second Life that enables its use in new and interesting ways.

Making systems and processes more protean

One of the core aims behind the design of the Webfuse e-learning system (Jones and Buchanan, 1996) was “flexibility and the ability to adapt to change”. In that paper I wrote

The one unchanging characteristic of the Internet, and the computing field in general, is that it never stops changing. This characteristic makes flexibility and adaptability essential features for any online or computing system. Without these characteristics an organisation runs the risk of either retaining an out of date system because it is too expensive to replace, or having to throw away the investment in a system because it has not kept up with change. This risk is demonstrated by the problems faced by Universities that have only recently stopped (and some who haven’t stopped) using mid-80s style, text based computer mediated communications systems.

As the system and the knowledge used to design and support it improved over time there were two main approaches used to make Webfuse “more protean”

  1. The technical design of the system enables and supports modification and enhancement.
  2. The design and support processes place emphasis on close contact between the system developers and the system users.

The trend is towards more protean technology

Originally the design of Webfuse, in terms of “technical design” was enabled by the amount of open source software associated with the Web and Internet that was available in the late 1990s. At any other time it would not have been possible.

Webfuse’s flexibility was based on provide an institutional specific wrapper around this open source software. The wrapper provides a standard interface for users and does some translation from institutional specific abstractions into the abstractions used by the open source software.

With this design, if the discussion was no good. We could find another discussion forum and incorporate it. However, this model was still based on the “run everything on our server” model.

The rise of “web 2.0” and in particular mashable services helps remove the need for this model and significantly increases the protean possibilities of the Webfuse model. The first experiment with these possibilities was the Blog Aggregation (BAM) Project. The second was the “web 2.0 course site” idea.

Both extended the Webfuse model beyond open source software to making use of RSS to mash up services.

The development of immersive 3d applications/worlds like Second Life continue this trend towards more protean software. Not only in that it continues the ability to mash up (to some extent) it also provides an environment more like the real world.

Conclusions

This has been a quick mind dump about the nature of digital computer systems as being protean and the implications for e-learning within universities. The basic outcome is I believe most institutional e-learning actively negates the protean nature of information systems and that e-learning is the worse for this.

This is not to say that enabling and supporting the protean features of e-learning is a silver bullet without any problems. There are issues, mostly associated with how you operate under a completely different set of assumptions that question accepted practice.

References

Patrick Danaher, Jo Luck, David Jones, Jeanne McConachie, Course management systems: Innovation versus managerialism, Proceedings of 11th International Conference, ALT-C 2004, 14-16 September 2004, University of Exeter, Devon, England, 2004. pp 23-35.

Jones, D. and R. Buchanan (1996). The design of an integrated online learning environment. Proceedings of ASCILITE’96, Adelaide.

Morgan, G. (2003). Faculty use of course management systems, Educause Centre for Applied Research: 97.

Truex, D., R. Baskerville, et al. (1999). “Growing systems in emergent organizations.” Communications of the ACM 42(8): 117-123.

On the silliness of "best practice" – or why you shouldn't (just) copy successful organisations

The very idea of “best practice” is silly. In any meaningful complex activity the idea of simply copying what someone else did is destined to fail because it doesn’t seek to understand the reasons why that best practice worked for them and what are the differences between “them” and “us”.

This post over at 37 Signals expounds a bit more on this and references an article titled Why your startup shouldn’t copy 37signals or Fog Creek. The article gives one of the explanations of why best practices are silly.

Dave Snowden has an article called Managing for Serendipity: why we should lay off “best practice” in Knowledge Management that takes the discussion even further. Some of the reasons he gives include:

  • Human beings naturally learn more effectively from failure than success.
  • There is only a very limited set of circumstances in which you are able to identify some “best way” of doing something (see wicked problems).
  • It’s very unlikely that we can codify this “best way” in a way that makes it possible for others to fully understand and adopt the practice.
  • People are unlikely to actually follow the best practice.

My favourite one, from a number of sources, is that “best” practice, “good” practice and even “bad” bad practice from somewhere else tends to be adopted because it is easier than attempting to really understand the local context and draw on expertise and knowledge to develop solutions appropriate to that context.

Doing that is hard. Much easier to see what “important organisation X” has done and copy them. This is where fads come from.

This is a small part of the argument made in the book Management Fads in Higher Education: Where They Come From, What They Do, Why They Fail by Robert Birnbaum that I’m currently reading. More on this soon.

Good teaching is not innate, it can be "learned" – and what's wrong with academic staff development

The title to this post is included in a quote from Kane, Sandretto and Heath (2004)

The research team, comprising two teacher educators and an academic staff developer, embarked upon this research confident in the belief that good teaching is not innate, it can be learned. With this in mind, the project sought to theorise the attributes of excellent tertiary teachers and the relationships among those attributes, with the long-term goal of assisting novice academics in their development as teachers.

This coincides nicely with my current task and also with an idea I came across on the week-end about deliberate practice and the work of Anders Ericsson.

The combination of these “discoveries” is also providing some intellectual structure and support for the REACT idea about how to improve learning and teaching. However, it’s also highlighting some flaws in that idea. Though the flaws aren’t anywhere near as large as what passes for the majority of academic staff development around learning and teaching.

The following introduces these ideas and how these ideas might be used to improve academic staff development.

Excellent tertiary teaching

Kane et al (2004) close the introduction of their paper with

We propose that purposeful reflection on their teaching plays a key role in assisting our participants to integrate the dimensions of subject knowledge, skill, interpersonal relations, research/teaching nexus and personality into recognised teaching excellence. We conclude with a discussion of the implications of our model for staff development efforts.

Their proposition about the role of reflection in contributing to excellent teaching matches with my long held belief and perception that all of the best university teachers I’ve seen have been those that engage in on-going reflection about their teaching, keep looking for new knowledge and keep trying (and evaluating) innovations based on that knowledge in the hope to improve upon their teaching.

The authors summarise a long history of research into excellent teaching that focused on identifying the attributes of excellent teachers (e.g. well prepared, stimulate interest, show high expectations etc.) but they then suggest a very important distinction.

While these, and other studies, contribute to understanding the perceived attributes of excellent teachers, they have had limited influence on improving the practice of less experienced university teachers. Identifying the elements of “good” university teaching has not shed light on how university teachers develop these attributes.

The model the develop is shown below. The suggest

Reflection lies at the hub of our model and we propose that it is the process through which our participants integrate the various dimensions

Attributes of excellent tertiary teaching

The authors don’t claim this model to have identified any novel sets of attributes. But they do suggest that

the
way in which the participants think about and understand their own practice through purposeful reflection, that has led to their development of excellence

What’s been said about reflection?

The authors have a few paragraphs summarising what’s been said about reflection in connection to tertiary teaching, for example

Day (1999) wrote “it is generally agreed that reflection in, on and about practice is essential to building, maintaining and further developing the capactities of teachers to think and act professionally over the span of their careers”

.

They trace reflection back to Dewey and his definition

“an active, persistent, and careful consideration of any belief or supposed form of knowledge in light of the grounds supporting it and future considerations to which it tends

The also mention a framework of reflection outlined by Hatton and Smith (1995) and use it to provide evidence of reflection from their sample of excellent teachers.

Expertise and deliberate practice

Among the many quotes Kane et al (2004) provide supporting the importance of reflection is this one from Stenberg and Horvath (1995)

in the minds of many, the disposition toward reflection is central to expert teaching

Another good quote (Common 1989, p. 385).

“Master teachers are not born; they become. They become primarily by developing a habit of mind, a way of looking critically at the work they do; by developing the courage to recognize faults, and by struggling to improve”

Related to this view is the question “Was Mozart, and other child prodigies, brilliant because of some innate talent?”. This is a question that this blog post takes up. The answer it gives is no. Instead, it’s the amount and quality of practice they engage in which makes the difference. Nurture wins the “nature versus nurture” battle.

The blog post builds on the work of Anders Ericsson and the concept of “deliberate practice”. The abstract for Ericsson et al (1993) is

The theoretical framework presented in this article explains expert performance as the end result of individuals’ prolonged efforts to improve performance while negotiating motivational and external constraints. In most domains of expertise, individuals begin in their childhood a regimen of effortful activities (deliberate practice) designed to optimize improvement. Individual differences, even among elite performers, are closely related to assessed amounts of deliberate practice. Many characteristics once believed to reflect innate talent are actually the result of intense practice extended for a minimum of 10 years. Analysis of expert performance provides unique evidence on the potential and limits of extreme environmental adaptation and learning.

Implications for academic staff development

If reflection or deliberate practice are key to developing mastery or expertise, then how do approaches to academic staff development and associated policies, processes and structures around university learning and teaching help encourage and enable this practice?

Seminars and presentations probably help those that are keen to become aware of new ideas that may aid their deliberate practice. However, attendance at such events are minimal. Much of existing practice seems to provide some level of support to those, the minority, already engaging in deliberate practice around learning and teaching.

The majority seem to be able to get away without engaging like this. Perhaps there’s something here?

References

Common, D.L. (1989). ‘Master teachers in higher education: A matter of settings’, The Review of Higher Education 12(4), 375–387.

Hatton, N. and Smith, D. (1995). ‘Reflection in teacher education: Towards definition and implementation’, Teaching & Teacher Education 11(1), 33–49.

Kane, R., S. Sandretto, et al. (2004). “An investigation into excellent tertiary teaching: Emphasising reflective practice.” Higher Education 47(3): 283-310.

Sternberg, R. and Horvath, J. (1995). ‘A prototype view of expert teaching’, Educational Researcher 24(6), 9–17.

Some possible reasons why comparison of information systems are broken

All over the place there are people in organisations performing evaluations and comparisons of competing information systems products with a strong belief that they are being rational and objective. Since the late 1990s or so, most Universities seem to be doing this every 5 or so years around learning management systems. The problem is that these processes are never rational or objective because the nature of human beings is such that they never can be (perhaps very rarely – e.g. when I’m doing it 😉 ).

Quoting Dave Snowden

Humans do not make rational, logical decisions based on information input, instead they pattern match with either own experience, or collective experience expressed as stories. It isn’t even a bit fit pattern match, but a first fit pattern match. The human brain is also subject to habituation, things that we do frequently create habitual patterns which both enable rapid decision making, but also entrain behaviour in such a manner that we literally do not see things that fail to match the patterns of our expectations”.

Dave also makes the claim that all the logical process, evaluations, documents and meetings we wrap around our pattern-matching decisions is an act of rationalisation. We need to appear to be rational so we dress it up. He equates the value of this “dressing up” with that of the ancient witch doctor claiming some insight from the spirit world leading him to the answer.

Via a Luke Daley tweet I came across a TED talk by Dan Gilbert that provides some evidence from psychology about why this is true. You can see it below or go to the TED page

As an aside the TED talks provide access to a lot of great presentations and even better they are freely available and can be downloaded. Putting them on my Apple TV is a great way to avoid bad television.

Somethings that are broken with evaluation of university teaching

This article from a Training industry magazine raises a number of issues, well known in the research literature, about the significant limitations that exist with the evaluation of university teaching.

Essentially the only type of evaluation done at most universities is what the article refers to as “level 1 smile sheets”. That is student evaluation forms that ask them to rank what they felt they learn, what they felt about the course and the teacher. As Will Thalheimer describes

Smile sheets (the feedback forms we give learners after learning events) are an almost inevitable practice for training programs throughout the workplace learning industry. Residing at Donald Kirkpatrick’s 1st level—the Reaction level—smile sheets offer some benefits and some difficulties.

His post goes on to list some problems, benefits and a potential improvement. Geoff Parkin shares his negative view on them.

The highlight for me from the Training mag article was

In some instances, there is not only a low correlation between Level I and subsequent levels of evaluation, but a negative one.

The emphasis on level 1 evaluation – why

Most interestingly, the article then asks the question, “why do so many training organisations, including universities, continue to rely on level 1 smile sheets?”

The answer it provides is that they are too scared to do them in case of what they find. It’s the ostrich approach of sticking the head in the sand.

What else should be done?

This google book search result offers some background on “level 1” and talks about the other 3 levels. Another resource provides some insights and points to other resources. I’m sure if I dug further there would be a lot more information about alternatives.

Simply spreading the above findings amongst the folk at universities who rely and respond to findings of level 1 smile sheets might be a good start. Probably necessary to start moving beyond the status quo.

Open source learning management systems – the latest fad in e-learning

The following is forming up as something to go into my thesis as part of the “Past experience” component of the Ps Framework.

Essentially, I’m going to try and

  • Suggest a timeline of e-learning within universities.
  • Contend that this history shows all the hall marks of fads and fashions in organisational decision making, aka the bandwagon effect.
  • Outline what I think is one contributing factor to this bandwagon effect.

The timeline

Still early days for this, but I think you can group the timeline for e-learning in universities into the following (the times I’ve put in place are based on my experience at CQUniversity:

  1. Pre-Interent – pre-1992 or so
    This is the use of bulletin board systems and proprietary systems run by telecommunication companies. Text-only, primitive and horrible user interfaces and really, really expensive charges for dialing up at speeds approach 2400bps.
  2. Individual Internet – 1992-~1996 (perhaps later in some places)
    Individual innovators adopting use of FTP, email, Gopher and the very early days of the Web for their own classrooms. Most access is still via text-based interfaces with very slow expensive charges. The Internet is not a widespread things.
  3. Cottage industry learning management systems – ~1995/6-late 1990s
    The mid to late 1990s saw widespread recognition that the majority of academic staff simply did not have the skills or time to individual design their use of Internet technologies (Goldberg, Salari et al. 1996; Jones and Buchanan 1996). Through this time a diverse collection of intranet-based systems , home-built virtual learning environments, off-the-shelf products and customized groupware solutions were developed by different schools and faculties (Dron 2006). Some of them are still being used today – but all are just about gone.
  4. Institutional and then Enterprise commercial learning management systems – late 1990s through to now
    Centralised systems start being put in place. Almost without exception these are commercial systems from vendors. In the early days there were a plethora of vendors. But eventually this is reduced through mergers and takeovers. Some time in the noughties these systems go for added “prestige” by calling themselves “enterprise” systems. They most are still based on dodgy bits of technology put together in an ad hoc manner.
  5. Open source learning management systems – mid-noughties and onwards
    For various reasons, mostly cost, increasingly universities are going for open source learning management systems. Essentially the same as the commercial systems in terms of functionality, but designed by an open source community.

In terms of Gartner’s technology hype cycle my guess is that we’re currently climbing the peak of inflated expectations in terms of open source learning management systems. I believe most of the universities in New Zealand have adopted Moodle. All of the Australian universities that have recently made decisions about changing their LMS, I believe, have gone open (or community) source.

I was going to include the Wikipedia image of the hype cycle, but the format is wrong for this task. So I searched Flickr and found the following. It’s one view of emerging technologies in education and elsewhere. Interesting to see where other people have put Web 2.0, e-portfolios etc. It doesn’t distinguish between open source and commercial course management systems though. (click on the image to go to the original photo or here for the original blog post the image is from).

http://fleeptuque.com/ version of the gartner hype cycle

I believe we’re also at the start of another stage in this development, the “post industrial” approach. But this one still hasn’t risen to the awareness of most organisations, at least not at senior management making a decision to base the institution’s approach on it.

  • Post-industrial – mid-noughties (perhaps later) –
    This stage includes various ideas from such concepts as e-learning 2.0 (or the original), personal learning environments and web 2.0. At a very simple level, the emphasis moves away from the industrial model where everything is supplied by the institution through “enterprise” systems. To a model where the services provided by the institution blend in with the services and requirements of the learners. Where learner is defined in the broadest possible sense.

The case for fads, fashions and band-wagons

Back in June, 2007 – when writing this paper I said

…the subsequent limitation of rationality is demonstrated by the “faddish” adoption of LMSs within universities. Surprise has been expressed at how quickly university learning and teaching, commonly known for its reluctance towards change, has been modified to incorporate the use of learning management systems (West et al., 2006). Pratt (2005) finds connections between the Australian university sector’s adoption of e-learning during the 1990s and the concept of management fashions, fads and bandwagons where a relatively transitory collection of beliefs can legitimise the exercise of mindlessness with respect to innovation with information technology (Swanson & Ramiller, 2004). In particular, given conditions of uncertainty about prevailing technologies organisations may rely on imitation to guide decision making (Pratt, 2005).

One of the reasons, I believe, that the timeline sequence above applies fairly well to e-learning, regardless of institutions or countries. Is because most have been influenced by the bandwagon effect. The people making decisions in most organisations have had so little understanding of how to go about e-learning they have followed what everyone else is doing.

Following on from the above quote I asked the following questions

Is the current trend amongst universities to move towards open source learning management systems (e.g. Moodle) the most recent e-learning fashion? Will an open source learning management system, especially one that is supported within an institution in the same way as a commercial product, really make a significantly different impact than use of a commercial product?

Guess which university has recently decided to adopt Moodle?

The source of the bandwagon – unquestioned assumptions

Information systems development (and other interventions in human organisations) are not rational, purposive or goal-driven processes, they are instead subject to human whims, talents and the personal goals of those involved (Truex et al., 2000). Decision making about the implementation of information systems is not a techno-rational process with many decision makers relying on intuitions or instincts and simple heuristics to simplify decision making (Jamieson & Hyland, 2006). People are not rational in that their decision-making is influenced by a range of cognitive and other biases.

In the words of Dave Snowden, “Human beings are pattern matching intelligences”. Or more specifically (Snowden, 2005)

This builds on naturalistic decision theory in particular the experimental and observational work of Gary Klein (1944) now validated by neuro-science, that the basis of human decision is a first fit pattern matching with past experience or extrapolated possible experience. Humans see the world both visually and conceptually as a series of spot observations and they fill in the gaps from previous experience, either personal or narrative in nature. Interviewed they will rationalize the decision in whatever is acceptable to the society to which they belong: “a tree spirit spoke to me” and “I made a rational decision having considered all the available facts” have the same relationship to reality

When it comes to e-learning, the same problem arises. No matter who looks at e-learning their answers to the question of “how to do e-learning” is almost always limited by their background and experience. They do not see the whole picture, they only focus on what they know. And the fashion of what all the other universities are doing become a narrative that is so strong it becomes the obvious way to go.

Related to this problem is that there are range of unquestioned assumptions underlying these decisions. I’ll try and use the the Ps Framework to represent these. (This is very much formative for the thesis).

  1. Past experience – there’s almost an ignorance of what has gone before and the limitations of previous attempts. My university has just gone through its 3rd selection process for an LMS. How many has yours done?
  2. People – there’s an assumption that people are rational. The above disproves some of that. Google – bounded rationality or cognitive bias. There’s an assumption that students and staff will adopt what ever is decided.
  3. Product – there’s almost an automatic assumption that e-learning means LMS.
  4. Place – people seem to think universities and the context they operate in are static. There seems to be an assumption that universities are ordered or complicated systems, rather than complex systems (see the Cynefin framework).
  5. Process – there is a complete bias to teleological design processes (Introna, 1996; Jones and Muldoon, 2007) when ateleological design is more appropriate. In the words of Snowden, “fail-safe” design versus “safe-fail” design.

Institutional implementation of e-learning can’t help but not be faddish or fashionable because many of the underlying understandings of the components that contribute to these decisions are unquestionably biased towards one understanding of those components. These understandings have become the single domineering, and usually unquestioned, understandings that underpin the organisational implementation of e-learning within universities.

Paraphrasing Truex et al (2000)

The adoption of such domineering understandings not only imprisons thinking about those understandings, but also thinking about think about those understandings.

References

Dron, J. (2006). Any color you like, as long as it’s Blackboard. World Conference on E-Learning in Corporate, Government, Healthcare and Higher Education, Honolulu, Hawaii, USA, AACE.

Goldberg, M., S. Salari, et al. (1996). “World-Wide Web – Course Tool: An environment for building WWW-based courses.” Computer Networks and ISDN Systems 28: 1219-1231.

Jamieson, K. and P. Hyland (2006). Factors that influence Information Systems decisions and outcomes: A summary of key themes from four case studies. 17th Australasian Conference on Information Systems, Adelaide, Australia.

Jones, D. and R. Buchanan (1996). The design of an integrated online learning environment. Proceedings of ASCILITE’96, Adelaide.

Introna, L. (1996). “Notes on ateleological information systems development.” Information Technology & People 9(4): 20-39.

Pratt, J. (2005). “The Fashionable Adoption of Online Learning Technologies in Australian Universities.” Journal of the Australian and New Zealand Academy of Management 11(1): 57-73.

Snowden, D. (2005). Multi-ontology sense making: A new simplicity in decision making. Management Today, Yearbook 2005. R. Havenga.

Swanson, E. B. and N. C. Ramiller (2004). “Innovating mindfully with information technology.” MIS Quarterly 28(4): 553-583.

Truex, D., R. Baskerville, et al. (2000). “Amethodical systems development: the deferred meaning of systems development methods.” Accounting Management and Information Technologies 10: 53-79.

West, R., G. Waddoups, et al. (2006). “Understanding the experience of instructors as they adopt a course management system.” Educational Technology Research and Development.

Why am I a ePortfolio skeptic?

Update: It’s not just eportfolios

16 September, 2012 – for some reason today has seen a fair bit of interest in this post from over 3 years ago. I though I’d take the opportunity to move this particular argument up a level.

Proposition: Attempts to improve or – heaven forbid – “innovate” university teaching and learning is largely driven by mindless innovation, fads and fashions. eportfolios was just one such fad. There have been and will be many more. Some of the more recent include: e-learning, the LMS, the enterprise LMS, the open source LMS (e.g. an earlier post where I proposed that the open source LMS was yet another fad), and more recently MOOCs.

@downes seemed slightly annoyed when I wrote in this earlier post

MOOCs are the latest fad to hit higher education.

. It was never my intent to denigrate the work he and others have done with cMOOCs. Rather it was to criticise how universities – especially Australian universities – were responding to the rise of MOOCs. Somewhat along the lines of what @bonstewart argues in Is MOOC more than just a buzzword?”.

My argument is that rather than mindfully innovating – or simply improving – learning and teaching (either with or without), Universities are driven by outside influences. By fads, fashions and buzz words.

When I say I’m an eportfolio skeptic, I’m not necessarily denigrating eportfolios (or MOOCs or the LMS or any other fad, fashion or buzzword). I’m critiquing the institutional leadership and management that continues to be driven by these fads, fashions and buzzwords and is apparently unaware of the problems this entails.

Both this original post and this post “Justificatory knowledge” use Swanson and Ramiller (2004) on innovating mindfully with technology. The “justificatory” post includes the following summary.

An organisation which is mindful in innovating with IT, uses reasoning grounded in its own organisational facts and specifics when thinking about the innovation, the organisation recognises that context matters (Swanson and Ramiller 2004). Within mindful innovation, management have a responsibility to foster conditions that prompt collective mindfulness (Swanson and Ramiller 2004).

Especially when how the institution adopts the latest fad ends up corrupting some of the fundamental underpinnings of the original idea. e.g. how the adoption of a specific eportfolio system seems to create the situation where universities are mandating that students create a portfolio in the institutional system. So much for individual choice.

Or when institutional MOOCs are no longer MOOCs, which as @downes defines

What makes a MOOC is the way it is designed – it supports thousands of users that fully interact because it is distributed. It’s not located in just one place, it is located in many places.

Original post starts here

Update: Donald Clark has 7 reasons why he doesn’t want one.

I am a skeptic when it comes to ePortfolios. I believe they are a waste of time. Another fad that will take attention away from activities that will actually improve learning and teaching at universities. I believe they embody many of the faulty assumptions and mistakes that underpin most of e-learning within universities.

So, why do I think that? Why am I using such strong language to describe it? Am I right? (Answers to these questions are always open to change).

That second last question is easy to answer, given a range of contextual factors I’m increasingly annoyed at people making, what I see as, the same mistakes again and again and again. I’m also simply in a grumpy mood today. It’s now over a month since I first wrote the start of this paragraph, that sentiment still stands. I also find myself in a position where I can be a little more critical.

I’m going to try and get this post published today to achieve something and also because I came across this presentation from the EDUCAUSE Mid-Atlantic Regional Conference. It’s called “Assessing Impact: E-Portfolios in Higher Education”.

Yet another fad – ignorance of place, emphasis on product

In an earlier post I started some of this complaint. In the session which that post reports on there were lots of gleams of interest from CQUniversity folk when e-portfolios and elgg were mentioned. I could see people, who had little or no idea about what e-portfolios or elgg were, becoming interested in looking at them. At the time I said

This is one example of how the “product” (in terms of the Ps Framework) overwhelms consideration of “place”, of context. This is exactly how fads and fashions arise in educational technology and organisational/management practice in general.

Here’s what Swanson and Ramiller (2004) said about fads and fashions.

Attention deferral and contextual insensitivity may appear to be unproblematic in the face of the overwhelming “proof” afforded by the larger community’s rush toward the innovation

i.e. since everyone else is doing it, you don’t have to worry about considering whether or not it actually makes sense for your context. In many cases you also don’t have to worry about whether or not everyone else that is already using the fad is succeeding or not.

The educational technology/curriculum design community and consequently the broader university community is talking about e-portfolios. In Australia there is the Australian ePortfolio Project which is raising the profile of e-portfolios in the sector. I predict that this project will further increase awareness of e-portfolios within the sector and before the year is out (if they haven’t already) some discussions will occur about their use at CQUniversity.

The EDUCAUSE presentation above suggests e-portfolios have been around for almost 10 years. So it’s taken a while. The same presentation also reports on some less than stellar results from students and staff. It seems the people aren’t that happy.

The “technologists” alliance – ignorance of people

I’m guessing that if we did an analysis of the literature around ePortfolios that we would find three categories of people making up the majority, if not the entirety, of the list of authors. Those categories are

  1. Vendors or developers of e-portfolio systems – the people that make and sell the systems.
  2. Institutional instructional designers or instructional technology folk – the people implementing e-portfolio projects within universities.
  3. Innovative academic staff – the people who adopt e-portfolios first.

In other words, a minority of the folk involved with education within universities.

In the words of William Geoghegan (1994) these three groups of folk are the “technologists” alliance. He had this to say about them (emphasis and some explanation added by me)

Those involved include faculty innovators and early adopters, campus IT (IT here is instructional technology – US phrase that includes instructional designers and information technology folk) support organizations, and information technology vendors with products for the instructional market. Ironically, while this alliance has fostered development of many instructional applications that clearly illustrate the benefits that technology can bring to teaching and learning, it has also unknowingly worked to prevent the dissemination of these benefits into the much larger mainstream population.

Geoghegan (1994) continued to say

There seems to have been a naive assumption on the part of all three communities that what worked for those who were already committed to the use of instructional technology, were actively applying it in their own work and were serving as evangelists to others would work equally well with those who had not yet committed.

Geoghegan works with Geoffrey Moore’s concept of a chasm that exists between the early adopters of a product (the enthusiasts and visonaries) and the early majority (the pragmatists).
Represented something like this.

The diffusion curve with the chasm

Figure 1 – Revised technical adoption cycle

This is not to suggest the elgg or e-portfolios are a bad idea, there is some value in them. If they are appropriate for the organisational context and they are adopted because of a large organisational need and not because someone heard or read something positive about the idea. They are a good idea if they are implemented in a way that engages with the reality of what the people within the place are dealing with and with what they are ready to do.

The following table is an example of the differences which Geoghegan talks about between the folk that work with the technologists’ alliance and those in the mainstream.

Early Adopters Mainstream
Like radical change Like gradual change
Visionary Pragmatic
Project oriented Process oriented
Risk takers Risk averse
Willing to experiment Need proven uses
Self sufficient Need support
Relate horizontally Relate vertically

The types of support and encouragement you give to the early adopters has to be radically different than that you give to the mainstream. Time and time again, I have heard senior university folk express the opinion “We’ll concentrate on the people that are keen”. This perspective only entrenches this gap, it only makes certain that the mainstream won’t engage. It’s a mistake.

There are many other differences between people that will impact upon perceptions and adoption. The process used to identify, design, develop and implement any technology, including e-portfolios, has to be aware of and work with these differences.

Failure to adopt or work – ignorance of process

E-portfolios, like many other fads, are becoming a solution looking for the problem. The process used to implement these fads within higher education goes something like this

  • There is a problem. A problem is identified and some one or some group is convinced that is important and requires that a project be set up.
  • Some analysis is performed. A small group of folk, generally from the technologists’ alliance with a few senior management folk added, go away and do some analysis. They may be helped by a consultant. Typically, the problem will have been framed so that the decision is a foregone conclusions. For example, rather than examining the question “How do we improve assessment?” the question will be “Which approach to e-portfolios should we take?”
  • A decision is made. That analysis will be used by some smaller group to make a decision. From now on, this is the goal towards which the organisation and its resources are focused.
  • The decision is implemented. A project group is set up to achieve the goal. The project group will be there to ensure that all work moves towards the goal. Than anything that is different is cut off.
  • Long period of support. To recoup the costs involved in making and implementing the goal the organisation then has to use they “system” for a long period. During this time organisational resources are focused on supporting the system (and little or nothing else).
  • Eventually it will start to drift. In some cases the people won’t want to use it, they aren’t convinced of the rationale. They may well appear to be working towards the goal, but they may simply be playing sufficient lip service so they don’t get into trouble. Then there will be the problem of “stable systems drag” (Truex, Baskerville et al, 1999) where the world has moved on and the goal no longer makes any sense. Alternatively, a new senior executive could arrive with different approaches and kill one set of fads for another set.
  • Another problem is perceived, and the process starts again.

You can see this with the notion of LMS churn going on with universities. In the 10 years since 1999 CQUniversity will have been through 3 separate process to replace an LMS.

The very nature of universities, the place, the people and of pedagogy makes this type of process completely and utterly inappropriate and destined to fail.

But that’s a general perspective that applies to just about anything within universities, what about e-portfolios in particular.

The wrong solution – the wrong product

The definition of e-portfolio used in the EDUCAUSE presentation is taken from Lorenzo and Ittelson (2005)

a digitized collection of artifacts including demonstrations, resources, and accomplishments that represent an individual, group, or institution.

For the majority of the technologists alliance within universities this means that the university must purchase or build and then support a software system (almost certainly referred to as an enterprise software system to give it that badge of respectability – even though it means nothing in terms of reliability, flexibility, suitability and probably more in terms of cost and constraints) that resides on university hardware and is badged with the university look and feel.

The major problem with this product approach is that it ignores One student = multiple learning experiences = multiple learning “institutions”.

In the typical e-portfolio product there is an assumption that the students’ only place for learning is the host institution. It ignores the observation that students attend multiple learning institutions (including work-place training) and it ignores that most learning is informal. In other words, an institution that plays a very small part in the learning of a student expects the student to place all of their “demonstrates, resources and accomplishments” onto the institution’s server.

What are we trying to solve – ignorance of purpose

The EDUCAUSE presentation gives the following primary uses of e-portfolios

  • Academic advising
  • Institutional accreditation
  • Curricular development at program level
  • Career planning and development
  • Alumni development

I have two main problems around these stated purposes

  1. One system, multiple tasks. One of my major problems with “enterprise systems” is that try to do everything. They try to be all things to all tasks and end up being really bad at all of them. I hear the information technology folk cry, “But they are all integrated!”. Yea, but no-one uses them because they are really horrible to use. The above list includes a number of very different tasks performed by very different people. The assumption that one system can perform all of these well, is somewhat questionable. Rather than put all tasks in one system, adopt a best of breed approach and put your effort into making sure they are integrated. This requires the IT folk (rather than the users) do some work. Of course, there is an alternate position that is based on a number of false assumptions, but that’s a story for another day.
  2. The institutional, not adopter focus. Take a look at the diffusion curve image above. The majority of people are not the innovators and early adopters. This applies to university academics. Most university academics are keen to do a good enough job in teaching their classes so they can concentrate on other pursuits. Look at the list of primary uses of e-portfolios. How many of them are going to be of direct interest to the majority of academics, most of the time? Perhaps academic advising, maybe institutional accreditation from time to time, but typically that’s only a small number of academics. Perhaps curricular development at program level? Most of the programs in my context don’t do this. Your context might be different. (Remember, place/context is important).

    How many of these tasks are seen as problematic by these academics? Remember the majority are very different from the innovators and early adopters. See the table above. Most people don’t want to radically change the way they are doing things. They will only consider radical change if there are huge problems with current practice.

    The same applies to students. How many of the above tasks are students directly involved with regularly? How many of these tasks do students currently have huge problems with?

Conclusions

There’s more that can be put in here. For those who are wondering, yes much of this content is related to my PhD. To a large extent I believe that there are large collection of mistaken assumptions that underpin most of the practice of e-learning within Universities. Many of the above complaints about e-portfolios can be easily applied to other technologies and how they are implemented within universities. More on this as the thesis progresses.

References

Geoghegan, W. (1994). Whatever happened to instructional technology? 22nd Annual Conferences of the International Business Schools Computing Association, Baltimore, MD, IBM.

George Lorenzo and John Ittelson, An Overview of E-Portfolios, ed. Diana Oblinger (Boulder, CO: EDUCAUSE Learning Initiative, July 2005), http://www.educause.edu/ir/library/pdf/ELI3001.pdf.

Truex, D., R. Baskerville, et al. (1999). “Growing systems in emergent organizations.” Communications of the ACM 42(8): 117-123.

Swanson, E. B. and N. C. Ramiller (2004). “Innovating mindfully with information technology.” MIS Quarterly 28(4): 553-583.

Page 1 of 3

Powered by WordPress & Theme by Anders Norén

css.php