The research "foundation" for the missing Ps

The work on the missing Ps is intended to contribute some research publications within the information systems field and also form the structure for chapter 2 of my PhD thesis. Consequently, it is important that the work have a good research foundation. This post is intended to provide that foundation, or at least document my development of the foundation.

The attempt is to develop a descriptive type of theory, a framework or taxonomy. Literature that talks about this type of research, appropriate processes and content include

The missing Ps – Process

The Missing Ps framework is my attempt to generate a way of identifying the flaws in the methods used by Universities to select an LMS.

In this section I’m expanding out my thoughts associated with process. It will include

  • The plan-driven assumption
    The almost automatic assumption that a plan-driven process will be adopted. The almost complete ignorance of the adaptive alternative.
  • IT governance model
  • Process and tool alignment
  • The importance of process change

The Plan-driven assumption

Over emphasis on plan-driven development at the cost of adaptive approaches
This is the one I’ve banged on about in previous papers (e.g. an early one, and the most recent one). The on-going acceptance of agile development methodologies, the Enterprise 2.0 meme incorporating emergence and the idea of rapid incrementalism from the two Johns make pushing this view a bit easier.

There are a number of reasons why this plan-driven approach is not appropriate to elearning

  • elearning, what it is and how best to do it are two very open questions.
    It is impossible to properly plan because what we know now is not going to be the best practice into the future. Examples include the arrival of blogs and social software onto the scene. Also LAMS and the learning design folks.
  • A large portion of any system cost is enhancement.
    This is a finding from software engineering research.

There are problems with the plan driven approach

  • The blind men and an elephant story
    The type of top-down design characteristic of plan driven development leads to the loss of the whole.

Some Peter Drucker quotes

from “The Effective Executive”

“Most discussions of the knowledge worker’s task start with the advice to plan one’s work. This sounds eminently plausible. The only thing wrong with it is that it rarely works. The plans always remain on paper, always remain good intentions. They seldom turn into achievement.”

“Innovation and Entrepreneurship”

“‘Planning’ as the term is commonly understood is actually incompatible with an entrepreneurial society and economy….innovation , almost by definition, has to be decentralized, ad hoc, autonomous, specific and microeconomic.”

IT Governance

The Wikipedia defintion

Corporate governance aims to:

  • align the actions of the individual parts of an organisation toward aggregate mutual benefit
  • provide the means by which each individual part of the organisation can trust that the other parts each make their contribution to the mutual benefit of the organisation and that none gain unfairly at the expense of others
  • provide a means by which information can quickly flow between the various stakeholders to ensure that the changing nature of both the stakeholder needs and desires and the environment in which the organisation operates get effectively factored into decision processes

Need to look more into the idea of IT governance and get some nice definitions.

How this is typically implemented in organisations is through a hierarchical committee structure. i.e. there is some central committee that meets to decide what should happen with IT. That committee has representatives from each part of the organisation. If someone at the coal-face identifies a way to make a potential large improvement their suggestion has to climb up the hierarchy, either within their part of the organisation or up the IT division, until it makes it to the committee. Along the way some small suggestions may get implemented, but anything that is large, crosses organisational boundaries or is a significant change from current practice will generally have to wend its way to the committee.

This is a rather negative description of the problem. On the surface this approach does appear to be somewhat logical if you buy into the rational model of people, organisations and decision making.

The problems with this model include

  • Even if it works it is incredibly slow.
    Often this committees meet no more than 4 times a year. Not exactly agile.
  • At any stage the idea can get knocked on the head.
    Any one of the people on the ladder up to the committee can kill off the idea, especially if it is perceived as significantly different, difficult or threatening.
  • It assumes the committee itself always acts rationally in the best interests of the organisation.
    e.g. the killing off of the AIS idea.
  • It assumes that the decisions of this committee will actually be followed and acted upon.
  • It assumes that people at the coal-face will even bother to start the ball rolling
    This working paper from the Harvard Business School reports on research that indicates that people don’t speak up.

    Qualitative data collected in 190 interviews with employees from all levels and functions suggest that fear of speaking up, even with pro-organizational suggestions, is pervasive and, for many, a source of intense negative affect.

Process and tool alignment

When you’ve been told to use a particular tool, you can’t use a process that doesn’t fit the tool. Or if you do there are going to problems of inefficiency or poor quality.

The implication for higher education when adopting an LMS (or any information system) is that how things are currently done and the system being adopted have to achieve some sort of alignment.

The problem with the implementation of many information systems is that it is assumed that it is the tool that cannot be changed and instead how things are done in the organisation must be changed to meet the tool. With enterprise resource planning systems this is seen as a good thing because these systems are meant to encapsulate “best practice”. But this assumption is highly questionable.

It also ignores the difficulty of forcing process change on an organisation. Especially when the organisation is full of knowledge workers, like a University. This difficulty often means that process change is often ignored or not completed and consequently leads to inefficiencies and poor quality.

Need to mention the idea that the organisation now becomes captive to the system. When the system changes, often due to the vendors needs, not the organisations, the organisation must go through yet another round of difficult process change (or simply ignore it).

There is an alternative. Modern information technology can be implemented in a way when it becomes significantly more malleable than previously thought. It is possible to mould the technology to fit the organisation. This makes it possible to enable a conversation between the organisation, it’s members and the technology where both are modified to provide significantly improved processes. .. this needs to be expanded.

The importance of process change

One way to build a comprehensive model is to place IT in a historical context. Economists and business historians agree that IT is the latest in a series of general-purpose technologies (GPTs), innovations so important that they cause jumps in an economy’s normal march of progress. Electric power, the transistor, and the laser are examples of GPTs that came about in the nineteenth and twentieth centuries. Companies can incorporate some general purpose technologies, like transistors, into products, and others, like electricity, into processes, but all of them share specific characteristics. The performance of such technologies improves dramatically over time. As people become more familiar with GPTs and let go of their old ways of thinking, they find a great many uses for these innovations. Crucially, general purpose technologies deliver greater benefits as people invent or develop complements that multiply the power, impact, and uses of GPTs. For instance, in 1970, fiber-optic cables enabled companies to employ lasers, which had already been in use for a decade, for data transmission. (McAfee 2006)

What should the institution provide in a Web 2.0 world?

Clarence Fisher has an interesting experiential post about the difficulties that start to arise when using a lot of cool, new “Web 2.0” apps in a teaching/learning context. Free(ish) services he lists are blogs, wikis, podcasts/vlogs, superglu, flickr etc.

But it’s not all free.

But we do pay, having to access different accounts across the web. The kids need to remember URLs, passwords, and how to navigate through different interfaces. They need to remember how to run WordPress, make a photostream in Flickr, add blogs to their aggregator, and format a wiki.

His reason for not using an integrated system?

I have consciously not worked with a single piece of software such as Moodle because I wanted kids to see the power of collecting resources across the web into a single environment

He sees the need for an interface that would combine the pieces.

When I thought/dribbled about Web 2.0 course sites one initial negative thought was, “why keep the whole course structure idea?”. Isn’t this an example of the horseless carriage problem. That’s how we did it the old way, so we must do it that way in the new Web 2.0 world?

Sort of, yes. Most people, students and staff, aren’t going to make that leap straight away. They need the comfort of the familiar to ease the pain.

As the above post points out. Integrating these disparate applications together is going to require some glue to reduce the difficulty of using it. Otherwise it won’t work.

Starting my "blogs for learning" article about BAM

blogs for learning is an “online resource about instructional blogging being set up by Michigan State University. I’ve volunteered/been asked to write an article about the use of blogs in COIS20025 and BAM.

This post where I’m going to start my writing.

Thinking about the article

Would be nice to link in with some of the previous articles on the site. Here’s a summary of articles I might link to and why

The site also contains various other posts, some of the ones to refer to include

  • Nicole Elison’s post on Non-Pseudonymous Student Blogging: Ethical and Legal Concerns
    The advantage of BAM is that students public blogs can be anonymous. The identification of which student owns which blog is only visible to CQU staff. Of course, in the absence of any specific advice students have tended to name their blogs using their student identifier.

What will I write about?

  • Why and what we did
  • How it worked out
  • What’s next

Additional improvements to BAM

It’s almost the end of term. We’ve started marking the blogs using BAM and not surprisingly some rough edges are showing. Things to look at

  • Merging feeds
    Currently BAM simply copies over the local mirror of the feed with the new one. This causes problems because WordPress, and likely other providers, only provide a feed with the last 10 posts. The solution would be to merge the new feeds with the local mirrored copy. XML::Feed, the Perl module used in BAM, offers the splice method to do this. So should be easy.
  • Making copy detection, pro-active
    I find it hard to believe but we have had students who have copied the blog entries of other students. At least half a dozen. Now that it is written, need to have this run as a more regular event and notify students as soon as it happens.
  • Increase visibility to students
    The students don’t really see anything from CQU’s end. This causes problems in terms of whether they’ve completed the questions etc. Implement a view and email notification system that gives the student a visible view of CQU’s summary of where they are up to. Notify students when the miss certain events.
  • Remove the special characters
    Students tend to cut and paste their posts from Word. This introduces special characters which, after the blog has finished with them, can cause problems.

Who is to blame for plagiarism: technology, lecturers or context?

At the Plagiarism Conference 2006 Baroness Ruth Deech, the first Independent Adjudicator for Higher Education gave a presentation which has been widely reported on the Web and in the press. It was interesting to note that the conference was co-sponsored by Turnitin’s UK arm. Rather than sound entirely negative the conference proceedings are online and, from the titles, there looks to be some good stuff.

Some of the comments online, especially from Digg and ArsTechnia which both include a fair number of student comments.

It is being widely reported, because of the title of the Times article, that plagiarism is the fault of indulgent lecturers. The Chronicle of Higher Education also carries the story with some interesting comments.

It suprises me that such views are getting such airplay as they miss the point entirely. My problems with these views include

  • The idea that there is a single cause of academic misconduct
  • That the primary cause is rooted in either the students or the teachers
  • The idea of placing blame is not exactly a helpful way to frame the conversation
  • That a change or introduction of technology will reduce academic misconduct

What was said

First, in the press Baroness Deech’s views are often reported along with a comment from Gill Clarke of the UK “education watchdog the Quality Assurance Agency”. From the Times article

Gill Clarke, of the education watchdog the Quality Assurance Agency, said that students should be made aware that most universities were using detection software, such as Turnitin, to spot plagiarism.

Baroness Deech’s views include the following, taken from an article in the Australian newspaper

“There is a culture of expectation among today’s students,” she says. “They just take whatever is put in their hands, be it a handout or a PowerPoint presentation. That way you end up boiling down complex things to three bullet points.

Taking down notes in longhand from a book is better than cutting and pasting from the internet, she says, because it requires students to digest material.

Taking notes is the solution

This point basically ignores most of what the educational literature has to say about learning. Learning isn’t enabled by copying content by hand. Information distribution is not learning.

The limited influence of copy detection software

In the course I’m currently teaching we have used copy detection on all assessment items. Part of the assessment is designed using strategies suggested as best practice for helping minimise plagiarism.

Teaching staff have repeatedly mentioned that copy detection software will be used on all assignments. It’s mentioned in the course profile and in the course discussion forum.

Currently (term not quite finished) the copy detection reports on the 3 assignments are showing about 40 students (about 14% of the class) have something that might be academic misconduct. The course staff already have 17 offences entered into the institutions system for managing academic misconduct. That 17 only includes assignment 1 and a small part of assignment 2.

For assignment 2 the students were provided with 3 references (including direct links to online versions) to help them with a report. The copy detection report shows students with reports where 20%, 23%, 49%, 58% and 70% of the report is a direct copy from one of the references we provided.

What should change?

Changing the technology doesn’t appear to make any difference.

Baroness Deech comes close to a possible solution when she says

Students need to be told that their own thoughts about a subject are very important. They need to be challenged to respond in their own way instead of downloading, cutting and pasting. The weighing up of views, the encouraging of nonconformity, the imbuing of intellectual tradition of inquiry are getting lost. If lecturers can imbue students with the view that they are searching rather than copying, then we might go some way towards tackling plagiarism.

The only way I see that we can “imdbue” our students with this view is if our primary form of teaching moves away from the information distribution model that emphasizes traditional lectures and textbooks and move towards more modern, effective pedagogies.

The Program in Course Redesign was a project that used technology as the basis for 30 different US Universities to redesign their instructional approaches to achieve cost savings and quality improvements.

The results from the last round included

  • Average savings of 39%
  • 8 of the 10 projects reporting improved learning outcomes, the other two reported no significant difference
  • 5 of the 10 projects showed improved course completion, 1 reported no change, 1 showed reduced completion (the other 3 did not measure course completion)

From the lessons page

All ten projects have effected significant shifts in the teaching-learning enterprise, making it more active and learner-centered. The primary goal is to move students from a passive, note-taking role to an active, learning orientation. Lectures are replaced with a wide variety of learning resources, all of which involve more active forms of student learning or more individualized assistance. In moving from an entirely lecture-based to a student-engagement approach, learning is less dependent on words uttered by instructors and more dependent on reading, exploring, and problem-solving undertaken actively by students.

Why isn’t it being done – who is to “blame”?

The type of change described by the Program for Course Redesign is large scale innovation within existing organisational settings. This type of thing is extremely difficult and time consuming and involves a great deal of battling against organisational norms. How is a poor academic expected to do something like this without organisational support at the highest levels?

For example, think of yourself as an academic who is thinking of implementing this sort of change. But your organisational context has the following characteristics

  • Increasing emphasis on research.
    The rhetoric from senior management is that research, in the form of top journal publications, is important. You hear from selection committees for new jobs at your institution using the publication/research record of new applicants as the main criteria.
  • On-going restructure and review
    For the last 4 or 5 years the academic unit and the institution as a whole has been either on-hold waiting for, or involved in organisational restructures and the introduction of new senior management. You’ve just heard that there are also going to a number of other reviews in the coming months.
  • Limitations of current models and processes
    The complexity of your institutions teaching means that there are a range of support units and other commercial organisations which manage aspects of teaching delivery. Those units have adopted specific models for the delivery of teaching that enshrine the traditional lecture/tutorial approach for face-to-face delivery. For a range of reasons they are very reluctant to change those models.

In this context, is it really the fault of the academic if he/she decides that it’s better to concentrate on the research than trying to actively engage with the large scale innovation around teaching and learning that is necessary to seriously address this issue?

Even if the issue is made to engage, how is it suggested that necessary organisational changes be made?

The missing Ps – People

LMS adoption decisions are made by people. LMSes are used by people. The nature of those people who they are, how they think and what they believe, amongst other characteristics, have a significant impact.

They myth that people are rational

Extensive research shows that our brains have certain hardwired propensities that might be exploited. For example, our brains tend to register frequently heard facts as true, even if they are patently false. As a result, our memories and beliefs are highly malleable and unreliable. We also tend, if unchecked by the conscious reasoning mind, to focus overly on risk, inconvenience, hassles—anything negative. And researchers have found that we all carry around an innate hostility toward “otherness,” which means anyone not like us. (Herbert, 2006)

An alternative is bounded rationality which suggests that people employ heuristics in decision making rather than strict optimization. This is in response to the complexity of the situation is to great and the individual is unable to process and understand all of the alternatives.

Various fields including cognitive science and social psychology have identified a range of cognitive biases that distort how people perceive reality. Wikipedia provides a list of cognitive biases. The following list includes some related to the LMS adoption decision – they are from the decision-making and behavioural biases section of the list

  • Bandwagon effect – tendency to do things because many others are doing it.
  • Bias blind spot – tendnecy not to compensate for one’s own cognitive biases
  • Choice-supportive bias – the tendency to remember one’s choices as better than they actually were
  • Confirmation bias – the tendency to search/interpret information to confirm your pre-conceptions
  • Congruence testing – the tendency to use tests that only confirm your hypotheses rather than disprove it, or prove another
  • Deformation professionnelle – the tendency to look at things through the conventions of your own profession with no thought to the broader view
  • Disconfirmation bias – tendency to critically examine information that contradicts your prior beliefs but accept uncritically information congruent with your beliefs
  • Endowment effect – the tendency for people to value something more as soon as they own it
  • Focusing effect – prediction bias when you place too much importance on one aspect
  • Hyperbolic discounting – tendency to have a stronger preference for immediate payoffs relative to later payoffs
  • Illusion of control – tendency to believe you can influence outcomes over which you have no control
  • Impact bias – tendency to overestimate the intensity of impact on future states
  • Information bias – tendency to seek information even when it cannot affect attention
  • Loss aversion – tendency to strongly prefer avoiding losses over acquiring gains
  • Outcome bias – tendency to judge a decision but its eventual outcome instead of on the quality of the decision at the time it was made
  • Planning fallacy – tendency to under estimate task completion times
  • Post-purchase rationalisation – tendency to persuade oneself through rational argument that a purchase was a good one.
  • Pseudocertainty effect – tendency to make risk averse choices if the expected outcome is positive, but make risk-seeking choices to avoid negative outcomes
  • Status quo bias – the tendency for people to like things to stay relatively the same
  • Zero-risk bias – preference for reducing a small risk to zero over a greater reduction in a larger risk

But let’s not leave out the social biases

  • False consensus effect – tendency to overestimate the degree to which others agree with you
  • Fundamental attribution error – tendency to over-emphasize personality-based explanations for the behaviour of others while under-emphasizing the role of power and situation influences
  • Halo effect – the tendency for a person’s positive or negative traits in one area to influence your perception of them in other areas
  • Illusion of asymmetric insight – the perception that your knowledge of your peers is greater than their knowledge of you
  • Illusion of transparency – you overestimate your ability to know others, and they overestimate their ability to know you
  • Ingroup bias – preferential treatement to people who you percieve to be part of your group
  • Project bias – tendency to unconsciously assume that others share your thoughts, beliefs, values or propositions
  • Self-server bias – tendency to claim more responsibility for successes than failures.

The missing Ps – Past Experience

While elearning is still relatively new there has been significant levels of implementation and research around elearning. It seems fairly obvious that any LMS adoption process should make use of the knowledge generated by this past experience.

The categories within this missing P include:

  • Quality benchmarks
  • eLearning Maturity Model

Quality benchmarks

A research project, funded by the National Education Assciation and Blackboard, from the Institute for Higher Education Policy in the USA drew on the experiences of pioneers in elearning from 6 institutions of higher education and developed 24 quality benchmarks spread across 7 categories (Phipps & Merisotis, 2000):

  1. Institutional support
  2. Course development
  3. Teaching and learning
  4. Course structure
  5. Student support
  6. Faculty support
  7. Evaluation and assessment

The specific benchmarks provide a very interesting set of questions to ask in the presentation.

eLearning Maturity Model

Work coming out of NZ and finding application in wider environments is application of the CMM into elearning. Stephen Marshall has a good summary of his work in this post.

In particular, it talks about the absence of process and the waste of money associated with the traditional, individual coordinator approach to elearning. Good stuff to include.

References

Phipps, R. and J. Merisotis. (2000). “Quality on the Line.” Retrieved 25 October, 2006, from http://www.ihep.org/Pubs/PDF/Quality.pdf

Moving beyond functional evaluation – scenario-based evaluation

Tony Byrne has posted an article entitled A Scenario-based approach to evaluating CMS vendors. It’s part of a website that examines/reviews content management systems. The post tells the story of how they evolved their evaluation of such systems from strictly functional evaluations to the addition of “vendor intangibles” and beyond.

Their reason for moving beyond this?

But still, the methodology remains incomplete, because it doesn’t explicitly answer key situational questions. When does a larger or smaller vendor make better sense? When does workflow matter more or less? More pointedly, when does a robust (read: complex) workflow subsystem become a drawback rather than an advantage?

Their solution is to add 12 common scenarios for implementing a CMS. Essentially 12 different type of contexts/organisations requiring a CMS and a list of the characteristics of those organisations that differ and may effect the selection of a CMS.

Obviously there’s a connection between this and the selection of a learning management system (LMS). Not all Universities are the same, not all Universities have the same aims or problems.

They need to think about their purpose and place

The missing Ps – Purpose and place

A continuation of the elements of the “missing Ps”.

This one looks at the Purpose and Place. The idea is that before a University starts thinking about selecting an LMS, or even starts to question whether or not an LMS is what it needs, it should think long and hard about the purpose to which it will put the LMS (need to try and avoid that label). In doing so it needs to think hard about what it considers to be the purpose of information technology and all of this needs to be driven by consideration of place, the context of the individual University.

Sections include

  • What’s the purpose of the tool
  • What’s the purpose of IT
  • Place – moving beyond purpose

What’s the purpose of the tool

Suppose I give you the choice of three vehicles

  1. Ferrari Enzo
  2. Humvee
  3. Toyota Prius

and I ask, “Which is the most appropriate car?”

The response of most people will be something like, “Most appropriate for what?”. An Enzo is a great car for completely different purposes than a Humvee and a Prius. The same applies to information systems.

You can’t choose the most appropriate tool unless you know what the purpose is.

For an elearning information system it’s often argued, quite logically, that the purpose/focus should be on learning. It also happens to be one of the most forgotten aspects.

One of the most crucial prerequisites for successful implementation of e-Learning is the need for careful consideration of the underlying pedagogy, or how learning takes place online. In practice, however, this is often the most neglected aspect in any effort to implement e-Learning. (Govindasamy 2002)

George Siemens (2006) suggests a review of LMSes should start with an organisational definition of learning, created through input from all stakeholders, as the foundation of decision making and the boundaries of platform selection.

A concern for learning is certainly an important consideration for an organisation making choices around technology to support education. But there are problems created by trying to start with an organisational definition of learning. Many of these problems are associated with this being an example of a plan-driven process. It assumes that it is possible for an institution to define a single view of learning and that such a single view is actually a good thing. Both are somewhat questionable and are (or will be) addressed in more detail in the process section of the missing Ps framework.

While an emphasis on education is essential it is not the only potential area of consideration for an organisation attempting to choose a technology platform for elearning. An organisation like CQU that has undergone considerable growth and expansion in the complexity of its offerings has a number of other problems and future aims to consider. There may be other business needs to consider.

Once the company’s business needs are clear, the technologies it requires will come into focus. (McAfee, 2006)

McAfee goes onto say in his Harvard Business Review article

I recommend that business leaders accept three IT-related responsibilities: selection (picking appropriate new tech investments), adoption (getting the technology ready for first use) and exploitation (maximizing value delivered once the system is up and running). Rules of thumb are:

  • During selection, work ‘inside-out’ by first determining what IT-based capabilities are required, then looking out at the technology landscape. This is in contrast to an ‘outside-in’ approach that attempts to determine IT needs by examining available technologies and deciding which ones to bring in.
  • During adoption and exploitation, put in place the organizational complements that will maximize the postive impat of the selected technology

The missing ingredients from most e-learning programs are clear and measurable objectives and cohesive strategies. Before an organization can evaluate any offerings from an e-learning provider or implement any internal initiative, it must first create a cohesive strategy that clearly defines and documents the value each program must deliver—before any program moves beyond the concept stage (Ismail 2002)

What’s the purpose of IT

Executives usually operate without a comprhensive model of what IT does for companies, how it affects organisations and what they must do to ensure that IT initiatives succeed (McAfee 2006).

Place, moving beyond purpose

Much of the enterprise systems IT world, including the LMS idea, is based on the assumption that a single piece of software and its encapsulated processes are equally applicable to just about every different organisation. This assumption ignores the central importance of context. In fact, it assumes that the type of organisation and the characteristics of that organisation and its environments are either the same or will exert no influence over the potential fit or success between the organisation and the enterprise system.

ERP systems grew out of the manufacturing industry. …need to say more about that type of organisation and then talk about the move of ERPs into other organisations eventually leading up to Universities and the characteristics of those institutional types. The various references I have will fit in nicely here.

Even within one organisation type there is significant difference due to a range of factors associated with the place and purpose of the University. For example, CQU is a very different organisation than MIT. What makes sense for MIT, does not make sense for CQU.

Scenario-based evaluation covers some aspects of this.

References

Govndasamy, T. (2002). “Successful implementation of e-Learning pedagogical considerations.” Internet and Higher Education 4(3-4): 287-299

McAfee, A. (2006). “Mastering the Three Worlds of Information Technology.” Harvard Business Review, November 2006.