LMS usage data and value

Warger (2003) (amongst others) makes the claim

Best yet, the learning materials and student progress data that accumulate through use of the CMS become an ever-increasing source value to institutions of higher education.

Questions that arise

  1. Does the progress data actually provide anything useful for organisations?
  2. Is it the only source of such data? What about logging into the computer network, using other administrative portals?
  3. How effectively can organisations gain access to this data? Are there tools available? Who do these tools “serve”, teaching staff, managers?
  4. Are there any organisations actually using this data?
    I know CQU isn’t really, at least not beyond the odd academic. Recently CQU discovered that some of the logging functionality in Blackboard hadn’t even been turned on. I have seen some papers (e.g. ASCILITE 2006) talking about how the data has been used post term, but haven’t yet seen anything about during term at an organisational level.


Tom Warger, Calling All Course Management Systems University Business, July 2003

Evaluating the drivers for LMS adoption

Coates, James and Baldwin (2005) identify/propose 5 drivers behind the adoption of LMS

  1. Means of increasing the efficiency of teaching
  2. Promise of enriched student learning
  3. The drive of new student expectations
  4. Competitive pressure between institutions
  5. A response to massive and increasing demands for greater access to higher education
  6. A culture shift towards the control and regulation of teaching

The questions around these include

  • How many, if any, institutions have stated these explicitly?
  • Have they attempt any sort of evaluation about how well their choice and implementation have achieved those goals?
  • Can we evaluate the current crop of LMS and their ability to respond to those drivers?

The last point is potentially the most interesting. Some initial thoughts

  • Do current LMS really provide the facilities that enable an institution to control and regulate teaching? How many actually use this facility?
  • Do current LMS really provide scope to improve T&L, provide greater access, or reduce costs?

In a recent survey of CQU staff we found the following somewhat relevant results

  • Enriched student learning/improved teaching and learning
    When asked if adoption of Blackboard would assist CQU become a flexible learning leader (one of its strategic aims) responses included

    • 56% of staff didn’t know if this was the case
    • 25% thought it unlikely
    • 17% thought it likely
  • Control and regulation of teaching
    Which organisational unit should be responsible for quality of the LMS

    • 61% the relevant faculty
    • 27% division of teaching and learning services
    • 12% information technology division

    “Do you believe that implementing Blackboard is a way to place additional controls on teaching and learning activities at CQU?”

    • 44.5% yes
    • 13.3% no
    • 10% no response
    • 32.3% no difference

Is pedagogical the only lens

Christian Dalsgaard in a 2006 paper suggests

Social software has initiated discussions about the extent to which tools should be separated or integrated in systems (see Levine 2004; Blackall 2005; Cormier 2005; Wilson 2005; Siemens 2005; Anderson 2006a; 2006b). However, the discussion will find no answer, unless it is placed within a context of pedagogy. Use and organization of tools within e-learning can be approached in different ways depending on the chosen pedagogy (Dalsgaard, 2005). Different pedagogies will have different things to say about the problem of integration vs. separation. A discussion of the educational value of different tools must use a pedagogy as a starting point.

There is certainly value in this perspective. But I’m not sure it is the whole picture and not sure it can even be used as the only criteria because

  • There is no one pedagogy that can be used for all courses at a University.
    Assuming of course that you are coming at this from an organisational perspective, I cannot see how any university could argue that there is one pedagogy that all staff in all courses would use. Or that there is “one right way” that should be used in all courses.
  • Pedagogy is not the only reason considered
    Again from an organisational perspective, pedagogy is not the only consideration. Certainly most decision making about technology adoption is not rational. For decision makers there will be other reasons why a technology is (or is not) chosen.

LMSes mostly for administrative purposes

The OECD report “E-learning in Tertiary Education: Where do we stand?” says that universities have primarily used LMSes for adminstrative purposes and that LMSes have, so far, had a limited impact on pedagogy.

There are three potential avenues for exploration in that finding

  1. Is it actually true?
    Some of the surveys I saw reported in the ASCILITE’2006 proceedings seem to offer supporting evidence for this. Though this might be a case of not getting to carried away with technological determinism. i.e. it’s not the LMS alone that is contributing to this. The way in which universities are introducing, supporting and encouraging T&L also play an influence, if not the biggest.
  2. Are LMSes actually well suited for adminstrative purposes
    Experience at CQU demonstrates that they are not particularly good at helping teaching staff perform their adminstrative purposes. e.g. early experience with a grade book limited to 999 students in classes with 1000+ students. On-going experience with their online assignment systems compared with CQU’s OASIS system.

    The design of these systems is essentially teleological. The design encapsulates some purpose. There’s a difference between the purpose of the Webfuse web team and that of the LMS development team. LMS development team is trying to design something so general that is widely usable by most universities, within the limited resources they have. The web team try to develop something that best suits the majority of staff at CQU within the limited resources.

    This difference in aims leads to a different type of outcome. There’s also something here about how IT people tend to want to design for the general case as early as possible.

  3. How easy is it to use current systems for pedagogically interesting innovation?
    Some of the discussion/literature indicates it is very hard. Even with Moodle, a tool designed to support a social constructivist approach to learnign

teleological and “unbiased design”

There’s been a bit of rhetoric from LMS vendors that their technology is pedagogically neutral. The socio-technical folk argue against that for a variety of reasons and have demonstrated it in numerous ways.

This could be worthy of further research. Something along the lines of

  • All systems are designed with a purpose – they are all teleological
  • The resulting system embodies that purpose because it drives the decisions that are made during the design and development
  • This can be demonstrated by comparing similar tools, design by different groups for different purposes. e.g. assignment submission in Webfuse versus Blackboard.
  • It can be extended to examining the stated purposes of the system owners. e.g. the intent behind the design of the ABM system within Webfuse driven by management and consequently rejected by academics.

Rationale for the Web 2.0 Course Site approach

Pittard (2004) talks about the UK government’s plans for e-learning (my bad description). It includes the following

It prioritises seven strategic action areas that are necessary to embed elearning
effectively across all sectors:

  • leading sustainable e-learning implementation;
  • supporting innovation in teaching and learning;
  • developing the education workforce;
  • unifying learner support;
  • aligning assessment;
  • building a better e-learning market;
  • assuring technical and quality standards.

The first two, but especially the first point, provide support for the “web 2.0 course site” idea – i.e. e-learning provided through using SaaS (software as a service) to bring together existing services on the web.

Sustainable means, at least in part, affordable.

Pittard (2004), herself, my emphasis

These represent the focus for central government action. However, actions, including many of those above, are necessary at all levels to ensure that the potential of ICT in education is realised. So the e-Learning Strategy in England is part of a larger policy landscape. It represents a leading voice in a dialogue about the actions required at different levels in order to realise the potential of ICT and to realise it efficiently – that is, add value to the system without incurring large extra costs.


Vanessa Pittard (2004). Evidence for E-learning Policy. Technology, Pedagogy and Education. 13(2): 181-194

Rationale for the Web 2.0 Course Site approach

Pittard (2004) talks about the UK government’s plans for e-learning (my bad description). It includes the following

It prioritises seven strategic action areas that are necessary to embed elearning
effectively across all sectors:

  • leading sustainable e-learning implementation;
  • supporting innovation in teaching and learning;
  • developing the education workforce;
  • unifying learner support;
  • aligning assessment;
  • building a better e-learning market;
  • assuring technical and quality standards.

The first two, but especially the first point, provide support for the “web 2.0 course site” idea – i.e. e-learning provided through using SaaS (software as a service) to bring together existing services on the web.

Sustainable means, at least in part, affordabl.

SaaS, Consumer products, shadow systems and e-learning

In a recent post I commented on the trend around how consumer driven computing is driving the development of “software as a service”. In particular, pointing to an article from the Economist that talked about how Arizona State University was using Google Apps to host email accounts for their students.

What I want to do here is to link this trend back to individual academics and how they may use these services to work around institutional structures and approaches to develop shadow systems.

A recent quote I came across from Somekh (2004) summarises why and links nicely with some work some colleagues and I have done previously.

But activity theory goes further to explain the way that institutional structures within national systems, with functions as diverse as education and the postal service (Engeström & Escalante, 1997), construct and constrain the interrelationship of humans and ICTs in mediated activity.

In that some work we built on work by Behrens and Sedera (2004) an examined how a particular shadow system rose and fell (it has since risen a bit further, but may well fall again). Behrens and Sedera (2004) generated a little framework which explains the factors that cause shadow systems to be created. It’s shown in the following figure. We attempted to show that rather than being objects of scorn, things to destroy. Shadow systems are actually important indicators, canaries in the coal mine, to demonstrate that there is a gap between user requirements and the service being provided.

Factors causing shadow systems in an ERP context

Given the diversity in universities there will always be a gap between what academics want to do with technology than can be provided by the support divisions at universities. More importantly, is that there may well be an even larger gap between what universities provide and what students want. Particularly, given the recent rhetoric around the net generation.

So, what’s new? That gap has always been there.

The difference is the growing influence of consumer computing, software as a service, websites/services such as YouTube, Google Video and the rest of the Web 2.0 avalanche, the increasing ease-of-use of personal computers and applications to create and manage multimedia resources and the growing capability of people to use these tools.

This is a significant change. In the early days of the Internet it was the universities that led the charge, that developed the innovations. Jones and Johnson-Yale (2005) give a brief overview of this in their introduction. An interesting aspect is that initially it was the staff driving the innovation, they then talk about the students going out and driving it further (e.g. Shawn Fanning and Napster, Brin and Page at Google).

John Pederson expressed this in a different, graphical form which was picked up by Scott McLeod which includes a Jack Welch quote.

All these changes increase the “resource” and “support” intervening conditions in the above figure. i.e. they make it easier for the staff and students to “fill the gap” between university provided IT services and what they want to do. University IT is no longer the only shop in town. Increasingly academics and students will ignore University IT services.

This is a change that needs to be countered. Initially, the university hierarchy will probably see this change as a “threat” and counter it by banning use of services. I can see university technical staff playing around with routing tables and firewalls to prevent use of sites. This will be counter productive. Obviously not an option that I would suggest.

Instead, university IT and related services should recognise that this change is coming and instead of banning it, they should on board. Recognise the potential benefits that this change may bring and figure out how they will operate in this new world.

This is especially important as this change may offer an opportunity to address the problem of funding, which is rated #2 in the Top 10 issues facing university IT support.

Potential University Responses

There are numerous approaches a university could take in responding to shadow systems and this particular issue. The following is a start of a spectrum of options:

  • Outlaw it
    All such systems are bad and should be crushed.
  • Ignore it.
    Shadow system, what shadow system?
  • Piecemeal adoption
    Some sections adopt and use it.
  • Adopt it
    Investigate and potentially adopt and support it centrally.

Collaboration, Design and Innovation

A nice quote for REACT comes from Campbell, Gibson and Grammlich (2005)

Like Albion & Gibson (1998), who maintain that individual faculty, sharing innovative teaching methods, can encourage others ‘to acquire the insights which will enable them to adapt their own practice’ (p. 1), we believe that faculty engagement in a design process provides a catalyst to change in understanding and practice. In fact, the sharing of experience through a social, relational process of collaborative conversation (Feldman, 2000) goes beyond the adaptation of practice to innovation, to the transformation of practice through innovation.

The resonance with REACT, or at least the future planned implementation of REACT, is with process

  • Get groups of staff, from diverse disciplines, wanting to develop elearning stuff together in REACT sessions
  • Structure the sessions around something like
    • Analysis – figure out what I want to do and what’s important
    • Design – figure out how I’m going to do it and evaluate it
    • Implementation – talk about how it’s going and what I might need to change
    • Analysis – talk about what happened and think about the next step
  • Have as the outcome from each session part of a REACT paper that is going to be finally published, well, at least submitted for publication.

As well as a diversity of academic staff, the sessions should have a diversity of support staff, including ICT, staff development, educational designers etc. This connects with the idea of connectivism. i.e. that knowledge is not resident in any one person, but in a range of people.

It also has connections with informal learning. In that the sessions would have worked into them a range of activities to maximise informal learning. i.e. minimise the formal talk head presentations, maximise ad hoc, informal discussion amongst attendees.

Might have to link this in with a prep session. A session intended to encourage staff to increase knowledge about themselves. Some components of such a session might include

  • Kiersey temprement tests
  • Learning styles
  • Cultural knowledge – i.e. a recognition that staff background and culture creates/comes with some baggage that needs to be recognised
  • Learning theory – connect all of the above with some basic learning theory. Not sure of this, but there is a need to attempt to provide participants with at least some common language.

Following on, another nice quote from the same source which summarises nicely the aim of the REACT sessions.

The purpose of conversation is to establish a community, the source of
power and meaning in society (Tannen, 1990; Minister, 1991). Lyotard
(1984) talks about the non-confrontational style of conversation that is in
contrast to a traditional, rational academic discourse. A design conversation,
possible in a relationship of mutual learning and trust, is one in which
‘nothing is at stake’ in terms of intellectual or moral authority. This powerfree
form supports and enhances the creativity of instructional design

And another

Conversation is
also a situated activity, as human activity is about striving to make meaning
out of the experiences of daily life, ‘to construct and reconstruct the
narratives of our experience’ (Goldsworthy, 2000, p. 101, after Bruner,
1990). By ‘writing ourselves into our own work as major characters’, we
challenge both ‘accepted views about silent authorship, where the
researcher’s voice is not included in the presentation of findings’ (Holt,
2003, p. 2), and the idea of instructional design as a rational, technical

And still some more

Katy and Sue describe characteristics of communities of practice that
include negotiation, intimacy, commitment and engagement (Kowch &
Schwier, 1997) and cross ‘the boundaries of formal power and status’
(Heckscher & Donnellon, 1994, p. 142, in Kowch & Schwier, 1997). In
this view the instructional design process is supported in a community in
which values and knowledge become aligned through the mutuality and
reciprocity of conversation. The design conversation is a negotiation of
personally held views that integrate and scaffold both social and cognitive
domains (Fosnot, 1996), and in which ideas are shared as a ‘form of
cultural learning or collaborative learning’ (Ewing et al, 1998, p. 10). As
knowledge is personal, through conversation we are able to share personal
representations of identity, values and intentions that will enable the work
of design to proceed.

And some more

Kowch & Schwier (1997) note that the ‘sharing of ideas, however they
may be expressed interpersonally or technologically’ (p. 3) define a
community of mind. These communities are often found in academic
contexts, where researchers come together in conversation to ‘grapple with
a shared research issue or problem’ (p. 3). Bruffee (1986) characterizes
such patterns of conversations as connected knowing and a site for
constructing knowledge. This aspect of conversation reflects its centrality
as a cognitive process in which we work together to decenter, moving
beyond personally held views to construct new and expanded
representations (Fosnot, 1996).


Kath Campbell, Susan Gibson, Catherine Gramlich (2005). On Conversation and Design: A socially constructed practice. Technology, Pedagogy and Education. 14(1): 9-24

Prototyping a Web 2.0 Course site

Goold and Auga (2006) describe an experiment in prototyping of a transformation of a face-to-face course into an online offering.

This looks like an interesting template to use in trying out the Web 2.0 Course Site idea. The process would be something like

  • Identify an existing course website or two
  • Figure out how to convert them to a Web 2.0 course site
    Which in itself encapsulates a range of interesting trade-offs. Do a purely technical translation? Aim for something more transformative?
  • Grab a collection of students and staff to trial it out
  • Use the feedback to return to step 2.

Software as a service – Google apps for your domain

The Economist magazine ran an article on Dec 19th, 2006 which is sub-titled – “Consumer technologies are invading corporate computing”.

The main story is about the head of IT at Arizona State University and his adoption of Google Apps for Your Domain which includes Google Apps for Education. The Apps For Education site has a page which includes quotes from various folk, including someone from Arizona State.

While that’s an interesting story, and one I think will be interesting to see play out at CQU (at least for those of us who work there), it’s only part of the bigger picture.

The bigger picture includes a continuing move up the abstraction layer, and as the sub-title of the article suggests, a change in where innovation is coming from.

The Abstraction Layer

The history of computing is replete with continual moves up (and across) the abstraction layer/spectrum.

The 50s saw very low-level hardware manipulation required to do useful things with computers. Slowly we moved up the abstraction layer, in programming with machine language, assembly language, Fortran and associates – slight move up to C++ and Java and then scripting languages.

The advent of component-based software, to me, is a sideways step, a dead-end. Why? Well let’s take a look at part of the Wikipedia page.

A simpler definition can be: A component is an object written to a specification. It does not matter what the specification is: COM, Java Beans, etc., as long as the object adheres to the specification. It is only by adhering to the specification that the object becomes a component and gains features like reusability and so forth.

To me that means, the component must be written in one of the specifications: COM, Java Beans, etc. Once written it cannot be easily reused in other “specifications”. To get full use of the component approach the organisation typically picks one of the component architectures/frameworks (i.e. COM, Java Beans etc.) and then, because of efficiency concerns, is stuck with that framework. It cannot make use of components written for any other framework.

The spectrum of choice available to the organisation is limited by a technical decision. The tail is wagging the dog.

I believe we’re reaching a stage up the abstraction layer (levels might be a better term) where the simplicity provided by having a single component framework is more costly than the benefits associated with being able to adopt any service that best suits business needs.

For me, that’s where scripting languages come in. Especially languages that focus on be eclectic, on being the glue that brings services/software together.

This approach is starting to be demanded because of the next observation.

The driving force behind innovation

In the good old days, as the above Economist article argues, all the innovation around computing and the Internet came from defence, research or higher education organisations. In the late 80s/early 90s, the cool Internet stuff was all happening in these organisations.

That is no longer the case. It’s coming from the consumer world. The consumer, and their needs, is driving the innovation and requirements of IT.

In the old days IT support divisions within organisations gained some leeway in terms of service, as they were the only show in town. Those days are long gone. There are many shows in town and that competition is driving ease of use and usefulness of the applications.

IT support divisions, especially those in Universities, cannot compete with Google. They don’t have the resources or skills. They never will.

IT support divisions will have to move up the abstraction layer, change what they do.