The following is a call for participation from folk interesting in writing a paper or two that will tell some real stories arising from LMS evaluations.
Alternatively, if you are aware of some existing research or publications along these lines, please let me know.
LMSs and their evaluation
I think it’s safe to say that the idea of a Learning Management System (LMS) – aka Course Management System (CMS), Virtual Learning Environment (VLE) – is now just about the universal solution to e-learning for institutions of higher education. A couple of quotes to support that proposition
The almost universal approach to the adoption of e-learning at universities has been the implementation of Learning Management Systems (LMS) such as Blackboard, WebCT, Moodle or Sakai (Jones and Muldoon 2007).
LMS have become perhaps the most widely used educational technologies within universities, behind only the Internet and common office software (West, Waddoups et al. 2006).
Harrington, Gordon et al (2004) suggest that higher education has seen no other innovation result in such rapid and widespread use as the LMS. Moodle or Sakai. Almost every university is planning to make use of an LMS (Salmon, 2005).
The speed with which the LMS strategy has spread through universities is surprising (West, Waddoups, & Graham, 2006).
Even more surprising is the almost universal adoption of just two commercial LMSes, both now owned by the same company, by Australia’s 39 universities, a sector which has traditionally aimed for diversity and innovation (Coates, James, & Baldwin, 2005).
Oblinger and Kidwell (2000) comment that the movement by universities to online learning was to some extent based on an almost herd-like mentality.
I also believe that increasingly most universities are going to be on their 2nd or perhaps 3rd LMS. My current institution could be said to be on its 3rd enterprise LMS. Each time there is a need for a change, the organisation has to do an evaluation of the available LMS and select one. This is not a simple task. So it’s not surprising to see a growing collection of LMS evaluations and associated literature being made available and shared. Last month, Mark Smithers and the readers of his blog did a good job of collecting links to many of these openly available evaluations through a blog post and comments.
LMS evaluations, rationality and objectivity
The assumption is that LMS evaluations are performed in a rational and objective way. That the organisation is demonstrating its rationality by objectively evaluating each available LMS and making informed decisions about which is most appropriate for it.
In the last 10 years I’ve been able to observe, participate and hear stories about numerous LMS evaluations from a diverse collection of institutions. When no-one is listening, many of those stories turn to the unspoken limitations of such evaluations. They share the inherent biases of participants, the cognitive limitations and the outright manipulations that . Stories that rarely, if ever, see the light of day in research publications. In addition, there is a lot of literature from various fields suggesting that such selection processes are often not all that rational. A colleague of mine did his PhD thesis (Jamieson, 2007) looking at these sorts of issues.
Generally, at least in my experience, when the story of an institutional LMS evaluation process is told, it is told by the people who ran the evaluation (e.g. Sturgess and Nouwens, 2004). There is nothing inherently wrong with such folk writing papers. The knowledge embodied in their papers is, generally, worthwhile. My worry is that if these are the only folk writing papers, then there will be a growing hole in the knowledge about such evaluations within the literature. The set of perspectives and stories being told about LMS evaluations will not be complete.
For years, some colleagues and I have regularly told ourselves that we should write some papers about the real stories behind various LMS evaluations. However, we could never do it because most of our stories only came from a small set (often n=1) of institutions. The stories and the people involved could be identified simply by association. Such identification may not always be beneficial to the long-term career aspirations of the authors. There is also various problems that arise from a small sample size.
Are you interested in helping solve these problems and contribute to the knowledge about LMS evaluations (and perhaps long term use)?
How might it work?
There are any number of approaches I can think of, which one works best might depend on who (or anyone) responds to this. If there’s interest, we can figure it out from there.
Coates, H., R. James, et al. (2005). “A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning.” Tertiary Education and Management 11(1): 19-36.
Harrington, C., S. Gordon, et al. (2004). “Course Management System Utilization and Implications for Practice: A National Survey of Department Chairpersons.” Online Journal of Distance Learning Administration 7(4).
Jamieson, B. (2007). Information systems decision making: factors affecting decision makers and outcomes. Faculty of Business and Informatics. Rockhampton, Central Queensland University. PhD.
Jones, D. and N. Muldoon (2007). The teleological reason why ICTs limit choice for university learners and learning. ICT: Providing choices for learners and learning. Proceedings ASCILITE Singapore 2007, Singapore.
Oblinger, D. and J. Kidwell (2000). “Distance learning: Are we being realistic?” EDUCAUSE Review 35(3): 30-39.
Salmon, G. (2005). “Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions.” ALT-J, Research in Learning Technology 13(3): 201-218.
Sturgess, P. and F. Nouwens (2004). “Evaluation of online learning management systems.” Turkish Online Journal of Distance Education 5(3).
West, R., G. Waddoups, et al. (2006). “Understanding the experience of instructors as they adopt a course management system.” Educational Technology Research and Development.