Data mining of online courses – dominant assumptions and innovation potential

For almost as long as learning management systems have been around their have been researchers and technologists investigating how the usage logs of these systems can be harnessed to inform and improve learning and teaching. For a little while I was sort of involved in a project that would look at some of this – interest has waned in line with organisational realignments.

A colleague, however, is continuing on with a project in this area. The following is an attempt to reflect about what little I know about the area and see if there is anything of interest to Col. In the following I am trying to identify the dominant assumptions underpinning this sort of work in order to see if there are any holes which might be ripe for exploitation.

Disclaimer: Until recently I haven’t paid much attention to the literature in this area and my recent reading was sparse, interrupted and incomplete. Hence the following could do with the insights of those with more knowledge of the literature to correct oversights, misunderstandings and naive assumptions.

What are we talking about

The common approach to this process uses automated evaluation of system logs and databases (e.g. Zorrila and Alvarez, 2008, Hung & Zhang, 2006; and Heathcote and Dawson, 2005) using some form of data mining or similar technology in order to provide additional information for teaching staff or Universities about the quality of the student experience. The typical form is take the usage logs and databases from a learning management system, put them into some sort of data warehouse system and generate reports for academics and management.

This type of work typically seems to differ on a few characteristics:

  • The technology/algorithms used to analyse the system logs and present the results.
  • The type of theory of learning or quality online presence that is used to “measure” the quality of the online course or the student experience.

I guess also there will be different communities within e-learning that take a different slant on this work. For example, I believe the intelligent tutorial system folk probably have a large amount of research and literature focused at analysing the student experience using data mining in order to make informed decisions about what to advise the student to do next.

Dominant assumptions and opportunity for innovation

Col is doing some of this work as part of a research project within his Masters study. So, I’m assuming he is keen to get into some innovation, to do something that hasn’t been done yet. So, in the following I’m trying to identify the dominant assumptions that, at least in my experience, seem to characterise this work. The idea is that over turning these assumptions might reveal an interesting line of work.

The assumptions that I’m thinking about (are there more?) include:

  • Limit and focus to the LMS logs.
    Analysis of LMS logs tells only a very small part of the story about students, staff and their interactions within learning and teaching. Even if the focus remains on the logs of computer systems, most students and staff will be leaving a fairly large trail of usage information through a range of other computer systems that could be combined with those in the LMS.

    This is especially important as the type of information gathered by mining LMS logs is limited by the nature and design of the LMS. For example, the Webfuse system (Jones and Gregor, 2006) has a design that by default allows open access to all resources. That is no requirement for users to login. Each system has its own design limits.

  • Kaplan’s law of instrument continued.
    Most of this work is done with similar tools. System logs and databases passed through statistics analysis and/or data mining technology to generate information. While useful, the tendency to focus just on these technologies has the potential to lead to flaws due to Kaplan’s law of instrument. Everything looks like a nail, if all you have is a hammer. Are there alternative tools or approaches that can be used, which may be base on different approaches and hence reveal different insight.
  • Emphasis on quantitative analysis alone.
    Simple log usage figures only tell a very small part of story of the student experience. Much of it is hidden within the words and emotions used and experienced by participants. It’s increasingly widely recognised that a multi-method approach for research is effective through each method covering limitations of other methods. Marrying quantitative analysis with textual analysis (e.g. Leximancer), or qualitative or specific feedback from students (e.g. course barometers; Jones, 2002) might be interesting.
  • Presenting information for staff and the institution, not the students.
    The aim for most seems to be to make the information available to the teaching staff so they can reflect and make improvements. Have seen little work where this information is made pro-actively available to the students so they can use it to guide their learning or student experience. For example, if a student were to see a huge spike in visitations to a particular page that they hadn’t visited, they may consider taking a look.
  • How is the insight generated used?
    Almost all of the research literature I’ve seen so far shows this information solely being used by researchers after the fact. i.e. after the term has finished they’ve analysed the data to see what lessons they can learn. The information may not even be shared with the teaching staff and they rarely seem to talk to academic staff. It would be interesting to see what would happen if you worked with academic staff during a term harnessing a range of different analysis/mining tools to provide information that they could respond to. This would allow tracking of what happens and the generation of insight and requirements within a real context of use, which is likely to be more useful and insightful than requirements gather out of context.


Heathcote, Elizabeth and Dawson, Shane (2005) Data Mining for Evaluation, Benchmarking and Reflective Practice in a LMS. In Proceedings E-Learn 2005: World conference on E-learning in corporate, government, healthcare & higher education, Vancouver, Canada.

David Jones, Student feedback, anonymity, observable change and course barometers, World Conference on Educational Multimedia, Hypermedia and Telecommunications, Denver, Colorado, June 2002, pp. 884-889.

What is a PLE? More than a suite of tools? More than social media?

Jocene and I are having a bit of a chat about PLEs and she raises a number of questions or perspectives in her last comment in that discussion that are worth of thought. So, I’m starting a new blog post here, rather than making a comment (the inequity in power and ease of use between the editing tools/interface used to create a post and those used to make a comment – very limited – make an interesting comment about the assumptions and affordances of a blog).

The issues raised by Jocene I want to consider include:

  • Is the notion of PLEs separate from those of Web 2.0/social media tools?
  • The connection of PLE with a set course.

A PLE is not a collection of Web 2.0 tools

In her comment Jocene makes the following point

I keep trying to separate the notion of PLEs and that of Web 2.0 social media tools. The latter may be used to construct various PLEs, but even the sum of these tools, in any PLE context, is still not the PLE itself. A suite of Web 2.0 tools is not a PLE.

My response/current belief can be summarised in two points:

  1. Yes I agree, but how else do you engage in this work?
  2. The tools have to come first, don’t they? No?

How do you do it? or Is a Web 2.0-based PLE better?

This section turned into something different as I was writing. The original point was to ask a question of what more could a University do to enable students’s use of PLEs without focusing on Web 2.0 based technologies. I’ll get to that, but first I’m reflecting on my experience and wondering whether or not a Web 2.0-based PLE is better than a traditional one.

A personal collection of tools to support your learning is nothing new. We’ve all done it. I had a collection of folders and loose leaf paper that I used at University back in the 80s to supplement my textbooks and handouts from the academics. Managing this collection of resources/tools effectively was half the battle of learning. I imagine I did some, perhaps many, of the tasks that Graham Atwell outlined in the slidecast that kicked off this conversation.

The following wasn’t planned. It arose out of thinking about this problem and the idea of comparing my old PLE with my new PLE using the tasks outlined by Graham Atwell in his slidecast.
Those tasks included:

  • access and searching;
    This was incredibly time consuming and poorly supported in the old style. The Internet, Google and my blog provide a much better set of tools to support this. Both individually and also socially, collaboratively with others.
  • aggregating and scaffolding;
    In the 80s this was called photocopying and placing in folders – rarely to be accessed again. What I did access was put into structures and frameworks that perhaps helped me understand.
  • manipulating;
    Much of the learning I have undertaken has always been around ideas and information – computer science, information systems, learning and teaching, philosophy etc. – part in due to the nature of the disciplines but also the nature of teaching/learning. Manipulating the artifacts associated with this learning in my old PLE was laborious. In fact, if I attempt to do this now – e.g. write more than my signature – my body rebels, aches and generally says “stop!”. As a techie the manipulations I can perform in my new electronic PLE is so much easier, powerful and interesting.
  • analysing;
    Given that I eventually graduated there must have been some analysis of the content of my old PLE. I must have worked out what some of it meant. I believe my new PLE is orders of magnitude better at helping me in this analysis. The ability to access hugely diverse opinions and have tools like Google, Wordle and many others to perform various forms of low-level analysis is a great help.
  • storing;
    The question of long-term viability is still open. Moving from my old website to this blog has probably led to some loss of information. But keeping information is getting easier. I certainly have very little of the content from my 80s PLE. The multimedia nature of the new PLE, however, is a significant improvement. On my laptop I have videos and audio that I consider important. I also think there is something to be said for the way that my new PLE makes it much easier to store information/learning in a fragmented form, which is a good thing, really it is.
  • reflecting;
    Did my old PLE help in terms of reflection. To some extent. But the private nature, difficulty of manipulating, storing, accessing and searching that old PLE certainly placed significant constraints. My new Web 2.0 PLE makes reflection much easier. It lets me find and link my thoughts together. The form of a blog and its connection to a diary also helps encourage reflection. (Not to say that the technology determines this, it takes discipline and motivation on my part – but the affordances of the new PLE help.)
  • presenting;
    My old PLE led to presentations only in the form of formal, necessary presentations. To some extent that remains true, but even this post is a form of presentation, perhaps of representation. Trying to show my understanding. Even this bit of reflection is available as a “presentation” for others. The combination of presentation and reflection add meaning, at least for me as the author.
  • representing;
    I’m not sure I got Graham’s meaning on this one. However, the word points to me about representing the meaning and identify I place on what I’ve learnt in my PLE. Have I got it wrong? My old PLE had very little connection with me. If someone picked up the collection of folders and textbooks there wouldn’t be a lot in it that represented me. The odd comment, not a lot of reflection. With my blog, it’s a different story. There are photos of what I experience, there are small personal storied intermixed with the work and learning. Does there need to be a separation between learning and others aspects of life? Certainly in my PLE (my blog) there isn’t one.
  • sharing;
    With my old PLE I could do little or none of this. Access to those folders and their contents was not readily available to folk (access) and the searching was poor. With my blog I’m currently averaging around 50 or so “sharing events” a day as people visit the resources on it. They generally come here through links on WordPress, elsewhere on the web or through google searchers.

However, during my undergraduate education I certainly didn’t engage in the move from “sanctioned knowledge” to “collaborative forms of knowledge construction”. I didn’t talk to many folk, worked on my own with the “sanctioned knowledge” and my own constructions. Almost certainly the poorer for it and am now engaging differently through the Web 2.0 tools. Why the difference?

I’m certainly more mature and open about learning, perhaps I just wasn’t ready for it as a kid. I also know that the Web 2.0 tools, like blogs, have a much greater affordance for the type of “collaborative forms of knowledge construction” that I prefer. i.e. I don’t particularly like synchronous, group-based, warm and fuzzy co-operation. I prefer to be on my own, considering what lots of others have said and done and working through my own ideas.

On the basis of the above, it looks like, at least for me, that a Web 2.0-based PLE is a tremendous improvement over a traditional non-Web 2.0 based PLE. Too many of the tasks which Graham Atwell suggests you want to do in a PLE are much easier, more effective with the assistance of Web 2.0 technology. If it is better, should we look at helping people use it.

Question: Is this part of the difficulty we face with PLEs? A Web 2.0/social media enabled PLE is, because of the affordances of the technology, a completely different kettle of fish. Think of the difference between written, personal communication implemented in the 17th century and implemented now in the 21st century. It’s a completely different ball game. Perhaps the whole PLE thing is getting too bogged down with the “yea, we’ve always done it stuff”. Perhaps we haven’t always done it, perhaps it so different that relying on the old patterns of thought is preventing innovation?

I’m still thinking about this myself.

Back to the original point I was thinking of making. If you decide that students have always made use of a PLE using traditional approaches, just like I did back in the 80s, then what more can we do to support students in using the traditional form of PLEs? If you assume that in some institutions, like CQUniversity, that more and more of the learning experience will be moving online, then are is there anything we can do?

I’ve always believed that it’s not the task of the university to build or specify a PLE for students. Whether it be “traditional” or Web 2.0. The services a university could perform to help students use PLEs, seem to me, to be:

  • Open up its learning activities, resources and services so the student can use the tools they select to perform the tasks Atwell points out.
  • Because this idea is somewhat novel, scaffold and aid the development of individual PLEs, in whatever form, to learn some lessons and see where things go.
  • Learn from what worked and from what didn’t and continue.

PLEs and university courses

In her comment Jocene makes the following point

Conceptually, there is no reason why my PLE needs to service, or make me accountable to a set course (in which I may be enrolled) if my way of knowing (principle 2) does not match that of the course designer. Conceptually, I will learn when I am ready to learn, and I will select the evidence I need from seemingly infinite data, to bring me to the realisation that I know something.

I agree entirely with this perspective. I believe that the amount of learning an individual will go through with a formal learning organisation (like a university) pales almost into insignificance against the amount of informal learning.

The “services” I listed at the end of the previous section seem to allow for this. The emphasis is on opening up the university’s courses to allow students to use the PLEs they chose. Eventually the assumption being that this is the same PLE students use to service the broader array of learning experiences they have. In addition, the “opening up” of university courses may also include developing and helping academics use course designs that allow for more freedom and diversity in how a student travels through a course.

Back to watching the Super Bowl.

Joining the landed gentry – getting a money burner

Sunset in January with a bit of rain

For the last 6 years or so the family and I have had the great pleasure of living on a couple of hundred acres in the “Rockhampton hinterland”. Ahh, the serenity. The view from the front veranda can be quite nice. Especially since we can’t see nor hear the neighbours.

Friends of ours from Europe have expressed surprise at the size of our place. Apparently, in Europe we’d have to be lords and ladies to afford such an expanse. I’m not sure our life-style quite fits with the stereotypes of the European upper-class. However, we seem to have taken the first step in putting on those types of airs (and going by someone’s current predilection for expensive house improvements it won’t be the last step).

Malina - the new money burner

Our first step toward joining the landed gentry has been to purchase a half-share in a brood-mare. Not content to simply own horses that we ride on our place for a hobby, the decision has been made to go into race horses. If you’re going to burn money, one may as well do it in style (and at a much faster rate). So, yesterday morning Malina (unsure of the spelling or of the full appellation, which is certain to be pretentious enough to justify the expense) joined our little crew of animals.

Malina is the one in front. Yes, there are two. Not content to have one it was decided we could also “host” another broodmare – one wouldn’t want one’s money burner to get lonely – owned by the culprit who connived and convinced my poor, deprived better half to enter into the money burning enterprise. So, Malina is joined by Hermione who probably has an equally long and expensive, quality name. The following photo is Hermione introducing herself to Scruff.

Scruff and Hermione

Disclaimer: It’s my role in our family to play the curmudgeon, the nay-sayer. Without this occasional brake on ambitions there is no telling what ideas my partner may develop. Having Malina and Hermione around will be just fine and possibly even positive. At least in an “ambience” kind of way.

Disclaimer’s disclaimer: I have yet to be told or request financial information about this transaction, which has almost certainly prevented a significant deepening in the level of my negativity around this happy event.