It seems to be the day for a backlash against learning analytics or its parent big data. This morning my PLN has filtered to the top Taleb’s “Beware the big errors of ‘big data'” and Why big data is not truth. Not that surprising to me given that I’ve argued that learning analytics in Universities has all hallmarks of yet another fad.

As I mentioned in this presentation my argument isn’t that “Learning analytics == crap”. My argument is that “How universities (and most organisations) for that matter implement learning analytics == crap”. After all – to paraphrase some Sir Ken Robinson schtick about lifting standards – no-one is arguing against data-driven decision making (which is a main claim of big data). All decisions should be based on data. The problem is the reality of most implementations will be that the data provided by learning analytics is likely to be horribly flawed, provided to the wrong people, and used to make decisions very badly.

As it happens, Damien, Col and I are all working on various related projects that are trying to figure out how to implement learning analytics within universities in a way that doesn’t equate to animal waste product. The following summarises some thinking out loud and the following of some initial ideas and will hopefully inform what I’ll be talking about at Moodlemoot’AU 2013 in a couple of weeks.

In the end, this is more just an early exploration of Performance Support Systems literature that has shown some level of support for a direction we’re exploring, i.e. embedding learning analytics into the LMS and other tools currently being used to better enable action.

Performance support systems

A common refrain heard when institutional folk get together and chat about learning analytics goes something like this

But we already have all this data, why don’t we use it?
Because it’s all in separate systems.
So we’ll give lots of money to Vendor X to implement another piece of technology that will bring all this data together and provide dashboards for people to look at the data.

The only winner out of this approach is the vendor who chalks up another sale. The data could have been brought together via any number of technical means, probably at much less cost, and in a way that doesn’t tie the organisation to accessing the data through the reporting tool provided by the vendor. Dashboards are generally a waste of time because people don’t use them. Especially teaching staff. Even if the dashboards can provide the sort of contextual information that will help a teacher intervene with a student, the information is provided in a system that is a million miles away from the system where the teacher will intervene.

I’ve long through that the obvious lens for looking at this is the long quoted idea of “Performance Support”. The following is an initial exploration of some of the literature.

Raybould (1995, p. 11) offers the following definition with some added emphasis from me

  • Encompasses all the software needed to support the work of individuals (not just one or two specific software applications).
  • Integrates knowledge assets into the interface of the software tools, rather than separating them as add-on components. For example, company policy information may be presented in a dialog box message rather than in a separate online document.
  • Looks at the complete cycle including the capture process as well as the distribution process.
  • Includes the management of nonelectronic as well as electronic assets.

While not without flaw, the definition does get at some of what I’m interested in, including

  • Integrating all of the knowledge required to complete a task into the tool I use to complete the task.
  • Consider the complete cycle.

    i.e. just don’t build the data warehouse and expect it to be used. Think about how people can be supported in using the data. After all the EDUCAUSE definition of learning analytics is (my emphasis added)

    the use of data, statistical analysis, and explanatory and predictive models to gain insights and act on complex issues.

  • All brought together with a focus on the “work of individuals”.

    i.e. don’t provide generic dashboards and leave it to the teachers to figure out what to do. Figure out what tools can be built to help teachers perform their work better.

Can’t see the trees for the forest

Which brings up the biggest barrier. The processes, systems, structures and people set up to implement “enterprise” systems like learning analytics focus on the forest. Or if they have any conception of the trees in the forest, the trees are all pine trees of identical size, shape and requirements. Being able to see the trees, to focus on the work of the individuals, is not easy. But that’s not the point. The point is that enterprise systems and processes can never effectively focus on the “work of individuals”. They don’t even try.

And it’s important, at least for this argument. Villachica et al (2006, p. 540) argue that the purpose of PSS is “expert-like performance from day 1 with little or no training” and that this can only occur within an appropriate zone – the “performance zone”.

The Performance Zone

Oh and here’s a good quote that reinforces my point above

There is also widespread
agreement that maintaining performance within this zone requires users to be able to learn, use, and reference necessary information within a single context and without breaks in the natural flow of performing their jobs. (Villachica et al, 2006, p. 540)

I can see myself using that a few times.

Kert and Kurt (2012, p. 486) cite Sleight (1998) to identify the following requirements (amongst others) of an EPSS

  • computer assisted.
  • accessible exactly at the time the task is realised.
  • in the study environment.
  • controllable by the user
  • ability to easily bring it up to date and fast access to information.


Kert, S. B., & Kurt, A. A. (2012). The effect of electronic performance support systems on self-regulated learning skills. Interactive Learning Environments, 20(6), 485–500.

Raybould, B. (1995). Performance Support Engineering : An Emerging Development Methodology for Enabling Organizational Learning. Performance improvement quarterly, 8(1), 7–22.

Villachica, S., Stone, D., & Endicott, J. (2006). Performance Suport Systems. In J. Pershing (Ed.), Handbook of Human Performance Technology (Third Edit., pp. 539–566). San Francisco, CA: John Wiley & Sons.

0 thoughts on “Learning analytics, intervention and helping teachers

  1. I think

    * in the study environment.
    * controllable by the user ”

    are the most important – and who’s the most important user? THE STUDENT!

    Strange how little of the learning analytics movement focuses on students using the data – so they can improve their learning. It would be a shame if LA was used as another force away from student centred learning and towards entrenching the ‘sage on the stage’ paradigm…

    1. I don’t disagree with the intent that the focus should be on student learning.

      But I’m not convinced that the most productive approach to improve student learning is to focus on “performance support systems” for students. Making a truly personal performance support system (i.e. one the student makes the choice to install) would be very difficult in terms of abstraction. Also, such a system is probably more useful in the context of “informal” learning, rather than institutionalised learning. Creating an institutionalised student system would likely raise adoption and implementation issues. e.g. requiring a standardised set of approaches to teaching etc…..

      This is not to say that exploring such a path isn’t important. I hope and expect others will explore it.

      But I’m a fairly concrete, limited sort of guy. Frankly, a lot of my rationale for focusing on improving the systems for teachers is selfish. I’m currently being hampered in what I can do in my own teaching because of the quality of the tools and information I have access to. A system like the above would help me create and support student centered learning environments and probably end up including some level of support for the student anyway.

      In fact, in pondering this reply, I’m wondering whether the concepts of “student” and “teacher” are part of the problem. Personally, I see my role as being a fellow learner at some level.

        1. As it happens, I came across that link via another part of my PLN today.

          I find myself in much agreement with the author re: paying the expert to provide the solution. Especially, as noted, some of these experts are somewhat lacking in background. Personally, I lay much of the blame at the feet of institutional leadership and their search for quick, successful and universal solutions to just about anything, including the quality of learning and teaching.

          I see what I do as different. I’m not interested in universal solutions. I don’t want to sell a product, but rather figure out what’s going to work for me, make my job easier, more effective, more enjoyable. To a large extent to make it possible to adopt some of the more radical perspectives on education within the current system. Actually, much, much more here. Keep throwing them at me. Appreciate it.

Leave a Reply

Your email address will not be published.