An overview of learning analytics

The following is a summary and some reflections upon

Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, (August), 1–13. doi:10.1080/13562517.2013.827653

A good overview of the learning analytics field as it stands. A good call to arms for teachers to engage with the fad and use it to improve learning and teaching.

Abstract

Learning analytics, the analysis and representation of data about learners in order to improve learning, is a new lens through which teachers can understand education. It is rooted in the dramatic increase in the quantity of data about learners and linked to management approaches that focus on quantitative metrics, which are sometimes antithetical to an educational sense of teaching. However, learning analytics offers new routes for teachers to understand their students and, hence, to make effective use of their limited resources. This paper explores these issues and describes a series of examples of learning analytics to illustrate the potential. It argues that teachers can and should engage with learning analytics as a way of influencing the metrics agenda towards richer conceptions of learning and to improve their teaching.

An abstract that has me very interested in the paper. Too much of analytics tends to lean toward the “management approaches that focus on quantitative metrics” and not on teaching staff engaging with learning analytics. Given the journal this is published in, the focus on teachers is not surprising.

Introduction

Identifies the tension teachers and learners face between economic activity and actual learning.

Quantitative metrics are increasing due to

  1. “theoretical framings” i.e. management etc.
  2. big data

    Like the idea of “big data” – the availability of data and the rise of computational techniques to handling that data – as an enabler.

Describes the scale of data in the sciences – e.g. 23 petabytes of data in 2011 generated by the Large Hadron Collider. Would like to compare the size of this data with the data available to a university on learners as a comparison of just how “big” big data is in tertiary education.

Makes an important point which I think most traditional researchers don’t get as a difference between traditional approaches to data and big data

The volume and scope of data can be so large that it is possible to start with a data-set and apply computational methods to produce results and to seek an interpretation or meaning only subsequently

Frames the rise of business intelligence and its focus on performance metrics as something encroaching into higher education through the “education as an economic activity”

What about the learners?

Moves onto learning analytics. Offers the following distinctions/definitions

  1. Academic Analaytics – the application of business intelligence to education.
  2. Educational Data Mining – development of methods for analysing educational data focusing “more on the technical challenges than on the pedagogical questions (Ferguson, 2012)”.
  3. Learning analytics – “first and foremost concerned with learning”.

But doesn’t mention teaching analytics – the “proposed subfield of learning analytics that focuses on the design, development, evaluation and education of visual analytics methods and tools for teachers in primary, secondary and tertiary educational settings”.

But does make the point that a key concern is generate “actionable intelligence” which informs appropriate interventions. And brings up the Campbell and Oblinger (2007) 5 step cycle:

  • capture
  • report
  • predict
  • act
  • refine

Good description of the “field”

Learning analytics is not so much a solid academic discipline with established methodological approaches as it is a ‘jackdaw’ field of enquiry, picking up ‘shiny’ techniques, tools and methodologies

Positioned as both strength (“failitates rapid development and the ability to build on established practice and findings”) and weaknesses (“lacks a coherent, articulated epistemology of its own”).

Predictive modelling

Defined as

a mathematical model is developed, which produces estimates of likely outcomes, which are then used to inform interventions designed to improve those outcomes.

Techniques: factor analysis and logistic regression

Applied to data including

  • prior educational experience and attainment
  • demographic information
  • information arising from the course – use of tools, assessment etc.
  • Whether the student went on to complete the course.

Similar to the teaching noticing a struggling student, but some practical differences

  1. Output is in the form of estimated probabilities, which many people struggle to understand.
  2. Output available to more than just the teacher.
  3. Interventions can be triggered without teacher involvement.

Predictive power of these models is not perfect. But they can be more often right than wrong and they can be used to improve student completion.

Pudue’s Course Signals project is the example used. Makes the point that

Course Signals are not used in a decontextualised
environment: the teacher is central to the process and uses their judgement to direct students to appropriate existing resources within the university.

Mentions other examples – OUUK’s Retain and S3 from D2L. Mentions the OUUK’s finding that level of activity was not a predictor, but a fall-off in activity was a clear indicator.

SNA

Use of network analysis to analyse connections between people. Can be interpreted both by eye and thorugh mathematical analysis.

SNAPP as the example. Mentions the trade-off. SNAPP is easy to use, but not as flexible/powerful as more general tools.

Mentions other more complex work, including an ANT informed analysis of Tapped-In.

Usage tracking

Tracking of student activity in computer-based environments. Raises questions about value of this for student learning, what sorts of feedback is helpful, and ethical questions.

Content analysis and semantic analysis

A move away from quantitative data generated by students toward the analysis of qualitative textual data.

Example of “Point of Originality Tool” that helps to “track how students develop originality in their use of key concepts over the course of a series of writing assignments”.

Also the OU work on exploratory talk, a more speculative example

These methods could be used to analyse the students’ contributions to an online forum, giving them feedback about the degree to which their online talk is exploratory (or matches other criteria for constructive educational dialogue), and offering suggestions for ways in which they might contribute more effectively.

Automated assessment not currently within learning analytics. But does appear to be a form for learning analytics.

Recommendation engines

Analyse behaviour to provide suggestions to individuals for items that may be of interest.

Could be used to suggest learning resources to a student. But may be difficult in the context of a set curriculum.

This is an approach I’d like to implement in BIM recommending to students either

  • Other class mates blogs they may wish to follow.
  • Useful links that have been mentioned in student posts.
  • Or perhaps specific posts from students that may be of interest.

Discussion

Engages in some implications for teacher in higher education

Ethics and privacy a bit one. Mentions common points.

Being open about learning analytics with students can improve perceptions.

The student knows more about their own learning situation than even the best teacher. Make it available to them.

Teachers have a professional responsibility to use appropriate means to improve student learning. Analytics raise ethical questions – do you allow a student to enrol if the predictive algorithm suggests they’ll fail.

Student feedback has to be handled well.

Learning analytics is often atheoretical or not explicit about its theoretical basis. Some authors have tried to ground it.

Makes the point that if assessment doesn’t reflect learning, then learning analytics that improves performance on assessment, doesn’t necessarily improve learning.

Conclusions

The promise of learning analytics is the empowerment of teachers and students to
understand the wealth of data that relates to their learning….to achieve institutional change, learning analytics data need to be presented and contextualised in ways that can drive organisational development (Macfadyen and Dawson 2012).

Makes the point that data and mathematics (and learning analytics) can be used to reinforce the status quo. And leads into this risk of learning analytics being driven by the world view of managers and the economic framing of education.

References

Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, (August), 1–13. doi:10.1080/13562517.2013.827653

The network challenge to the LMS mindset

It’s been an intellectually draining few days with a range of visitors to the institution talking about a range of different topics. It started on Monday with George Siemens giving a talk titled “The future of the University: Learning in distributed networks” (video and audio from the talk available here). A key point – at least one I remember – was that the structure of universities follows the structure of information and the current trend is toward distributed networks. On Tuesday, Mark Drechsler ran a session on Extending Moodle using LTI which raises some interesting questions about how the LMS mindset will handle the challenge of a distributed network mindset.

LMS != distributed network

The LMS is an integrated, enterprise system sourced from a single vendor. The institution as a whole decides upon which of the different available systems it will choose and then implements it on a server. The students and staff of that institution must now make use of that system. Changes to that system are controlled by a governance structure that may/may not approve the addition or removal of functionality into the enterprise system. Only a designated IT group (outsourced or local) are actually able to make the change. The “network” has a single node.

The typical mindset encouraged by an LMS when designing learning is not what is the best way to engage student learning. It’s the best way to engage student learning within the constraints of the functionality currently provided by the LMS. I wrote more about the limitations of this model in Jones (2012) and almost incessantly over the last few years. Chapter 2 of the PhD thesis a fair bit of this argument.

Over recent years most institutions have realised that a single node network consisting of the LMS isn’t sufficient for their needs. The network has had a few new nodes such as a lecture capture system, a content repository, an eportfolio system and a range of others. However, this “network” of services isn’t really a distributed network in that it’s still only the institution approved processes that can add to the network. I as an academic, or one of my students, can’t decide we’d like to add a service that is integrated into this institutional network.

Sure we can use standard hyperlinks to link off to Google docs or any one of the huge array of external services that are out there. An extreme example is my kludge for using BIM this year. Where I’m hosting a version of BIM on my laptop because for various reasons (including many of my own making) BIM couldn’t get installed into the institutional version of Moodle in time.

The trouble is that these kludges are not members of the distributed learning systems network associated with the institution. The most obvious indicator of this is the amount of manual work I need to engage in to get information about students from the institutional system into my local install of BIM and then to get information out of my local install of BIM back into the institutional ecosystem.

To have seamless integration into the institutional LMS network requires going through the institutional governance structure. Now there are good reasons for this, but many of them arise from the problem of the LMS not being a network. Some examples include

  • a “bad” addition to the LMS could bring the system down for everyone;

    If the LMS were a network, then this wouldn’t happen. The additions would be on another node so that if the addition was “bad” only that node would be impacted. If nodes could be added by individuals, then only that individual’s applications would be impacted.

  • not enough people are going to use the addition;

    To make it worthwhile to integrate something into the system, there has to be the likelihood that a large number of people are going to use it. Otherwise it’s not worth the effort. The cost of adding something to an integrated system is high. With a network approach the cost of adding a new feature should be low enough to make it economical for only one person to use it.

  • who’s going to help people use the new addition;

    Since a large number of people have to be able to use the system, this raises the question of who is going to support those people. In a network approach, there isn’t this need. In fact, I may decide I don’t want other academics using the service I’ve added.

  • the inertia problem;

    The other major impact of this high cost of integrating a tool into the LMS is inertia. The cost of making changes and the cost of negative impacts means great care must be taken with changes. This means that rapid on-going improvement is difficult leading to inertia. Small-scale improvements suffer from a starvation problem.

  • the trust problem;

    Since it’s a high cost, high risk situation then only a very limited group of people (i.e. central IT) are allowed to make changes and only after approval of another limited group of people (the governance folk).

  • vanilla implementation.

    All of the above leads to vanilla implementations. It’s too difficult to manage the above, so let’s implement the system as is. I’ve heard stories of institutions moving away from more flexible systems (e.g. Moodle) back toward more constrained commercial systems because it removes what element of choice there is. If there’s no choice, then there’s no need for complex discussions. It’s easier to be vanilla.

The LTI Challenge

The Learning Tools Interoperability standard, or more precisely it’s integration into various LMS offer a challenge to this LMS mindset. LTI offers the possibility – at least for some – of turning all this into more of a network than an integrated system. The following will illustrate what I mean. What I wonder, is how well will the existing governance structures around institutional LMS – with their non distributed network mindset – respond to this possibility?

Will they

  1. Recognise it as a significant advantage and engage in exploring how they can effectively encourage and support this shift?
  2. Shut it down because it doesn’t match the LMS mindset?

BIM and LTI

In the very near future, BIM will be installed into the institutional Moodle install for use by others. I have always feared this step because – due to the reasons expressed above – once BIM is installed I will not be able to modify it quickly.

LTI apparently offers a solution to this via this approach

  1. I set up a version of Moodle on one of the freely available hosted services.

    This would be my install of Moodle, equivalent to what I run on my laptop. No-one else would rely on this version. I could make changes to it without effecting anyone. It’s a separate node in the network relied upon by my course. I can install a version of BIM on it and modify it to my hearts content confident that no-one else will be impacted by changes.

  2. Install the Moodle LTI Provider module on my version of Moodle.
  3. Set up a course on my version of Moodle, create a BIM activity and add it to the LTI provider module.

    This allows any other LTI enabled system to connect to and use this BIM activity as if it were running within that system, when it is actually running on my version of Moodle. Of course, this is only possible when they have the appropriate URL and secret.

  4. Go to the institutional version of Moodle and the course in which my students are enrolled and add an “External Tool” (the Moodle name for an LTI consumer) that connects to BIM running on my version of Moodle.

    From the student (and other staff) perspective, using this version of BIM would essentially look the same as using the version of BIM on the institutional Moodle.

LTI allows the institutional LMS become a network. A network that I can add nodes to which are actually part of the network in terms of sharing information easily. It’s a network where I control the node I added, meaning it no longer suffers from the constraints of the institutional LMS.

The downsides and the likely institutional response

This is not an approach currently in the reach of many academics. It’s not an approach required by many academics. But, that’s the beauty of a network over an integrated system, you don’t need to be constrained by the lowest common denominator. Different requirements can be addressed differently.

In terms of technical support, there would be none. i.e. you couldn’t expect the institutional helpdesk to be able to help diagnose problems with my Moodle install. I would have to take on the role of troubleshooting and ensure that the students, if they have problems, aren’t asking the helpdesk.

Perhaps more difficult are questions around student data. I got in trouble last year for using a Google spreadsheet to organise students into groups due to students entering their information onto a system not owned by the institution (even though the student email system is outsourced to Google). I imagine having some student information within a BIM activity on an externally hosted server that hasn’t been officially vetted and approved by the institution would be seen as problematic. In fact, I seem to recollect a suggestion that we should not be using any old Web 2.0 tool in our teaching without clearing it first with the institutional IT folk.

Which brings me back to the my questions above, will the organisational folk

  1. Recognise the LTI-enabled network capability as a significant advantage and engage in exploring how they can effectively encourage and support this shift?
  2. Shut LTI down (or at least restrict it) because it doesn’t match the LMS mindset?

    How long before the LTI consumer module in the institutional LMS is turned off?

LTI seems to continue what I see as the inexorable trend to a more networked approach, or as framed earlier as enabling the best of breed approach to developing these systems. LTI enables the loose coupling of systems. Interesting times ahead.

References

Jones, D. (2012). The life and death of Webfuse : principles for learning and leading into the future. In M. Brown, M. Hartnett, & T. Stewart (Eds.), Future challenges, sustainable futures. Proceedings ascilite Wellington 2012 (pp. 414–423). Wellington, NZ.

Neither strategy nor "space" to innovate is enough

Read this piece – Why CIO’s, IT and Faculty need to find common ground on technology – by David Wiley yesterday as it did the rounds. The article argues that the focus of the CIO/IT on highly reliable systems is a mismatch for the needs of innovation in learning and teaching. It brings up the tension between standard systems and rogue systems (aka shadow systems). The solution to this is a space – a policy space – that enables innovation.

In general, I am sympathetic to this argument, but I also don’t think it truly captures what is required for innovation in contemporary learning and teaching.

I then had a real laugh when I read the comments. The comments suggested that there was a misconception underpinning Wiley’s argument. In particular, that it’s not the responsibility of the CIO to maintain uptime. That’s the CTO’s job. The CIO should be innovative, they should be focused on strategy and on how to enable innovative ways to use technology.

I’m not convinced that either side has found the solution.

The idea that strategic thinking by a CIO will result in innovation in learning and teaching is, based on the literature and research I’ve seen, essentially wrong. That’s not where innovation originates. There are a variety of reasons, but one of the main ones is that being strategic inevitably leads to large scale projects and projects are based on incorrect understanding of the world. This essay – The seemingly peculiar property of projects – explains these problems and offers the solution – Tinkering.

And tinkering is where the “space” for innovation faces a challenge. I haven’t gotten to the stage of abstract principles, so instead I’ll talk about specifics. I’m currently using BIM in my teaching. To tinker with BIM while I’m using it, BIM needs to both have access to the institutional data/systems and yet be in a space where I can play. Without the institutional data I can’t use it effectively within the rest of the organisation. But without the space to play, I can’t tinker. There’s more, but taxi duty awaits.

Aligning learning analytics with learning design

The following is a summary and some initial responses to this paper

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing Pedagogical Action: Aligning Learning Analytics With Learning Design. American Behavioral Scientist. doi:10.1177/0002764213479367

The abstract for the paper is

This article considers the developing field of learning analytics and argues that to move from small-scale practice to broad scale applicability, there is a need to establish a contextual framework that helps teachers interpret the information that analytics provides. The article presents learning design as a form of documentation of pedagogical intent that can provide the context for making sense of diverse sets of analytic data. We investigate one example of learning design to explore how broad categories of analytics—which we call checkpoint and process analytics—can inform the interpretation of outcomes from a learning design and facilitate pedagogical action.

I’m interested by the focus on moving from “small-scale practice to broad scale applicability”, but I wonder about how broad scale pedagogical practice can be given the inherent diversity/complexity.

Thoughts

The paper describes how learning design and learning analytics could be merged in a way that would provide better understanding of what is going on around student learning and perhaps allow demonstration of learning quality. To do this, it is assumed that the teacher or teaching team

  1. Makes explicit their pedagogical intent using an existing learning design.
  2. Then analyses this learning design to identify where analytics can provide checkpoint (i.e. have the learners performed certain necessary steps) and process (what’s going on during the learning process) insights.
  3. These insights are then used to either make interventions during learning or to redesign for the next time.

What I particularly like about this is the recognition that the real value of learning analytics arises from when it is applied within the learning process with knowledge of what is intended. As opposed at some abstract higher level.

But I wonder is the following are problems?

  • How many University academics have explicit pedagogical intents?

    Let alone make use of learning designs. How many think beyond the number of weeks of semester, the number of hours of lectures/tutorials, the topics to be covered and the assessment?

    Especially in a higher ed sector increasingly reliant on casualisation?

  • How many University academics or even support staff know enough about the data stored and the available learning analytics tools to be able to identify, design and implement creative and appropriate combinations of checkpoint and process learning analytics specific to a particular learning design?

    Especially given the learning designs at a certain level are generic. Once I start implementing a learning design in a particular course with a particular system there is going to be wide diversity of possible data and tools that I might like (or more correctly am only allowed) to draw upon.

    Beyond the contextual diversity, there is the idea of diversity in learning designs. Both these would require creative and informed manipulation of systems. Systems that aren’t designed for this sort of manipulation and in an environment where it is known that most people have troubles interpreting available visualisations, let alone creating new ones.

  • How many could actually get access to the data that could be used to support this?

    For example, Moodle activity completion offers some support for checkpoint analytics, but the reports are difficult to read, let alone create linkages between them.

    Damien Clark I think there’s some potential here for MAV or at least something based on its architectural approach.

  • Which leads to questions about the affordances of the available tools and the systems within which they are used.

    The LMS as an enterprise system is not flexible nor agile. It can change rapidly. Consequently, most of the existing LMS learning analytics work is focused on common tasks. Tasks that the majority of users of such systems might find useful. This is one of the reasons for the focus on engagement as measured by clicks, rather than better support for specific learning designs.

    A learning design is only going to be used by a subset of users of the LMS. Perhaps a very small subset of the LMS. Which makes it unlikely that the LMS will provide the affordances necessary. Raising the question of how to go about this?

    Do you

    1. Go down the personal route?

      Each academic (or at least those keen) have their own analytics tool that operates outside the LMS. This is much the approach that browser-based tools like SNAPP and others uses. But perhaps this needs to be expanded more.

    2. Add the capability for “pedagogical skins” to existing LMS tools.

      e.g. a way to say to the discussion forum tool that my pedagogical intent with this forum is X, provide me with the analytics and other affordances that make sense for pedagogical intent X.

    3. Or something else?

Some other misc thoughts follow and then a summary of the paper.

Analytics and course redesign

The paper argues that

Analytics can also help with course redesign. Traditionally, educators draw on their past experience when they teach a course or when they are designing a new course. For many teachers this may be an informal process that relies on recall, student surveys, and/or teacher notes recorded during or after the teaching session. Revisiting the learning analytics collected during the course can support teachers when they are planning to run the course again or when the learning design is being applied to a different cohort or context.

I think this would be an interesting area to explore. In part because I don’t think most academics have the time or inclination to do active redesign. It’s all tweaks.

While there are significant flaws with recall and student surveys, the data provided by learning analytics is not without fault. Especially given the fairly low level of the data that is available.

Learning design reuse

Learning design aims for reusability across educational contexts. Hence repositories of learning designs and tools that make learning designs machine readable and adaptable. But how have these been adopted? Who has adopted them? What are the barriers?

Relationship between learning design and the tools?

The paper claims that learning design can be used “as a framework for design of analytics to support faculty in their learning and teaching decisions”. But given the current nature of the tools available, just how realistic is this?

On a more personally interesting note, is the idea that learning designs describe pedagogical intent, but not how students are engaged in that design during or post-implementation. Perhaps giving a focus for how learning analytics can be integrated into a tool like BIM.

The paper argues that the critical step is a marriage between learning design and learning analytics? I wonder if a more concrete step is the marriage between learning analytics and the e-learning tools being used to implement the learning designs? If there is difficulty in understanding, then surely the tools can/should help? Wonder if they address this?

What if the learning tools themselves provided specific checkpoint and process analytics specific to the types of learning intent most common with a tool. Wouldn’t that help? After all, shouldn’t the technology be invisible?

The tool focus of course narrows applicability. A learning design might encompass a whole range of tools and getting those to play nicely together could be interesting, but also more useful. It gets at some of the same integration questions with BIM and EDC3100.

Of course, given the current nature of the tools available, I don’t think there are many people that could easily engage in the sort of creative combination of checkpoint and process analytics and learning design embodied here.

Link between learning outcomes and analytics

Some suggestions here of a link between learning outcomes and analytics which may link with work I need to complete locally. It’s underpinned by the same desire to demonstrate the quality of learning and teaching processes and both plagued by the difficulties of developing anything such thing that is meaningful or broadly applicable.

Self-reflection reading

Self-reflection requires strong metacognitive capacities that have been demonstrated to be essential for developing the skills necessary for lifelong learning (Butler & Winne, 1995).

What follows is a summary of my reading of the paper.

Introduction

Definitions

  • Learning analytics – “the collection, analysis, and reporting of data associated with student learning behavior”
  • learning design – “the documented design and sequencing of teaching practice”

The aim is how “together these may serve to improve understanding and evaluation
of teaching intent and learner activity”

Makes the point that learning analytics doesn’t suffer from the difficulties of survey groups of focus groups

which rely on participants both opting to provide feedback and accurately remembering and reporting past events

But the difficulty with LA is “interpreting the resulting data against pedagogical intent and the local context to evaluate the success or otherwise of a particular learning activity”

The claim is

learning design establishes the objectives and pedagogical plans, which can then be evaluated against the outcomes captured through learning analytics.

Overview of Learning analytics

Starts with the accountability interest around “indicators of the quality of learning and teaching practices” and “the development in the education sector of standardised scalable, real-time indicators of teaching and learning outcomes”. And links to the difficulty of doing so and the need for “a thorough understanding of the pedagogical and technical context in which the data are generated”.

Uni QA data generally derived from student experience surveys and measures of attrition etc. The rise of the LMS has led to work around learning analytics.

LA research interrogates data to create predictive models of performance, attrition, as well as more complex dimensions as dispositions and motivations. In turn to inform future L&T practice.

To date the focus is on predictors of student attrition, sense of community and achievement and ROI of technologies. But by providing measures of the student learning process it can help teachers design, implement and revise courses. Much work to be done here.

Overview of learning design

Apparently learning design had two main aims

  1. to promote teaching quality
  2. to facilitate the integration of technology into L&T

A development of the 2000s in response to the capability of the Internet to share examples of good practice – related work includes learning design, pedagogical patterns, learning patterns and pattern language (that last one is drawing a long bow). A learning design

  • describes the sequence of learning tasks, resources and supports constructed by a teacher
  • captures the pedagogical intent of a unit of student
  • a broad picture of planned pedagogical actions, rather than detailed accounts (as per a traditional lesson plan).
  • provide a model for intentions in a particular context

The proposition is that a learning design can be used “as a framework for design of analytics to support faculty in their learning and teaching decisions”

Focus on reusability, repositories (e.g. http://www.learningdesigns.uow.edu.au/ and http://193.61.44.29:42042/ODC.html).

Most based on constructivist assumptions.

Many formats, general or specific, very short or very long, but essential elements

  • Key actors
  • What they are expected to do
  • What resources are used
  • sequence of activities

The fit with learning analytics is that learning designs describe pedagogical intent, but not how students are engaged in that design during or post-implementation.

Using learning analytics

Lots of data provided. LMS provides some date. But it is under-used. Why? Suggestion is

This is largely the result of the lack of conceptual frameworks and resulting common understanding of how to use and interpret such data, and models that can validly and reliably align such data with the learning and teaching intent (Ferguson, 2012; Mazza & Dimitrova, 2007). At present, the available LMS data are neither easily understood by teachers as they align with individual and group student engagement behaviors (activity patterns) nor presented in ways that provide easy interpretation.

The following starts to get interesting

It is the conceptual bridging and understanding between the technical and educational domains that remains problematic for learning analytics (Dawson, Heathcote, & Poole, 2010). This leads to questions surrounding how analytics can begin to bridge the technical–educational divide to provide just-in-time, useful, and context-sensitive feedback on how well the learning design is meeting its intended educational outcomes.

The suggestion is that the critical step is “the establishment of methods for identifying and coding learning and teaching contexts”. i.e. the marriage of learning design and learning analytics.

Learning analytics to evaluate learning design

The aim here is to explore the importance of understanding pedagogical intent (i.e. learning design) for accurately understanding exactly what learning analytics is showing (in this case, a social network diagram).

In this case, they are using a diagram that shows a facilitator centric pattern. Pedagogical intent is shown as helping understand the value of the pattern

  • If it’s a Q&A forum on course content, then there’s alignment with the results and the intent.
  • If the instructor is absent and there’s a student answering questions, this might show successful delegation or it might show an absent instructor.
  • Might show early stages of a student introducing a topic
  • If the aim was to promote learner-to-learner interaction, then doesn’t seem to fit.

Not sure that this is capturing learning design, or simply learning intent.

Perhaps more interestingly

learning analytics may allow us to test those assumptions with actual student interaction data in lieu of self-report measures such as post hoc surveys. In particular, learning analytics provides us with the necessary data, methodologies, and tools to support the quality and accountability that have been called for in higher education.

Aligning learning analytics with learning design

This is where the propose two broad categories of analytics applications as a way to “align the two concepts”.

  1. Checkpoint analytics.

    Snapshot data indicating a student has met the pre-reqs for learning by accessing the relevant resources. e.g. downloading required files etc.

    The value here “lies in providing teachers with broad insight into whether or not students have accessed prerequisites for learning and/or are progressing through the planned learning sequence”.

  2. Process analytics.

    These “provide direct insight into learner information processing and knowledge application (Elias, 2011) within the tasks that the student completes as part of a learning design”. e.g. SNA of a student discussion activity gives data on level of engagement, peer relationships and therefore potential support structures. Adding content analysis offers the potential for determining the level of understanding etc.

    Not sure I get this “The articulation of the nature of support available within learning designs helps to interpret process learning analytics”.

Learning design and analytics investigation

This is where a theoretical scenario is used to illustrate the potential of the above ideas. Takes a learning design from the UoW learning design repository and looks at the type of analytics that might inform the teacher about how students are learning during implementation and how the design might be adapted.

Adapts the learning design figure with the addition of checkpoint and process learning analytics. Then describes these

  • Have they accessed the cases via the website, generate reminder alerts (teacher generated or automatic) to prompt late starters.
  • SNA to evaluate the effectiveness of each small groups discussion.
  • Similar with a teacher led, whole class discussion with some expectation of evolution over time.
  • Similar in small group project discussions, with an emphasis on identifying outliers etc.
  • Checkpoint and process at the end, including content analysis.

Conclusions

Next stages of research has several parallel directions, several of which I don’t grok at the moment.

  • Engaging teachers and students in understanding and using visual patterns of interaction to encourage learning activity

    What about improving the affordances of the tools to aid in this?

Mentions the rise of various predictive models for student performance and progression. These provide the opportunity to establish pedagogical recommendations. But there remains the need to inform these with contextual insight.

References

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing Pedagogical Action: Aligning Learning Analytics With Learning Design. American Behavioral Scientist. doi:10.1177/0002764213479367

Bugger analytics, just give me useful information

In a a prior post I wondered about the “end in mind” for university analytics projects. What follows outlines why I’m skeptical of such projects.

Starting about 15 years ago, most Australian Universities started spending tens of millions of dollars on Enterprise Resource Planning systems. These were systems intended to solve the information problems of universities. I remember the great promises proposed by Vice Chancellors about how much time and money would be saved through the adoption of these systems. How much easier it would be to get access to correct information and consequently improve university operations.

I’m especially interested in hearing whether there are other universities that have solved these problems and/or that this same problem exists elsewhere.

Student list by date enrolled

This morning I’d like to get a list of students in my course sorted by the date they enrolled. I need to modify my actions based on how late they enrolled. I can’t get this information at my current institution.

At my previous institution, I could not only get this information, I could also then contact students based on the date they enrolled. As per the following image.

Class list by enrol date

Of course, this wasn’t provided by the ERP system (my previous institution and my current institution have the same ERP). It was a “shadow system” I implemented.

Other missing information

The class list I can access from the current ERP doesn’t include the following information: email address, GPA, campus and a few others.

In terms of the courses I teach, it doesn’t appear that I can query the historical data and see what the distribution of grades were in previous offerings. This is somewhat problematic because the institutional processes require me to compare the current course results with the historical performance as part of the results moderation process. Just to make the point again

The organisation expects me to carry out a task, but doesn’t provide the information necessary to perform it

I’m sure I could go on. What about your institution? What information do you want/need, but can’t get through institutional provided systems?

Actually, if you’re interested, add your list of information to this Google doc.

Note: I’m not even going to comment on how difficult it is to get the information it does provide out of the enterprise system.

Analytics as a repeat of the ERP mistakes?

I wonder if the analytics push is simply the next phase in the ERP fashion process? Are Universities destined to make the same mistakes with analytics that they made with the ERP?

How can this be avoided? Is any institution likely to even be aware of this problem, let alone do something about it?

Which brings me back to the prior post. If the “end in mind” with learning analytics is the provision of useful information to students and teachers in ways that action can be taken, then it might be okay. But if the “end in mind” becomes installing Vendor Tool X because everyone else is doing it…..

What’s especially troubling is that I know of at least one Australian University that is explicitly heading down the “install tool X” approach since it is seen as saving money. i.e. if you implement the tool as vanilla and then focus on changing practices to suite the tool, it’s better and cheaper. Good luck with that.

Statistics in Education – Week 1

Have signed up for another MOOC – Statistics in Education for Mere Mortals – to fill a hole. What follows is the diary of the first week.

According to the syllabus, I’m already a bit behind.

So the instructor is doing some research around participation – one of the motivations for offering the course. Google map of participants. Seems I’m #2 for Queensland. Concentration seems to be Eastern US.

Canvas

The MOOC is being run using the Canvas LMS. Second time using the system. Am finding it interesting that there seems to be in-built support for the idea of a learning path. The series of activities/resources is sequential and the system seems to support that. The lack of support for this type of functionality in Moodle is something I’ve missed. Finding the ability to step through each step sequentially appealingly efficient.

Research questions

Will be interesting to see the research that comes of this. Have to admit to some of the questions leaving me a little underwhelmed.

First presentation

And the content begins. A 20 minute video. Lecture with a talking head in the bottom left hand corner, which disappears when the slides start. No annotation of the slides during the lecture, might have helped in places.

🙂 A “wii play station” as a type of video game console.

Good quote

Measurement is limiting the data … so that those data may be interpreted and, ultimately, compared to an acceptable qualitative or quantitative standard

Data limited by: measurement construct; instrument capability; amount of raw information we are prepared to deal with

Need to think about this applies to analytics. Data mining has approaches to get around this limitation of statistical approaches.

Metaphor

Isolating meaningful data when conducting most research studies is like …. filling a tea cup with a fire hose

Four scales of measurement

Different scales suggest different operations are possible.

  1. Nominal scale – Frequency distribution

    Nominal == name. Numbers are used as a name, not as a quantity. Doing arithmetic on these numbers is nonsensical.

  2. Ordinal scale – median and percentiles

    Ordinal == order/ranking. e.g. ranking preferred candidates.

  3. Interval scale – add or subtract, mean, standard deviation, standard error of the man
    • Has equal amounts of measurement.
    • Zero point established arbitrarily.

      e.g. temperature and 0 degrees.

    • Can determine, mean, standard deviation, and product moment correlation.
    • Can apply inferential statistical analysis.
  4. Ratio scale – ratio
    • Equal measurement units.
    • Has an absolute zero point.
    • Expresses values in terms of multiples and fractional parts and the ratios are true ratios (e.g. ruler )
    • Can determine geometric mean and percentage variation
    • Can conduct virtually any inferential statistical analysis

Measuring temperature is given as example of interval scale. Where you can’t say 40o is twice as hot as 20o. It’s just an interval, not a ratio. Where as length is.

Types of Statistics

Presents two

  1. Measures of central tendency – first module
  2. Measures of variability – second module

Measures of central tendency

  • Mean – average of a set of numbers
  • Median – the number at the midpoint of a set of numbers.
  • Mode – the most popular number.

All are the same in a normal distribution. But not in a skewed distribution.

So the statistics in this course assume a normal distribution. Seems limiting.

In passing the central tendency gets defined as the number that best represents a group of numbers. The explanation of median/mean would have been better illustrated visually, rather than by narration of a text-based powerpoint slide.

Normal distribution

Woo hoo. Narrated lecture + graphics tablet.

As the description of the normal distribution proceeds, I’m wondering how on earth I would ever be doing anything that would have data in a normal distribution? But perhaps just indicates the value of “central tendency”.

The Galton Machine

A bit of fun. Link to Java applet.

Computing the man of a set of scores

It appears that Excel will be the statistical software of choice. Perhaps including some auto marking of student work. The first is a simple task to test this out. Apparently going to take an hour to do. 11 minutes in, not sure how it could drag on that long.

It will be interesting to see how many questions arise from simple technical issues – like using different versions of Excel. Shall also be interesting to see how the difficulty of the activities grows.

Interesting that the quiz self evaluation asks for results in tenths, but the quiz system wants to add a couple of zeros to the end. I can see that throwing a few people off.

Woo hoo. 100%.

The next page after the quiz is a discussion forum for general help. There are a few folk reporting problems. Especially with the second and third questions. I’m guessing this arises from this combination of factors

  1. The video creating the spreadsheet only rounded off results in set of averages, and not the averages that would be used for the second and third questions. It didn’t need to, because that data meant the averages were rounded to a tenth
  2. The new data doesn’t result in data rounded to the tenth.
  3. The quiz question asks for results rounded to the tenth.

In entering the new data, I added the rounding. But I imagine others didn’t.

Appears that other folk didn’t modify their existing spreadsheet.

Ahh, other folk from China and Pakistan reporting being unable to access the YouTube videos.

Descriptive statistics – Standard Deviation

Onward and upward.

Statistic has two meanings: a description or an estimate about population.

Mm, the video didn’t do a great job of clearly defining the difference betwen sample and population. Use an example, but didn’t clearly define it. Google is my friend.

And here comes the terminology

  • Population – “a set of entities concerning which statistical inferences are to be drawn.
  • Sample – a subset of the population.

The questions

  1. Collecting data on a population?

    Describe the population using a parameter.

  2. Want to know about a population, but can only collect data on a sample?

    Sampling gives the sample and then we do a description using a statistic, which is then used to make an inference about the parameter for the population.

  3. Only collecting data on a sample?

Mmm, seems Broad is cleaning up in Durham.

Here comes the maths and symbols.

Population is mu. Sample is Summation of X i.e. X bar (a bar over the top of X).

So, we did central tendency above. That’s one type of descriptive statistic. Now it’s the other main type of descriptive statistics. Measures of variability. Kurtosis – shape of the curve, it’s peakedness.

Three methods to measure variability

  1. Range – Difference between high and low scores.

    Only tells the difference between two scores. Ignore the others.

  2. Deviation scores – Computer the difference between each score and the grand mean.

    Now take the average. Well the average of this always 0.

    This is described, could have been much better with a visual.

  3. Standard deviation

    Square the difference scores first (gets rid of the negatives). Then take the square root.

 by kxp130, on Flickr
Creative Commons Attribution-Noncommercial 2.0 Generic License  by  kxp130 

When estimating the population parameter for a sample has a N-1, rather than N. Why? Apparently a rule thumb developed over time, imagine there must be some research behind it. Instructor admits not a good explanation. It’s a problem area.

At this point, interesting that there hasn’t been much placement of why we’re doing this.

Another example with this table where some visualisation accompanying the verbal explanation would have helped.

And now the difficult stuff. What does a SD of 8.66 mean?

If all sold the same then SD = 0. If all sold about the same, then SD should be small. But what is considered small?

And now another Excel exercise, apparently 26 minutes of video == 90 minutes worth of activity. I don’t think so….really dragging now. Small win. Picked up an Excel tip.

2nd quiz done. I think I’ll stop there for the night. Time to watch some cricket and read a book.

Learning analytics – What is the "end in mind"?

Learning analytics is one of the new management fashions sweeping higher education. As a consequence, every Australian University I know has some sort of project around learning analytics underway. Some of these projects are actually considering how to help the folk at the coal face – students and teaching staff – use and benefit from learning analytics.

(Aside: It feels like the focus is more on the students than the teaching staff. This feels like a continuation of a recent trend where central institutional L&T is increasingly focusing on working directly with the students and bypassing the teaching staff, perhaps because it’s easier. But that’s a topic for another time).

The “end in mind” – a success factor

The broader business analytics field is starting to identify what they consider to be success factors. For example, this recent post titled “12 predictive analytics screw-ups” which lists 12 major mistakes (the inverse of a success factor?).

Number 1 on their list is

1. Begin without the end in mind

You’re excited about predictive analytics. You see the potential value of it. There’s just one problem: You don’t have a specific goal in mind.

I wonder if this is going to be a source of problems for learning analytics when applied to teaching staff (or perhaps students)?

How can you have a common “end in mind” for such a diverse set of people?

The problem of diversity

Is there any university, anywhere in the world, where there is a consistent pedagogical practice across all of its courses (units) and all of its teaching staff? I think not.

The “end in mind” is going to be different for different academics. The information they most need at any point in time is going to be different than others. In fact, it’s likely to be different across time.

Generic indicators such as engagement may be useful at a certain level. But when they are based on standard assumptions about the use of certain types of LMS tools at certain levels there are problems. For example, the pedagogy in the course I’m currently teaching is trying to push most of student engagement out onto their own blogs hosted on their choice of blog provider.

Retention: the solution

Most University learning analytics projects seem to focus on retention. i.e. the bottom line. There is some value in this, but the on-going focus on that is going to cause problems. For example, this recent story about a University considering the inclusion of student grades as part of staff appraisals. Goodhart’s law anyone?

How do you succeed?

If having an “end in mind” is a success factor for analytics projects, then what is an appropriate “end in mind” for learning analytics within a University?

What impacts will arise from the chosen “end in mind”? Will adoption be high? Will task corruption be high?