Month: March 2012

Following on from recent posts around learning analytics, this post is going to try and drill down a bit further on one of the useful questions around analytics identified by Shelia MacNeil’s summary of a Gardner Campbell talk. The questions

What are the really useful “analytics” which can help me as a learner, teacher, administrator, etc? Which systems provide that data just now ? How can/do these stakeholders access and share the data in meaningful ways? How can we improve and build on these systems in ways which take into account the complexity of learning? Or as Gardner said, how can we start framing systems and questions around wisdom?

I’m going to focus on “What are the really useful “analytics” which can help me as a learner, teacher, administrator, etc?” and in particular on the “useful analytics” – I’ll call them patterns – that the Indicators project has identified so far.

So far, the Indicators project has really only looked for these useful patterns within one institution, though with a fairly large sample. The definition of useful here is something like “potentially of interest to learners, teachers or administrators”. In the short term, I’m hoping we can extend both the sample and also the number of institutions.

A side step into purpose

Before getting into that, a few words about the purpose of learning analytics. Barneveld et al (2012) suggest that there are some common business reasons behind analytics

: increasing financial/operational efficiency; expanding local and global impact; establishing new funding models during a changing economic climate; and responding to the demands for greater accountability

which appears to be taken from an IBM white paper

There follows an argument and some references that suggest analytics can help management a great deal to “cut costs and improve teaching and learning”. So much so that the following sentence repeats the mantra “improving efficiencies to saving money to enhancing student achievement”

While that’s certainly a potential application of analytics, like others I find this a limiting perspective. It almost automatically entails the sort of simplification that Gardner Campbell was arguing against.

Instead, I’m starting to lean more toward the idea of analytics being yet another “blind man” available to people to discover what is going on around e-learning/learning. Like the other blind men of e-learning research (I’ll use that term for want of a better label) – surveys, interviews etc. – it has its limitations (Gardner expressed a big picture limitation) but perhaps combined they can help.

Perhaps the indicators project is more about helping combine the perspectives blind men in order to better understand what is going on. At this stage, just maybe the managers can get their hands on it. But not until we’ve tested the “sight” of the analytics blind man. Which leads nicely into the next section.

The patterns

Much of the initial work of the indicators project has simply been to investigate what the analytics blind man can actually see. What were the patterns in the usage data? Some of this exploration was inspired by the work of others. We were asking, “Does that same pattern hold here?”. We were particularly interested in seeing if what patterns emerged from cross-LMS, cross-institutional, and longitudinal data. Here’s a quick list of what we’ve looked at so far, more explanation following.

First, the list

• Does LMS feature adoption change over time and between systems?
• Is there a link between LMS activity and student grades?
• Is there a link between LMS activity and external factors (e.g. staff participation, staf background, instructional design involvement, student age)?
• Investigating critical success factors.
• Differences between LMSs.

Feature adoption

An LMS comes with a host of features. Which features are used? Does feature adoption change over time? Based on the work of Malikowski et al (2006, 2007, 2008) we initially examined feature adoption within a locally grown LMS, Blackboard use at the same institution, and eventually Moodle feature adoption.

Malikowski et al identified five categories of LMS features. Our initial exploration of four predominant features at the one institution from 2005-2009 revealed some widely different results as summarised in the following graphs. Some explanation of the graphs

• The green and purple lines represent the top and bottom ranges found by Malikowski et al.
• The black continuous line represents feature adoption with this institution’s version of the Blackboard LMS.
• The black dashed line represents feature adoption with a locally produced “LMS”.

The transmit content usage for Wf (locally produced LMS) only includes optional content distribution. Wf automatically produced course websites with a range of information. Academics could optionally add more material.

Use of class interaction features were significantly higher in Wf than Blackboard and what was found by Malikowski et al.

Student activity and grades

Dawson et al (2008) found significant differences between low and high performing students in the quantity of online sessions times, total time online and the amount of active participation in discussion forums. i.e. the more usage of the LMS, the better the grade for students.

We found that this generally applied for our students whether “usage” was measured by visits to the LMS, posts to discussion forums, or replies to discussion forums. However, it didn’t apply to all groups of students. The institution had three types of students. The following graph shows how one group (the dotted line) don’t show this pattern. With this group of students, students who got high distinctions and distinctions had less LMS usage than the students with a Credit.

Impact of external factors

We then explored whether other factors influenced this link between student usage and grade.

For example, would online courses taught by the education academics (teacher education) be any different than the other courses. The following graph shows somewhat similar trends, but it also shows that education (EDED) courses have discussion forums that are, on average, visited less than other courses.

And when looking at the average number of posts/replies by students the expected pattern breaks down. Especially for replies.

We found that staff involvement with course sites made a difference. The following graphs show the usage/grade pattern for courses where staff accessed the course website less than 100 times during the course.

We also found that age changed the pattern and level of usage. Older students used the LMS more. Younger students less.

Critical success factors

The choice of external factors was informed by Fresen’s (2007) work, which identified a “taxonomy of factors to promote quality web-supported learning”.

Differences between LMSs

Beer et al (2010) compared various patterns between Blackboard (pre-2010) and Moodle (2009 and beyond). Findings included

• Number of clicks on Moodle course sites somewhat lower than on Blackboard.
• Average time on site was about the same between the two LMS.
• The average number of pages per visit on Moodle was less than for Blackboard (11.37 versus 24.51).

References

Barneveld, A. V., Arnold, K., & Campbell, J. P. (2012). Analytics in Higher Education: Establishing a Common Language. Business (pp. 1-11). Retrieved from http://net.educause.edu/ir/library/pdf/ELI3026.pdf

Beer, C., Clark, K., & Jones, D. (2010). Indicators of engagement. Curriculum, technology and transformation for an unknown future. Proceedings of ASCILITE Sydney 2010 (pp. 75-86). Sydney. Retrieved from http://ascilite.org.au/conferences/sydney10/procs/Beer-full.pdf

Dawson, S., McWilliam, E., & Tan, J. P. L. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. Melbourne. Retrieved from http://www.ascilite.org.au/conferences/melbourne08/procs/dawson.pdf

Fresen, J. (2007). A taxonomy of factors to promote quality web-supported learning. International Journal on E-Learning, 6(3), 351-362.

Malikowski, S. (2008). Factors related to breadth of use in course management systems. Internet and Higher Education, 11(2), 81-86.

Malikowski, S., Thompson, M., & Theis, J. (2006). External factors associated with adopting a CMS in resident college courses. Internet and Higher Education, 9(3), 163-174.

Malikowski, S., Thompson, M., & Theis, J. (2007). A model for research into course management systems: bridging technology and learning theory. Journal of Educational Computing Research, 36(2), 149-173.

Continuing a re-engagement with analytics I spent some time listening to Gardner Campbell’s talk to the LAK’12 MOOC – Here I Stand and from there followed various links.

Gardner captures one of my major concerns with how analytics may proceed, especially within institutions that are increasingly driven by accountability, efficiency and other concerns. Concerns that they are responding to with top-down management. Gardner’s uses the metaphor of the human mind/learning being as complex as M-Theory (actually more complex) and that learning analytics as commonly thought of is equivalent to measuring M-Theory using a simple cartesian graph.

The end result is that it simplifies learning and how we treat to an extent that is meaningless.

He connects this view of analytics with the LMS approach to e-learning and the traditional nature of curriculum that are all in the simple domain. Learning analytics just continues this. Lots of imagery with school as a feedlot or a Skinner box.

Gardner talks about four strong cautions for analytics
Four strong cautions

1. “Student success”
   Typically defined within analytics as the student passing doesn’t mean the same as succeeding in life. e.g. given of high performing high school student with no idea of what to do next.
2. Complexity
   A lot on this that resonates, more below.
3. Points of “Intervention”
   Just one idea is that of an analytics system that, rather than intervening just before the student fails (as most current analytics projects are trying to achieve), intervenes just as the students begins to understand.
4. The “Third wave”

Draws on John Naughton – From Gutenberg to Zuckerberg: What you really need to know about the Internet (2012) – to illustrate “Complexity is the new Reality”

1. Non-linear
2. Feedback matters – a lot
3. Systems demonstrate self-organisation
4. EMERGENCE – synergies – new phenomena

Naughton also talks about double loop learning where “success means more than positive outcomes “relative to pre-established targets” – Which sounds very much like learning objectives – Instead it means that learners “need to reflect on the appropriateness, in the light of unfolding events, of the assumptions (the mental model) used to set up those actions and targets”

Gardner’s quote (or close to it somewhere in here) is “A real learning analytics system must be able to learn.”

Also mention of the Pardox of the Active User

Other links

Shelia MacNeil offers another summary of Gardner’s talk and points to other work. It was from Shelia’s post that I came across Exploiting activity data in the academic environment which is a somewhat broader example of analytics including some useful insights into privacy etc issues around the data.

Shelia identifies some very useful questions

What are the really useful “analytics” which can help me as a learner, teacher, administrator, etc? Which systems provide that data just now ? How can/do these stakeholders access and share the data in meaningful ways? How can we improve and build on these systems in ways which take into account the complexity of learning? Or as Gardner said, how can we start framing systems and questions around wisdom?

An earlier blog post from Gardner that arose out of reading this book (really learning to dislike book’s that aren’t available on Kindle).

Implications for the indicators project

The types of questions identified are exactly the areas which the Indicators Project was (and is about to start again) attempting to explore. The point about complexity is also timely, as that is the perspective that will underpin our work. Consequently I will be reading a bit more of Naughton.

I especially like the point about double loop learning. For three main reasons

1. It captures one important distinction between traditional business intelligence approaches and what we hope to do with the indicators project.
2. It highlights how we’d like to use analytics, i.e. to help university academics engage in double loop learning about how and why they teach.
3. It frames a concern I have out the outcomes focus of much university education, i.e. we’re measuring students against outcomes we think are important and we’ve established ahead of time, rather than asking them to reflect on their assumptions and mindsets.
   In particular, I’m thinking this might be an interesting point of departure for thinking about how courses I’m responsible for might evolve.

It looks like we’re going to start playing around with the Indicators project again. So it’s time to start reading up a bit on the learning analytics literature and see where the interesting work is. First up is

Phillips, R., Maor, D., Cumming-Potvin, W., Roberts, P., Herrington, J., Preston, G., Moore, E., et al. (2011). Learning analytics and study behaviour: A pilot study. In G. Williams, P. Statham, N. Brown, & B. Cleland (Eds.), ASCILITE 2011 (pp. 997-1007). Hobart, Tasmania.

The following is a summary/reflection of reading through it.

Abstract

Describes a study that looks closely at analytics data of four students – interacting mainly (it appears) with lecture capture software and then interviewed them to find check out the assumptions from analytics. It found that the analytics had some limitations which was supplemented nicely by the qualitative data. Makes some suggestions for further research around the methodology.

Aim is to find out how students engage with and study in e-learning environments. But this study focuses on the lecture capture tool

Background/intro

Makes mention of “Much e-learning research over the years has been based on quantitative data largely derived from the perceptions of students” which has some limitations. Other work is qualitative around individual contexts. This work is seeking to combine descriptive/qualitative with learning analytics.

Analytics has a history, but the meaning of the data is not always clear. usage logs record behaviour, without explaining why suggesting taking care to analyse and interpret the data.

Prior work

Some mention of the lecture capture work. Which is seen to focus on the technology, not the learning environment as a whole.

This was the impetus for our current work, which holistically examines a unit of study, and uses learning analytics to gain a richer understanding of what students actually do in a technology-enhanced learning environment.

The focus being on learning processes used by students, rather than learning outcomes.

Lectopia usage patterns

Summarises prior work around analytics from lecture capture system and identifying “types” of student users: conscientious, good-intentioned, repetant, bingers, crammers, one-hit wonders, and random users.

This work reports on a pilot study interviewing students with diverse usage patterns to find out what’s going on.

Method

..pragmatic, mixed-methods paradigm of inquiry using a modified design-based research approach..

Data sources: their analysis tool, SNAPP, standard LMS usage reports, assessment results, attendance logs for lectures, interviews with teachers, and semi-structured interviews with students.

Received ethics approval for an approach that included identifying students and handling this appropriately. Some difficulties getting a good sample of students to interview – became a “convenience sample”

Students drawn from a 3rd year sociology of education course. ~150 students on main campus, ~50 students on regional campus, ~100 DE students. 1 hour lecture and 2 hour workshop for internals. Lectopia plus discussion forum activities for DEs. Numerous readings.

109 students accessed lecture recordings

Recording usage appears to match the expected peaks (lead up to assessment) and troughs (other times).

Interviewed students were all internal.

Discussion

Interview data from 4 students is summarised. And shows very different approaches to study.

the four cases reported here start to illustrate some of the complexity of the modern, technologyenhanced learning environment.

A range of limitations of the study are identified and then

two major implications arose from this analysis: a need for the broadening of the mechanisms for identifying student behaviour patterns; and the application of the methodology to other contexts.

Student don’t visit the lecture capture system enough to give useful data. Broader LMS usage needs to be examined.

Suggests the addition of a mid-term survey of students about their perceptions of technology use and wider range of data.

Talks about selecting more units to include by reviewing courses for alignment.