Some ideas for e-learning indicators – releasing half-baked ideas

The following is a quick mind dump of ideas that have arisen today about how you might make use of the usage data and content of course websites from course management systems (CMS) to find out interesting things about learning and teaching. i.e. Col is aiming to develop inidcators that might be of use to the participants associated with e-learning – management, support staff, academics, students etc.

This post is an attempt to live up to some of the ideas of Jon Udell that I discussed in this post about getting half-baked ideas out there. Col and I have talked a bit today and I’ve also regained some prior thoughts that I’m putting down so we don’t forget it.

The major benefit of getting these half-baked ideas out there are your thoughts. What do you think? Want to share your half-baked ideas?

The fundamental problem

How do you identify/generate useful indicators that might be harnessed to act as weak signal detectors? How can we use all of this data about e-learning to generate something useful?

Disclaimer

It is fully understood that drawing simply upon usage data and other electronic data can never tell you the full story about a student’s learning experience or the quality of the teaching. At best it can indicate that something might be there, in almost all cases further investigation would be required to be certain.

For example, lots of discussion on a course discussion forum with lots of people responding to each other might be indicative of a good learning experience. It might be indicative of an out of control flame war.

However, knowing a little bit more about what’s going on and applying it sensible will helpfully be of some use.

The following are propositions about what might be interesting indicators. These need to be tested, both quantatively and qualitatively.

Content correlations?

It’s fairly widely accepted that most CMSes are generally used primarily as content repositories. Academics put lots of documents and other content into them for students to access. In some cases the ability of the CMS to act in some other purpose (e.g. to encourage discussion and collaboration) is significantly limited by the quality and variety of the tools they provide for these services and also some of the fundamental assumptions built into the CMSes (e.g. you have to login to acces the services).

If content is the primary use, is there anything useful that can be gained from. What I can think of includes:

• If there is no or little information then this is bad.
  If the course site doesn’t contain anything, that’s probably a sign of someone who is not engaging with teaching. Courses with little or no content could be a negative indicator.
• If the information is structured well, then it is good.
  Poor structure again may indicate some less than knowledgable. Almost all CMSes use a hierarchical structure for information. If all the content is located within 1 of 7 parts of the hierarchy, things may not be good.
• If the content is heavily used, then this might imply usefulness.
  If students are using content heavily and that heavy use is consistent across most content this might indicate well designed content, which might be a good thing.
• If the content is primarily the product of publishers, then this might be bad.
  A course that relies almost entirely for content from a textbook publisher might suggest an experience that is not customised to the local context. It might suggest an academic taking the easy way out. Which might indicate a less than positive outcome.
• A large average # of hits on course content per student, might be positive indicator.
  If, on average, all of the students use the course content more, this may indicate more appropriate/useful material which might indicate a good learning experience.

Looking for particularly strong courses (see images below) might lead to the following being of interest

• Percentage of total content per course.
  See images below. Essentially, courses with a greater percentage might be better.
• Percentage of total requests.

Percentage of staff using the system

A simple one, the greater the percentage of the employed teaching staff using the system, the better.

An example

The following images illustrate how this was used in this presentation to compare and contrast usage of Webfuse after a period using the wrong development process and then after a period of using a better development process (remember the disclaimer above).

Results of “bad” process

Results of “good” process