Category: lmsEvaluation Page 1 of 2

As part of the thesis I’ve been engaging with some of the literature around LMS feature usage to evaluate usage of Webfuse. A good first stab of this was reported in an earlier post. There were a number of limitations of that work, it’s time to expand a bit on it. To some extent for the PhD and to some extent because of a paper.

As with some of the other posts this one is essentially a journal or a log of what I’m doing and why I’m doing it. A permanent record of my thinking so I can come back later, if needed.

There’s even an unexpected connection with power law distributions towards the end.

Content distribution

In that previous post I did not include a graph/any figures around the use of Webfuse course sites to distribute content or files. This is because Webfuse had a concept of a default course site. i.e. every course would have the same basic default site created automatically. Since about 2001 this meant that every course site performed some aspect of information distribution including: the course synopsis on the home page, details about the course assessment, details about course resources including textbook details and a link to the course profile, and details about the teaching staff.

Beyond this staff were able to upload files and other content as they desired. i.e. moving beyond the default course site was optional and left entirely up to the teaching staff. Some of us, perhaps went overboard. Other staff may have been more minimal. The aim here is to develop metrics that illustrate that variability.

Malikowski et al (2007) have a category of LMS usage called Transmitting Content. The LMS features they include in this category include:

• Files uploaded into the LMS.
• Announcements posted to the course site.

So, in keeping with the idea of building on existing literature. I’ll aim to generate data around those figures. Translating those into Webfuse should be fairly straight forward, thinking includes:

• Files uploaded into the LMS.
  Malikowski et al (2007) include both HTML files and other file types. For Webfuse and its default course sites I believe I’ll need to treat these a little differently:
  • HTML files.
    The default course sites produce HTML. I’ll need to exclude these standard HTML files.
  • Other files.
    Should be able to simply count them.
  • Real course sites.
    Webfuse also had the idea of a real course site. i.e. an empty directory into which the course coordinator could upload their own course website. This was usually used by academics teaching multimedia, but also some others, who knew what they wanted to do and didn’t like the limitations of Webfuse.
• Announcements.
  The default course site has an RSS based announcements facility. However, some of the announcements are made be “management”. i.e. not the academics teaching the course but the middle managers responsible for a group of courses. These announcements are more administrative and apply to all students (so they get repeated in every course). In some courses they may be the only updates. These announcements are usually posted by the “webmaster”, so I’ll need to exclude those.

Implementation

I’ll treat each of these as somewhat separate.

• Calculate # non-HTML files.
• Calculate # of announcements – both webmaster and not.
• Calculate # HTML files beyond default course site (I’ll postpone doing this one until later)

Calculate # non-HTML files.

Webfuse created/managed websites. So all of the files uploaded by staff exist within a traditional file system. Not in a database. With a bit of UNIX command line magic it’s easy to exact name of every file within a course site and remove those that aren’t of interest. The resulting list of files is the main data source that can then be manipulated.

The command to generate the main data source goes like this

find T1 T2 T3 -type f | get all the files for the given terms
grep -v ‘.htm$’ | grep -v ‘.html$’ | remove the HTML files
grep -v ‘CONTENT$’ | remove the Webfuse data files
grep -v .htaccess | remove Apache access restriction file
grep -v ‘updates.rss$’ | remove the RSS file used for announcements
grep -v ‘.ctb$’| grep -v ‘.ttl$’ | grep -v ‘/Boards/[^/]*$’ | grep -v ‘/Members/[^/]*$’ | grep -v ‘/Messages/[^/]*$’ | grep -v ‘/Variables/[^/]*$’ | grep -v ‘Settings.pl’ | remove files created by discussion forum
sed -e ‘1,$s/.gz$//’

The sed command at the end removes the gzip extension that has been placed on all the files in old course sites that have been archived – compressed.

The output of this command is the following

T1/COIT11133/Assessment/Assignment_2/small2.exe
T1/COIT11133/Assessment/Weekly_Tests/Results/QuizResults.xls
T1/COIT11133/Resources/ass2.exe

The next aim is to generate a file that contains the number of files for each course offering. From there the number of courses with 0 files can be identified, as can some other information. The command to do this is

sed -e ‘1,$s/^(T./………/).*$/1/’ all.Course.Files | sort | uniq -c | sort -r -n > count.Course.Files

After deleting a few entries for backup or temp directories. We have our list. Time to manipulate the data, turn it into a CSV file and into Excel. Graph below, fairly significant disparity in number of files – the type of curve looks very familiar though.

In total, for 2005 there were 178 course sites that had files. That’s out of 299 – so 59.5%. This compares to the 50% that Col found for the Blackboard course sites in the same year.

Calculate # of Announcements

The UNIX command line alone will not solve this problem. Actually, think again, it might. What I have to do is:

• For each updates.rss
  • count the number of posts by webmaster
  • count the number of posts by non-webmaster
  • output – courseOffering,#webmaster,#non-webmaster

Yep, a simple shell script will do it

echo COURSE,ALL,webmaster
for name in `find T1 -name updates.rss`
do
  all=`grep '' $name | wc -l`
  webmaster=`grep 'webmaster' $name | wc -l`
  echo "$name,$all,$webmaster"
done

Let’s have a look at the 2005 data. Remove some dummy data, remove extra whitespace. 100% of the courses had updates. 166 (55%) had no updates from the teaching staff, 133 (45%) did. That compares to 77% in Blackboard. Wonder if the Blackboard updates also included “webmaster” type updates?

In terms of the number of announcements contributed by the teaching staff. The following graph shows the distribution. The largest number for a single offering was 34. Based on a 12 week CQU teaching term, that’s almost, on average, 3 announcements a week

Power laws and LMS usage?

The two graphs above look very much like a power law distribution. Clay Shirky has been writing and talking about power law distributions for some time. Given that there appears to be a power law distribution going on here with usage of these two LMS features, and potentially that the same power law distribution might exist with other LMS features, what can Shirky and other theoretical writings around power law distributions tell us about LMS usage?

References

Malikowski, S., Thompson, M., & Theis, J. (2007). A model for research into course management systems: bridging technology and learning theory. Journal of Educational Computing Research, 36(2), 149-173.

This post follows on from a previous post and continues an examination of some papers written by Malikowski and colleagues examining the adoption of features of an LMS/VLE/CMS. This one focuses on the 2006 paper.

External factors associated with CMS adoption

The abstract of this paper (Malikoski, Thompson and Theis, 2006) is

Course management systems (CMSs) have become a common resource for resident courses at colleges and universities. Researchers have analyzed which CMS features faculty members use most primarily by asking them which features are used. The study described builds on previous research by counting the number of CMS features a faculty member used and by analyzing how three external factors are related to the use of CMS features. The external factors are (a) the college in which a course was offered, (b) class size, and (c) the level of a class—such as 100 or 200. The only external factor showing a statistically significant relationship to the use of CMS features was the college in which a course was offered. Another finding was that CMSs are primarily used to transmit information to students. Implications are described for using external factors to increase effective use of more complex CMS features.

Implication: repeat this analysis with the Webfuse and Blackboard courses at CQU We can do this automatically for a range of external factors beyond those.

Point of the research

Echoing what was said in the 2008 paper and one reason I am interesting in this work

Faculty members often receive help in using a CMS (Arabasz, Pirani, & Fawcett, 2003; Grant, 2004). This help typically comes from professionals who focus on instructional design or technology. Information about which features are used most could provide these professionals with a starting point for promoting effective use of more complex CMS features. Information about how external factors influence use could identify situations in which more complex features can be successfully promoted.

Prior research

Points of difference between this research and prior work are listed as:

• Few studies have focused on the use of the CMS in resident courses.
• Generated from surveys.
• Morgan’s suggestion that faculty use more features over time may only be partially correct.
• Only one study used a statistical analysis.
• Previous studies analyse usage for all staff or for a broad array of staff – focusing on a few factors might be a contribution.
• Lastly, add to research by including examination of how people learn.
  

  Currently, research into CMS use has considered CMS features, opinions from teachers about these features, and student satisfaction with CMS features. Gagné, Briggs, and Wager summarize the importance of considering both learning psychology and technology, which they refer to as “media.” They emphasize “the primacy of selecting media based on their effectiveness in supporting the learning process. Every neglect of the consideration of how learning takes place may be expected to result in a weaker procedure and poorer learning results.” (Gagné, Briggs, & Wager, 1992, p. 221). For decades, researchers have studied how teaching methods affect learning outcomes. Several recent publications describe seminal research findings, research that has built on these findings, and learning theories that have emerged from this research (Driscoll, 2005; Gagné et al., 1992; Jonassen & Association for Educational Communications and Technology, 2004; Reigeluth, 1999).

  That is as may be. But given my suspicion that most academic don’t really make a rational judgements about how they teach based on educational literature – would such an analysis be misleading and pointless?

They argue that the model from Malikowski, Thompson and Theis (2007) is what they use here and that it combines both features and theory and can be used to synthesise research.

Methodology

Interestingly, they have a spiel about causation and relationship

An important point to clarify is that the method applied in this study was not intended to determine if external factors caused the use of CMS features. Identifying causation is an important but particularly challenging research goal (Fraenkel &Wallen, 1990). Instead, the current method and study only sought to determine if significant relationships existed between external factors and the adoption of specific CMS features.

Looks like basically the same methodology and perhaps same data set as the 2008 paper. They do note some problems with the manual checking of course sites

This analysis was a labor intensive process of viewing a D2LWeb site for a particular course and completing a copy of the data collection form, by counting how often features in D2L were used. In some cases, members of the research team counted thousands of items.

Results

The definition of adoption used is different than that in the 2008 paper

In this study, a faculty member wasconsidered to have adopted a feature if at least one instance of the feature was present. For example, if a faculty member had one quiz question for students, that faculty member was considered as having adopted the quiz feature

Only 3 of the 13 LMS features available were used by more than half the faculty – grade book, news/announcements, content files. Also the only 3 features where the percentage of adoption was greater than the standard deviation.

Implication: A comparison of Webfuse usage using different definitions of adoption could be interesting as part of a way to explore what would make sense as a definition of adoption.

In some cases STDDEV was twice as large as the percentage of faculty members using a feature.

They include the following pie chart that is meant to use the model from Malikowski et al (2007). But I can’t, for the life of me, figure out how they get to it.

Found that only the college (discipline) could be said to be the only external factor that was a significant predictor of feature usage.

Discussion

Raises the question of norms and traditions within disciplines driving CMS feature adoption. I’m amazed more isn’t made of these being residential courses. This might play a role.

Implication: It might be argued that norms and tradition are more than just discipline based. I would argue that at CQU, when it comes to online learning that there were three main traditions based on faculty structures from the late 1990s through to early noughties:

1. Business and Law – some courses with AIC students, very different approach to distance education and also online learning. Had a very strong faculty-based set of support folk around L&T and IT.
2. Infocom – similar to Business and Law in terms of AIC courses and distance education. But infected by Webfuse and similar to BusLaw had a strong faculty-based set of support folk around L&T and IT.
3. Others – essentially education, science, engineering and health. Next to no AIC students. Some had no distance education. No strong set of faculty-based support folk around IT and L&T> Though, education did have some.

Would be interesting to follow/investigate these norms and traditions and how that translated to e-learning. Especially since the faculty restructure around 2004/2005 meant there was a mixing of the cultures. BusLaw and large parts of Infocom merged. Parts of Infocom merged with education and arts….

Limitations

Study involved 81 faculty members as opposed to 862, 730, 192 and 191 in other studies. Argument is that those other studies used surveys, not the more resource intensive approach used by this work.

The recognise the problem with change

The current study analyzed CMS Web sites when they were on a live server. The limitation in this case is that a faculty member can change a Web site while it is being analyzed. Fortunately, the university at which this study occurred has faculty members create a different CMS Web site each time a course is offered.

References

Malikowski, S., Thompson, M., & Theis, J. (2006). External factors associated with adopting a CMS in resident college courses. Internet and Higher Education, 9(3), 163-174.

Malikowski, S., Thompson, M., & Theis, J. (2007). A model for research into course management systems: bridging technology and learning theory. Journal of Educational Computing Research, 36(2), 149-173.

Malikowski, S. (2008). Factors related to breadth of use in course management systems. Internet and Higher Education, 11(2), 81-86.

As a step towards thinking about how you judge the success of an LMS/VLE, this post looks at some work done by Steven Malikowski. Why his work? Well he is co-author on three journal papers that provide one perspective on the usage of features of an LMS, including one that proposes a model for research into course management systems. A list of the papers in the references section.

This post focuses on looking at the 2008 paper. On the whole, there seems to be a fair bit of space for research to extend and improve on this work.

Factors related to breadth of use

The abstract of this paper (Malikowski, 2008) is

A unique resource in course management systems (CMSs) is that they offer faculty members convenient access to a variety of integrated features. Some featurs allow faculty members to provide information to students, and others allow students to interact with each other or a computer. This diverse set of features can be used to help meet the variety of learning goals that are part of college classes. Currently, most CMS research has analyzed how and why individual CMS features are used, instead of analyzing how and why multiple features are used. The study described here reports how and why faculty members use multiple CMS features, in resident college classes. Results show that nearly half of faculty members use one feature or less. Those who use multiple features are significantly more likely to have experience with interactive technologies. Implications for using and encouraging the use of multiple CMS features are provided.

Suggests that cognitive psychology is the theoretical framework used. In particular, the idea that there are discrete categories of learning goals ranging from simple to complex and that learners that don’t master the simple, first, will have difficulties if they attempt the more complex. An analogy is made with the use of a CMS, there are simple features that need to be learned before using complex features.

In explaining previous research on adoption of features of an LMS (mostly his own quantitative evaluations) the author reports that the college/discipline an academic is in explains most variation.

How to use these findings

The point is made that a CMS is used to transmit information more than twice as much as it is used for anything else. Also, that there are cheaper and better ways to transmit information.

The suggestion is then made that

Instructional designers, researchers, and others interested in increasing effective CMS use can use the research just summarized to emphasize factors that are related to the use of uncommon CMS features and deemphasize factors that are not related to increased use.

But the best advice that is presented is that if you wish to promote use X, then encourage it in discipline Y first since they have shown interest in related features. Then, after generating insight, seek to take it elsewhere…….??

Use of multiple features

Only a small number of studies have focused on use of multiple features. Most achieved by asking academics how they use the CMS. Suggests that a second way is to visit course sites and observe which features are used. Suggests that observing behaviour is more accurate than asking them how they behave.

IMPLICATION: the approach Col and Ken are using for Blackboard and what I’m using for Webfuse is automated. Not manual. A point of departure.

Methodology

Three bits of data were used

1. Usage of 6 common CMS features
   • Random sample of 200 staff at US institution using D2L were asked to participate – 81 chose to participate.
   • 154 D2L sites were analysed as staff teach more than one course a semester
   • 2 research team members visited and manually analysed each course site – repeating until no discrepancies.
2. External factors: class size, the college/discipline and class level (1st, 2nd year etc)
   Gathered manually from the course site.
3. 10 internal factors focused primarily on the faculty members’ previous experience with technology.
   Gathered by surveying staff.

Limitation: I wonder if D2L has any adaptive release mechanisms like Blackboard. Potentially, if the team member visiting each course site has an incorrectly configured user account, they may not be able to see everything within the site.

Purpose was to determine if internal or external factors were related to adoption of multiple CMS features. Established using a regression analysis with the dependent variable being the number of features adopted and the independent variables being the 3 external and 10 internal factors.

What is adoption?

This is a problem Col and I have talked about and which I’ve mentioned in some early posts looking at Webfuse usage. The definition Malikowski used in this study was

In this study, adopting a feature was defined as a situation where a D2L Web site contained enough instances of a feature so this use was at or above the 25th percentile, for a particular feature. For example, if a faculty member created a D2L Web site with 10 grade book entries, the grade book feature would have been adopted in this Web site, since the 25th percentile rank for the grade book feature is 7.00. However, if the same Web site contained 10 quiz questions, the quiz feature would not have been adopted since the 25th percentile rank for quiz questions is 12.25

I find this approach troubling. Excluding a course from adopting the quiz feature because it has only 10 questions seems harsh. What if the 10 questions were used for an important in class test and was a key component of the course. What if there are a few courses that have added all of the quiz questions provided with the textbook into the system.

Implication: There’s an opportunity to develop and argue for a different – better – approach to defining adoption.

Sample of results

• 36% of sites used only 1 feature
• 72% of sites used 2 or less features
• 0% of sites used all 6 features
• Only four of the external/internal factors could be used to predict the number of CMS features adopted
  1. Using quizzes
  2. College of social science
  3. Using asynchronous discussions
  4. Using presentation software (negative correlation)

Discussion

Suggests that the factors found to predict multiple feature use can be used to guide instructional designers to work with these faculty to determine what works before going to the others.

Limitation: I don’t find this a very convincing argument. I start to think of the technologists alliance and the difference between early adopted and the majority. The folk using multiple LMS features are likely to be very different than those not using many. Focusing too much on those already using many might lead to the development of insight that is inappropriate for the other category of user.

Implication: There seems to be some research opportunities that focuses on identifying the differences between these groups of users by actually asking them. i.e. break academics into groups based on feature usage and talk with them or ask them questions designed to bring out differences. Perhaps to test whether they are early adopters or not.

References

Malikowski, S., Thompson, M., & Theis, J. (2006). External factors associated with adopting a CMS in resident college courses. Internet and Higher Education, 9(3), 163-174.

Malikowski, S., Thompson, M., & Theis, J. (2007). A model for research into course management systems: bridging technology and learning theory. Journal of Educational Computing Research, 36(2), 149-173.

Malikowski, S. (2008). Factors related to breadth of use in course management systems. Internet and Higher Education, 11(2), 81-86.

My PhD is essentially arguing that most institutional approaches to e-learning within higher education (i.e. the adoption and long term use of an LMS) has some significant flaws. The thesis will/does describe one attempt to formulate an approach that is better. (Aside: I will not claim that the approach is the best, in fact I’ll argue that the notion of there being “one best way” to support e-learning within a university is false.) The idea of “better” raises an interesting/important question, “How do you measure success with institutional use of an LMS?” How do you know if one approach is better than another?

These questions are important for other reasons. For example, my current institution is currently implementing Moodle as its new LMS. During the selection and implementation of Moodle there have been all sorts of claims about its impact on learning and teaching. During this implementation process, management have also been making all sorts of decisions made about how Moodle should be used and supported (many of which I disagree strongly with). How will we know if those claims are fulfilled? How will we know if those plans have worked? How will we know if we have to try something different? In the absence of any decent information about how the institutional use of the LMS is going, how can an organisation and its management make informed decisions?

This question is of increasing interest to me for a variety of reasons, but the main one is the PhD. I have to argue in the PhD and resulting publications that the approach described in my thesis is in some way better than other approaches. Other reasons include the work Col and Ken are doing on the indicators project and obviously my beliefs about what the institution is doing. Arguably, it’s within the responsibilities of my current role to engage in some thinking about this.

This post, and potentially a sequence of posts after, is an attempt to start thinking about this question. To flag an interest and start sharing thoughts.

At the moment, I plan to engage with the following bits of literature:

• Malikowski et al and related CMS literature.
  See the references section below for more information. But there is an existing collection of literature specific to the usage of course management systems.
• Information systems success literature.
  My original discipline of information systems has, not surprisingly a big collection of literature on how to evaluate the success of information systems. Some colleagues and I have used bits of this literature in some publications (see references).
• Broader education and general evaluation literature.
  The previous two bodies of literature tend to focus on “system use” as the main indicator of success. There is a lot of literature around the evaluation of learning and teaching, including some arising from work done at CQU. This will need to be looked at.

Any suggestions for other places to look? Other sources of inspiration?

Why the focus on use?

Two of the three areas of literature mentioned above draw heavily on the level of use of a system in order to judge its success. Obviously, this is not the only measure of success and may not even be the best one. Though the notion of “best” is very subjective and depends on purpose.

The advantage that use brings is that it can, to a large extent, be automated. It can be easy to generate information about levels of “success” that are at least, to some extent, better than having nothing.

At the moment, most universities have nothing to guide their decision making. Changing this by providing something is going to be difficult. After all, providing the information is reasonably straight forward. Changing the mindset and processes at an institution to take these results into account when making decisions…..

Choosing a simple first step, recognising it’s limitations and then hopefully adding better measures as time progresses is a much more effective and efficient approach. It enables learning to occur during the process and also means if priorities or the context changes, you lose less as you haven’t invested the same level of resources.

In line with this is that the combination of Col’s and Ken’s work on the indicators project and my work associated with my PhD provides us with the opportunity to do some comparisons of two different systems/approaches within the same university. This sounds like a good chance to leverage existing work into new opportunities and develop some propositions about what works around the use of an LMS and what doesn’t.

Lastly, there are some good references that suggest that looking at use of these systems is a good first start. e.g. Coates et al (2005) suggest that it is the uptake and use of features, rather than their provision, that really determines their educational value.

References

Behrens, S., Jamieson, K., Jones, D., & Cranston, M. (2005). Predicting system success using the Technology Acceptance Model: A case study. Paper presented at the Australasian Conference on Information Systems’2005, Sydney.

Coates, H., James, R., & Baldwin, G. (2005). A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning. Tertiary Education and Management, 11(1), 19-36.

Jones, D., Cranston, M., Behrens, S., & Jamieson, K. (2005). What makes ICT implementation successful: A case study of online assignment submission. Paper presented at the ODLAA’2005, Adelaide.

Malikowski, S., Thompson, M., & Theis, J. (2006). External factors associated with adopting a CMS in resident college courses. Internet and Higher Education, 9(3), 163-174.

Malikowski, S., Thompson, M., & Theis, J. (2007). A model for research into course management systems: bridging technology and learning theory. Journal of Educational Computing Research, 36(2), 149-173.

Malikowski, S. (2008). Factors related to breadth of use in course management systems. Internet and Higher Education, 11(2), 81-86.