Category: lmsreview

Messiness of information systems – another reason institutional e-learning struggles

On March 12, 2009

In Chapter 2, design theory, elearning, lmsreview, thesis, Uncategorized

My current disciplinary home is within the information systems community, which, not surprisingly, concerns itself with research and practice around information systems. This begs the question, “What is an information system?”. This post provides one answer to this question and in doing so suggests another reason why most institutional, university-based e-learning implementations enjoy less than stellar success. Of course, it’s not limited to just e-learning, but that’s what my focus is.

What is an information system?

DuPlooy (2003) describes an information system as consisting of three subsystems: the hardware, software and “otherware” and uses the following figure to represent their relationship.

The hardware component is the computer hardware, the processing units, printers, network equipment, monitors etc. The software component, as you may expect, are the software applications that make use of the hardware to help users of the system to perform various tasks. “Otherware” is defined as including the system’s goals, the owner, users, operational procedures, and the tasks and responsibilities of the people involved.

DuPlooy (2003) makes the point that the hardware and software components are deterministic. That is, given the same inputs, the outputs of these components will generally be the same. On the otherhand, he makes the point that otherware is not deterministic. One reason why otherware is not deterministic is the involvement of people and the observation by Markus (1983) that people may have agendas and goals that differ vastly from those of the organization.

Given this fundamental difference in the nature of otherware when compared to the other two components, I believe DuPlooy’s (2003) representation is somewhat less than effective. I suggest the following representation is more appropriate.

This better captures the messy, non-deterministic nature of otherware. It illustrates that otherware is different, that it can’t be treated the same as software and hardware.

It is the consideration of all three subsystems, and in particular the addition of “otherware”, which differentiates information systems from other related disciplines such as computer science and information technology. The inadequacy of computer science in addressing problems associated with the use of computers in organisational contexts has played a large part in the emergence of the IS discipline (Fitzgerald and Adam 1996).

The inadequacy of computer science and information technology in addressing problems with organisational information systems is the main reason I’ve moved from the information technology discipline to that of information systems. Of course, it must be said that much of the research into information systems doesn’t fully engage with the messiness of otherware. For one example take a look at Behrens (2007).

What’s wrong with e-learning?

Essentially, the vast majority of practice in university-based e-learning radically under-estimates the messiness of the otherware invovled within a university context. Within the pantheon of organisations, I believe that universities are amongst the elite in terms of just how messy their otherware can be.

Whether it be a computer scientist developing an “adaptive LMS” by applying some algorithm and mathematics, the central information technology support division applying some project management methodology or management applying some top-down management approach, they all under-estimate the messiness of the otherware and the impact this has on their nice, neat plans and assumptions.

In some cases, it appears that the neatness of the software, hardware or of the traditional methodologies is used as a haven from the messiness of the context. The messiness is too hard, so one must ignore it and focus on the neatness of the process or the product.

For example, selecting one piece of software (e.g. an LMS) from a set of such software, which all have a pre-defined, limited set of features and expect those limited set of features to fulfill all the requirements of a messy “otherware”. Rather than recognise that the otherware is inherently messy, will continue to be messy, and will continue to change the nature of its messiness over time and subsequently adopt an approach that is able to respond to that messiness. Institutions insist on attempting to limit the messiness within the confines of the product. The phrase “we’ll implement a vanilla version” in connection with an enterprise system, including learning management systems, sums this up very nicely.

It shows a perspective that thinks it is too hard to change the information system, so the solution is to force the messiness of the otherware to comply with the confines of the information system. This perspective assumes that you can force messy otherware to conform.

Personally, I don’t think this is possible. It is possible to create the illusion that it is conforming, but scratch the surface and you will find it isn’t. Personally, I think it is better to engage with, seek to understand and respond to the messiness of the otherware.

References

Behrens, S. (2007). Diversity in IS Research; Metaphor, Meaning and Myth. ICIS’2007, Montreal, Canada.

duPlooy, N. F. (2003). Information systems as social systems. Critical Reflections on Information Systems: A Systematic Approach. J. Cano. Hershey, IDEA Group Inc.

Markus, M. L. (1983). “Power, politics and MIS implementation.” Communications of the ACM 26: 430-440.

Some possible reasons why comparison of information systems are broken

By admin

On January 27, 2009

In Chapter 2, design theory, elearning, lmsEvaluation, lmsreview, missingPs, thesis, Uncategorized

All over the place there are people in organisations performing evaluations and comparisons of competing information systems products with a strong belief that they are being rational and objective. Since the late 1990s or so, most Universities seem to be doing this every 5 or so years around learning management systems. The problem is that these processes are never rational or objective because the nature of human beings is such that they never can be (perhaps very rarely – e.g. when I’m doing it 😉 ).

Quoting Dave Snowden

Humans do not make rational, logical decisions based on information input, instead they pattern match with either own experience, or collective experience expressed as stories. It isn’t even a bit fit pattern match, but a first fit pattern match. The human brain is also subject to habituation, things that we do frequently create habitual patterns which both enable rapid decision making, but also entrain behaviour in such a manner that we literally do not see things that fail to match the patterns of our expectations”.

Dave also makes the claim that all the logical process, evaluations, documents and meetings we wrap around our pattern-matching decisions is an act of rationalisation. We need to appear to be rational so we dress it up. He equates the value of this “dressing up” with that of the ancient witch doctor claiming some insight from the spirit world leading him to the answer.

Via a Luke Daley tweet I came across a TED talk by Dan Gilbert that provides some evidence from psychology about why this is true. You can see it below or go to the TED page

As an aside the TED talks provide access to a lot of great presentations and even better they are freely available and can be downloaded. Putting them on my Apple TV is a great way to avoid bad television.

The model underpinning blackboard and how ACCT19064 uses it

By admin

On November 19, 2008

In elearning, lmsEvaluation, lmsreview, Uncategorized

As proposed earlier this post is the first step in attempting to evaluate the differences between three learning management systems. This post attempts to understand and describe the model underpinning Blackboard version 6.3.

Hierarchical

Blackboard like most web-based systems of a certain vintage (mid-1990s to early/mid 2000s) tend to structure websites as a hierarchical collection of documents and folders (files and directories for those of us from the pre-desktop metaphor based interface days). This approach has its source in a number of places, but most directly it comes from computer file systems

Webopedia has a half way decent page explaining the concept. The mathematicians amongst you could talk on in great detail about the plusses and minuses of this approach over other structures.

In it’s simplest form a hierarchical structure starts with

A single root document or node.
Underneath this will be a sequence of other collections/folders/drawers.
- Like this one
- And yet another one
  Each of these collections can in turn contain other collections.
  - Like this sub-collection.
  - And this one.
    This hierarchical structure can continue for quite some time. Getting deeper and deeper as you go.
- And probably another one.
  Best practice guidlines are that each collection should never contain much more the 7+/-2 elements as this is a known limitation of short term memory.

Blackboard’s idea of hierachy

One of the problems with Blackboard is that it’s underpinning models don’t always match what people assume from their what they see in the interface. This applies somewhat to the hierarchical model underpinning Blackboard.

Normally in a hierarchical structure there is one root document or node that is at the “top” of the pyramid of content. What Blackboard does is that each course site has a collection of content areas and then you nominate one of those as the “home” page. i.e. the one that appears when people first login. It’s not really the top of the pyramid.

Let’s get to an example, the image is the home page for the ACCT19064 course.

Note: I currently have “admin” access on this installation of Blackboard. Some of what appears in the interface is based on that access and is not normally seen by student or other staff users.

The links in the course menu on the left hand side of the image are (mostly) the top levels of the hierarchical structure of the Blackboard course. There are 13 main areas

Machinimas
Hird & Co
Feedback
Notice Board
Discussion Board
Group Space
Resources Room
Assessment Room
Instructor Resources
Teaching Team
External Links
Library
Helpdesk

The “home page” in the large right hand frame, which would be at the top of the hierarchy if Blackboard followed this practice, is the Announcements page. The link to the announcements page in ACCT19064 is provided by the “Notice board” course menu link.

The other complicating factor is that the course menu links for Helpdesk and Library aren’t really part of the Blackboard course site. They are links to other resources.

Feature: Blackboard allows top level folders to be links to external resources and also ad hoc elements within the course site.

The bit above is a new strategy. Everytime I come to something that I think is somewhat strange/unique or a feature of Blackboard I am going to record it in the text and also on this page.

Feature: The course home page can be set to a selection of “pages” within the site.

Other bits that don’t fit

Underneath the course menu links there are a couple of panels. The content of some of these can be controlled by the coordinator. In the example above the designer has removed as many of the links on these panels as possible. Normally, there would be two links

Communication; and
Links to a page of the default communication tools Blackboard provides each course including: announcements, collaboration, discussion board, group pages, email and class roster.
Course tools
Links to a page of the default course tools (not communication tools) Blackboard provides including: address book, calendar, journal, dictionary, dropbox, glossary….. This list can be supplemented.

The links to these tools are not part of the hierarchical structure of the course. They are always there, though the designer can remove the links. Confusingly, most staff leave these links and so students waste time checking the tools out, even if they aren’t used in the course (and most aren’t).

Feature: Blackboard does not maintain the hierarchy metaphor at all well. Confuses it with “tools” which sit outside the hierarchy.

The course map feature

To really reinforce the hierarchical nature of a Blackboard course site, Blackboard provides a course map feature which provides a very familiar “Windows explorer” link representation of the structure of a course website. The following image is of a part of the course map for the ACCT19064 course site.

What’s do the course menu links point to?

The links in the course menu can point to the following items

Content area
This is the default content holder in Blackboard. If the designers wants to create a collection of content (HTML, uploaded files, tools etc.) they create a content area. More on these below.
Tool link
This is a link to one of the communication or course tools mentioned above.
Course link
A link to some other page/item within the course, usually within a content area.
External link
A URL to some external resource.

The course menu link for ACCT19064 points the following

Machinimas – content area with 5 elements
Hird & Co – content area with 3 elements
Feedback – content area with 3 elements
Notice Board – link to the announcements tool
Discussion Board – direct link to the course level discussion conference
Group Space – a content area with 5 elements
Resources Room – a content area with 15 or so elements
Assessment Room – a content area with 6 elements
Instructor Resources – a content area with 1 element (see below)
Teaching Team – a link to the “Staff Information” tools
External Links – a content area with a number of links
Library – a direct link to the Library website.
Helpdesk – a mailto: link for the helpdesk

The number of elements I mention in each content area might be wrong. Blackboard supports the controlled release of content in a content area. Some people may not be able to see all of the content in a content area – explained more in the “Controlling Access” section below.

What’s in a content area?

A content area consists of a number of items. The items are displayed one under the other. The following image is of the Assessment Room in the ACCT19064 course site. It has 6 items. Not the alternating background colour to identify different items.

The edit view link in the top right hand corner appears when the user has permission to edit the content area. This is how you add, modify or remove an item from the content area.

An item in a content area can be one of the following

A “content” item
i.e. something that contains some text, perhaps links to an uploaded file.
A folder
This is how you get the hierarchical structure. A folder creates another level in the hierarchy within/underneath the current content area. This folder contains another content area.
An external link.
A course link.
A link to various tools or services.
e.g. to tests or a range of different services and tools provided by Blackboard and its add ons.

Each item is associated with a particular “icon”. A folder item will have a small folder icon. A content item will have a simple document.

Feature: The icon associated with each item cannot easily be changed, especially for an individual course. It can also not be made invisible (easily) and causes problems for designers.

For example, the following image is from what was intended to be the home page for a Blackboard course. A nice image and text ruined by the document icon in the top left hand corner.

Controlling access

Blackboard provides a facility to limit who can see and access items within a content area and also what links can be seen in the course menu. However, it’s done consistently.

Feature: Different approaches with different functionality is available to restrict access/viewing of the course menu links (very simple) and individual items in content areas (very complex and featured). Restrictions on discussion forums also appear some what limited.

The following image is of the “Instructor Resources” content area of the ACCT19064 course site. It is being viewed as a user who is not a member of the staff for this course. Actually not a member of the blackboard group “Teaching Staff”.

What follows is the same page with the same content area. However, it is now viewed as a user who is a member of the “Teaching staff” group.

Access to items can be restricted in the following ways

Visible or not
A simple switch which says everyone can see it, or they can’t.
Date based ranges
Specify a date/time range in which it is visible.
Group based membership
You can see it if you are part of the right group or in a specified list of users.
Assessment related
You can only see if if you have attempted a piece of assessment or achieved a grade within a specific range.

The specification of rules to restrict access can be combined.

Feature: Access to items can be restricted based on simple on/off, date, group membership, assessment.

A description of the ACCT19064 site

At initial look the course site is designed as a container for the content and tools used within the course. The design of the course site itself does not inherently provide any guidance to what the students are meant to do. i.e. there is no study schedule or similar showing up in the course menu links.

However, looking at the announcements for the course. It appears that this type of guidance and support for the students is given by an announcement from the course coordinator on the Sunday at the start of each week. This announcement is very specific. It outlines the individual and team-based tasks which the on-campus and off-campus students are meant to complete. There is also some additional comments, sometimes errata and sometimes the odd big of advice.

Interestingly, these guidance announcements didn’t link students directly to the tasks.

Breaking down the content of the site

The following describes in more detail the content within each of the course menu links, at least those that point to content areas.

Machinimas – content area with 5 elements. There is no adaptive release.
- Description of the machinimas and their purpose.
- Four external links to web pages that contain video of the machinima.
Feature: Blackboard uses breadcrumbs for navigation. Including external web pages can be made more transparent if the breadcrumbs can be re-created on those external pages.

The machinima pages, with the video playing, look like the following image
Hird & Co – content area with 3 elements. No adaptive release
This is meant to represent the imaginary audit company used throughout the course
- External link to a “intranet” site for an imaginary audit company.
- External link to an external discussion forum used by AIC campuses for discussion about questions prior to face-to-face classes.
- A link to a Blackboard discussion forum used by the Audit Partner (the coordinator)
Feedback – content area with 3 elements
Actually it’s 4 elements. One is not visible due to adaptive release. The missing element is a course experience survey. This section is entirely set aside to getting feedback from students about the course. I’m guessing it was added late in the term.
- Content item linking to an information sheet
- Content item linking to a consent form
- A link to survey tool for the actual survey
- An external link with help on an issue with the survey.
Notice Board – link to the announcements tool
Discussion Board – direct link to the course level discussion conference
Group Space – a content area with 5 elements
- Link to a folder for the “Group Space for Audit Teams” – content area with 3 elements
  - Link to the groups page for the course
    This is one of the Blackboard communication tools that is supported by a group allocation/management system. It allows each group to have a collection of pages/tools which are unique and only accessible by members of the group.
    
    Typically this includes a group discussion conference, collaboration, file exchange and email.
  - Content item containing an announcement about groups
  - Content item containing details of group allocation – group names and student members
- Content item linking to a document explaining problems faced by Vista users
- Link to the Blackboard drop box tool
- Folder containing feedback for submitted tasks
- Folder containing declarations for each quiz
Resources Room – a content area with 15 or so elements
Apart from a content item describing solutions to problems for users of Vista, this content area consists entirely of folders used to group resources associated with a particular week or activity. There are
- 12 folders for each week of term
  These are a collection of folders and content items providing access to various weekly materials such as: eLectures, powerpoint lecture slides, activity sheets as Word docs, weekly quizzes and solutions (available only after a specified time).
- one containing feedback and facts from previous students,
  A simple collection of content items with pass results and qualitative student feedback.
- one containing general course materials (e.g. course profile and study guide),
  Two external links to the course profile and study guide.
- one on auditing standards
  Two external links to the auditing standards applicable to the course.
- One containing a course revision presentation
Assessment Room – a content area with 6 elements
- Team membership – content item links to a word doc that students must complete and return
- Personal journal – link to Blackboard journal tool that is used for personal reflection and integrated into assessment and weekly activities.
- Resources for assessment items 1, 2a and 2b
  Each contains a range of content items, folders and external links pointing to resources specific to each assessement item.
- Exam information – collection of information about the exam
Instructor Resources – a content area with 7 elements, a number under adaptive release
- Staff discussion forum – external link to an external discussion forum
- Snapshots for teaching team – collection of word documents explainining activities/tasks for the various weeks.
- Teaching materials for lectures – collection of materials for staff to give lectures
- Teaching materials for the tutorials – ditto for tutorials
- Teaching materials for the workshops – ditto for workshops
- Information on Assessment item 2a – misc additional background on assessment item
- Information on assessment item 2b
Teaching Team – a link to the “Staff Information” tools
External Links – a content area with a number of links
Library – a direct link to the Library website.
Helpdesk – a mailto: link for the helpdesk

Overview

A fairly traditional hierarchical design for a course website. Students receive direction on what tasks to do from weekly announcements, not from some fixed “schedule” page.

Heavy use is made of groups.

There are some significant differences between tasks/activities for on-campus versus off-campus students. While educationally appropriate this does tend to make things more difficult for the coordinator and students. i.e. there has to be two sets of instructions created by the coordinator and students have to discern which they should follow.

Some use of external discussion forums. Probably due to the limitation in how Blackboard allows discussion forums to be configured. i.e. one discussion conference per course and one discussion conference per group.

Staff information not integrated with CQU systems, require duplication of effort. Same applies for the provision of links to course profile and to some extent lectures.

Evaluating an LMS by understanding the underpinning "model"

By admin

On November 18, 2008

In Chapter 2, elearning, lmsEvaluation, lmsreview, Uncategorized

Currently, CQUni is undertaking an evaluation of Sakai and Moodle as a replacement for Blackboard as the organisation’s Learning Management System. The evaluation process includes many of the standard activities including

Developing a long list of criteria/requirements and comparing each LMS against that criteria.
Getting groups of staff and students together to examine/port courses to each system and compare/contrast.

Personally, I feel that while both approaches are worthwhile they fail to be sufficient to provide the organisation with enough detail to inform the decision. The main limitation is that neither approach tends to develop a deep understanding of the affordances and limitations of the systems. They always lead to the situation that after a few months/years of use people can sit back and say, “We didn’t know it would do X”.

A few months, at least, of using these systems in real life courses would provide better insight but is also prohibitively expensive. This post is the start of an attempt to try another approach, which might improve a bit on the existing approaches.

What is the approach?

The approach is based somewhat on some previous ramblings and is based on the assumption that an LMS is a piece of information technology. Consequently, it has a set of data structures, algorithms and interfaces that either make it hard or easy to perform tasks. The idea is that if you engage with and understand the model, you can get some idea about what tasks are hard or difficult.

Now there is an awful lot of distance between saying that and actually doing it. I’m claiming that the following posts are going to achieve anywhere near what is possible to make this work effectively. My existing current context doesn’t allow it.

At best this approach is going to start developing some ideas of what needs to be done and which I didn’t do. Hopefully it might “light” the way, a bit.

Using the concept elsewhere

We’ve actually been working on this approach as a basis for staff development in using an LMS. Based on the assumption that understanding the basics of the model will make things work somewhat easier for folk to use. The first attempt at this is the slidecast prepared by Col Beer and shown below.

Blackboard@CQ Uni

View SlideShare presentation or Upload your own.

What will be done?

Given time constraints I can only work with a single course, from a single designer. More courses, especially those that are different would be better. But I have to live with one. I’ve tried to choose one that is likely to test a broader range of features of the LMSes to minise this. But the approach is still inherently limited by this small sample set.

The chosen course is the T2, 2008 offering of ACCT19064, Auditing and Professional Practice. For 2008 this course underwent a complete re-design driven by two talented and committed members of staff – Nona Muldoon (an instructional designer) and Jenny Kofoed (an accounting academic). As part of this re-design they made use of machinima produced in Second Life. The re-design was found to be particularly successful and has been written about.

The basic steps will be

Explain the model underpinning Blackboard and how it is used within the course.
Seek to understand and explain the model underpinning Moodle and then Sakai.
Identify and differences between the models and how that might impact course design.

Hopefully, all things being equal, you should see a list of posts describing these steps linked below.

The dissonance gap in systems and LMS evaluations

By admin

On November 17, 2008

In Chapter 2, elearning, lmsEvaluation, lmsreview, thesis, Uncategorized

Ania Lian writes in the paper Knowledge transfer and technology in education: toward a complete learning environment

It is argued that technology itself is neither liberating, empowering nor enabling one to be with other people but that it will serve whichever goals motivate its incorporation.

In a couple of papers (e.g. this one) I’ve paraphrased this as

Technology is not, of itself, liberating or empowering but serves the goals of those who guide its design and use (Lian, 2000).

This posts tries to explain why this is the case and also to explain what relevance this has to an institution when it attempts to evaluate and select a new learning management system.

Why is this the case?

A learning management system is a piece of information technology. As a piece of information technology it has been designed by a small group of expert designers. Typically a company or open source community (individual).

This group of expert designers analyse the problem, identify some sort of solution and then turn that solution into code. They do not do this is a purely sequential nor objective manner. Their past experiences, lessons learned and new knowledge all impact on this process.

However, at some stage they must eventually design and implement algorithms, data structures and interfaces. These artifacts will all embody the view of the world they have formed during the above process. This view of the world will impact upon the final system.

For example, a system designed with an emphasis on being “enterprise ready”, on being “scalable” will have a very different set of facilities, structure and “feel” than a system designed with an emphasis on a particular educational approach.

This influence of the world view of the designers is present at all levels in a system. Not just the high level perspective of the system. For example, the Peoplesoft ERP had its origins as a human resource management system. As such one of the early and important identifiers for that system was “EMPLID” i.e. employee identifier. In a HR system you have employees.

Later on Peoplesoft entered the field of providing ERPs to universities. As part of this they had to add a student records database. In part, to record things like which courses a student was enrolled in.

To achieve this you have to have some form of unique identifier for each student. Most universities do this through a unique student number. I’ve seen other student records system give the student number labels like STUD, STUD_NUM etc.

Guess what label Peoplesoft uses? (Remember it’s origins and the impact of the original designers). Yep, that’s right. EMPLID – employee id.

What relevance does this pose?

When a university is seeking to select a new learning management system it is selecting an enterprise information system. As such any selection will have embedded in it a certain world view. Even the notion of an LMS embodies a certain world view. That of the “big” system. This world view brings certain positives and negatives. The impact of the original designers will play a part in how effective the choice is, it will limit or enable what learning and teaching can occur, what staff and students can do.

Discussion forums in Blackboard

As one example, let’s take the Blackboard LMS (version 6.3). Like any LMS it provides support for discussion forums. The model, or at least my current understanding of the model, is that a course website can have

One “Course Conference” for the course.
A course conference can consist of many separate discussion forums. This “course course conference” is usually used for everyone in the course. For many CQU courses, this is the only course conference used.
Each group of students/staff can have a course conference.
Blackboard allows you to create groups of students and staff. Each of these groups can have a single course conference. Again the single course conference can contain many different discussion forums, but they are all located in the single course conference.

So what are the limitations of this approach? Essentially, it doesn’t support each group having multiple course conferences.

One of the designs by an instructional designer at CQU had each group having a part of the blackboard course site that contained a number of rooms. Sections of their group site set aside specifically for different activities, topics or times during the term. Each room might include a number of services including a discussion forum, a wiki, sharing of resources etc.

The design made sense from the perspective of keeping everything associated with a particular activity in the one place.

This could not be done with Blackboard. The discussion forums would have had to be within the single course conference allowed for each group. Taking students out of the activity into another part of the course site.

Assignment submission in Blackboard

The assignment submission process in Blackboard 6.3 allows students to submit any type of file, performs no checking of the files and provides only ad hoc support for marking staff.

Conseqently, it is well known that you would be really stupid to use the Blackboard assignment submission system for a class with more than about 20 students. To do so is opening you up for a huge amount of extra work.

Dissonance

A dissonance between the needs of a systems users and its embedded world view lead to a number of problems. The dissonance becomes a gap between the users and the system. A gap that prevents the adoption of certain approaches or which creates additional workload as people attempt to work around the gap.

Eventually, this dissonance gap will lead users to attempting to use alternate means. Of creating shadow systems.

Understand the gap as part of the valuation

Consequently, it seems to be sensible for any evaluation of new systems would include an attempt to actively identify the models and purpose underpinning the different systems and attempting to understand the size of the dissonance gap.

All things being equal (and there’s a real good chance that they won’t be), this is something we’ll be attempting over the next few days as CQU attempts to move from Blackboard to either Sakai or Moodle.