Assembling the heterogeneous elements for (digital) learning

Month: March 2009

Further evidence of problems with evaluation of teaching

In a couple of recent posts (and this one) I’ve essentially been arguing that most evaluation of teaching at universities is actually worse the useless. The following quote from White (2006) provides some additional ammunition.

Research has suggested that grade leniency is the most significant factor in positive evaluation of teaching (Greenwald & Gilmore, 1997; Marsh & Roche, 2000). These pressures may be resulting in grade inflation by some teachers and departments, and concomitantly, unrealistic student expectations with respect to the marks they should be awarded.

References

Greenwald, A. G., & Gilmore, G. M. (1997). ‘No pain no gain’: The importance of measuring course workload in student ratings of instruction. Journal of Educational Psychology, 89(4), 743–751.

Marsh, H. W., & Roche, L. A. (2000). Effects of grade leniency and low workload on students’ evaluations of teaching: Popular myth, bias, validity or innocent bystanders? Journal of Educational Psychology, 92(1), 202–228.

White, N. (2006). “Tertiary education in the Noughties: the student perspective.” Higher Education Research & Development 25(3): 231-246.

Some potential updates to BAM – a step towards breaking the LMS/CMS orthodoxy

The initial design and use of the Blog Aggregation Management (BAM) system was, in part, designed to try out approaches that leverage the protean nature of information technology. A major part of this is a move to something different, and hopefully better, than the current, broken e-learning orthodoxy within universities that is stuck on the idea of course management systems (CMS – aka learning management systems) as the only possible solution.

The vast majority of what BAM does was designed and implemented over a couple of months almost 3 years ago. Since then we’ve learned a bit about using BAM and also have some time to extend BAM in appropriate ways. This post seeks to explain the next major expansion of BAM, which will see it move further away from CMS orthodoxy. In particular, the plan to expand BAM’s generation of RSS/OPML feeds so academic staff can avoid badly designed web-based management interfaces and use an RSS reader of their choice as the major interface to BAM.

Current limitations of BAM

One of the assumptions underpinning BAM was to significantly question the ability for a university to provide a blogging service that could compete with existing free blog services in terms of reliability, quality of features and quality of support services and resources. This is an extension of one of the principles behind the design of the Webfuse e-learning system (Jones and Buchanan, 1996) within which BAM is currently implemented. This principle is talked about under the heading “Flexibility and don’t reinvent the wheel”

The design of the M&C OLE (online learning environment) will attempt to maximise adaptability by concentrating on providing the infrastructure required to integrate existing and yet to be developed online learning tools. The M&C OLE will provide the management infrastructure and consistent interface to combine existing tools such as WWW servers, online quizzes, assignment submission etc. into a single integrated whole. While a number of the component systems will be developed at CQU, the emphasis is on integrating existing tools into the OLE.

At the moment, BAM provides a management interface for academic staff around existing blogging engines. Actually it is designed so that students can maintain a reflective journal in anything that will produce an RSS feed. The only direct interaction with BAM by students is at the start of term when they register their blog using the interface shown in the next image.

BAM blog registration

Academic staff currently use a web-based interface provided by BAM to track student blog registration and posts, view student posts and mark student posts. See the screenshots in this paper for what they look like. That is, BAM is still stuck in the CMS orthodoxy.

Moving to RSS readers and OPML feeds

Late last year there was a simple extension of BAM to allow academic staff to obtain an OPML feed pointing to all their students’ blogs. This could be imported into an RSS reader of their choice, in order that they could keep a track of posts by their students. There were a number of limitations of this approach:

  • It wasn’t automated.
    Someone had to run a script, generate the OPML feed, send it to the staff member who could then import it. They should be able to do it themselves.
  • It only provided access to the student posts, none of the other BAM services.
    The OPML was using the RSS feeds straight from individual student blogs. It did not go through BAM and consequently could not provide any additional BAM/institutional related information. For example, which students hadn’t yet registered their blog, no direct access to the BAM marking interface, etc.

Implementing BAM “cooked” feeds

The premises on which this extension of BAM is based are:

  • Increasingly people will have an application they use to access, manipulate and store RSS, OPML and other feeds.
    e.g. I believe the Outlook, the spawn of the devil, even supports feed reading now.
  • Using this application(s) to track information of interest will become part of their daily life.
    The “come to me” web will become increasingly important. It’s certainly part of my everyday life at the moment and it is an improvement over the “I go get web”. These two assumptions were the basis for the work in this presentation aimed at adding RSS feed generation to discussion forums in Blackboard course sites.
  • BAM’s interface does provide some additional information about the student, the course etc. that isn’t provided in the “raw” RSS feeds from each students’ blog.

The fundamental idea is that BAM will generate “cooked” RSS feeds and that academic staff will be able to access the feeds for their students via their choice of RSS reader. The outstanding questions are:

  1. What ingredients need to go into the cooking?
  2. What’s the best (and easiest) technical approach to implementing “cooked” feeds?
  3. Why are academics going to use this?
    This is the big one, if they don’t want to use it, then there’s no point doing it. The aim will be that this will be easier and more effective than using the BAM interface.

What the cooked feeds need to do

At the very least cooked feeds will need to support all of the existing functionality provided by BAM and where possible provide additional functionality.

Existing functionality

  • Which students have registered their blogs and which haven’t.
  • A method to view photos and details about the students who fall into either group (registered or not).
  • Provide a link to a “mail merge” facility for those students who fall into either group.
  • View statistics about student blogs – e.g. how many posts in total, when was the last time they posted an entry and a link to the student blog.
  • A marking interface for each post.
  • A question allocation interface for each post.
    BAM was originally designed to implement individual student reflective journals where students are expected at fixed times during a term to answer specific questions. BAM automatically examines each student post and attempts to determine if it is a response to one of these fixed questions.
  • Information about whether the post has been marked or allocated.
  • Whether or not a student has answered all of the necessary questions.
  • Ability for the course coordinator (academic in charge of a course) to view and track student posts for other teaching staff and also the staff’s marking progress.

Potential new features

  • Indication of what new posts there have been since the academic last visited.
    This is essentially what would be provided by an RSS reader.
  • Addition of institutional/student based information to individual blog posts.
    Currently a post to a student blog does not include any information about who the student is, their institutional student number, whether or not the post is a match for one of the required questions they must answer, a link to the marking and question allocation interfaces for BAM posts
  • On the fly copy detection of student posts.
    Currently there is a half-baked script that will compare all student posts against each other to check of copying. There’s questionable educational value for this, but something that is perceived to be useful by staff.
  • A daily summary of activity by related staff and students.
    Each staff member could see a single post that summarises activity by their students. For example, who posted, which questions they answered, who still hasn’t registered, who did register, what copy detection incidents were identified etc. In addition, staff who are supervising other staff could recieve a daily post on the progress of staff. For example, how many of each staff member’s students haven’t registered, haven’t posted, haven’t been marked etc.

    This idea could serve the basis for a broader service associated with courses and perhaps attached to some current work around indicators

Technical implementation

Initial quick ideas might include

  • The provision of two top level feeds for each staff member.
    1. Activity summary – this is the daily summary of activity by related staff and students. Staff and students might be updated as separate feed items. Non-supervisory staff simply wouldn’t see an item of that type. Alternatively, the would see such a feed. It would simply summarise what they did over the last day or so. Staff who are supervising other staff, would also see posts summarising the activity of the staff they are supervising.
    2. Student posts – similar to the existing feed, this would consist of numerous feeds (one per student) summarising what they have posted to their blog.
  • Perhaps a “staff activity” feed.
    Supervisory academic staff might also have a third collection of feeds. The “Staff student activity” feed would include one collection of feeds for each staff member being supervised. This collection of feeds per staff member would be exactly the same as “Student Posts” feed that the supervised staff member would see. This would allow supervisory staff to see the detail, if they wanted to.
  • Cook the individual student blogs
    The individual student feeds would not be from the blog feeds. They would be cooked versions from BAM that will have added a range of additional institutional and BAM related links and information.

Initial implementation ideas might include:

  • Each of the individual feeds would be implemented as simple RSS files on the institution’s server
    i.e. static files that are updated by BAM, but staff are requesting the static files, not a script or similar. The drawback here is that a Perl access module will have to be written to control access to the appropriate folk. The advantage is that some of the “cooking” will require some significant processing (e.g. copy detection). Also the OPML feeds that bring these feeds together for staff could be implemented in a simple hierarchical file system.

    For example, BAM/YEAR/PERIOD/COURSE/Staff/username/{all.opml|summary.rss|students.rss}. And for each student …./COURSE/Students/STUDNUMBER.rss

  • Updating of these feed files would be done at the end of the current BAM “mirror” process.
    Every hour or so BAM goes out and checks each student’s blog for any new entries. If it finds any, it updates a local mirror of the raw RSS file. Could add to the end of this process all of the necessary steps required to “cook” the feeds.

Questions

  • What features are missing?
  • What potential implementation approaches have I missed?
  • What problems exist with the above implementation plan?
  • Is the cost/benefit ratio sufficient for me to implement these plans given the PhD etc.?

Reflective problematisation – description of reflection in "reflective alignment"?

Thinking about reflective alignment, I came across the following quote in Booth and Anderberg (2005). Thought it might be useful so am saving it here.

the equally important notion of reflective problematization – deliberately distancing oneself from the familiar, deliberately avoiding the taken-for-granted and considering the alternatives that might be at hand, relating to theories and experience and reaching an analytical insight into productive change.

The connection with “reflective alignment” is that this is a pretty good description of the type of reflection which I observe in the “good” teachers. It’s the type of reflection “reflective alignment” would seek to encourage and enable.

References

Booth, S. and E. Anderberg (2005). “Academic development for knowledge capabilities: Learning, reflecting and developing.” Higher Education Research & Development 24(4): 373-386.

Another spectrum for using indicators to place course websites

This post adds another perspective borrowed from Gonzalez (2009) as a framework to report or evaluate findings from Col and Ken’s indicators project. Col added an update on his work recently. Like previous post this one borrows a table of dimensions around conceptions of online learning because it may be helpful.

First the table and then how it might be used.

Dimensions

Dimensions delimiting approaches to online teaching – (Gonzalez, 2009: p311)
Informative/individual learning focuses Communicative/Networked learning focused
Intensity of use Small range on media and tools used to support learnign tasks and activities (mainly sources of information with small opportunities for interaction and communication) Wide range of media and tools used to support learning tasks and activities (with emphasis on interaction and communication)
Resources Web pages with information. Lecture notes. Links to websites. Web pages with information. Lecture notes. Links to web sites. Discussion boards. Chat. Blogs. Spaces for sharing. Animations. Videos. Still images.
Role of the learner Select and present information Design spaces for sharing and communication. Support the process.
Role of the students Study individually information provided Participate in a process of knowledge building

How might it be used

The above dimensions could be used to develop “analysis routines” that would place courses within these dimensions. Some potential approaches:

  • Variety and use of tools and media within a course site. (Intensity of use and Resources)
    Group the different tools available in the course management system into different types. e.g. those used for information distribution and those for interaction/communication. Count the number of different types of tools present in a course site and the level of usage.

    The difficulty here is the increasing use of non-CMS based tools for communication. e.g. I know of an increasing number of staff and students who are using external tools such as Messenger to work around the limitations of CMS services.

  • Measure student and staff activity (Role of the lecturer/students)
    I believe Blackboard, the main CMS at our institution, tracks the activity in some detail of each course site participant. If the type of activity can be categorised into groups (e.g. adding information to the site, using information on the site, posting to a discussion forum, responding to a post in a discussion forum etc.) then analysis could be run against the activity of all participants. This would identify the type of role the main groups are taking on.

What’s the value of this?

I can hear some thinking, “so what!”. What is the value of this sort of thing? A couple of thoughts.

  • As a framework to help make sense of the data.
    From my perspective it appears that the project is “drowning in data” and could use some sort of reviewed framework with which to organise or structure their investigations. These dimensions might provide it.
  • Enable institutions to get a handle on what is happening.
    Most of it ain’t great. The combination of the dimensions and the data potentially enable institutions, that are spending a lot of money on course management systems, to improve the awareness they have of what is actually happening. At the very least some sort of indication of where online courses site within the institution, as imperfect as it will be, sit within the dimensions might start some conversations about online practice that is actually somewhat informed by the reality of what is going on.
  • As a demonstration of building on the work of others.
    It is possible to argue with the value/validity of the knowledge generated by Gonzalez (2009) – but then it’s possible to argue against the validity of just about the knowledge generated by any research project depending on your perspective. However, this work is in a fairly prestigious journal, so it comes with a certain stamp of approval. This will help Col and Ken.
  • Perfect opportunity for a publication.
    Building on the last point, suggests that complimenting the qualitative nature of Gonzalez (2009) with some more quantitative measures from a broad collection of students and courses sounds like a pretty good publication opprotunity (or three).

It’s the potential for discussion within the organisation that is, I believe, of potentially the most beneficial for the most people.

The potential for publication is probably the most interesting to the project participants and frankly by far the easiest.

Further publications

The publication idea would be strengthened if previous work in this area (e.g. the recent ALTC project Learning and teaching performance indicators report either doesn’t do something like this or uses a different set of dimensions.

In addition, Gonzalez interviewed only 7 academics within a single discipline within a single institution. Chances are the results and dimesions identified in the paper are going to exhibit some sort of limitation, potentially caused by the nature of the context. Using a different approach in a different context will at least compliment/reinforce the findings and potentially identify additional dimensions.

References

Gonzalez, C. (2009). “Conceptions of, and approaches to, teaching online: a study of lecturers teaching postgraduate distance courses.” Higher Education 57(3): 299-314.

Page 3 of 3

Powered by WordPress & Theme by Anders Norén

css.php