Validity is subjective

Just another quote, but a good one. And one that connects with a recent post about the difficulty of getting agreement around learning and teaching and the difficulty this creates within universities when you want to try and improve the quality of that learning and teaching.

Validity is subjective rather than objective: the plausibility of the conclusion is what counts. And plausibility, to twist a cliché, lies in the ear of the beholder.
Cronbach (1982)

Perhaps then, it is no surprise to see where the quote comes from? This book on the evaluation of educational programs.


Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco, Jossey-Bass.

Initial steps toward an education aggregation taxonomy – community versus individual?

In a previous post I talked about the rationale and need for thinking about a taxonomy of educational aggregation projects. Something that I haven’t really given a lot of thought to, just yet.

About 15 minutes ago I posted this about “cooked” course feeds.’s “possibly related posts” feature (where it automatically appends 3 or 4 other posts from blogs which it deems to possibly related) included a link to this post entitled “Prologue as an eLearning Blog Portal”.

It’s a post by Mike Bogle in which he outlines one of the key problems we saw with using blogs for individual student reflective journals.

the main challenge for educators using blogs in their courses is how to keep track of them all. By nature blogs function completely independently of one another, so the prospect of monitoring the individual blogging activities of several dozen or more people presents a substantial time investment for instructors and students alike.

In the post, Mike outlines some problematic solutions before suggesting that the use of Prologue as a way to create a blog portal. The aim being create a “dynamic, interactive student network”.

It’s this purpose and some of the features Prologue provides and the possible applications these features can support that seem to suggest another place in a taxonomy of aggregation projects.

The “portal/community” aggregator

What Mike is suggesting could perhaps fit under the title of “portal aggregator”. Perhaps community rather than portal might be a better label – portal has too many connotations from the late 90s and corporate Web 1.0 centralised IT constraints. After all, the purpose of the suggestion is to enable the creation and maintenance of a community.

Community aggregators tend to provide many of the features Mike mentions including various filtering options and the idea of social tagging of posts.

This seems to be the category into which EduGlu would fit. There is now even a EduGlu sandbox which allows you to play. The workflow image in that post provides a good representation.

Perhaps this type of workflow might be the way to differentiate different aggregation projects?

Based on the little I know of it, I believe that Cloudworks probably belongs in this category as well.

What’s the difference? Is there one?

Suggesting a need for a taxonomy of aggregation projects assumes that there are different types of projects. The “community aggregator” approach seems to be fairly general, perhaps general enough to provide all of the necessary features. At the moment, I think not. I think some aspects of BAM identify a potential difference, though there is also some overlap.

So what’s the difference between BAM and tools like Cloudworks, EduGlu and Prologue? Well, apart from the quality of code and implementation, I believe the main differences are

  • An initial focus on the individual student.
  • The integration of institutional needs.

A focus on the individual student

The original rationale and design for BAM was to support and improve student usage of individual reflective journals and enable academic staff to be aware of student progress. i.e. the only people directly and regularly reading the student blog posts was their tutor. Hence the social aspects of filtering and tagging aren’t in BAM, because they weren’t immediately useful.

There are plusses and minuses to this approach. There would certainly have been benefits for student reflections to be seen and commented upon by other students. But there also would’ve been negatives.

In terms of Mike’s post the emphasis in BAM was on helping staff keep track of individual student blogs, rather than creating a “dynamic, interactive student network”.

That’s not to say that BAM can’t help support this approach. It was used to implement a simple form of the community approach for the portfolio and weblog networks on the Creative Futuring course site. But without the tagging and filtering.

Integration of institutional needs

D’arcy Norman in his post on EduGlu describes the magic combination of features for EduGlu as

Aggregation of feeds + Groups + Social Rating + Tagging

A summarised version of a similar magic combination for BAM would be

Aggregation of feeds + institutional needs

Where “institutional needs” gets expanded out into:

  • Institutional data;
    Students register their blog with BAM. BAM associates their student number and course with their blog. This then allows BAM to determine which staff member (actually its which hierarchy of staff membrs) is responsible for the student. It associates the use of the blog with the specific assignment for the course. This is all drawn from existing institutional systems and processes.
  • Institutional services; and
    The institution already has services such as a staff portal, online assignment submission and results uploading. BAM integrates the use of the blogs into these systems. BAM provides a marking interface for the student blog posts. The staff member accesses this interface through the staff portal. The marks are available via the online assignment submission and then at the end of term are integrated into the end of term results processing system.

    All this minimises work for the staff.

  • Institutional requirements.
    One of the most common concerns raised about using a blog service that is not serviced and hosted by the institution is the “dog ate my homework” fear. i.e. what happens if the external blog service disappears and the students’ blog posts are lost? To address this concern, BAM maintains a mirror of the RSS feed from student blogs on a institution system.

Where does BAM fit?

Does this mean that BAM is an “individual” aggregator? An “institutional” aggregator?

That type of labeling doesn’t seem to work all that well. Suggestions?

Cooked course feeds – An approach to bringing the PLEs@CQUni, BAM and Indicators projects together?

The following is floating an idea that might be useful in my local context.

The idea

The idea is to implement a “cooked feed” for a CQUniversity course. An RSS or OPML feed that either students or staff or both can subscribe to and receive a range of automated information about their course. Since some of this information would be private to the course or the individuals involved, it would be password protected and could be different depending on the identity of the person pulling the feed.

For example, a student of the course would receive generic information about the course (e.g. any recent posts to the discussion forums, details of resources uploaded to the course site) as well as information specific to them (e.g. that their assignment has been marked, or someone has responded to one of their discussion posts). A staff member could receive similar generic and specific information. Since CQU courses are often offered across multiple campuses staff and student information could be specific to the campus or the sets of students (e.g. a tutor would receive regular updates on their students – have they logged into the course site etc)

A staff member might get a set of feeds like this:

  1. Student progress – perhaps containing a collection of feeds. One might be summary that summarises progress (or the lack thereof) for all students and then one feed per student.
  2. Course site – provides posts related to the course website. For example, posts to discussion forums, usage statistics of resources and features etc.
  3. Tasks and events – updates of when assignments are due, when assignments are meant to be marked, when results need to be uploaded. These updates would not only contain information about what needs to be done, but also provide links and advice about how to perform them.

The “cooked” adjective suggests that the feeds are not simply raw data from original sources. But that they undergo additional preparation to increase the value of the information they contain. For example, rather than a single post simply listing the students who have (or have not) visited a course site the post might contain the students GPA for previous courses, some indication of how long into a term they normally access a course site, when they added the course (in both date and week format – i.e. week 2 of term), links back to institutional information systems to see photos and other details of the students, links to an email merge facility to send a private/bulk email to all students in a particular category, a list of which staff are responsible for which students etc.

The point is that the “cooking” turns generic LMS information into information that is meaningful for the institution, the course, the staff, and the students. It is this contextual information that will almost always be missing from generic systems, simply because they have to be generic and each institution is going to be different.


The PLEs@CQUNi project already has a couple of related sub-projects doing work in this area – discussion forums and BAM.

Discussion forums. The slideshow below explains how staff can currently access RSS feeds generated from the discussion forums of CQU’s current implementation of Blackboard version 6.3. A similar feature has already been developed for the discussion forum used in the other “LMS” being used at CQU.

The above slideshow uses the idea of the “come to me” web. This meme is encompasses one reason why doing this might be a good thing. It saves time, it makes information more visible to the staff and the students. Information they can draw upon to decide what to do next. Information in a form that allows them to re-purpose and reuse for tasks that make sense to them, but would never be apparent to a central designer.

BAM. The Blog Aggregation Management (BAM) project now generates an OPML feed unique for each individual staff member to track their students’ blog posts. The slidecast below outlines how they can use it.

The indicators project is seeking to mine usage logs of the LMS to generate information that is useful to staff. I think there is value in this project looking at generating RSS feeds for staff based on the information it generates. Why depends on the difference between lag and lead indicators.

I’ve always thought that too much of the data generated at Universities are lag indicators. Indicators that tell you how good or bad things went. For example, “oh dear, course X had a 80% failure rate”. While having this information is useful it’s too late to do anything. You can’t (well you shouldn’t be able to) change the failure rate after it has happened.

What is much more useful are lead indicators. Indicators that offer you some insight into what is likely to happen. For example, “oh dear, the students all failed that pop quiz about topic X”. If you have some indication that something is starting to go wrong, you may be able to do something about it.

Aside: Of course things brings up the problematic way most courses are designed, especially the assessment. They are designed in ways such that there are almost no lead indicators. The staff have no real insight into how the students are going until they hand in an assignment or take an exam. By which time it is too late to do anything.

Having the indicators project generating RSS posts summarising important lead indicators for a course might encourage and help academics take action to prevent problems developing into outright failure.

This is also encompassed in the idea of BAM generating feeds and the very idea of BAM in the first place. It allows staff to see which students are or are not progressing (lead indicator) and then take action they deem appropriate.

It’s also a part of the ideas behind reflective alignment. That post also has some suggestions about how to implement this sort of thing.