Choosing your indicators – why, how and what

The unit I work with is undertaking a project called Blackboard Indicators. Essentially the development of a tool that will perform some automated checks on our institution’s Blackboard course sites and show some indicators which might identify potential problems or areas for improvement.

The current status is that we’re starting to develop a slightly better idea of what people are currently doing through use of the literature and also some professional networks (e.g. the Australasian Council on Open, Distance and E-learning) and have an initial prototype running.

Our current problem is how do you choose what the indicators should be? What are the types of problems you might see? What is a “good” course website?

Where are we up to?

Our initial development work has focused on three groupings of category: course content, coordinator presence and all interactions. Some more detail on this previous post.

Colin Beer has contributed some additional thinking about some potential indicators in a recent post on his blog.

Col and I have talked about using our blogs and other locations to talk through what we’re thinking to develop a concrete record of our thoughts and hopefully generate some interest from other folk.

Col’s list includes

  • Learner.
  • Instructor.
  • Content.
  • Interactions: learner/learner, learner/instructor, learner/content, instructor/content

Why and what?

In identifying a list of indicators, as when trying to evaluate anything, it’s probably a good idea to start with a clear definition of why you are starting on this, what are you trying to achieve.

The stated purpose of this project is to help us develop a better understanding of how and how well staff are using the Blackboard courses sites. In particular, we want to know about any potential problems (e.g. a course site not being available to students) that might cause a large amount of “helpdesk activity”. We would also like to know about trends across the board which might indicate the need for some staff development, improvements in the tools or some support resources to improve the experience of both staff and students.

There are many other aims which might apply, but this is the one I feel most comfortable with, at the moment.

Some of the other aims include

  • Providing academic staff with a tool that can aid them during course site creation by checking their work and offering guidance on what might be missing.
  • Provide management with a tool to “check on” course sites they are responsible for.
  • Identify correlations between characteristics of a course website and success.

The constraints we need to work within include

  • Little or no resources – implication being that manual, human checking of course sites is not currently a possibility.
  • Difficult organisational context due to on-going restructure – which makes it hard to get engagement from staff in a task that is seen as additional to existing practice and also suggests a need to be helping staff deal with existing problems more so than creating more work. A need to be seen to be working with staff to improve and change, rather than being seen as inflicting change upon them.
  • LMS will be changing – come 2010 we’ll be using a new LMS, whatever we’re doing has to be transportable.

How?

From one perspective there are two types of process which can be used in a project like this

  1. Teleological or idealist.
    A group of experts get together, decide and design what is going to happen and then explain to everyone else why they should use it and seek to maintain obedience to that original design.
  2. Ateleological or naturalist.
    A group of folk, including significant numbers of folk doing real work, collaborate together to look at the current state of the local context and undertake a lot of small scale experiments to figure out if anything makes sense, they examine and reflect on those small scale experiments and chuck out the ones that didn’t work and build on the ones that did.

(For more on this check out: this presentation video or this presentation video or this paper or this one.)

From the biased way I explained the choices I think it’s fairly obvious which approach I prefer. A preference for the atelelogical approach also means that I’m not likely to want to spend vast amounts of time evaluating and designing criteria based on my perspectives. It’s more important to get a set of useful indicators up and going, in a form that can be accessed by folk and have a range of processes by which discussion and debate is encouraged and then fed back into the improvement of the design.

The on-going discussion about the project is more likely to generate something more useful and contextually important than large up-front analysis.

What next then?

As a first step, we have to get something useful (for both us and others) up and going in a form that is usable and meaningful. We then have to engage with them and find out what they think and where they’d like to take it next. In parallel with this is the idea of finding out, in more detail, what other institutions are doing and see what we can learn.

The engagement is likely going to need to be aimed at a number of different communities including

  • Quality assurance folk: most Australian universities have quality assurance folk charged with helping the university be seen by AUQA as being good.
    This will almost certainly, eventually, require identifying what are effective/good outcomes for a course website as outcomes are a main aim for the next AUQA round.
  • Management folk: the managers/supervisors at CQU who are responsible for the quality of learning and teaching at CQU.
  • Teaching staff: the people responsible for creating these artifacts.
  • Students: for their insights.

Initially, the indicators we develop should match our stated aim – to identify problems with course sites and become more aware with how they are being used. To a large extent this means not worrying about potential indicators of good outcomes and whether or not there is a causal link.

I think we’ll start discussing/describing the indicators we’re using and thinking about on a project page and we’ll see where we go from there.

Getting started on Blackboard indicators

The unit I work for is responsible for providing assistance to CQUniversity staff and students in their use of e-learning. Which currently at CQUni is mostly the use of Blackboard.

The current model built into our use of Blackboard is that the academic in charge of the course (or their nominee) is responsible for the design and creation of the course site. In most instances, staff are provided with an empty course site for a new term at which stage they copy over the content from the previous offering, make some modifications and make the site available to students.

Not surprisingly, given the cruftiness of the Blackboard interface, the lack of time many staff have and a range of other reasons there are usually some fairly common, recurrent errors that are made. Errors which create workload for us when students or staff have problems. In many cases it may even be worse than this as students become frustrated and don’t even complain, they suffer in agony.

Most of these problems, though not all, are fairly simple mistakes. Things that could be picked up automatically if we had some sort of automated system performing checks on course sites. The fact that Blackboard doesn’t provide this type of functionality says something about the assumptions underlying the design of this version of Blackboard – a very teaching academic focus, not so much on the support side.

Developing this sort of system is what the Blackboard Indicators project is all about. It’s still early days but we’ve made some progress. Two main steps

  • Developed an initial proof of concept.
  • Started a literature, colleague and literature search.

Initial proof of concept

We currently have a web application up and running that, given a term, will display a list of all the courses that are meant to have Blackboard course sites and generate a number between 0 and 100 summarising how well a site has meant a particular indicator.

Currently, the only indicator working is the “Content Indicator”. This is meant to perform some objective tests on, what is broadly defined as, the content of the course. Currently this includes

  • Is the course actually available to students?
    The score becomes 0 automatically if this is the case.
  • Does the the site contain a link to the course profile?
    20 is taken off the score there isn’t one.
  • Is the course profile link for the right term?
    50 taken off if it’s wrong.

At the moment, we’re planning to put in place three indicators, the content indicator plus

  • “Coordinator Presence”
    How present is the coordinator of the course? Have they posted any announcements? Are they reading the discussion forum? Posting to it? What activity have they done on the site in the last two weeks?
  • “All interactions”
    What percentage of students and staff are using the site? How often? What are they using?

It’s still early days and there remain a lot of questions, which we hope will be answered by our searching and some reflection.

Literature, web and colleague search

We’ve started looking in the literature, doing google searches and asking colleagues what they are doing. Have some interesting information already.

What we do find will be discussed in our blogs, bookmarked on del.icio.us (tag: blackboardIndicators) and talked about on the project page.

Alternate foundations – the presentation

A previous post outlined the abstract for a presentation I gave last Monday on some alternate foundations for leadership of learning and teaching at CQUniversity. Well, I’ve finally got the video and slides online so this post reflects on the presentation and gives access to the multimedia resources

Reflection

It seemed to go over well but there’s significant room for improvement.

The basketball video worked well this time, mainly because the introduction was much better handled.

What was missing

  • Didn’t make the distinction between safe-fail and fail-safe projects.
  • Not enough time on implications, strategies and approaches to work with this alternate foundation.
  • The description of the different parts of the Cynefin Framework were not good

The second point about strategies of working within this area is important as the thinking outlined in the presentation is hopefully going to inform the PLEs@CQUni project.

The resources

The video of the presentation is on Google Video

The slides are on Slideshare

Some alternate foundations for leadership in L&T at CQUniversity

On Monday the 25th of August I am meant to be giving a talk that attempts to link complexity theory (and related topics) to the practice of leadership of learning and teaching within a university setting. The talk is part of a broader seminar series occurring this year at CQUniversity as part of the institution’s learning and teaching seminars. The leadership in L&T series is being pushed/encouraged by Dr Peter Reaburn.

This, and perhaps a couple of other blogs posts, is meant to be a part of a small experiment in the use of social software. The abstract of the talk that goes out to CQUniversity staff will mention this blog post and some related del.icio.us bookmarks. I actually don’t expect it to work all that well as I don’t have the energy to do the necessary preparations.

Enough guff, what follows is the current abstract that will get sent out.

Title

Some alternate foundations for leadership in L&T at CQUniversity

Abstract

Over recent years an increasing interest in improving the quality of university learning and teaching has driven a number of projects such as the ALTC, LTPF and AUQA. One of the more recent areas of interest has been the question of learning and teaching leaders. In 2006 and 2007 ALTC funded 20 projects worth about $3.4M around leadership in learning and teaching. Locally, there has been a series of CQUniversity L&T seminars focusing on the question of leadership in L&T.

This presentation arises from a long-term sense of disquiet about the foundations of much of this work, an on-going attempt to identify the source of this disquiet and find alternate, hopefully better, foundations. The presentation will attempt to illustrate the disquiet and explain how insights from a number of sources (see some references below) might help provide alternate foundations. It will briefly discuss the implications these alternate foundations may have for the practice of L&T at CQUniversity.

This presentation is very much a work in progress and is aimed at generating an on-going discussion about this topic and its application at CQUniversity. Some parts of that discussion and gathering of related resources is already occuring online at
http://cq-pan.cqu.edu.au/david-jones/blog/?p=202
feel free to join in.

References and Resources

Snowden, D. and M. Boone (2007). A leader’s framework for decision making. Harvard Business Review 85(11): 68-76

Lakomski, G. (2005). Managing without Leadership: Towards a Theory of Organizational Functioning, Elsevier Science.

Davis, B. and D. Sumara (2006). Complexity and education: Inquiries into learning, teaching, and research. Mahwah, New Jersey, Lawrence Erlbaum Associates

A new "all-in-one" introduction to OASIS

A new introduction to OASIS was delivered in a 90 minute session today to a collection of CQUni staff from a number of campuses. The 100+ slides and accompanying video provide an introduction to all aspects of OASIS which are covered in more detail in separate slidecasts available from the main OASIS page.

PLE drivers being considered in the corporate IT world?

The PLEs@CQUni is being driven, in part, by a range of external factors around the practices, availability and affordances of information technology, especially those associated with Web 2.0 and social software. We’ll be looking at this means for the use of educational technology within universities, not to mention the practice of learning and teaching.

Obviously these same drivers are going to have some interesting implications for the practice of the broader problem of how to IT is supported within organisations. There’s sure to be much work in this area and it will be important to keep an eye on it for what they find and subsequently work out what it means for the PLEs project.

Susan Scrupski drops a few hints about some research she is involved with and provides a link to a video commentary from one of the US news/business programs talking about one aspect of the problem. The problem talked about in this commentary is one which faces how universities practice of e-learning.

Of course, there will be some argument about all this being just the latest fad being beaten up by various academics and commercial consulting firms. But I’m not sure that this is a fad, it seems to be a key shift in IT and one that will need to be addressed. Accompanied, of course, by a good dose of cynicism.

For example, I’m not sure that the broad generalisation of “younger workers” used in the video is broadly true. Do all young workers really want that? Do all CQU students really want that?

The real questions become, just how do you address this problem, provide the type of resources that they younger workers expect within the constraints of existing organisations, especially in terms of resources? For example, at my institution there seems to be growing concern about the cost of Internet usage. Usage is growing, it is costing more and this at a time when minimising cost is important.

Creating a voice thread presentation

The following is step 2 in getting organised for a trial of VoiceThread as part of the PLEs@CQUni project. The background was given in a previous post.

This post tries to summarise what’s been found about create a presentation in Voice Thread. It’s more a work in progress and a way of saving what I’m finding, rather than any particular use for anyone else.

It appears that a presentation (i.e. like Powerpoint) might be one approach to support the development of an online research poster. Much of this information is taken from the VoiceThread Tutorials example on VoiceThread Presentations.

The example is a nice example, includes the talking head and some doodling.

It’s that hard a process, apparently includes the following steps

  • Create the presentation file.
    VoiceThread suggest that the presentation file should be PDF and 1024×768 or larger with a 4:3 aspect ratio. Powerpoint, at least on the Mac, has it as an export option.
  • Upload it to Slideshare.
    This seems a fairly simple process, need to set the options. The 1 minute VoiceThread tutorial gives a good introduction.
  • Set some options – collaboration
    Has the ability to invite people, make it public, moderate comments and include the voice thread in public list. Will need to include this in a screen cast.

  • Record the narration.
    Using voice thread’s 5 methods of recording.

Comments on VoiceThreads appear to require that you have a VoiceThread account. This will be a bit of a limitation when it comes to having visitors to the research poster session comment on the posters.

Interesting that Slideshare does offer a “guest comment” facility that requires you are able to read a captcha, rather than login.

Voice Thread for Research Posters

This post describes some very early thinking about a trial that forms part of the PLEs@CQUni project. The trial seeks to support the use of Voice Thread to allow students to share research posters they prepare as part of the CQUni course PSYC13021, Special Topic in Psychology. The aim here is to summarise the project and describe the first steps.

What is voice thread?

According to them, Voice Thread is

..online media album that can hold essentially any type of media (images, documents and videos) and allows people to make comments in 5 different ways – using voice (with a microphone or telephone), text, audio file, or video (with a webcam) – and share them with anyone they wish. They can even be exported to an Archival Movie for offline use on a DVD or video-enabled MP3 player.

The most interesting aspect of what VoiceThread is, in terms of this trial, is encapsulated in the following quote

A VoiceThread allows group conversations to be collected and shared in one place, from anywhere in the world.

The following embedded voice thread is the example.

How is it going to be used?

The existing assignment for the course is for the students to prepare a research poster. The type of poster traditionally seen at academic conferences. A poster summarises a research project, usually in the form of a couple of posters, and is then presented in a poster session in some big hall. Conference attendees file past, stop and view those that interest them and talk with the poster presenters.

The special topic course, this term, is looking at the topic “Public health: A psychologist’s playground” and the offering’s synopsis is

The health of an individual constitutes a central foundation of what this person can achieve. Yet, everybody’s health is influenced by the environment and society in which they live. Public health is described as the art and science of preventing disease and promoting health through efforts focused on individuals, local communities or entire nations. Many efforts aim at (1) changing individuals’ and groups’ attitudes, beliefs, and behaviours, (2) at providing interventions to support individuals and groups in preventing disease and promoting their health, and (3) at providing the evidence base for the need and efficacy of any Public Health activities.

Psychological theories and tools form an important part of Public Health and psychologists are called to continually contribute to the development and improvement o f Public Health measures. This course will provide an introduction to Public Health, critically examine key issues in Public Health, and explore the roles and remits of psychologists and other professionals in this field.

Why use VoiceThread?

Well 89.7% of the students in this special topic are external students. They are likely never to set foot on a CQUni campus. Traditionally, external (AKA distance education, flexible learning) students would prepare their poster, probably as a powerpoint, and submit it for marking. There might be some audio narration attached.

The only feedback the students would get would be from the markers. In fact, the markers would become the only folk who ever see the research posters.

The aim for this offering is to use VoiceThread to host the students’ posters so that people out on the Web will be able to see the posters and hopefully comment on them. It’s also hoped to run a session on the Rockhampton campus of CQUni later in the year and invite local psychologist to view the research posters. The posters will be shown on stations that will allow the attendees to view the VoiceThread posters and also to make comments on the posters.

Hopefully this will increase the feedback the students receive on their posters and perhaps provides greater motivation.

What needs to be done

We need to provide some support to the students about how to use voicethread to share their research poster. To reduce the perception of difficulty and wasted time they will initially perceive this approach to have. To help them see that we’re not simply using technology for technology’s sake.

Simple, eh?

Gathering principles for Web 2.0 – PLEs

The PLEs@CQUni project is attempting to figure out how/if social software, web 2.0 etc can be effectively used at CQUni to improve learning and teaching. I’m part of a group attempting to figure out how we can do this, figure out what works, what doesn’t and get these technologies/ideas used effectively.

As part of this process we need to think about the principles that underpin these technologies and identify how they can used, what problems they will pose and how we can investigate further what they mean for the students and staff of CQUni. This post is, hopefully, the start of a gathering of and perhaps some reflection on what others have already written about these principles.

Web 2.0

In this Slideshare presentation ???? lists the following principles of Web 2.0 (no references)

  • No products but services – “There are no products, only solutions”
    Which seems to focus on simple solutions to customer needs identified through a problem solving approach. This has some implication for the processes to be used, a sense-making, adopter focused approach could be argued to be more appropriate.
  • Customisation
    Allow the user to choose, don’t force them to use what you have made. Allow them to incorporate what we provide into their “home” in a way that they choose. This is exactly the opposite of what happens in traditional IT divisions where the focus is on providing one way to do things as that is cheaper and easier to support.
  • Focus on the “long tail”
    Don’t focus on just the majority, look to all of the folk. Look to leverage customer self-service?
  • Harness collective intelligence
    Make use of network effects, wisdom of crowds etc. Make the results of that collective knowledge available to the user. Try to encourage participation, easier said than done because only a small percentage of folk will contribute. Implications about openness and trust which may prove challenging in an organisational setting.
  • Specialised databases
    Claims that every significant Web 2.0 application has been backed by a specialised database (e.g. Google, Amazon, eBay etc.). Potential connection here, the specialised database for PLEs@CQUni would be the CQU specific data: course content, discussions, staff and student knowlege about the learning experience etc.
  • Who owns the data
    A particularly interesting question in this context where information sharing isn’t typically near the top of the agenda.
  • Perpetual beta
    No more version numbers. It’s always being improved in small ways. This has interesting implications when an organisation is using traditional project based development approaches. i.e. where the developers only get to work on specific projects selected by their management. An approach that generally has to have version numbers.
  • Software above the level of a single device
    Move away from the computer focus, support different devices, allow use on any.

The honeycomb approach Gene Smith has taken identifies seven building blocks for social software. Those seven are:

  • Identity – a way of uniquely identifying people in the system
  • Presence – a way of knowing who is online, available or otherwise nearby
  • Relationships – a way of describing how two users in the system are related (e.g. in Flickr, people can be contacts, friends of family)
  • Conversations – a way of talking to other people through the system
  • Groups – a way of forming communities of interest
  • Reputation – a way of knowing the status of other people in the system (who’s a good citizen? who can be trusted?)
  • Sharing – a way of sharing things that are meaningful to participants (like photos or videos)

Creating slidecasts on Slideshare – e-learning support?

The problem

The unit I work with is responsible for helping staff (and to some extent students) of CQUniversity with their learning and teaching. This is traditionally a fairly difficult task which is made more difficult at CQUniversity by a number of contextual factors. Perhaps the largest is the fact that staff and students are spread across at least 9 Australian campuses spread across almost the full length of the Australian east coast, a couple of overseas campuses and partners and throughout the world via distance education.

Face-to-face support sessions, either one on one or in groups, is somewhat difficult when our small unit is entirely based on one of the campuses. We have to look at using technology and other strategies to address this geographic distribution. We’ve been slowly developing our website using a Wiki and other Web 2.0 tools. This post talks about our early attempts at using Slidecasts – simple powerpoint presentations with a narration. In our case designed to be short and sharp and focus on a particular need.

In some vague, nascent and emergent way this work also links into and aims to continue the growth of our PLEs@CQUni project which seeks to develop insights into how CQUniversity can effectively make use of the increasing number of social media services available out on the net. We hope, that if this approach proves useful, we will develop much simpler and easier ways in which CQUniversity staff and students can make use of this sort of approach.

What we’ve done

Our initial experiments with slidecasts have been around the use of CQU’s online assignment submission system – OASIS – and have been implemented using Slideshare.

The slidecasts that are in place at the moment cover the following topics

How we did it

The process currently being used involves the following steps

  1. Prepare a powerpoint presentation with an emphasis on showing what happens.
  2. Use the “record narration” facility of either Powerpoint 2007/2008 to “deliver” the session in a room.
    This is currently done

    • Using a simple usb headset/mic in my office.
    • Without linking the audio files (this makes sure that Powerpoint creates a separate wav file for each slide – which is important for following steps).
    • On a Windows box – the times I’ve tried this on my Mac Powerpoint has cut arbritary lengths of audio off the end of each slide’s narration.
  3. Save a copy of the presentation (without audio) in Office 2004 format and upload it to Slideshare.
  4. Unzip the Office 2007 version of the presentation (with audio) and create a single MP3 file containing the whole narration.
    This is done using the following steps (on the command line on my Mac)

    • Rename the audio files for the first 9 slides
      Powerpoint names the first 9 slides audio1.wav audio2.wav etc. This throws out the order of the slides. i.e. echo audio*.wav – results in the following order of files: audio1.wav audio10.wav audio11.wav……audio2.wav audio20.wav audio21.wav. Which is not good when you want to concatenate the files together in slide order. I do this with a simple shell script

      for name in 1 2 3 4 5 6 7 8 9
      do
        mv audio${name}.wav audio0${name.wave
      done
      

      Appropriate use of sort could probably achieve the same thing, but I’m not keeping the unpacked files, so no problems here.

    • Concatenate the individual wav files into one using SoX
      Very simply using “sox audio*wav all.wav”
    • Convert the wav file into mp3
      SoX should be able to do this, but I haven’t had time to nut it out (busy, busy) so I’ve fallen back on the approach I know that works – iTunes.
  5. Upload the mp3 file to a public website
  6. Use the audio linking slidecast facility on Slideshare to link slides to the corresponding bits of the audio
    The command soxi gives information about individual sound files. I use that to identify the end of each individual slide audio which helps make using the Slidecast audio linker quicker.

    for name in *.wav
    do
      echo $name
       soxi $name | grep Duration
    done
    
  7. Long term aim

    Much of the above process can be automated. I can see a process by which someone gives a presentation with the narration feature of Powerpoint turned on. They then upload the complete (and usually very large) file to a CQUniversity web site (the size of the file and the specific requirements for CQUni would require an institutional system). The CQUni system could then extract the audio, produce the mp3, upload to a public website, upload a version of the presentation to Slideshare and connect the audio with the slides.

    Perhaps the current major limitation with this idea, at least the last time I checked, is that the Slideshare API doesn’t/didn’t appear to provide support for providing timing data for the slides so that the MP3 audio synchronisation could be automated.

    The other major problem is whether or not this approach is actually useful, usable and used by staff and students.