BIM – tidy up #2

Some more tidying up of BIM to get it ready for release.

Error messages on student details

When one student views the details, we’re getting error messages.

Okay, need another check. That’s done.

Check the group usage

Have got the allocate groups stuff working, is it actually being used by the support functions that return lists of students associated with markers?

No.

So, the get_records_select for the tmp table bim_markers_students needs to get replaced with something that goes via bim_group_allocation and then group_members to get the userids.

That means changing the signature of the bim_get_markers_students function – take bim now, not course. My, the comments actually said course was temporary.

Update the calls to records select to go via the other tables.

Done.

Fix up the display of last post in the student details screen

This is a hang over of the change from datetime in BAM to unix timestamps in Moodle. There are at least 2 contributing problems to this :

  • Test data from BAM into BIM put in just a year, not the unixtime.
    So need to modify the lastpost entry in my test bim_student_feeds to be using unixtime. All entries changed to the same value.
  • The display of the timestamp in BIM isn’t doing the right conversion.
    Already doing this in the allocation and marking form using the PHP date function. Done.

Ahh, but that’s the rub. What is actually needed here is how long ago the last post was. That’s a different method.

Used function from here to implement.

Removed sortable links on tables

Haven’t groked how to do the flexible_table sorting. Will leave that for later.

Some reasons why business intelligence tools aren't the right fit

The following started as an attempt to develop an argument that business intelligence/data warehouse tools are not a perfect fit for what is broadly called academic analytics. In fact, as I was writing this post I realised that it’s actually an argument that business intelligence tools aren’t a perfect fit for what I’m interested in doing and that is not academic analytics.

The following is by no means a flawless, complete argument but simply an attempt to make explicit some of the disquiet I have had. Please, feel free, and I actively encourage you, to point out the flaws in the following. The main reasons for writing this are:

  1. see what form the final argument might take;
  2. see if I can convince myself of the argument; and
  3. see if others can see some value.

Background – the indicators project

Some colleagues and I are currently involved in some work we’re calling The Indicators Project which

aims to build on and extend prior work in the analysis of usage data from Learning Management Systems

In our first paper we presented some findings that contradicted some established ideas around LMS usage due to the differences in our student body, the breadth and diversity of the data we used.

The indicators project is especially interested in how what we can find out about LMS usage that can help improve learning and teaching. We’re especially interested in what analysis across time, across LMS and across institutions can reveal that we don’t currently know.

It’s somewhat related to the idea of academic analytics

Background – academic analytics

According to Wikipedia academic analytics is “the term for business intelligence used in an academic setting”. In turn, business intelligence is described as “skills, processes, technologies, applications and practices used to support decision making”.

In an environment that is increasingly demanding techno-rational approaches to management, especially the requirement for universities to quantitatively prove that they are giving value for money, universities have started to go in for “business intelligence” in a big way. For most, this foray into “business intelligence” means setting up a data warehouse and the accompanying organisational unit to manage the data warehouse.

The “intelligence” unit is often located either within the information technology grouping or directly within the senior management structure reporting to a senior executive (i.e. a Pro or Deputy Vice Chancellor within an Australian setting). This location tendency arguably reveals the focus of such units on either the technology or servicing the needs of the senior executive they report to.

With the increasing use of technology (especially Learning Management Systems – LMS/VLE) to mediate learning and teaching there becomes available an increasing volume of data about this process. Data which may (or may not) reveal interesting insights into what is going on. Data which may be useful in decision making. i.e. Academic Analytics (aka business intelligence for learning and teaching) has arrived.

In those universities where data warehouses exist, an immediate connection is made between analysing and evaluating LMS usage data and the data warehouse. It is assumed that the best way to analyse this data is to put it into the data warehouse and allow the “intelligence” unit to do their thing.

I’m not convinced that this is the best approach and the following is my attempt to argue why.

Business Intelligence != Data Warehouse

van Dyk (2008) describes “a business intelligence approach is followed in an attempt to take advantage ICT to enable the evaluation of the effectiveness of the process of facilitating learning” and argues that the context that leads to effective data for decision making “can only be created when a deliberate business intelligence approach if followed”. The paper also contains a description of a data warehouse model that accomplishes exactly that. The framework is based on the work of Kimball and Ross (2002) and is shown below

The business intelligence framework

As you can note this business intelligence framework includes as a core, and very important, part a data warehouse. Not surprising as it is based on a book about data warehouses.

However, drawing on the Wikipedia business intelligence page (my emphasis added)

Often BI applications use data gathered from a data warehouse or a data mart. However, not all data warehouses are used for business intelligence nor do all business intelligence applications require a data warehouse.

I’m trying to develop an argument that a data warehouse, defined as a tool/system that sold as a data warehouse tool, may not be the best fit for supporting decision making based around LMS usage data. In particular, it may not be the best fit for the indicators project.

But don’t you need a data warehouse

In the early days of the indicators project we developed an image to represent what we were thinking the project would do. It’s shown below.

Project Overview
Overview of indicators project

There is certainly some similarity between this image and the business intelligence framework above. Both images encapsulate the following ideas:

  • You take data from somewhat, do some operations on it and stick it into a form you can query.
  • You take some time to develop queries on that standardised data that provide insights of interest to people.
  • You make that information available to folk.

So you’re doing essentially the same thing. A lot of people have spent a lot of time on the math and the IT necessary to create and manage data warehouse tools. So why wouldn’t you use a data warehouse? What’s the point of this silly argument?

The problems I have with data warehouses

My problem with data warehouses is that the nature of these systems and how they are implemented within organisations are a bad fit for what the indicators project wants to achieve. From above, the indicators project is especially interested in finding out what analysis of LMS usage data across time, LMS and institution can reveal beyond what is currently known.

The nature of data warehouses within universities and the tools and processes used to implement them are, from my perspective, heavy weight, teleological, proprietary and removed from the act of learning and teaching. These characteristics get in the way of what the indicators project needs to do.

Heavyweight and expensive

Institutions generally spend a lot of money on the systems, people and processes required to set up their data warehouses. Goldstein (2005) reports key findings from an ECAR study of academic analytics and suggests that more extensive platforms are more costly. When systems cost a lot they are controlled. They are after all an expensive resource that must be effectively managed. This generally means heavyweight processes of design and control.

While a significant amount of work has been done around evaluation LMS usage, there’s still a lot to discover. The very act of exploring the data – especially when going cross institutional, cross LMS and cross time – will generate new insight that will require modification to how data is being prepared for the system and how it is being reported.

This level of responsiveness is not a characteristic of heavyweight processes and expensive systems. Especially when the systems main aim and use is focused at other purposes.

Not focused on L&T

Goldstein (2005) reports that few institutions have achieved both broad and deep usage of academic analytics with the most active users coming from central finance, central admissions and institutional research. In fact, the research asked respondents to evaluate their use of academic analytics within seven functional areas. None of those seven areas involved teaching and learning.

This seems to suggest that the focus of data warehouse use within universities is not focused on L&T. The expensive resources of the data warehouse is focused elsewhere. Which suggests that resources available to tweek and modify reports and data preparation for learning and teaching purposes will be few and far between.

Proprietary

Due to the expense of these systems universities will sensibly spend a lot of time evaluating which systems to go with. This will lead to differences in the systems chosen for use. Institutional differences will also lead to differences in the type of data being stored and the format in which it is stored.

The indicators project has an interest in going across institutions. Of comparing findings at different universities. While a data warehouse approach might work at our current institution, it probably won’t be easily transportable to another institution.

This is not to suggest that it wouldn’t be transportable, but that the cost of doing so might exceed what is politically possible within current institutions.

Not located within L&T

It is well known that the two most important factors contributing to the adoption (or not) of a piece of technology are:

  • Ease of use; and
  • Usefulness.

Academic staff at universities are not rewarded (by the institution) for spending more time on their learning and teaching. They do not received any, let alone significant, encouragement to change their practice. Academics are generally given enough freedom to choose whether or not they use a tool and always have the freedom to choose how they use a tool. i.e. if they are forced to use a tool that is not easy to use and/or useful, they will not use it effectively.

The reports and dashboards associated with data warehouse tools do not live within the same space that academic staff use when they are learning and teaching. E-learning for most university staff means the institutional LMS. Systems that are not integrated into the every day, existing systems used by academic staff are less likely to be used.

The usefulness of these reports will be governed by how well they are expressed in terms that can be understood by the academic staff. Goldstein (2005) reports on there being a two-part challenge in providing effective training for academic analytics. I’m going to divide those two into three challenges (in the original the last 2 in my list were joined into one):

  1. help users how to learn the technology;
  2. help users understand the underlying data; and
  3. envision how the analytical tools could be applied.

To me, the existence of these challenges suggest that the technology being used is inappropriate. It is too hard or different for the users to understand and the information being presented is also too far removed from their everyday experience. i.e. if they need training in how to use it, then the tool is too far removed from their existing knowledge.

Given that Goldstein (2005) found these difficulties for the “sweet spot” of business intelligence (i.e. “business and finance”, “budget and planning”, “institutional research” etc.). Imagine the difficulties that will arise when attempting to apply the same technology to learning and teaching. Learning and teaching itself is inherently diverse, while the perspectives of learning and teaching held by the academics doing the teaching is several orders of magnitude more diverse.

The key point here is the “build it and they will come” approach of putting this data into a data warehouse will not work. The academic staff will not come. A large amount of work is required to develop insights into how to identify and integrate the knowledge that arises out of the LMS data in a form that encourages adoption.

Getting academic staff to meaningfully adopt and use this information to change – hopefully improve – their teaching is much more important, difficult and expensive than the provision of a data warehouse. The wrong tool – e.g. a data warehouse – will significantly limit this much more important task.

So what

I believe any approach to use data warehouse tools to provide “dashboards” to coal face academics so they can see information about the impact of their teaching and their students learning, will ultimately fail, or at the very least be very expensive, difficult and be used in limited ways.

Is there any institution doing just this know that can prove me wrong?

What’s the solution?

That’s for another post. But what I’m thinking of is :

  • Much cheaper/simpler technology.
  • Lightweight methodology.
  • Research and coal-face informed development and testing of useful measures/information.
  • Design of additions to institutional LMS and other systems that leverage this information.

References

Goldstein, P. (2005). Key Findings. Academic Analytics: The Uses of Management Information and Technology in Higher Education. ECAR Key Findings, EDUCASE Center for Aplied Research: 12.

Kimball, R. and M. Ross (2002). The data warehouse toolkit: The complete guide to dimensional modeling, John Wiley and Sons.

van Dyk, L. (2008). “A data warehouse model for micro-level decision making in higher education.” The Electronic Journal of e-Learning 6(3): 235-244.

BIM – Tidying up – Part 1

Apart from some playing around with the gradebook, the new development for BIM is complete. What is left to do is the various cleaning up, double checking, documentation and other tasks necessary to turn BIM from essentially functional to something reasonably okay to install. This is the first post describing that process. It and the following ones will essentially be working through the to do list.

Created locallib.php

Theory is if you have a number of local functions in lib.php, you should move them to locallib.php – done.

Fix coordinator mark page

Coordinator should be able to use the same code to generate a page to mark a post as used by the marker. Main difference is that the coordinator has some extra tabs. As it stands, any attempt to make a post displays the “Configure BIM” output. Fix it.

Diagnose the problem:

  • URL is /mod/bim/view.php?id=71&screen=MarkPost&markingId=538
  • However, because the use has the coordinator capability, this goes to coordinator/view.php:show_coordinator.
  • It appears that there should be a tab parameter in the URL
  • Ahh, the tab parameter gets set in an if, based on the value of screen, but MarkPost wasn’t included.
  • Add that in, all fixed.

Get auto-discovery working

This essentially means identifying the right PHP library to use to do this for me. Based on discussions in Moodle, it looks like SimplePie is the go.

Got an initial script using SimplePie going.

And got the auto discovery think working. Looks like there is more work to do there. Also need to discover whether simplepie is now coming with Moodle or whether I’ll need to include it in BIM, at least initially.

BIM – sending results to the gradebook

Almost forgot to add this into BIM. BIM is used by staff to manage and mark feed/blog posts from students. Eventually, those marks have to be put into the Moodle gradebook so they can form part of the students final grade. This post summarises a search for information about how this can be done within Moodle and some design thinking about how to get it done in BIM.

Activity modules sticking stuff into the gradebook

Initially, I’m using the “raw” version of the Moodle extension development book to get this information. Chapter 4 of the book covers activity modules, including talking to the gradebook. First a quick summary of what I’m reading before I start trying to understand it.

Page 60 talks about a modgrade element of formslib for creating a menu for the user to select a point value for the activity. May need something like this in the coordinators view.

gradelib.php seems to be the API. Misc ideas from the gradebook for further lookup

  • specify maximum scale.
  • grade_update accepts an array of grades
    $grades = array(‘userid’=>$userid, ‘rawgrade’=>$grade);
  • parameters for grade update, based on activity module instance
    $params = array(‘itemname’=>$foo->name, ‘idnumber’=>$foo->id);
  • setting $params values for ‘gradetype’ ‘grademax’ ‘grademin’ ‘reset’
  • Call the grade_update function
    return grade_update(‘mod/foo’, $foo->course, ‘mod’, ‘foo’, $foo->id, 0, $grades, $params);

So essentially, the overview is that there’s a single grade_update method that takes a collection of parameters.

I probably could of gotten that from the Moodle docs and the library file. Time to look at those next.

Moodle resources

Docs on grades which includes mention of the limited public API used by activities to sent grades. Modules do not read grade tables or use the internal API. And the docs on the API.

Functions of interest in the API include:

  • grade_is_locked($courseid, $itemtype, $itemmodule, $iteminstance, $itemnumber, $userid=NULL)
    To tell whether a grade or grade item is currently locked. i.e. (I believe) the module can’t update it. Seems to be confirmed by this. Related is if grades are manually changed in the gradebook, they become read only in the module.

This section talks about functions required in lib.php of a module. Suggesting that there are “many examples in official modules” with assignment being the most advanced.

There’s also some general docs on Grades which suggests three building blocs of the Moodle gradebook:

  • the grade category
    Can have its own grade formed by calculating the grades from individual items within the category. Categories may belong to other categories. All categories belong to the course category.
  • the grade item
    An item can only belong to a single category. Become columns in the grader report. May refer to course activities or manual grades.

    Modules generate grade items through the Gradebook API to generate a matching grade item.

    The gradebook interface allows a number of settings to be edited, including a multiplicator. Important: This may imply that BIM only has to add up the total marks and leave the multiplicator to the gradebook. This same editing interface allows specification of whether the item is hidden or locked.

  • The grade

Questions

  • Is there a need to provide specific functions in mod/bim/lib.php? If so, what do they need to do and how do they relate to the interface for the coordinator to control this?
  • Need to learn more about the parameters and implications for the functions in the API.
  • How does a item for a specific activity get added to the gradebook? Can you simply add a student’s grade and the item for all students will be added OR do you need to create the item and then add student grades?
  • It appears scale is related to the grademenu and specifies the type of scale being used?

lib.php required functions

It appears necessary to create some functions within the lib.php file for the module. Need to find out more. This is not something that the example from the book uses. No mention of “update_grades” in the book. It creates a new function (no required by Moodle) called “Foo_grade”.

A search of module code for “update_grades” finds elements in modules such as: quiz, scorm, lesson, assignment. The subcourse module also provides some examples, but not using the standard lib.php type stuff.

Ahh, a
new NEWMODULE.zip
includes the functions related to gradebook, at least some. Having a look now. Not all that complete, just stuff around scale. Points to the forum module.

Forum module includes the following functions in lib.php which appear to be somewhat common:

  • forum_get_user_grades( $forum, $userid=0)
    Return grade for given user or all users.

    Based on type of forum aggregation, do different selects into forum database to get different recordset (get_records_sql) and apply some scaling.

  • forum_update_grades( $forum=null, $userid=0, $nullifnone=true)
    Update grades by firing grade_updated event.

    Based on various conditions call various other grade update functions.

  • forum_grade_item_update( $forum, $grades=NULL)
    For a given forum, update grade item.

    Wrapper around grade_update that modifies the params based on various forum settings.

  • forum_grade_item_delete( $forum )
    Another wrapper to use grade_update to delete grade item for the forum.

Current situation

It appears that there are two separate “APIs” at play here:

  • The minimalist gradebook API in gradelib.php that provides the basic operations for a module to update grade items.
  • The API/collection of functions that can be defined in the lib.php file of a module so that use of the gradebook API works with the module user interface.

It doesn’t appear that it is necessary for you to actually use the module API. The example from the extension development book didn’t. But I expect it would be good practice to do so.

I’ve not been able to find documentation that explains either very well. Beyond of course the actual working code of existing modules.

If you use the “module” API then you should also probably include some options in the mod_form.php for the type of grading you want for the specific activity to be set.

Then there is this quote from the development grades docs

Modules usually store raw grades internally and pass them into gradebook every time they change. Gradebook may also request activities to resend the grades.

This suggests that the module API functions may be important so that the gradebook can “request” the resending of grades.

Playing with Foo

I’m going to install the Foo activity from the extension development book and observe what happens. In particular to answer question #3 from the questions section above.

  • Copy the code into the moodle tree.
  • Visit the site as the admin user. Upgrade/install of foo works.
  • Go back to the course and can now add “Activity Foo!”
  • Does include a grade menu in the mod_form.php.
    It seems to follow a fairly standard structure. Ahh, this is the modgrade element. Need to find out more about how this works and what it means for users and coding.
  • Foo is created, but nothing is added to the gradebook at this stage.
  • Use the foo. Ahh, the coordinator/teacher has no access to submit.
  • What about a student? Yes, they have access. Submit.
  • Has the gradebook changed? No!!!!
  • Was viewing the simple “Grader report”. Perhaps change has happened, but I just can’t see it. No!! Try full view.

Playing with forum

That hasn’t answered any questions. Time to comment the forum grades functions and create a new one. Submitting the form has resulted in an entry being added and also some of the “comments” being displayed (to fast to see though).

Add in a die. It seems to be a call to forum_grade_item_update with grades NULL and the params set to (
[itemname] => Another one
[idnumber] =>
[gradetype] => 2
[scaleid] => 1
) Where “Another one” is the name of the forum, and a believe the name for the grade item.

forum_grade_item_update is only called from with lib.php. One of which is within add_instance and which passes only the name of the new instance. Also called in update_instance

Fixing foo

So, is the problem with the foo module that it isn’t creating this grade item in the gradebook before trying to add individual items for students? What does gradelib.php say that grade_update should do?

From the comments

Submit new or update grade; update/create grade_item definition. Grade must have userid specified, rawgrade and feedback with format are optional. rawgrade NULL means ‘Not graded’, missing property or key means do not change existing.

. So it does both. The question is whether or not foo is doing it properly.

Let’s do a comparison of the parameters being passed.

Here’s the important bit – in gradelib.php the $grades parameter is set to NULL if you are updating the grade_item definition.

By adding a call to grade_update in the add_instance of the foo activity. I’ve got it adding the item to the gradebook.

Some other limitations of the module are preventing me from going any further. I think I have a handle on this. Time to submit the errata.

Suggestion for process

Based on what I’ve learned so far, the following is what I’m thinking of implementing.

  • Modify mod_form.php to ask for a scale and whether or not this should be graded.
  • Modify lib.php functions and database to store the scale.
  • Modify add_instance and other related functions to modify the gradebook in appropriate ways.
  • Think about whether we need a special function to use the grade book API.
  • Think about how the updating of the gradebook should occur
    • Grade should be just an addition of the marks to questions.
    • Should gradebook be updated when marked or when released (released?
  • Have BIM simply set the raw grade, no support for scales etc, at least not initially.
    Which means the form option will be “gradebook” or not.

Implementation will have to wait till another time.

Increasing weak ties, not strong – improving learning and teaching

I’m wondering if too much of the focus on improving teaching within universities is focused on groups within strong(er) ties at the cost of increasing the weaker ties.

Strong(er) ties

It’s my observation that attempts within institutions of higher education to improve learning and teaching generally focus their strategies on discipline groups. At my current institution strategies currently being adopted include:

  • Seeking to employ strong leaders within discipline/school groupings.
  • Increase the power of those discipline/school groupings, and subsequently reduce the power of broader multi-discipline groupings.
  • Align curriculum designers along school/discipline groupings.

There are also some traditional attempts at cross-disciplinary, cross-institution approaches to improving learning and teaching. For example, running special presentations/sessions by visiting or local experts and setting up communities of practice around certain specific interests. It could be argued that these are focused on encouraging weak ties. However, I would argue that the type of people who participate in these approaches are generally the small percentage of university academics who are inherently interested in teaching and learning and improving their own teaching. i.e. a group that’s likely to create strong ties and a group that represents a very small percentage of the overall academic population.

In addition, interaction between support folk (e.g. curriculum designers, information technology folk etc) are increasingly being limited through governance structures. That is, you can only talk to someone who knows about Moodle and get them to do something if you’ve gone filled out various forms, it’s been discussed at various project boards, received approval, been run through the project manager/analyst….

Weak ties

This post doing the rounds this morning examines the question of Dunbar’s number. A theoretical cognitive limit on the number of strong ties that anyone person can handle. Many of the above approaches, appear to suffer from this limit (as well as other problems).

The post goes onto argue, based on this book, that the real value of collaboration arises from weak ties and that weak ties can “beat” Dunbar’s number.

A major flaw in improving teaching

It’s my current thought that a major flaw in attempts to improve learning and teaching at universities, especially the ones I’m currently seeing within my institution, is that they focus on creating formal, strong ties between people. Ties that are limited in terms of size but more importantly in terms of variety.

There’s a need to focus more on enabling and encouraging weak ties that cross disciplinary and organisational boundaries. In particular, in enabling them to creating meaningful innovation and change and also to hold them accountable in someway.

BIM – Allocating markers

I had sort of hoped that most of the new BIM development was finished. However, after a bit of research at the local institution it becomes obvious that I’ll have to implement a marker allocation screen. Hopefully, it won’t require that much work.

Requirement

The aim is to allow the coordinating teacher to specify which groups of students will be marked by which staff. In my head the aim will be to:

  • Get and show a list of all the teaching staff currently allocated to a course.
  • Get a list of all the current groups allocated for a course.
  • Show a list of the groups in a multi-select box against each staff member.
  • Allow the coordinator to select and update as required.
  • Provide a pointer to the Moodle tools/links to allocating users and adding/managing groups.

The plan is to implement this as another tab for the coordinator.

Once this is implemented will need to modify the existing bim_get_markers_students and related functions to use the new method.

Should also look at providing away to make teaching staff coordinators – or is this simply done by capabilities?

groupslib

Developement docs for groups in Moodle. Somewhat limited, but does appear to have some important point re: the use of groupings.

Would appear that groupslib.php is the place to find functions associated with Moodle groups. The following list summarises some of those that might be of interest to this task.

  • groups_get_groupname( $gid )
  • groups_get_group_by_name
    The implementation of this suggests that the groups table includes courseid as a field.
  • groups_get_group( $gid ) - return the record for the group
  • groups_get_all_groups( $courseid, $userid=0, $groupingid=0 ) - will use this one
  • groups_get_user_groups( $courseid, $userid )
  • groups_get_members( $gid...)

There is something called groupmode applied to activity and course setting. Will need to look at that.

There are also some functions apparently creating menus

  • groups_print_course_menu( $course, $urlroot, $return=false)
  • groups_print_activity_menu( $cm, $urlroot, $return=false, $hideallparticipants=false )

Get the data

To implement this will need to retrieve:

  • Details of all teaching staff in a course.
  • List of all groups associated with a course.
  • How best to do multi-select boxes in Moodle.
  • Eventually, how to get members of a group.

Implementation

Add the extra tab

Get the structure going and start doing some testing. Added new allocate_markers.php. Basically done.

Testing with groups

My initial though was a simple moodleform to implement this. However, to some extent the existing functions that show menus, might be the way to go?

groups_get_user_groups is currently returning an empty array. Probably because there are no groups configured for this course. Yep, no groups created. Let's create one and see what happens.

Group created, but no-one assigned to it. It still shows up as empty. But I am asking specifically for groups that the user is a member of. What if I go more general?

Yes, groups_get_all_groups for the course, gives me all the groups. Let's allocate the user and see that get_user_groups change. Yep, when allocated, groups_get_user_groups returns the id of the group, but no other information. What is returned from groups_get_all_groups includes full information keyed on id.

I think I will try the Moodle form approach.

Getting all teaching staff

In reality, the point is not to get all the teaching staff, but to get all the staff that have the capabilities associated with marking students. For BIM, I've specified these as mod/bim:marker and mod/bim:coordinator. I'm interested in all users with those capabilities.

There's a get_users_by_capability function in accesslib.php - but there's a warning about performance. It seems the advice is to use it just once per page load.

Nutting out the life-cycle

In terms of the life-cycle for group allocation for BIM, I can see the following stages:

  • No groups at all in the course.
    Need to advise the coordinator to go and create some groups.
  • No teaching staff won't happen, at least the coordinator should be listed.

Add the database table

The aim of this form is to allocate staff members to particular groups of students for marking purposes. Am going to need a table to track which groups a staff member has been allocated to. Am thinking of something fairly simple.

CREATE TABLE mdl_bim_group_allocation (
    id BIGINT(10) unsigned NOT NULL auto_increment,
    bim BIGINT(10) unsigned NOT NULL,
    userid BIGINT(10) unsigned NOT NULL,
    groupid BIGINT(10) unsigned NOT NULL,
CONSTRAINT  PRIMARY KEY (id)
);

Creating the form

The form will essentially be a list of the markers with a multi-select box containing course groups. From here it becomes a moodle form process.

Basically done, but am not pleased with the layout. Will eventually have to play with the CSS. It's workable as it stands.

Processing the form

  • Set the values from the existing allocations.
  • Display the form.
  • On submission, compare the allocations with those already in the database.

Question here about the best way to proceed:

  • Simply delete all entries for the bim and insert the new ones.
  • Loop through each marker and add or delete as required?

I'm leaning towards the latter. The plan is

  • Loop through each marker
    • if there's an allocation in the form, that's not in the dBase - add it.
    • if there's an allocation in the dBase, but not in the form - delete it.

The question now is whether there's a way in Moodle to execute multiple SQL statements, or whether I should do it one, by one?

delete_records_sql seems to be the way to delete records and can do multiples. insert_record will insert one at a time.

Okay, after some "issues" with setting the data. All is done.

What was done

The marker allocations screen is complete.

Thoughts on DEHub research themes

My current institution is currently a member of the DEHub – “A community for learning and teaching, research and Innovation in Distance Education”. The exact connection with my current institution and more importantly the folk doing work in distance education/learning and teaching remains somewhat vague. However, it is probably time for me to actively see what connections exist with my work and interests, there should probably be a lot.

DEHub – Research themes

It would seem the folk at DEHub have done a fair bit of work generating a collection of research themes. It seems 15 were identified, with 7 prioritised for action in 2010. Those 7 are:

  1. Theories and models – trying to examine various theoretical frameworks that underpin distance education and what these might imply for future directions of distance education.
  2. Globalisation of education and cross-cultural aspects – i.e. global environment, need to consider different cultures etc.
  3. Access, equity, social inclusion and ethics – how can “e-learning” & high quality education be made available to those with limited resources.
  4. Professional development and faculty support – prof development seen as a pre-requisite to innovation and change (big assumption) and primary importance placed on identifying competencies of online teachers and how to develop them.
  5. Learner support and development approaches – how to support learners from marketing through alumni relations
  6. Curriculum design – focus on issues of educational design process for learning and course development. – emphasis on pedagocial approaches, how learning is achieved and assessed, design of culturally appropriate learning materials and opportunities for development of new educational technologies
  7. Interaction and communication in learning communities – connected with the curriculum design stage, this seeks to examine design that fosters engagement with an empahsis on online communities, gender differences and cross-cultural aspects of online communication.

Some immediate reactions

The following are some knee-jerk reactions to these research themes.

What about frameworks and models from further afield?

In trying to answer the question what distance teaching theories and models are required to meet future needs the range of identified research questions seem somewhat limited. There are a few questions around OERs (a recent fad), a couple about government, some connection with work integrated and lifelong learning but nothing really “out” there. I get the impression of a group stuck in a paradigm.

What about the impacts of different theories of learning (e.g. connectivism) on the assumptions of distance education? There is perhaps a bit of this in the question about models of education related to life-long learning. Rather than simply focus on OERs – an end product – what about the implications of openness on the processes associated with learning and distance education? What barrier/benefits exist from a truly open course or open curriculum design?

What about the very different insights that arise from complexity for the practice of distance education? A lot of the distance education literature (and a lot of the literature around L&T in higher education) strikes me as being based on assumptions of order, rationality and predictability that I simply don’t see as existing in organisations. In fact, I’d argue that the focus on order, massification and industralisation of print-based distance education was a direct contributor to its inability to adapt to the online world. It’s also a flaw in what I see around current practices of university e-learning with LMSes.

And that’s just drawing on a couple of my interest areas. There’s a lot more that I’m not all that familiar with.

Perhaps I’m just being a bit hard, or not seeing past the less confronting language used to phrase the questions.

The on-going separation of curriculum design and staff development

Through personal experience I’ve observed the silliness of making curriculum design separate from academic staff development/support. Based on that experience and some theory from complex systems and ateleological design I believe this is a fundamental flaw in much of higher education.

Curriculum design should not be separate from professional development nor from the support structures that enable academic to do the teaching. Instead, the network of people, knowledge, processes and systems that enable and support an academic to learn and teach needs to include significant capability within curriculum design. The processes involved in an academic actually doing the learning and teaching needs to include activities and events the encourage and enable the on-going evaluation and re-design of teaching.

More on this below, perhaps.

Effectiveness and efficiency of design methodologies

The guiding question for the curriculum design theme is

What design methodologies are effective and efficient for the design, development, implementation and evaluation of effective teaching and learning for social media-enabled environments?

In the subsequent discussion what I was interested in was exactly how would you measure the effectiveness and efficiency of instructional design methodologies? One of my bug bears is that most existing approaches to instructional design require quite significant involvement of an instructional designer. This is in part due to the academics not having expertise in instructional design but also largely because most instructional design methodologies require a fair bit of expertise.

<p At one stage, we had 2 and a bit curriculum designers responsible for helping in the improvement/re-design of a thousand or so courses. Heavy-weight instructional design processes simply could not scale in such an environment. The artificial separation of professional development and curriculum design at the organisational level also significantly hindered this work.

To some extent this is a symptom of the overly mechanical/techno-rational approach taken to curriculum design. The JISC project, Transforming curriculum design and curriculum delivery through technology, seems to promise something different in some of its blurb. But on further reading it’s heading down what might be an even more techno-rational approach that adopts an almost obsessive compulsive approach to tidiness and neatness.

Further work

There’s obviously connections between what I’m meant to be doing and DEHub. I’ll need to look further into how and what form this connection can make, what connection can be formed with other members of the unit I work with, and how some of the disagreements may be discussed, researched and harnessed.

Plans for a AJET paper around the indicators project

The following is an attempt to make concrete some idea and tasks associated with writing a journal paper for the Australian Journal of Educational Technology (AJET) based on the ASCILITE’09 paper based on the work of the Indicators project. The paper will be co-authored by a group and the aim of this post is to start discussion about content of the paper and the tasks we need to do to get it written.

The purpose/abstract of the paper

One argument is that as a group of authors we should have a common sense of the purpose of the paper. We should have an elevator pitch about the paper, a 30 second spiel that summarises what we’re trying to achieve/show. Any improvements?

The use of Learning Management Systems (LMS) is almost ubiquitous within universities. However, few institutions are actively using the data generated by student and staff use of these systems to guide future work. This paper seeks to highlight:

  • Some limitations of existing attempts to leverage LMS usage data.
  • Illustrate knowledge about e-learning within universities that, because of these limitiations, appear not to be as certain or clear as existing literature or common-sense might suggest.
  • Indicate how the work within this initial paper can be used to design further work that can improve our understanding of what is happening with e-learning, why it is happening and how it can be improved.

This might become a useful tag line for the paper:

  • What is happening.
    Refers to the patterns we find in the usage data.
  • Why is it happening.
    Refers to additional research with different methods and theories that would be required to suggest and test reasons why these patterns might exist.
  • How to change practice.
    Refers to further research and insight that seeks to combine the what and why information with other theory to change how learning/teaching practice is changed.

Ideas for titles

The title of a paper is, from one perspective, a summary of the elevator pitch. It should attract the reader and inform them what they might find out. What follows are some initial ideas.

  • The indicators project: Improving the what, why and how of e-learning.
  • Student LMS activity, grades, and external factors: Identifying the need for cross-platform, cross-institutional and longitudinal analysis of LMS usage.
    This title is based on a narrowing down of the “what” to be shown in the paper to just the linkage between LMS activity, grades and external factors. i.e. exclude the feature adoption stuff.

The structure of the paper

At this stage, the structure of the paper for me is heavily based around the three aims of the paper outlined in the purpose/abstract section above. The basic structure would be:

  • General introduction.
    Explain the background/setting of the paper. i.e. widespread use of LMS as implementation for e-learning. Start of people doing academic analytics. The identification of some known patterns. The importance of knowing what is going on and how to improve it. The diversity of universities and how one size might not fit all. i.e. different universities might have different experiences.
  • Limitations of existing work.
    Seek to provide some sort of framework to understand the work that has gone before in terms of academic analytics and/or LMS usage log analysis.
  • Different perspectives on what is happening.
    Examine and explain how we’ve found patterns which seem to differ or provide alternative insights to what has already been found. The idea here is to establish at least 2 groupings of patterns that illustrate some differences between what we have found and what has been reported in the literature. Each of the groupings could have multiple patterns/findings but there would be some commonality. More on this below.
  • Further work.
    Argue that these differing findings suggest that there is value in further work that:
    • addresses the limitations identified in the 2nd section and,
      i.e. cross platform, cross-institutional and longitudinal.
    • also expands upon the findings found in the 3rd section.
      Moves from just examining the “what” into the “why” and “how”.
  • Conclusions.

The work to be done

Now time to identify the work that needs to be done.

Limitations of existing work

The basic aim of this section is to expand and formalise/abstract the knowledge/opinion about the existing literature around LMS usage analysis expressed in the ASCILITE paper.

The draft ASCILITE paper – prior to compaction due to space limitations – was working on a framework for understanding the literature based on the following dimensions:

  • # of institutions;
  • # of LMS;
  • time period;
  • Method.

Work to do:

  • Ensure that we have covered/gathered as much of the relevant literature as possible.
  • Examine that literature to see how it fits within the framework.
  • Identify from the literature any additional dimensions that might be useful.
  • Identify if there is any findings that support or contradict the findings we want to introduce in the next section.

Different perspectives on what is happening

This is the section in which we draw on the data from CQU to identify interesting and/or different patterns from what is found in the established literature. The biggest question I have about this section is, “What patterns/groupings do we use?”. The main alternatives I’m aware of are

  1. Exactly what we did in the ASCILITE paper.
  2. The slight modification we used in the ASCILITE presentation
  3. Drop the feature adoption stuff entirely and focus solely on the correlation between student activity, grade and external factors. Perhaps with the addition of some analysis from Webfuse courses.

Whichever way we go, we’ll need to:

  1. Identify and define the patterns we’re using
    e.g. The correlation between level of participation, the grade achieved and some external factors.
  2. Identify literature/references summarising what is currently known about the pattern.
    e.g. The correlation that suggests the greater the level of participation in a LMS, the better the grade.
  3. Identify ways in which the pattern can be measured (see below).
  4. Use the measure to examine the data at CQU.
    e.g. many of the graphs in the ASCILITE paper/presentation.
  5. Look for any differences between expected and what we see.
    e.g. LMS usage by HD students is less than others for AIC students and Super low courses
  6. Establish that the differences are statistically significant.
  7. Perhaps generate some initial suggestions why this might be the case.

The patterns

The patterns we’ve been using so far seem to fit into one of two categories. Each of these categories has a definition of the pattern and how we’re actually measuring it. This is also an area of difference, i.e. there could be different ways of measuring.

The patterns we’ve used so far:

  1. % of courses that have adopted different LMS features.
    This is the comparison of feature usage between Blackboard and Webfuse during the period of interest. It shows that different systems and different assumptions do modify outcomes.

    • Current measurement – Malikowski et al
    • Alternative measurement – Rankine et al (2009)
  2. The link between LMS activity, student grades and various external factors.
    We’re currently measuring this by
    • # of hits/visits on course site and discussion forum.
    • # of posts and replies to discussion forum.

    The external factors we’ve used in papers and presentations are:

    • mode of delivery: flex, AIC, CQ.
    • Level of staff participation.
    • Different staff academic background.
    • Impact of input from instructional designer.
    • Age for FLEX students.

    I think there are a range of alternative measures we could use, need to think more about these.

We need to come to some consensus about the patterns we should use.

Statistical analysis

In the meantime, however, I think that we will end up using the activity/grades correlation at least for:

  • Mode of delivery.
  • Level of staff participation.
  • Age for FLEX students.

I would suggest that having the statistical analysis done and written up for these three would be a good first step. At least while we talk about the other stuff.

Further work

In terms of the further work section of the paper I think we need to:

  • Summarise the recommendations from the literature.
  • Identify where we might disagree and why.
  • Identify what we think should be done.

In terms of further work, my suggestions would be:

  • What
    • Testing the existing patterns in cross-platform, cross-institutional and longitudinal ways.
    • Establishing and testing alternate and additional patterns. e.g. the SPAN work
    • Establishing and testing alternate measurements for these patterns
    • Testing and developing alternate methods to make these patterns and enable different institutions to use them.
    • Identifying theories that would suggest other patterns which might be useful.
    • How to make these patterns available to front-line teaching staff and academics.
  • Why
    • Lots of work seeking to explain the differences in patterns found.
    • Identifying theories that help explain the patterns.
  • How
    • How to make these patterns available to staff and students in a way that encourages and enables improvement.
    • How to encourage management not to use these patterns as a stick.

Work to do

What follows is a list of tasks, by no means complete:

  • Everyone
    • Read and comment on the elevator pitch. Does it sound right? Can it be made better? Is there a different approach?
    • Does the proposed structure work? Should there be something more?
    • Suggestions for a title.
    • Thoughts about what patterns we use in the paper.
  • Stats guy
    • Have done the analysis on the “mode of delivery”, “level of staff participation” and “Age for flex students” patterns.
    • Be able to help us write a blog post that summarises/explains the analysis for each pattern in a way that would be suitable for the journal paper.
  • Non-stats guys
    • Combine existing literature around LMS usage analysis into a single bibliography, actively try and fill the holes.
    • Analyse the literature to help develop the framework for understanding and comparing the different approaches.
    • Pull out any interesting patterns, measures or findings that either support or contradict what we’ve found.
    • Writing
      • Introduction
      • Limitations of existing work
      • Future work.

Questions about alternatives to curriculum mapping

A small group of colleagues are discussing possible ideas around a project to engage in curriculum mapping. Possibly with the aim of going for an ALTC grant. As part of that process one of my colleagues has raised a range of questions about the project (we’re still at the storming and norming phase). The following is my attempt to answer some of those questions. The responses I give are obviously influenced by my beliefs about learning and teaching in higher education and how it does or does not work.

It is also influenced by existing writing/thinking I’ve done around curriculum mapping, including:

The following will build on and refer to the above.

Main challenges

The question

What are the main challenges for the teaching teams (and students) using mapping technologies?

My current response

The main challenge I see is staff engagement. Student engagement could possibly be grouped within this if the particular approach to curriculum mapping taken requires additional work from the students.

People will not take on new tasks simply because people outside think it’s a good idea. Teaching academics will not change their teaching practice within the current university environment. This is the core argument in my presentation Herding cats and losing weight: How to improve learning and teaching.

Unless curriculum mapping is seen and supported as a valuable and on-going part of how teaching and learning occurs, they won’t do it. At best they will be seen to do it.

This is especially true given my belief that most universities are currently at the level 2 of 3 levels of improving teaching. In particular, I would suggest that curriculum mapping, like most large-scale suggestions for improving teaching, are not being driven directly by the interests of academics or students, but by the following groups:

  • Managers of academic units who see the implementation of a unit-wide improvement as a way to show how they are good leaders. i.e. the focus is on what management does.
  • Accrediting bodies (government, professions etc) who wish to see evidence of the quality of the learning and teaching.
  • Vendors who have software or other products to sell to institutions that will help them achieve some widespread improvement.
  • Technologists in terms of information technology, but also education researchers and designers, who have some wonderful theory based idea that will radically improve learning and teaching, if only the recalcitrant academics would adopt it.

Now, the above might come across as to overly negative for some. However, it is my belief that this malaise underpins much of what goes on in universities at the moment and is either not recognised or widely ignored. It could be that I am wrong, however, there are some other folk who have made similar comments. For example, Cavallo (2004) that I’ve talked about here and here.

Disciplinary differences

The question

Are particular discipline groups engaged more with curriculum mapping than others? If so, what might be the reason for this and what are the impacts/outcomes?

My response

As I’ve written here, from my reading of the curriculum mapping literature most of the work seems to being done within the health disciplines, some within engineering and a bunch related to the idea of generic skills. I would suggest that most of these support the idea above about the important role of accrediting bodies in the drive towards curriculum mapping. In health, engineering and around generic skills there are bodies wishing to see proof of the quality.

Given that the major aim is to fulfill the purpose of the accrediting body, it is to me, no surprise that coal-face teaching academics aren’t exactly keen on the idea. e.g. the following quote from this post

Driven by government requirements, university initiatives around graduate attributes remain patchy with the best outcomes being the production of “curriculum maps” which have the potential to foster superficial and ineffective approaches to the development of graduate attributes (Green, Hammer et al. 2009).

Effective frameworks

The question

What frameworks and approaches can be developed to support effective, engaging and transformative usage of curriculum mapping?

My response

Given that I’ve argued above that the major challenge I see with curriculum mapping is that it is being done to academics rather than with them, I guess it is no surprise that I think the frameworks/approaches that have the best chance of working are those that seek to achieve level 3 of the 3 levels of improving teaching which I’ve described as

The focus is on what the teacher does to design and deliver their course. The aim is to ensure that the learning and teaching system, its processes, rewards and constraints are aiming to ensure that the teacher is engaging in those activities which ensure quality learning and teaching. In a way that makes sense for the teacher, their course and their students.

In this post I argued that around curriculum mapping such an approach would be much more bottom-up than top-down. An approach that started with what staff were already doing and seeking to build on it in interesting, useful and transformative ways.

I have previously, briefly described the mechanics of such a solution. There is more to it than just this, but it provides one concrete example. The theory behind bottom-up or emergent approaches is discussed in more detail in a sequence of posts: overview, teleological and ateleological.

This approach was inspired by the following quote from Gers Tielemans outlined in this post

The point is that Moodle works – by design – bottom-up while most curriculum tools work top-down.

  • In the top-down approach you have first to fill a cascade of questions before you can enter one single resource or activity (teachers hate this!)
  • In Moodle you start on course level and even there it is a free form design, teachers become “bricoleurs” (French for handyman).

Institutional changes

The question

What are the institutional changes needed to support curriculum mapping initiatives?

My response

The main change, and I think the most difficult, is a move away from “what management does” (this includes the tools of managers including IT, quality and educational designers) towards a focus on “what the teacher does” and how what they can do can be moved forward. The change needed is to create an environment that encourages and enables (i.e. does not force or require) academics to improve their practice. This is the argument I made in the herding cats presentation. That presentation also started making some recommendations for what changes would be necessary.

Pedagogies

The question

What pedagogies and approaches are needed to make the use of curriculum mapping applications most effective and engaging, and what evidence exists to support such approaches?

My response

Which ever pedagogies and approaches make sense to the individual academics that are working on the project. It may be that some interesting and useful appraoches arise out of the work that could be re-used. However, I don’t think you can identify one pedagogy or approach before the project starts. Once you have academics engaging effectively with curriculum mapping, then such things might be able to be identified.

As the project progress you will learn. Something from Cavallo (2004)

As we see it, real change is inherently a kind of learning. For
people to change the way they think about and practice education, rather than merely being told what to do differently, we believe that practitioners must have experiences that enable appropriation of new modes of teaching and learning that enable them to reconsider and restructure their thinking and practice. The limitations inherent in existing systems based upon information transfer models are as impoverished in effecting systemic development as they are in child development.

What value?

The question

What value can curriculum mapping add to conventional methods of educational delivery and what evidence exists to support such propositions?

My response

Again, this is a problem with ROI and teleological processes. In a truly complex environment, of which I think learning and teaching within a university is an example, you cannot predict what might happen. You can’t predict what values may arise.

You can fall back onto the traditional value attributed to curriculum mapping in the literature. Here’s a couple of paragraphs from this post

Curriculum mapping is a representation of the different components of the curriculum in order that the whole picture and the relationships between the components of the curriculum can be easily understood (Harden 2001). Curriculum mapping displays the essential features of the curriculum in a clear and succinct manner (Prideaux 2003) and provides a context for planning and discussing the curriculum (Holycross 2006).

Curriculum maps ensure that decisions about the curriculum are not made in a vacuum. Curriculum maps are a key requirement for curriculum development, which includes identifying professional development needs of academic staff, as curriculum maps provide the necessary information to identify the requirements of a particular course and program. Curriculum mapping provides an object that can help academic and other staff communicate about the curriculum of a course (Holycross 2006). Mapping is a remarkable tool for communication among teachers (Jacobs 2003).

You might even achieve some of these. But I’m more interested in the value/benefits that arise unexpectedly.

In terms of arguing the case for this project, if pushed, the specific values/benefits I’m most interested in are:

  • Reducing the workload and increasing the validity of the processes currently used by programs that are preparing to go for accreditation.
  • Leveraging that work in such a way as to actively increase visibility of the curriculum so that academic staff and students are more aware of the strengths and weaknesses and subsequently engage in communication about the curriculum.

In terms of that last benefit, I’m not sure what the outcomes of that communication might be. I’m only interested in enabling and encouraging it to happen as I believe it’s a necessary first step in encouraging academics to increase their reflection on practice.

In terms of the first benefit, that becomes the main concrete aim of the project, at least in my head and why I’ve suggested working with the accrediting bodies. If we can reduce the workload and increase the quality of existing accreditation processes, then that’s a nice concrete aim. The rest is probably more important, but it’s more uncertain and icing on the cake.

How will things change?

The question

Will the use of curriculum mapping approaches change how teaching and learning occur OR how people learn? If so, what evidence exists to support such a claim?

My response

I hope it’s becoming clear that, at least for this project, I’m not much interested in predicting what changes might occur. In fact, I hope you can see that I don’t think you can predict what changes might occur and any attempt to do so is wasted effort and at worst a case of you fooling yourself.

What change I think I’d like to aim for out of this project is tied to the values outlined in the last section:

  1. Reduce the workload and increase the quality of the accreditation process through having curriculum mapping that is much more closely aligned with what happens and also has concrete links to what has happened within the LMS.
  2. Investigate and provide services that increase the visibility of that curriculum while staff are teaching the courses that in turn encourage greater communication around the curriculum.

As I’ve said, it’s the increasing visibility that is, I think, the most important contributor to improvements, but improvements that will be unexpected.