Exploring course site resource usage using MAV

The following starts with the following question raised from a colleague about a Moodle course site they have designed.

The tabs mentioned in the above aren’t standard to Moodle. They are an institutional addition and a follow up tweet illustrates

The tabs have been added (I believe) as they capture important information that students should be able to find easily on every course site from this institution. The consistency == quality argument of which I’m not a fan.

The actual problem @chalkhands is having arises from a number of clashing perspectives/models for creating a Moodle course site. The following is not going to tackle that issue. Instead, this discussion has sparked an interest in exploring just how important those tabs and the information held there is to students. Or more correctly, how much have they been used in the courses I have been responsible for (and can I find out).

You can’t find out?

The “institutional tabs” are supported by some local additions to Moodle that provide the functionality. It appears that the usage of some of these tabs can’t be tracked by the standard Moodle logs. In particular, it appears that the Assessment, Study Schedule, and Teaching Team can’t be tracked on a standard site. At least can’t seem to find how to find this information via the Moodle logs report.

The benefits of hacking (this time)

In this particular case, it’s lucky that I have been guilty of “hacking” the site. Instead of the institutional specific method for these particular tabs they are pointing to more traditional Moodle plugins, which play with the Moodle logging facility. In turn, this allows me to use the Moodle Activity Viewer (MAV) script to find out how these things are being used.

For example, I can explore usage of the Assessment, Study Schedule and Teaching Team tabs by students in the S1, 2016 of the 300+ undergrad course I taught.

I can see how many times they clicked on the resources. (Click on the images to see larger versions)

EDC3100 2016, S1, Clicks

I can see the number of students who clicked on the resources

EDC3100 2016, S1, Students

Unsurprisingly, the Assessment tab was the most used. The following table summarises.

Resource Students Clicks Clicks/Student
Assessment 308 27,976 90.8
Study Schedule 274 2988 10.9
Teaching team 190 1061 5.6

What is surprising is just how much the Assessment tab is used. In theory, students can print/download a copy of this information. In spite of that, students are averaging around 91 clicks on that information during a semester.

I wonder why? Could they not figure out how to print the information? Was the printed version of insufficient quality?

Given that just under 50% of students never clicked on the teaching team information, I wonder what that says about the value of the tab? Or how it compares with other courses?

What parts of the assessment information was useful?

Using a standard Moodle plugin and combining that with MAV allows me to quickly get an indication of just which parts of the Assessment information was being used. The assessment information was implemented using the Moodle Book module, which produces a table of contents. The following images show the MAV modified table of contents for the Assessment book from the same offering.

Number of clicks on each section.

Screen Shot 2017-01-25 at 1.11.05 pm

Number of students using each section.

EDC3100 Assessment contents by student

The most clicked on information in this book are the three pages specifying what the students had to do for the three assignments. The next closest was the “learning journal” information which outlines one of the different practices in the course. The nature of which causes some consternation early in the course. But even with that, a good 10% of students never visit that information.

Also a bit interesting, less than half the enrolled students ever visit the information about how to query the marking of their assignments.

Formulating a Trello process

I’ve been using Trello for a couple of years to track manage personal projects/tasks. I’ve done this in an ad hoc way. I’ve even occasionally attempted using it (badly) for small groups. I now need to get a bit more professional about it. The following is an attempt to formulate an initial plan for testing out the use of Trello for a work project. If only so we can escape the desolate, unproductive waste that is Sharepoint and email attachments….more on this at the end.

What are others doing

This post gives an overview of a range of different ways that different groups have made use of Trello. It also gives a reasonable overview of the major components of Trello, including some mention of automation using Zapier.

Trello offers this collection of inspirations (including links to sample Trello boards and explanations of structure) for how Trello is used for purposes ranging from agile approaches, production workflows, six sigma, Kanban workflow, conference planning.

This offers explicit advice on how to progress from an empty board to something that can be used.

Kanban?

Kanban as an inspiration seems the simplest. This explanation of Kanban mentions that it’s “a lot more laid back” than other methods and outlines the following tasks

  • Define the workflow stages.
  • Set up how tasks move between stages.

and pillars of Kanban

  1. Each task has a card that includes all information about the task.
  2. There is a cap on how many tasks can be worked on.
  3. There is a continuous flow through the backlog in order of importance so that something is always being worked upon.
  4. There is a focus on analysing the flow to enable constant improvement

This for more background and detail on Kanban and a list of 8 Kanban board apps.

A local plan

This is an initial test directly involving myself and one other person. The aim is to gain more insight into using Trello in this context and in a more disciplined way. We’re working on the development of a proposal document. In the future I wonder whether this task might be worth of it’s own board, but for now it will. KISS.

So a plan might be

  • Set up a team for the group

    Not mention in the above.

  • Set up a board on Trello.
  • Configure the board
    Based on suggestions from here

    • Add a background image (but only if business class)
    • Add five lists
      • Three of the lists To do, Doing, and Done hold cards for specific tasks that someone has worked upon.
      • Current deliverable

        Holds information about the specific deliverable that the tasks in to do, doing and one are helping produce. This is held in a deliverable card, indicated by a red label.

      • About

        About provides background/guidance on how the board works. It also has a card that summarises the purpose of the board. In some cases, the key outcome. Perhaps one specified by a supervisor/client.

        It also contains a number of subsequence deliverable cards. The idea is that we know the next deliverable for the project and we have a place to put some thinking/detail. Eventually the next deliverable card will move to the Current deliverable list.

    • Invite the other person.
    • Start filling in the to-do list.
    • Pick a simple card/task to use as an example/demo of some practices.

    This plan is for a broader project with multiple stages. I’m thinking that each of the roles/people in the team might also have a board related to specific other tasks. Such a board wouldn’t necessarily have a current deliverable. Longer term we might also have templated boards that are set up to guide completion of recurring tasks.

    Early reflections

    It’s a day later and this set up is being used by the other member of the team. Early signs are that it’s being seen as a big step forward. Why?

    IMHO, it’s in part because the information technologies currently available are generic and have models focused on supporting more technical tasks. e.g. sharepoint. Sharepoint “helps” you save and perhaps share files. Maybe a little bit more than that, but because it is such a generic tool it doesn’t offer a lot of specific scaffolding.

    Trello is designed with a model based on boards, lists and cards. Not files, folders and generic technical objects. Boards, lists and cards and the functionality provided by trello align directly with the major tasks in a Kanban like process. The technology is actively supporting that process of organising projects in a way that is collaborative and transparent to the team. It offers functionality that helps perform these tasks.

    One of the problem with Sharepoint (and service based on a computer filesystem) is that information is organised into folders and files. The organisation of folders into files tends to be a fairly individually unique activity. I’m certain the absolutely logical and obvious way I organise folders and files is not something you would find logical or obvious. Hence finding a file would would require that you grapple with and understand the mental model I employed when organising the files and folders.

    While we may now still save the documents we use as part of this process, we can now link to them from Trello. Hence access to these documents is now (in addition to other means) available via the projects and tasks associated with the information. Hopefully a mental model that we all share and thus makes it easier to find information.

    Time will tell how well this works in reality and how/if the above plan scales.

Early thoughts on the new year and the new job

Some time off doing too little and eating too much allows me to stumble out of 2016 into a new year and a new job. Still at USQ, but I’m leaving the School of Teacher Education & Early Childhood and joining the Advancement of Learning and Teaching (ALT) – not sure if it’s a unit, department, office etc. Yes, for the second time I’m leaving the land of the faculty-based academic and venturing into the wilds of central learning and teaching.  I’ll leave reflection on the wisdom of that move to another post.

Today is my first day as part of a group with responsibility for “educational excellence and innovation” (EEI). Quoting from the position description I applied for, this group is

dedicated to improving students’ learning experiences by promoting academic development and learning, reward and recognition of good teaching, scholarly of learning and teaching and educational leadership development

ALT is the result of a recent organisational restructure, which means that both ALT and EEI are at some level figuring out how and what they should be doing. Given previous findings that 70% of centres like ALT are less than three years old (Challis, Holt & Palmer, 2009), it’s fairly important we develop some good answers. The following is the first step in thinking about how we might develop such answers.

Vision Deployment Matrix (VDM)

Late last year I did attend a leadership and strategic planning session organised for folk within the broader division to which ALT belongs. The session was run by Neil Carrington and was well worth it (not something that can always be said about such sessions). One of the tools/ways of thinking mentioned in the session was Kim’s Vision Deployment Matrix (VDM). While I have my reservations about aspects of the matrix, it does appear to provide a useful way to develop some shared understanding of why, how and what a group might be doing. The following is an attempt to make explicit some thinking about how it might be employed by EEI and as part of that dig a bit deeper into VDM.

The following table is one representation of the matrix. What isn’t captured by the table is the idea of increasing leverage and the idea that the “ability to influence the future increases as we move from the level of events to vision” (Kim). i.e. responding to event is not as likely to impact the future as re-thinking the mental models underpinning what is being done/seen. (Due to experience, I struggle a little with the idea of “vision” having an impact)

Level of perspective (Action mode) Desired future reality Current reality Gaps, open issues & questions Action steps Indicators of progress Timeline
Vision (Generative)
Mental models (Reflective)
Systemic structures (creative)
Patterns (Adaptive)
Events (Reactive)

The task set for EEI is difficult. I see the potential value of the VDM as a tool for mapping out what is being done at the moment at the moment, surfacing some of the limitations, and identifying tasks to do.  Kim provides a couple of “pocket guides” (shifting from a reactive to a generative orientation, and crossing the chasm from reality to vision) to how this might be done.

The idea is that you start with the vision you’d like to create and work down getting each to align. My first problem with VDM was the chicken and egg problem. VDM suggests that once you have your vision, you can then look for the beliefs and assumptions embedded within that vision and required to achieve it. My problem is that those formulating the vision have a range of mental models that influence the formulation of the vision.

Since I’m new to the role and the group has recently been restructured I see value in ignoring the vision and instead starting with the immediate reality and questions such as

  1. What are the events we should be involved with this year?
  2. What are the observable patterns arising from/contributing to those events?
  3. What are the systemic structures that we have (or don’t have) that support and produces these patterns/events?
  4. What are the mental models that underpin those structures?

Working through this with the group and then sharing this more broadly would appear to help those involved work towards a vision and what is needed to achieve it.

An example

So translating the abstract potential into reality, needs to be doable and I initially struggled with what I might use as an example. A sign of difficulties for the VDM? It won’t be easy, but in this case I think it’s my inexperience in the role.  Parts of the following are based on brief conversations last year and may have no connection to institutional reality at all.

Event: Contact a new casual teaching staff member and pass along pointers about where they can find out more about teaching at the institution.

Patterns: This might/should help the new staff member figure out how to teach, reduce any uncertainty they have. It might increase use of those services.  There’s liable to be a fairly short time frame between appointment and when they need to perform the task (sometimes it’s a negative time frame). This task will peak at certain times (just before start of teaching semester perhaps).

Systemic structures:  At the moment, the identity of new casual teaching staff members is done via an email from HR to another member of staff. This person then sends out a standard email.  The list of new staff members doesn’t always capture prior teaching roles. Starting to reveal some holes here.

Mental models:  There’s a growing recognition of the importance and value of casual teaching staff, but at the same time a recognition of the growing complexity of the role and how limited prior support might have been.  Casual staff members have limited time (Do their contracts fund training?). An email with links is a useful practice.

What next?

I think there might be some value here, especially if done by the group and beyond in an on-going, iterative and open method. Some next potential steps

  1. Identify a list of potential events that the group is involved with in coming months.
  2. Become more familiar with the current espoused visions and plans for the institution and and broader groups.
  3. Revisit the relevant literature for visions, mental models, systems, and events.
  4. Personally try out a few more examples using the VDM
  5. Share the approach with others and learn.
  6. Figure out how and if this should become something we focus time on.

References

Challis, D., Holt, D., & Palmer, S. (2009). Teaching and learning centres : towards maturation Teaching and learning centres : towards maturation. Higher Education Research & Development, 28(4), 37–41. http://doi.org/10.1080/07294360903067021

Kim, Daniel H. “Vision Deployment Matrix I.” Systems Thinker 6.1 (1995).