Strategies for curriculum mapping and data collection for assuring learning

The following is a summary of and some reaction to the final report of an OLT funded project titled “Hunters and gatherers: strategies for curriculum mapping and data collection for assuring learning”. This appears to be the project website.

My interest arises from the “Outcomes and Analytics Project” that’s on my list of tasks. That project is taking a look at Moodle’s new outcomes support (and perhaps other tools) that can be leveraged by a Bachelor of Education and try to figure out what might be required to gain some benefit from those tools (and whether it’s worth it).

Reactions

The recommended strategies (holistic, integrated, collaborative, maintainable) could form a good set of principles for some of what I’m thinking.

In terms of gathering data on student performance, assessment and rubrics appear to be the main method. Wonder if analytics and other approaches can supplement this?

It would appear that no-one is doing this stuff very well. The best curriculum mapping tool is a spreadsheet!!! And the data gathering tools are essentially assignment marking tools. Neither set of tools were well evaluated in terms of ease of use.

Lots of good principles and guidelines for implementation, but crappy tools.

Student centered much?

AoL is defined as determining program learning outcomes and standards and then gathering evidence to measure student performance. use for curriculum development, continuous improvement and accrediatation – but no mention is made of helping students develop eportfolios for employers. Especially in professional programs with national standards, this would seem an obvious overlap. Student centered much?

Interesting given that AoL is meant to be based on student centered learning.

Staff engagement

Is positioned as a difficulty.

Facilitating

AN earlier project (on which this was built) was titled “Facilitating staff and student engagement with graduate attribute development”. Apparently limited to helping staff design criteria to assess GAs and students self-evaluate against those. Seems to be only a small part of the facilitation process.

GAs versus professional standards

I have to admit to being deeply skeptical around the notation of institutional graduate attributes. Always struck me as a myth created by high-priced senior management to justify what was unique about the vision they were creating and hence with little connection to the reality of teaching at a university. In particular, that the standards/attributes set by the professional bodies associated with certain disciplines would always count for more.

Of course institutional GAs have been apparently required by the government since 1992. I wonder if this is like some of those other legal requirements that have been discovered to have never existed, ceased to exist or which were misinterpreted?

Executive summary

Assurance of Learning (AoL) “evaluates how well an institution accomplishes the educational aims at the core of its activities”…AoL provides “qualitative and quantitative indicators for the assessment of the quality of award courses” Thus be used for

  • institutional/management ends: strategic directions, priorities, quality assurance, enhancement processes.
  • individual curriculum development.
  • Valid evidence to external constitutents.

Focus of this project on

  1. mapping program learning outcomes.
  2. Collecting data on student performance against each learning objective.

Aside: interesting that they’ve already used both outcome and objective. Does this mean these are different concepts, or just different labels for the same concept?

Investigation was done via

  • Exploratory interviews with 25 of 39 associate deans L&T from Oz Unis.
  • 8 focus groups with 4 good practice institutions, 2 at each institution – one with a senior leader, the other with teaching staff.
  • Delphi method – but who with?
  • Interviews with experts.
  • online survey.

Recommended good practice strategies

  • holistic – whole of program, to ensure student’s progress and introduction of GAs prior to demonstration.
  • integrated – GAs must be embedded in the curriculum and links to assessment.
  • collaborative – developed with teaching in an inclusive – not top down – approach to engage staff.
  • maintainable – must not be reliant on particular individuals or extra-ordinary resources.

They then proceed to mention typical cultural change strategies.

Also did an independent review of existing tools. Interestingly the Blackboard 9.1 goals/standards service gets a mention.

Chapter 1 – Project overview

Identifies 7 key stages in assuring learning from an AACSB White Paper

  1. establishing graduate attributes and measurable learning outcomes for the program;
  2. mapping learning outcomes to suitable units of study in the program (where possible allowing for introduction, further development and then assurance of the outcomes);
  3. aligning relevant assessment tasks to assure learning outcomes;
  4. communicating learning outcomes to students;
  5. collecting data to show student performance for each learning objective;
  6. reporting student performance in the learning outcomes;
  7. reviewing reports to identify areas for program development (‘Closing the Loop’).

Explains the growing requirement for this, lots of acronyms and literature.

Mentions their prior project which development the ReView online assessment system to help staff develop criteria that assessed GAs within the set assignments. Students have self-evaluations.

Project aims to inform strategy to identify efficient and manageable assurance mechanisms (effective not important?).

Chapter 2 – methodology

Key guiding questions

  1. How is mapping of GAs being done?
  2. How is the collection of GA data being done?
  3. What are the main challenges in mapping and collecting?
  4. Are there identifiable good practice principels?
  5. What are the tools currently being used?

Chapter 3 – Literature review

here come the standards

Standards are defined as “the explicit levels of attainment required of and achieved by students and graduates, individually and collectively, in defined areas of knowledge and skills” (TEQSA, 2011, p. 3)……Academic standards are learning outcomes described in terms of core discipline knowledge and core discipline-specific skills, and expressed as the minimum learning outcomes that a graduate of any given discipline (or program) must have achieved (Ewan, 2010).

TEQSA is apparently requiring academic standards “be expressed as measurable or assessable learning outcomes”.

Determining standards and then collecting data against those is complex. Coates (2010) acknowledges the complexity and suggests a need for cultural change. And there is apparently an urgent need for “new, efficient and effective ways of judging and warranting” (Oliver, 2011, p. 3).

Extant literature

AoL finds its pedagogical basis in student-centered learning.

Curriculum mapping in AOL is the process of embedding learning outcomes related to GAS into units of study where these are introduced, developed and then assured.

AUQA required curriculum mapping as do most professional accrediting bodies – hence the 2009 observation from Barrie (et al 2009) that most Australian universities have some sort of strategic project underway.

The higher ed mapping literature is scant but suggests it’s useful for (all backed up with citations)

  • identifying gaps in a program
  • monitoring course diversity and overlap
  • providing opportunity for reflection and discourse
  • reducing confusion and overlap and increasing coherence

There are more, but there does seem to be some overlap.

Mentions the problem of the compliance culture, others include

  • difference between the intended and the enacted curriculum from the students’ perspective
  • how to contextualise GAs into a discipline.
  • mapping seen as threatening, as a course cutting exercise, criticisms of teaching material etc.
  • labour intensive exericse.

Staff engagement is seen as the key and current suggestions for improvement include

  • develop a conceptual framework for developing GAs, including 3 elements
    1. clear statement of purpose for curriculum mapping.
    2. a tool that allows an aggregate view of a course.
    3. a process for use of the tool
  • map GAs using extensive audits of each course.
  • a cyclical process including visual representations to enable a fluid/adaptable curriculum
  • availability of sufficient resources.
  • use of alignment templates (isn’t this a tool?)
  • professional development to integrate and contextualise GAs.
  • having specialists who can teach a particular attribute.
  • whole of program approach, focus on team co-operation and more time spent on design.
  • staff support where workloads increase.
  • linkages between GA development and professional development.

Embedding versus standardised testing

Mention of various standardised tests at the end of study. Talks about plusses and minuses.

Data collection for AoL

Focused on entering student performance outcomes against each learning objective. Need a “systematic method to collect data and explore the achievement levels of students in each of the selected attributes” to inform on-going development.

There are challenges in collecting and providing evidence – highlighting the need for efficiency and streamlining the process.

Assessment rubrics (formative and summative) are key. But there are challenges. Don’t want a “tick list”. Some skills are ill-defined, overlapping and difficult to measure. And the question of standardisation – homegenisation or pursuit of common goals. Multiple interpretations of criteria.

Rubrics can become to be used for comparison between institutions; assurance of content/process/outcomes across courses.

Continuous improvement/closing the loop

Apparently the “raison d’etre for assessing student learning” and also something that institutions are “most confused about how to go about closing the loop (Martell, 2007)”…..”integration of the assessment of learning outcomes into developmental approaches in the classroom has been somewhat intangible (Tayloer et al, 2009)”.

Curriculum mapping

important features for selecting a CM system to support AoL

  • support an inclusive and participatory process;
  • foster a program-wide approach to produce a mapped overview;
  • map by assessment task;
  • develop student awareness of attributes and their distribution within the program;

The standout tool was a spreadsheet!

Data collection

Important features for a data collection system included

  • implement a consistent criteria for attributes across programs;
  • extract outcome-specific data;
  • embed measurement in the curriculum
  • produce built-in reports;
  • conduct analysis for closing the loop;
  • implement multiple measures of AoL for program wide view.

ReView was seen as the stand out. But then, it arose from the last OLT project. But then that makes this comment interesting

ReView does not rate that well on ‘ease of use without the need for much supplementary professional development

0 thoughts on “Strategies for curriculum mapping and data collection for assuring learning

Leave a Reply

Your email address will not be published. Required fields are marked *

css.php