Implications of cognitive theory for instructional design

The following is a summary/reflection of Winn (1990), the abstract follows

This article examines some of the implications of recent developments in cognitive theory for instuctional design. It is argued that behavioral theory is inadequate to prescribe instructional strategies that teach for understanding. Examples of how instructional designers have adopted relevant aspects of cognitive theory are described. However, it is argued that such adoption is only a first step. The growing body of evidence for the indeterminism of human cognition requires even further changes in how instructional designers think and act. A number of bodies of scholarly research and opinion are cited in support of this claim. Three implications of cognitive theory for design are offered: instructional strategies need to be developed to counter the reductionism implicit in task analysis; design needs to be integrated into the implementation of instruction; designers should work from a thorough knowledge of theory not just from design procedures.

Summary

Suggests problems arise when decisions within instructional design are driven by cognitive theory, not behavioural. Mostly around the assumptions of rationality and predictability and the subsequent appropriateness of the traditional teleological design process used by instructional design. Suggests some approaches/implications that might help address these somewhat.

Reflection

The ideas expressed here offer support for the ideas I’ve been formulating about how to improve learning and teaching at Universities. Which obviously means I think it is an important bit of work by an intelligent person. It probably does have flaws. Will need to read and reflect more.

Still not sure that these principles have been applied broadly enough (though the conclusion seems to indicate yes). Winn has focused on changes to the practice of instructional designers in how they approach design without talking about how they may have to change how they work with the academics. Instructional design, for me, is as much about staff development as it is about design, at least within the current university context. Instructional design within universities can’t scale unless it builds capacity amongst the academic staff and the system to help in design.

Many of these limitations of instructional design are similar to those I’ve been trying to push around the institutional implementation of e-learning and more generally about approaches to improve learning and teaching e.g. graduate attributes.

Introduction

Starts with a definition of instructional design from Reigeluth (1993) – essentially it is a set of decision-marking procedures which, given the outcomes to be achieved and the conditions under which they are to achieve them, develops the most effective instructional strategies.

Generally done with analysis of outcomes/conditions, selection of strategies, iterative testing until some level of success achieved. The decisions are guided by instructional theory.

Gives examples of instructional design processes informed by cognitive theory.

Suggests evidence that cognitive theory is impacting thinking/actions of instructional designers, however, suggests that cognitive theory requires further changes in the way they think/act. Has problems with the analysis and selection/testing stages. Current approaches are not sufficient.

Suggests that instructional design should be driven by an understanding of theories of learning and instruction, rather than mastery of design techniques.

I’m assuming here he means that the type and nature of the steps within design process itself should be informed by these, not what he also recognises is that the decisions made within these steps are already driven by this. I’m a bit slow this morning.

Instructional design and behavioural theory

Supports/explains the notion that instructional design originated in behavioural theory, the dominant learning theory of the time when ID originated. Shows how instructional design processes evolved to fit the needs of behavioural theory. Examples include the reductionist nature of task analysis and pilot testing being sufficient to debug instruction that consisted of stimulus-response prescriptions. i.e. behavourists did not consider that there were “mental operations” within the learner that might mediate between stimulus and response. This resulted in design being separated from implementation.

If instruction can be developed to the point where acceptable student performance is likely to occur, then it matters little whether instruction is implemented immediately after the designer has attained this standard, or at some later time and in some other place.

Connects with literature that acknowledges the separation (Richey, 1986), thinks it creates problems (Nunan, 1983, Streibel, 1989, Winn, 1989) and others which think it desirable (Heinich, 1970 and 1984). Desirable because “it allows instruction to be brought up to a high standard, and then to be distributed far and wide so that all students can enjoy the advantages of top-rate teaching”.

Lastly, suggests the idea that instructional design can be “done by the numbers” also arises from the behavioural tradition. The idea is that any novice designer can be successful if they just follow the process – do it by the numbers.

In summary, 3 important areas where behaviourism still exerts power over instructional design:

  1. Reductionist premise that you can identify the parts, then you can teach the whole.
  2. Separate design from implementation.
  3. Assumption that following good procedures, applied correctly results in good instruction.

Sticking with the behavioural traditions, suggests that these 3 are not a problem, if you’re limiting yourself to low-level skills. The problems arise when you go to high levels of cognitive processing.

Cognitive theory

The aim here is to explain why the three assumptions are problematic as informed by cognitive theory – the obvious though here is what would constructivism or connectivism suggest.

The description of cognitive theory is

Changes in behavior are seen as indirect rather than direct outcomes of learning. Observable behavior is mediated and controlled by such mental activities as the acquisition, organization and application of knowledge about the world (Neisser, 1976; Rumelhart and Norman, 1981); by the development of skills that allow the encoding, storing and relrieval of information (E. Gagne, 1985; Shuell, 1986); by people’s motivation (Keller, 1983); their perception of what a task requires of them (Salomon, 1983a); and their perception of their likelihood of success (Salomon, 1983b; Schunk, 1984). Consequently, students are seen as active in the construction of knowledge and the development of skills, leading to the conclusion that learning is a generative process under the control of the learner (Wittrock, 1979, 1982).

To my somewhat untrained ear, this sounds like it has aspects of constructivism.

References Bonner (1988) as identifying a number of the differences between traditional designers and those informed by cognitive theory including:

  • task analysis;
    Traditionally aims to identify directly observed behaviours. Cognitive theory requires that “unobservable” tasks be analysed. i.e. the mental tasks to be mastered prior to observable performance being possible. examples including identifying declarative and procedural knowledge or schemata required to perform. Also recognition that novice to expert involves many steps that need to mastered.
  • objectives;
    Statements of what the student is to accomplish under what conditions and to what criterion is a behaviourist approach. Cognitive objectives are schematic representations of the knowledge to be acquired and procedures to apply.
  • learner characteristics;
    Focus on the schemata/mental models students bring to instruction, not their behaviours. May not be a clear line between what they need to know and what they know – learner as dirty slate.

    This acknowledges the importance of current knowledge of the world, represented in mental models, for the acquisition of new knowledge and skills. Research (De Kleer and Brown, 1981; Larkin, 1985; and authors contributing to Gentner and Stevens, 1983) has shown that learning occurs as students’ mental models acquire refinement and accuracy.

  • instructional strategies;
    Behaviourism selected instructional strategies based on the type of learning to take place, the type of learning outcome.

    But because the cognitive conception of learning places so much importance on the student’s development of adequate knowledge structures, cognitive procedures and mental models, the designer should create opportunities for what Bonner calls a “cognitive apprenticeship” centered around problem-solving rather than prescribe strategies apriori.

    Some general principle may determine aspects of the strategy, however, it evolves like a conversation.

Rieber (1987) point out, is that instruction that is designed from cognitive principles can lead to understanding rather than just to memorization and skill performance.

This speaks to me because too much of what passes for improving learning and teaching strikes me as most likely to create memorisation and skill performance, not long term change.

The need for further change

While instructional designers are adopting principles from cognitive theory, the idea is that recent thinking in cognitive psychology and related fields brings into question some the assumptions of cognitive theory as currently accepted. Moving onto the reasons:

  • metacognition;
    Metacognition research shows that students have or can be trained to acquire the ability to reflect on their performance and adopt/adapt different learning strategies. This means that the intent of a instructional design can be circumvented if the student finds the chosen strategy problematic. If the instruction is not adaptable or the student doesn’t choose a good strategy, then the instructional design is compromised.

    I wonder what implications this has for constructive alignment and its idea of forcing the student to do the right thing?

  • dynamic nature of learning;
    Very interesting. As the learner learns, they develop knowledge and skill that is different from the start. The analysis performed at the start to select the instructional strategy no longer holds. If the analysis was done now, a different strategy would be required.

    Nunan (1983) develops this line of reasoning in his argument against the value of instructional design, drawing on arguments against the separation of thought from action put forward by Oakshotte (1962) and Polanyi (1958).

  • emergent properties of cognition;
    This is the argument against reductionism. Emergence is defined as the idea where the properties of the whole, cannot be explained solely be examining the individual parts of the whole. The nature of the whole affects the way elements within them behave. The suggestion is that a number of people have claimed that the actions of the human mind exhibit emergent properties (Churchland, 1988; Gardner, 1985).

    The reductionism that underpins task and learner analysis “acts counter to, or at best ignores, a significant aspect of human cognition, which is the creation of something entirely new and unexpected from the “raw material” that has to be learned.

  • plausible reasoning;
    A designer informed by cognitive theory assumes that the thought processes of a student will be as logical as the instruction itself. In order to learn from a machine, the student has to think like a machine (Streibel, 1986). There is lots of evidence to suggest people are not logical. “Plausible reasoning” is Collins (1978) idea that people proceed on hunches and incomplete information. Hunt (1982) suggests plausible reasoning has allowed the human species to survive.

    If we waited for complete sets of data before making decisions, we would never make any and would not have evolved into intelligent beings.

  • situated cognition; and
    Somewhat related to previous. Streibel (1989) argues that “cognitive science can never form the basis for instructional design because it proceeds from the assumption that human reasoning is planful and logical when in fact it is not”. References Brown, Collins and Duguid (1989); Lave (1988) and Suchman (1987) – i.e. situated cognition folk – to argue that the way we solve problems is dependent on the situation in which the problem occurs. We do not use formal or mathematical reasoning.
  • unpredictability of human behaviour.
    The 5 previous points suggest that human behaviour is indeterminate. Csiko (1989) gives 5 types of evidence to argue that the unpredictability and indeterminism of human behaviour is central to the debate concerning epistemology of educational research. Winn (1990) suggests it applies equally well to instructional design:
    1. Individual learner differences interact in complex ways with treatments which make prediction of performance difficult.
    2. Chaos theory suggests the smallest changes in initial states lead to wild and totally unpredictable fluctuations in a systems behaviour. Something that is more pronounced in complex cognitive systems.
    3. Much learning is “evolutionary” in that it arises from chance responses to novel stimuli
    4. Humans have free will which can be exercised and subsequently invalidate any predictions about behaviours made deterministically from data.
    5. Quantum mechanics shows that observing a phenomenon, changes that phenomenon so that the results of observations are probabilities, not certainties.

Though eclectic, this body of argument leads one to question seriously both the assumption of the validity of instructional prescriptions and the assumption that what works for some students will work for others.

While prediction may not be part of instructional design, it is of the theories it depends upon and Reigelluth (183) points out that any theory of instruction, while not deterministic, does rely on the probability that prescriptions made form it for its validity. Without such validity, you may as well rely on trial and error.

Conclusions

Cognitive theory has been incorporated into instructional design, but behaviourism influence remains and that causes problems.

Cognitive task analysis to develop objectives is just as reductionist as behaviourist approaches.. The whole approach designers take needs to be re-examined. Three directions might include:

  1. Analysis and synthesis;
    Addressing reductionist analysis – instructional strategies need to ensure knowledge/skill components are put back together in meaningful ways….e.g. Reigeluth and Stein’s (1983) use of “summarisers” and “synthesizes” in elaboration theory.

    Balance analysis as a design procedure with synthesis as an instructional strategy. Such prescriptions should exist in instructional theories.

  2. Design and implementation;

    For instruction to be successful, it must therefore constantly monitor and adapt to unpredicted changes in student behavior and thinking as instruction proceeds……To succeed, then, instructional decisions need to be made while instruction is under way and need to be based on complete theories that allow the generation of prescriptions rather than on predetermined sets of prescriptions chosen ahead of time by a designer. (p64)

    Requires the teacher to monitor and modify strategies as they’ve been prescribed. Requires teachers to be well schooled in instructional design and a solid knowledge of theories of learning and instructions – so that they can respond in some sort of informed way. e.g. need methods that will allow them to invent prescriptive principles when the need arises.

    Second recommendation is that the designer needs to monitor the actual use of the instructional system during implementation, or for the designer to make provision for the use to change instruction strategies.

  3. Theory and procedure.
    Decisions about instructional strategies need to be based on more than just the application of design procedures. Rather than techniques being taught, the principles should be.

    This problem is made worse by researchers who are content to identify strategies that work on single occasions rather than determine the necessary conditions for their success (Clark, 1983).

Reservations about instructional design

The following is at first a rambling diatribe outlining some of my reservations with instructional design as it is practiced. Then it is a summary/reflection on Winn (1990) – “Some implications of cognitive theory for instructional design”. The abstract for Winn (199)

This article examines some of the implications of recent developments in cognitive theory for instmctional design. It is argued that behavioral theory is inadequate to prescribe instructional strategies that teach for understanding. Examples of how instructional designers have adopted relevant aspects of cognitive theory are described. However, it is argued that such adoption is only a first step. The growing body of evidence for the indeterminism of human cognition requires even further changes in how instructional designers think and act. A number of bodies of scholarly research and opinion are cited in support of this claim. Three implications of cognitive theory for design are offered: instructional strategies need to be developed to counter the reductienism implicit in task analysis; design needs to be integrated into the implementation of instruction; designers should work from a thorough knowledge of theory not just from design proceduts.

Actually, I’m running out of time, this post will be just the diatribe. The summary/reflection on Winn (1990) will have to wait till later.

Some context

The following line of thought is part of an on-going attempt to identify potential problems in the practice of instructional design because I work within a Curriculum Design & Development Unit at a University. I am trying to identify and understand these problems as an attempt to move toward something that might be more effective (but would likely have its own problems). The current attempt at moving toward a solution will hopefully arise out of some ideas around curriculum mapping.

The diatribe

Back in the mid-1990s I was being put in charge of my first courses. The institution I worked at was, at that stage, a true 2nd generation distance education provider bolted onto an on-campus university (the university was a few years old, having evolved from an institute of advance education). Second generation distance education was “enterprise” print distance education. There was a whole infrastructure, set of processes and resources targeted at the production of print-based study guides and resource materials that were sent to students as their prime means of education. A part of the resources were instructional designers.

From the start, my experiences with the instructional designers and the system they existed within was not good. The system couldn’t see it was increasingly less relevant through the rise of information technology and the instructional designers seemed more interested in their knowledge about what was the right thing to do, rather than recognising the realities of my context and abilities. Rather than engaging with me and my context and applying their knowledge to show how I could solve my problems, they kept pushing their own ideal situations.

Over 15 years on, and not a lot has changed. I still see the same problem in folk trying to improve learning and teaching at that institution. Rather than engage in an on-going process of improvement and reflection, it’s all about big bang changes and their problems. Worse, then as now, only the smallest population of the academics are being effectively engaged by the instructional designers. i.e. the academics that are keen, the ones that are willing to engage with the ideas of the designers (and others). This is perhaps my biggest concern/proposition, that the majority of academics are not engaging with this work and that a significant proportion of them (but not all) are not improving their teaching. But there are others:

  • Instructional designers are increasingly the tools of management, not folk helping academics.
    In an increasingly managerialist sector, the “correct” directions/methods for learning and teaching are increasingly being set by government, government funded bodies (e.g. ALTC and AUQA) and subsequently the management and professionals (e.g. instructional designers, staff developers, quality assurance etc.) that are institutionally responsible for being seen to respond effectively to the outside demands.

    There are two problems with this:

    1. the technologists alliance; and
      The professionals within universities, because of their interactions with the external bodies and because their success depends on engaging with and responding to the demands of the external body, start to think more like the external body. For example, many of the folk on the ALTC boards/etc are from university L&T centres. Their agenda internally becomes more about achieving ALTC outcomes, rather than outcomes for the academics. Geoghegan (1994) identified the technologists alliance around technology, it is increasingly in existence for L&T.
    2. do what management says.
      Similarly, because senior management within universities are being measured on how well they respond to the external demands. They to are suffering the same problem. In addition, because they are generally on short-term contracts there’s increased demand to respond via short-term approaches that show short-term gain but are questionable in the long-term. Instructional designers etc are then directed to carry out these short-term approaches, even if they will hurt in the long term are or seen as nonsensical by academics.

    The end result is that academics perceive instructional designers as people doing change to them, not doing change with them or for them. Not a good foundation on which to encourage change and improvement in something as personal as teaching.

  • Traditional instructional design is not scalable.
    My current institution has about 4 instructional designers. The first term of this year sees the institution offering 400+ courses. That means somewhere around 800 courses a year. That’s 200 courses a year per instructional designer. If you’re looking at each course being “helped” once every two years, that means each course gets the instructional designer for 2 days every 2 years, at best.

    In this environment, traditional ADDIE type big-bang approaches can’t scale.

  • Instructional design seems informed by a great knowledge of ideal learning and teaching, but none of how to effectively bridge the gap between academics and that ideal.

References

Geoghegan, W. (1994). Whatever happened to instructional technology? 22nd Annual Conferences of the International Business Schools Computing Association, Baltimore, MD, IBM.

Winn, W. (1990). “Some implications of cognitive theory for instructional design.” Instructional Science 19(1): 53-69.

Fixes to BIM

Prior to BIM being installed and used at my current institution it had to be checked by folk at NetSpot. The following details what I’ve had to do in response to that.

All of the changes are fairly minor, some won’t be changed as I don’t see the point and don’t have the time. Shall be interesting to see how that goes down.

Especially, given that the evaluation response to the following two questions

  • Can the plugin be deployed in its current state without first applying fixes?
  • Considering any Security/Maintenance issues identified above would the decision to install and support this plugin contain acceptable risks to NetSpot or the client?

Was YES!

Left over debug stuff

  • references to localhost
  • reference to brisbane

Getting a list of students

The advice is

Getting a list of students should be extended to a selectable select of roles instead of hard code role

Actually moved this to using get_users_by_capability.

Paging

Essentially, where lots of database records are going to be displayed the request is to “page” the results to minimise load.

In particular, the suggestion is to introduce paging for

  • Find student – There is a hard-coded limit within the code that restricts find student to getting 200 student records.
  • Your student – Limited to what staff are allocated to. Could, theoretical be the entire course, which could be large.

Will leave this as is. This is only getting a few records of data per student. Shouldn’t be a great load and will take a fair bit to modify the code.

Logging

Apparently my use of “add_to_log” is somewhat off. Not surprisingly as it was a learning process and a bit of cut and paste went on.

Two problems are currently apparent:

  • a missing $ sign for cm->id
  • possible wrong parameters being sent to the URL related to this.

Custom HTML

A couple of the forms I use have some custom HTML in them. i.e. rather than use the Moodle forms constructs it outputs some HTML.

I can see the point in having this, but it’s not a strong point. It’s using HTML tables for layout which does break one of the aims of Moodleform. But I also would like to save the time it takes me to figure out how to achieve the look I need for these forms with just the Moodleform stuff for other tasks that are a little important.

However, I can see some folk being anal about this. May have to come back to it.

Backup and restore

As per an earlier post this is not 100% complete. Will have to complete it, eventually. At the moment back up of questions does work. This is the main functionality required to roll over courses, the immediate need.

The zip file

The zip file I uploaded was a bit confused in terms of having two copies of the bim code, one in a sub-directory.

The current state of curriculum mapping in/with Moodle

As part of looking into a project around curriculum mapping I need to take a look at the current state of play around curriculum mapping in the Moodle community. This is a summary of what I can find at the moment.

Moodle is used in both schools and universities. My impression is that schools have a longer history and broader use of curriculum mapping, however, mapping in universities is hotting up.

In summary, there are a few examples of Moodle “tools” that claim to do some aspect of curriculum mapping. However, none of them seems to provide the services I’m suggesting. There does appear to be some call for this type of service.

UPDATE: More recent exploration is based around Moodle’s existing outcomes support which might offer a foundation on which to build curriculum mapping.

Discussion threads

The Moodle community generally has discussions within Moodle-hosted discussion forums on the Moodle.org site. The threads I could find looking at curriculum mapping include (you will have to create an account on the Moodle.org site to view these):

  • Curriculum mapping with Moodle
    Starts with a general question back in September 2007 that never really went anywhere. Some 2009 posts that include a pointer to a more recent thread, a pointer to the curriculum module for Moodle and a pointer to a US-based school’s curriculum mapping done with drupal.
  • Curriculum mapping
    A more recent and complete thread. Includes a pointer to the module, and a pointer to a Moodle doc on curriculum mapping. There’s also some feedback from a user of existing mapping tools and summarises one of the aims of the project being thought about here

    Atlas curriculum mapping is time-consuming and frustrating for many, in part because it is divorced from the courses it maps. Since Moodle houses a school’s actual courses, it would be most excellent if the mapping could be a layer on top of what’s already there.

    . There’s also a pointer to the announcement of a Moodle block covered in the next point.

  • Announcement of a Moodle block – to facilitate the design of a competency based curricula.
  • Curriculum patch for Moodle.
    This is a “patch” to the core of Moodle. It only works with a small subset of Moodle versions. My take on it, is that this is not really mapping curriculum but providing a way to prescribe a fixed hierarchy for a course (tree, parallel or serial) or groups of courses and ensure students move within it.

    Doesn’t appear to be what I’m interested in. Also demonstrates some of the potential problems that arise when a “patch” needs to change lots of the Moodle core. This needs to be kept to a minimum.

  • Moodle docs on curriculum mapping
    Essentially a page dedicated towards spawning the development of a Moodle component for curriculum mapping.

Other moodle.org resources

  • New moodle block ‘curriculum design tool’
    Surpisingly not linked into the above discussions, found via a google search. It does seem to be the moodle block that was announced above though.

    There is project page (or this page which seems to be running the block) which I didn’t find last year when I looked a bit. The block is meant to provide functions to

    • Design curriculum maps relating to competencies (educational goals), professional abilities (learning objectives), learning activities (resources) and assessments.
    • Submission of learning activities by educators with associated searchable metadata
    • Review submitted materials
    • Search and browse published curriculum maps and get related learning object repositories.
    • Rate learning objects
    • Checking out materials from the learning object repositories via a shopping cart model (free shopping)
    • Gap analysis for selected learning object items
    • Packaging of selected curricula items with a dynamically created curriculum description including links to learning activities and assessments.
    • Tracking of downloads and follow up surveying of users.

    This seems to be somewhat broader than what I had in mind. As implemented it seems to implement a repository for learning objects that can be peer-evaluated against some criteria. Not curriculum mapping amongst existing courses.

Web resources

  • A discussion thread on a school educators network Ning forum.
    Describes one person’s aim to move a school from another mapping platform into using Moodle. Has various reactions and ideas in response. A number of the responses are talking about problems of limited use of specific purpose curriculum mapping software. There is also a description of how one school “kludged” around the direct absence of mapping in Moodle, which seems to be what the original author was going to do. i.e. no direct support for mapping in Moodle, just use Moodle’s existing features to approximate it.

    There is a post from a Moodle user/admin arguing why the stand alone mapping software is good because it does provide these features. In particular, the post mentions the inability to identify the overlaps or holes.

  • Enterprise Learning Intelligence Suite (ELIS)
    An “integrated stack of technologies designed to provide end-to-end management of an online learning program”. Includes curriculum mapping and a range of other services. Apparently being released under the GPL.

    Based on this presentation it appears that “curriculum mapping” is really just grouping courses together as a curriculum – similar to the “patch” mentioned above. Not what I had in mind. Another presentation seeks to reinforce this.

How to make curriculum mapping useful to university academics

The following is an attempt to make concrete various ideas that have been floating around about a project to take a very different approach to curriculum mapping. There’s a small glimmer that these ideas may form the basis for an ALTC grant application

The following includes:

The idea

The outline of the idea is:

  • Implement the following changes by working closely with the academics and changing the project, its processes, aims etc in response to the learning that occurs.
  • Modify Moodle (this is the LMS my current institution uses) so that it is possible that all activities and resources within a course site can be linked to course learning outcomes, institutional graduate attributes and other criteria/categories (outcomes etc).
  • For courses within a program that has been through a recent accreditation, use those resources to add the outcomes etc. to existing courses, resources and activities.
  • Have this process involve collaboration between the academics and people who can help with and change the system, explain the meaning of learning “terminology” and generally make it a helpful and positive experience.
  • Work with academics to ensure that current Moodle courses have outcomes etc populated appropriately.
  • Modify Moodle so that when an activity or resource changes, there are visible reminders that the outcomes etc associated with that activity/resource should also be changed. Make it simple for the academic to change these.
  • Ensure that when a course site is copied from one term to another, the outcomes etc are part of that copy process.
    The intent is that once the outcomes etc are in place, the academics only need to modify those bits that have changed and they are supported and encouraged to do so during what they normally do.
  • Draw on this information to develop different curriculum maps for different purposes. The maps can draw on the fact that there are links from the outcomes etc to actual data in the LMS (e.g. student posts on the discussion forum, assignments, quizzes etc.)
  • Work within a program with the teaching academics to make the use of the curriculum maps a useful and important part of what they do in the process of their normal course delivery. i.e. make it part of the way we do things around here. (This does not mean writing policies.)

The main part of the project would be having a group of people with the right mix of technical and educational knowledge actively working with the academics to identify how this information could be made useful for the academics. Some work would also be done for other stakeholder (e.g. accreditation bodies).

The actual uses of the information would arise from this collaboration, but some possible examples might include:

  • Access to examples of implementing an attribute or outcome within the program or the institution.
    Problem: The assumption is that given an outcome staff will pick a learning design that will help the students achieve that outcome. Most academic staff don’t have the abstract knowledge to do this. Seeing concrete examples might help.

    1 solution: Provide an interface that matches the current outcome of interest for the academic, with a other similar outcomes in other courses. Allow the academic to drill down and see the actual activities and resources mapped to those.

  • Representation of the holes and duplications in a course.
    Problem: Generally, the people teaching courses in a program don’t know what’s going on in another course or in the program as a whole.

    1 solution: Provide a program level summary that identifies the holes and duplications in attributes, outcomes, activities and resources.

What is curriculum mapping

Just to be sure that we’re talking about the same thing, the following offers some definitions of curriculum mapping

Curriculum mapping is a representation of the different components of the curriculum in order that the whole picture and the relationships between the components of the curriculum can be easily understood (Harden 2001). Curriculum mapping displays the essential features of the curriculum in a clear and succinct manner (Prideaux 2003) and provides a context for planning and discussing the curriculum (Holycross 2006).

If you want more information, the above quote is taken from this report.

Problems with current practice

The following list is based on my observations, the literature and anecdotal reports from others:

  • Staff don’t engage meaning it is unlikely to change practice.
    The curriculum mapping process is seen as an add-on, it solves someone else’s problem or something that someone else does for them. With the lack of engagement, it becomes questionable whether these considerations become a key part of day to day thoughts and subsequently makes any long-term change in understanding and practice.
  • Divorced from practice leading to unreliability of what is reported.
    Completing a curriculum map is done either at the beginning or the end of a course. It’s not done during a course. This separation leads to the reliability of what is included in a curriculum map being highly questionable. One reason for this is that memory is not perfect, what is recalled and reported may not be what went on. Then there’s the whole question to task corruption and compliance.
  • Complexity leads to unreliability.
    The task of mapping out an entire course is complex. Most curriculum mapping requires that it be done as one task after the course is complete. The complexity of the task leads to mistakes, either through academics rushing it or the inevitable problem of chinese whispers when the academics are communicating information to a third party.
  • Tools that are not integrated into practice leading to duplication and unreliability.
    Curriculum mapping is usually done with pen and paper, an excel spreadsheet or perhaps a commercial stand alone tool. Yet another tool for academics to use. Each of these tools have their limitations. But perhaps the most important is that the new curriculum mapping tool is not the LMS or any of the other systems the academic users for learning and teaching. It’s something else to learn and doesn’t even connect with other tools. These tools don’t actively reduce the workload for the academic or provide additional functionality. It’s all cost and no benefit.

Current representation

In order to implement curriculum mapping across a program or institution, you have to design how you are going to do this. How you understand or represent the problem significantly impacts upon how you design your solution. Representation has a profound impact on design work (Hevner et al., 2004), particularly on the way in which tasks and problems are conceived (Boland, 2002). The formulation of the initial state into an effective representation is crucial to finding an effective design solution (Weber, 2003).

I suggest that the process widely used to implement curriculum mapping is similar to most projects within organisations and universities. It is a teleological process. Truex et al (2000) identify a shared assumption about teleological design processes involving a three-stage rational sequence: “(1) determine goals, (2) determine steps and events that lead to these, (3) follow the steps and generate the events”.

In curriculum mapping this means some group, typically not coal-face teaching academics, identify the need for curriculum mapping. Common groups include accreditation agencies, quality assurance groups and other business and government bodies. In response to this need another group, generally some sort of central learning and teaching group, decides on a process to perform the curriculum map and then engages with and encourages academics to complete the curriculum map.

The focus – the central/core aim – of the institution then becomes of completing this project. The focus has moved away from improving learning and teaching but to actually getting the forms filled in. The questions academics are asked become, “have you completed the form yet?”. The academics start complying and not engaging.

If done well, the project will achieve its aim of getting completed curriculum maps. However, the quality of those maps will be questionable and there’s a good chance the majority of academics are annoyed at having to waste more time and teaching and learning when they all know that they get recognition and promotion for research.

One alternate representation – changing thinking

Oliver et al (2007) describe the practicalities of curriculum mapping as (emphasis added)

far from simple and require a shift in educational thinking

The “shift in thinking” is the foundation of the representation of curriculum mapping that informs the following idea.

The core aim of this project is to change the educational thinking of academics and consequently improve learning and teaching.

Representing the problem this way means that different ideas and approaches to complete the problem. For example:

  • You only change what people think by changing what they experience day to day.
  • You only change what people do day to day if it provides them with some demonstrable benefit. That the choice architecture around what they do is such that they voluntarily make a good choice.
  • You only know what will give them demonstrable benefit by really understanding their experience and if they trust you.
  • You only know what they experience and have their trust if you are interacting with them throughout the process and providing them real assistance.

This is why the above basic outline of an idea cannot be implemented as a traditional project, with set goals and outcomes. It has to be implemented as a learning project. The following from Cavallo (2004) captures this well

As we see it, real change is inherently a kind of learning. For people to change the way they think about and practice education, rather than merely being told what to do differently, we believe that practitioners must have experiences that enable appropriation of new modes of teaching and learning that enable them to reconsider and restructure their thinking and practice. The limitations inherent in existing systems based upon information transfer models are as impoverished in effecting systemic development as they are in child development.

References

Boland, R. J. (2002). Design in the punctuation of management action. Frontiers of Management Workshop, Weatherhead School of Management.

Cavallo, D. (2000). “Emergent design and learning environments: Building on indigenous knowledge.” IBM Systems Journal 39(3&4): 768-781.

Hevner, A., S. March, et al. (2004). “Design science in information systems research.” MIS Quarterly 28(1): 75-105.

Oliver, B., S. Jones, et al. (2007). Mapping curricula: ensuring work-ready graduates by mapping course learning outcomes and higher order thinking skills. Evaluations and Assessment Conference. Brisbane.

Weber, R. (2003). “Still desperately seeking the IT artifact.” MIS Quarterly 27(2): iii-xi.

BIM 1.0alpha is released

It is with a significant sense of relief that I announce the release of version 1.0 alpha of BIM. You can see the code and what passes for documentation on this page. The main BIM page gives more background on the rationale and design of BIM.

BIM is meant to be used in action by 5 courses at CQUniversity in the term starting next week. Before that happens, it is being given the once over by the folk at Netspot. Shall be interesting to see what they find, BIM is my first real taste of Moodle development and it probably shows.

I’m hoping that BIM 1.0alpha will become BIM 1.0 before the week is out.

What next

The rest of this week I’ll be producing various resources to help the students and staff of the 5 courses using BIM. These will be freely available.

At some stage in the next week, I’m hoping to start the process of contributing the BIM code to the broader Moodle community.

After than, I hope to get into some reflection on the process and get a paper ready for submission to MoodleMoot AU 2010.

At some stage, the various ideas that have not made it into v1.0 will have to thought through.

More immediately, it’s time for some rest and relaxation.

BIM – Final Tidy up

Today is the day that BIM get’s handed off to the folk of my institution’s central IT division and from there onto Netspot who will be ensuring that it is “safe” for IT to install it on the institution’s live Moodle instantiation. In preparation for the hand over, I’ve done one last test and identified some issues to fix. This post documents that work.

Process student feed on view – DONE

Historically, BAM, from which BIM grew, only mirrored students’ feeds once every hour. Essentially to be a good citizen on the server. But the delay causes problems. Students don’t understand the need to wait.

So the plan it to retain the hourly update, but to add a process student feed for a single student as the first thing that happens when that student attempts to view the details of their BIM activity. This removes the delay. In addition, because of caching and the minimal work involved does not place undue load on the server or the blog services.

Implementation includes:

  • How is processing done now?
    2 main functions
    1. bim_process_feed – checks the RSS feed and processes new posts.
    2. bim_process_unallocated – checks entries in bim_marking to see if they match any questions (some might have been added).
  • Can it be done for a single student/feed (register?)
    Yes.

    Both functions take the params $bim, $feed, $questions ) – $feed is for single student.

  • Implement it.
    Put a call to the two functions in the show_student_details method. But only call them if there are actually questions (allocation doesn’t work if there are no questions to allocate to).
  • Should we show a message?

Wrong count on student details – DONE

If there are no questions currently allocated, the student details table is still showing 1 as the count of required answers.

Strange problem with PHP count, added check for empty and working now.

Tell markers configuration – DONE

Currently coordinators and students get messages if they can’t register etc. The poor marking staff don’t. Add one.

Suspended post, no description – DONE

Student can see a message about the status of a post in BIM. If the post is suspended, they can’t see anything.

Left message the same as marked.

Changes to released post, no gradebook – DONE

A released post that has the mark changed, is not updating in the gradebook. Have any change to a released post (mark or comments) set the post back to Marked.

BIM – Ideas for changes arising from user testing

Today BIM was tested by an intelligent, but when it comes to BIM a clueless, user. I sat back and watched the assumptions and mistakes that were made because the system design was not well thought out enough or was missing some functionality. The following is a list of the ideas that arose from that session.

Suspend student posts – DONE

Need the ability for the coordinator and perhaps marker to suspend a particular post so that it cannot be released.

Planned implementation:

  • Add a suspend state into bim_marking table.
  • Show post details page needs to indicate suspension.
  • Add entry for suspend states in Marking overview
  • Add a suspend checkbox on the marking page.
    • Add the checkbox
    • Get it being set if suspended
    • Add notify box to reinforce that it is suspended.
    • Add the ability to change it
    • Ensure a suspended post is never set to mark by simple changes in mark
    • Make sure suspended is excluded from release
  • ?? add a suspend checkbox on the allocate page? NO
  • Allocate page does need to show that a post is suspended

Show student details

  • Trying to get last post to be colourful.
  • Also – #entries is still 0, not being updated.
    Should be set by process feed.

Help and support

Identify the places and ideas for additional support and help resources that can be added into various screens to get folk going – ultimate aim to have it self sustaining.

Some functions from weblib.php might help here includeing :link_to_popup_window, element_to_popup_window, print_heading_with_help, print_box( $message, $classes ), print_container( $message, $clearfix);, helpbutton, notice, notify (bold message in optional colour – $message, $style =’green’), page_doc_link( ), doc_link(),.

  • Marker – show student details
  • Marker – show post details
  • Marker – mark
  • Marker – allocate
  • Coord
    • mod_form
    • Configure BIM
    • Manage questions
    • Allocate markers
    • Manage marking
    • Find student
    • Your students?

Modify the help file that shows links to all help files for BIM.

See student report

Add a feature that allows markers to see exactly what one of their students sees when they visit BIM. Enables them to at least start from the same place in discussions.

Mark post

  • Next and previous student and post
    This was in BAM. When marking an individual student post, have links to be able to go to the next or previous post for this student or for this question.

    This is a fairly major task, I think. In BAM when you were marking a student post there were the following :

    For this student: previous post | next post
    For this question: previous student | next student

    Where the next/prev text were links that would take you to the next/prev post for the student or to the next/prev student’s answer to the current question.

    Implementation for BIM should probably be:

    • Questions
      • bim_get_question_hash returns list of questions, answerd by the student— $marking_details
      • find current question in that hash
      • get the id for next and prev (loop around at the end OR don’t show?)
      • construct the URL based on those ids and the current student/bim
    • Students
      • Get a list of students who have answered the current question.
      • Find the current student in the list
      • Get ids for the next/prev
      • construct the URL.
      • Re-arrange some of the feedback when submitting so that it shows up more obviously and in places folk more likely to look.
      • Change some of the naming of “Answer” and “Submitted” etc into more action oriented easier to understand words.
      • Screen casts for help on marking.

      Student instructions for registering

      • Reduce the size of the hard-coded instructions for “how to register” to nothing.
      • NO Provide some initial default values for the “intro/about”
      • Include pointers to the screencast and other help documents for the student.
      • Change the label for the submit button to “register blog”.
      • Make it clearer that once submitted, this is the detail you will see.
      • Add some advice about what to do if “question not allocated”
      • Give some indication of when the last “mirror” was and when the next one might be.
      • Add some additional links to explain specific terms.

      Some form of email group – DONE

      Mainly for unregistered students, some way of communicating with these students en mass to tell them to register. Some flexibility in what to send and where to send it.

      A fall back might be to simply show a list of email addresses for those students that can be cut and paste into an email client and/or a mailto link.

      Links for student and post details – DONE

      Re-work these navigation links to that they are situated more appropriately and more obvious in what they should do.

      Remove “change registration” for marker – DONE

      Only coordinator should be able to do this.

      Look at the ordering of the questions in columns

      Currently post details order of questions is somewhat arbitrary. Get it ordered on the question being entered?

      Roll overs? – NO

      Any chance of putting in place roll overs that provide some additional pointers on what a particular link will do.

      Show questions to users and markers?

      Let the students and the markers see the list of questions.

Implications arising from the absence of the "sameness of meaning"

A few days ago Stephen Downes – a little unusually – made a sequence of comments/tweets on Twitter around the “sameness of meaning” and its impossibility. Since then I’ve had a number of experiences and discussions that suggest some of the problem associated with learning and teaching policy, process and structure within universities arises because too many people assume that there is sameness of meaning.

Communication and the commonality of meaning

Let’s start with this tweet

Communication isn’t about commonality of meaning. That’s impossible. It’s about being able to at least approximately predict the response.

Further explanation flows from this tweet

What happens is, the word we transmit (and maybe gestures, etc) forms only a small part of the other person’s understanding of what you said

and this one

Your word is only a stimulus; Most of the person’s understanding is based on his prior knowledge, and that is what produces the response

And I particularly like this one as a guideline for how to move forward

“Get me a gazelle” would work just as well if your listener understood that he should deliver a Heineken; meaning doesn’t matter, results do

Implications for design

This talks to me because much of what I do could be broadly called “design”. Mostly it’s around the design of information systems. This means much more than technology. Information systems (in the meaning I am using) also embodies all the other “wetware” (i.e. people and organisational) stuff required for the technology to be used and used effectively.

This definition means that I include the following as design:

  • The design and implementation of training and support for the system.
  • The creation of the policies and procedures around the system.
  • The design of the organisational structures and positions within those structures that will impact on the system.
  • How people are encouraged to make decisions about the system.

As I wrote previously (Jones, 2004) – and really just repeating what others had already said – about the impact of representation/meaning

The formulation of the initial state into an effective representation is crucial to finding an effective design solution (Weber, 2003). Representation has a profound impact on design work (Hevner et al., 2004), particularly on the way in which tasks and problems are conceived (Boland, 2002). How an organisation conceptualises the e-learning problem will significantly influence how it answers the questions of how, why and with what outcomes

The answers that a university arrives at in terms of the how, why and with what outcomes end up embodying a collection of meanings. When the organisation and its members implement e-learning they too often assume that there is a commonality of meaning. Commonality of meaning is a key part of how they represent the system. Consequently, their design is fundamentally based on the idea of commonality of meaning. I think that this is a fatal flaw for much of what is designed.

What follows are some examples of where it doesn’t hold.

Software design

My main current task is the design of BIM (code should be out by Monday at the latest) and today was a day to watch a “clueless user” (she’s actually quite intelligent she just knows little about computers and BIM) interact with BIM. BIM is designed by me. It embodies the meaning that I have formed about BIM and its task over the 3 years or so I’ve been working on it. It also embodies meanings/ideas/understandings that have formed over the last 12/15 years of doing e-learning and developing e-learning systems. That same meaning is informed by my experiences in social media (e.g. this blog)

The “clueless user” is a sessional teaching academic in management/human resources. She’s done a bit of e-learning and used BAM. She doesn’t have very much in the way of detailed mental models about how her computer works, how the Internet works or how Moodle, BIM, blogs and feeds work (or even mean).

Needless to say, having observed the user and the meanings she has demonstrated of BIM, I have a long list of improvements for the interface and operation of BIM. If some of them aren’t made, the other academics going to be using BIM are going to struggle. Understanding her meaning and responding to it has been helpful. It has forced me to reconsider and hopefully improve BIM to better fit with other meanings. It should improve BIM.

Of course, 1 person does not make a universe. But that 1 person being very different from me will help a bit.

Downes tweeted

… and my take-away is that we should be careful not to assume that people see things the same way we do, because invariably they don’t

If I’d assumed the same meaning and left BIM as, there would have been trouble for someone. The academics using the unmodified BIM would have suffered increased levels of frustration of dealing with a new system for which they did not understand the embodied meaning. There will still be some of that, but hopefully not as much.

Downes, on the implications of this

there’s so much room for error in communication we don’t notice that we mean different things, usually, and then it surprises us when we do

I’m a little bit surprised by the level of changes needed in BIM. It’s based on a system that’s been used for 3 years, that has been used by this same “clueless user”.

But by engaging in what I’ve done I’ve opened myself up to that surprise at a stage much earlier where it is simpler for me to respond. Too much of how e-learning is implemented in universities does not allow itself to be exposed to “good” surprise, instead they get “bad” – often hugely problematic – surprise.

Minimum service standards

I know of an institution that has implemented minimum service standards for course websites. The standards have been approved at all the right committees, the designers of the standards have written a paper about it, there has been mention of it some of the training sessions for staff and it is now a couple of weeks out from the start of the first term using these standards.

The meaning being heard from the designers of these standards, at least until very recently, has been “it’s all good”. The meaning being heard from the academic staff now being required to fulfill the standards and complete the accompanying checklist includes: “Where did this come from?” and “How do we comply with it?”

Even some of the designers and promulgators of the standards have different meanings. Perhaps the two extremes of those meanings are:

  • The standards are a stick with which to identify the bad teachers.
  • The standards provide a scaffold within which to have discussions about the design of learning experiences within the LMS.

Now, will this difference of meaning result in a “bad” surprise. I’m not so sure. I think organisations and how they choose to perceive the world has a lot in common with what Downes says about communication

In fact, we mostly don’t detect the errors, there’s a huge tolerance for error in communication, that’s why it works

LMS training

I would characterise the standard approach used to “train” academics how to use a new LMS – or any new system – as:

  • In the months before the release of the system hold numerous training sessions in places and at times that suit the academics.
  • Have the supervisors of the academics, and especially the senior management of the institution, reinforce how important it is to attend these sessions.
  • Within the session seek to get the academics to understand the meaning embodied into the system so they can interact with it.
  • Provide these sessions at a time and place removed from the normal context within which the academics will use the system.
  • Employ a range of technical folk who can easily understand the meaning of the system to explain it to the academics in a way that is very similar to how the technical folk learned it.
  • Assume that at the completion of the training they only need a much lower level of support and training. Generally limited to repeating the original training for new staff and providing front-line helpdesk staff to explain how any problems are due to the academic misunderstanding the meaning embodied in the system.

Can you see how the lack of a commonality of meaning is going to cause problems here. To me it’s obvious that the academics will not get the meaning embodied in the system.

The “clueless user” I mentioned above expressed this understanding of the training she experience.

I did the training in the first batch. Over 6 months ago. I haven’t touched the LMS since. How much do you think I remember?

If there is no commonality of meaning, then what?

Downes suggestion is (remember he’s thinking/tweeting in a different context, but I think it applies)

You need to experiment- Wittgenstein called it a game – to test and feel to see what word evokes what response- there is no common ‘meaning’

Given the impossibility of any commonality of meaning and the huge complexity and diversity of the meaning associated with e-learning, learning, teaching, universities, people and technology, the processes within universities and e-learning should be aimed much more at experimentation, at sharing of meaning, at encouraging surprise and enabling effective response and interaction.

What if the assumption of commonality of meaning remains? You keep operating as if there was commonality of meaning? Downes

if ‘sameness of meaning’ were required, communication would grind to a halt.

BIM – misc changes from moodle-train

Currently testing BIM on the training/testing environment of my current institution. This is where I find all of the problems from the institution’s non-vanilla implementation of Moodle.

Manage marking – not all markers – DONE

Not showing details for all of the markers that have been allocated students. This appears to be because the institution has introduced a new role called “Course coordinator”. Checks:

  • Change on of the teachers to “teacher” more standard role.
  • Then change the access.php file in bim to map the new role appropriately.

Part of this problem is due to students not being allocated to groups TO DO: Implies a need to identify those students who are in the course, but not in groups.

Actually, the problem was a misplaced add_data_keyed that was outside, instead of inside, a loop.

Find student view details

Parse error: syntax error, unexpected ‘;’ in /Applications/XAMPP/xamppfiles/htdocs/moodle/mod/bim/marker/view.php on line 968

Need to remove mysql_real_escape – DONE

This is a hang over from early development and not removed. Won’t work on server environments if they aren’t using MySQL. — marker/view.php line 900

Manage marking – counting all marked posts – DONE

Only 1 post marked, but release is showing 3 marked!!

This is because it’s simply adding the total marked for each marker. If there is overlap in students between markers, then the # marked is counted more than once.

Fix needs completion locally before putting onto -train

Marking form – min/max display – DONE

Need to change hard code 0 and 55 to the real min/max values.

Student viewing details – DONE

The table with student details always shows the progress report. Even for marked posts that have not been released to the student. Initially this table was only for staff. But it is also being used on the students view page. Need the progress mark to only include released posts in its total.

Calculation done in bim_generate_student_results – do it here.

All calls to this function, need to pass in cm

Breadcrumbs for new functions – DONE

The “change blog” feature doesn’t have good bread crumbs. May also be a problem with other new functionality. Check and fix.

Mark post

Add “register blog” – DONE

Just as marker can change existing feed for a student, add the option to “register a blog” for unregistered students.

cron job not completing – DONE

Call to undefined function bim_get_mirrored() in /Applications/XAMPP/xamppfiles/htdocs/moodle/mod/bim/lib.php on line 176

Get this by running the admin/cron.php.

Setting up help – DONE

Did some minor updates, rest will have to wait