Category: addie

The alignment project as leadership

On April 29, 2010

In addie, alignment, curriculumMapping, Uncategorized

The following signals a slight change in direction around the curriculum mapping project. First, the project is now going under the label “alignment project” (curriculum mapping is just one aspect of the project). Second, the project is likely to be re-framed as an application for an ALTC leadership grant. This post is an attempt to begin this re-framing. It’s really just thinking out loud.

As a result, I am very interested in suggestions and criticisms. In terms of suggestions, I’m particularly keen for insights onto better/alternative theoretical frames. It has become a bit confused as I’ve tried out different lines of approach.

Note: In the following a “course” typically means the smallest unit of teaching offered by an institution. A “program” is a group of courses that form a degree or perhaps a major.

Summary of the alignment project

Alignment is an increasingly core component of teaching in Australian universities. At a basic level, alignment is where the learning resources, learning activities and assessment of learning all align with the stated outcomes or aims of teaching (in a course/unit or program/course). Such alignment is a core component of various “movements” within university learning and teaching including: graduate attributes, quality assurance, and improvements in learning and teaching. However, consideration of alignment is not a regular, everyday part of teaching or learning at universities.

The majority of academic teaching practice involves the teaching of an existing course, one the academic has usually taught before, and as such most “teaching practice” revolves around making minor modifications to material or content (Stark, 2000). Academics are not often required to engage in the development of new courses or major overhauls of existing courses (Stark and Lowther 1988). Alignment is most considered during the development of new courses, major overhauls of existing courses or in response to external quality assurance needs.

The following two sub-sections try to summarise the alignment project. The first is a more concrete description, the second more abstract or theoretical. The thinking behind this project has changed many times, the following are likely to change. Suggest away.

Process and intent

As currently thought, the alignment project can be described as four main tasks:

Modify Moodle to allow mapping of alignment.
Moodle has been chosen because it is the institutional e-learning system at both institutions that are part of the project. As such, support for Moodle is embedded into the institution and will continue to be supported. In addition, Moodle is an increasing part of the everyday practice of teaching academics. Lastly, Moodle, for a number of reasons, is very flexible and easy to modify and any modifications made could be usable by other institutions. The point of this project is not the Moodle modifications, it’s simply the best solution for embedding these changes into the institutions.
The changes will focus on enabling the alignment of outcomes (be the course learning outcomes, program learning outcomes, those from accrediting bodies, graduate attributes etc) with the assessment, activities and resources within a course site. Having this functionality is a foundation for the rest of the project.
Work with teaching staff to map their courses.
Mapping the alignment of a course within Moodle is not going to be straight forward. Teaching staff are likely to be busy and may not be entirely familiar with the concepts around alignment and mapping. The content and design of a Moodle course site may not be currently appropriate in terms of developing a useful mapping of alignment. Using the alignment mapping functionality added to Moodle may not be straight forward.
For these and other reasons staff from the L&T support services will have to work collaboratively with academic staff to overcome these problems. This will be the first cycle of action research and will identify specific problems and insights into potential solutions.
Work with teaching staff to embed alignment into everyday practice.
Once the initial mapping of alignment is complete, the focus moves onto helping academics maintain and reflect on the level of alignment within their course and programs. On helping them embed alignment considerations into everyday practice.
This is the second cycle of action research and will likely include the development of models, tools and processes that address questions such as:
- How do you encourage reflection and action based on the everyday consideration of alignment?
  Identify the support, processes and tools do teaching academics and program leaders need to encourage and enable reflection and action?
- How do you measure and give feedback upon action based on the everyday consideration of alignment?
Develop on-going and embedded institutional processes that take the lessons learned from the everyday consideration of alignment and use it to remove barriers within the institutional context.

Theoretical perspective

The alignment project is seeking to take the first steps towards what Biggs (2001) called the reflective institution. It seeks to do this by modifying the institutional systems around learning and teaching in ways suggested by Biggs (2001). These are:

Make obvious the quality model.
Most institutions espouse the theory of alignment, however the teaching systems and processes employed by institutions do not make this theory explicit. The first step is to modify these systems so that a focus on alignment is made explicit as a part of every day teaching practice.
This is partly achieved through the modifications to Moodle to enable mapping of alignment. But more importantly it is achieved through the changes in L&T support, systems and processes that support academic in using those new Moodle capabilities. In part, these changes are the next step.
Provide appropriate support for quality enhancement.
Further modify these systems and process to enable and encourage teaching staff to reflect and improve their teaching through a focus on alignment. The modifications build on the changes in the previous stage to enable this support to be highly contextualised to everyday teaching practice. A focus on helping in what academics do most often, the fine-tuning of existing courses.
Inform the quality feasibility process.
Quality feasibility is the removal of factors in the teaching environment that are not conducive to good L&T. The everyday consideration of alignment will identify a range of barriers in the institutional setting, many of which will require the engagement of institutional leadership to remove.

ALTC leadership grants

The guidelines for the ALTC leadership grants scheme describe the grants as being for (my emphasis added)

projects that build leadership capacity in ways consistent with the promotion and enhancement of learning and teaching in contemporary higher education, and which reflect the ALTC’s values of excellence, inclusiveness, diversity and collaboration, and its commitment to long-term, systemic change.

The scheme has three priorities which can be summarised as being focused on: institutional leadership capacity building; disciplinary and cross-disciplinary leadership; and building on earlier projects. The alignment project seems to be best suited to Priority one

institutional leadership to enhance learning and teaching through leadership capacity-building at the institutional level.

Funding range: $150,000 to $220,000

Project duration: up to 2 years

In completing a lit review around leadership for the ALTC, Southwell and Morgan (2009) make the observation that

Leadership for Excellence in Learning and Teaching Program is expected to be a ‘demonstrable enhancement of learning and teaching through leadership capacity building’

What is leadership?

It seems required when discussing leadership to make the observation that “leadership eludes comprehensive definition” (Southwell and Morgan, 2009). Southwell and Morgan (2009) reference Marshall (2006) and Jameson (2006) as folk who have made that observation. Having made this claim, the idea is that you then define your familiarity with the broad array of perspectives, understandings and definitions of what leadership is. I’ll postpone that bit for now.

Southwell and Morgan (2009) draw on Leithwood and Levin (2005) who suggest that the core of most conceptions of leadership are two functions generally considered to be indispensable:

Direction-setting: helping members of the organization establish a widely agreed on direction or set of purposes considered valuable for the organization; and

Influence: encouraging organizational members to act in ways that seem helpful in moving toward the agreed on directions or purposes

Leithwood and Levin (2005) arrive at these two functions by adopting a definition where “the primary effect of organisational leadership would be significant change in a direction valued by the organisation”. In defending their definition or understanding of leadership, Leithwood and Levin (2005) agree that this may not be a precise definition, but that attempts to too narrowly define a complex topic like leadership is “more likely to trivialise than help bring greater clarity to its meaning”.

How the alignment project fits

Taking the emphasised phrases from the above purpose of the ALTC projects, I’d suggest/argue that the alignment project fits in the following ways:

leadership capacity;
In terms of the above set of functions, the project is aiming to build into the systems and processes of each host institution the capacity to make consideration of alignment an every day part of practice. It helps establish a widely agreed direction for L&T and helps influence organisational members in moving towards that agreed direction.
promotion and enhancement of L&T;
The educational literature is replete with evidence that consideration of alignment changes the conception of L&T held by academics and that it also results in improvements in student learning outcomes.
contemporary higher education;
The project recognises and seeks to fulfill the increased requirements for accountability from a range of diverse source, however, it seeks to achieve it in a way that offers significant greater benefits that existing methods. As part of this the application seeks to engage with the on-going argument over centralised or devolved L&T support services by aiming for a focus on an approach to L&T support services that seeks to contextualise such support into the every day practice of teaching academics.
inclusiveness, diversity and collaboration; and
The action research process suggested for use by the project is largely based on recognition within the project that engaging with the full diversity of higher education is essential. It recognises that this diversity will result in different approaches and benefits and that the process needs to enable this to happen. Similarly, collaboration is seen as essential to the project. Not just in the process used in this project, but in the aims of the project. An important aim of the project is to increase the collaboration around consideration of alignment in teaching.
long-term, systemic change.
The project aims to embed consideration of alignment into the everyday practice of teaching staff. i.e. the aim of the project is long-term, systemic change. The process and approach being used is designed to achieve that aim.

The alignment project as leadership

Contemporary higher education is placing increasing importance on the concept of alignment in learning and teaching. In terms of quality assessment, program accreditation, graduate attributes and generally improving L&T alignment is broadly seen as a necessary component. However, many university courses are not all that well aligned and one explanation for this is that consideration and discussion of alignment is not a regular part of everyday teaching practice. Alignment is often only considered at the time of course and program reviews or accreditation.

The aim of the alignment project is to build leadership capacity into the system and processes of education within a university so as to encourage and enable effective and informed consideration of alignment as part of everyday teaching. This embedding of alignment into everyday practice then serves as the foundation for a range of other possibilities.

The alignment project is an example of leadership as it is attempting to encourage significant change – in the form of increased consideration of alignment at all levels – that is valued by the individual universities and the broader higher education sector. In addition, there is broad agreement in the education literature that alignment has significant positive effects on student learning outcomes.

The alignment project intends to fulfill the two indispensable functions of leadership identified by Leithwood and Levin (2005):

direction-setting; and
By making considerations of alignment a visible and hopefully key aspect of everyday teaching, there should be an increased emphasis placed on learning outcomes, graduate attributes and other “outputs of learning”. This should encourage and assist academics teaching the same course or in the same program to increase discussion of these outcomes. To increase discussion of the purpose or direction of a course or program.
influence.
A specific aim of the alignment project is to modify the teaching environment so that considerations, discussions and reflection upon alignment are directly encouraged and enabled. It encourages and enables them to think about how to move towards the stated directions or purpose.

Still not happy with that division. First attempt to make it concrete. Another query that arises from this is whether or not this project builds leadership capacity at two or more levels. For example, one approach is that it builds capacity at both the:

Institutional level; and
At an institutional level it enables curriculum/program alignment, but also accountability etc.
Instructional level.
This is the level between student and course/teacher. It makes alignment an more regular component of instruction.

Outcomes

If successful, the project should result in the consideration of alignment should becoming an every day component of teaching practice. This should/could generate the following outcomes:

Changes in the conceptions of L&T held by teaching staff;
Being required to consider alignment encourages a different way of looking at teaching. This should encourage changes in the conceptions of L&T held by teaching staff towards those considered more appropriate.
Improvements in student learning outcomes;
Effective considerations of alignment should increase the alignment within courses. The educational literature suggests that increases in alignment will result in improve student learning outcomes. In addition, it is likely that one of the likely additional outcomes will be making alignment more visible to students. Which should also encourage improved student learning outcomes.
Improvements in the quality and timeliness of quality assurance; and
In terms of demonstrating alignment against outcomes or attributes, current quality assurance practices rely on special “mappings” that are held every few years. By embedding alignment considerations and mapping into every day teaching practice, the there is no longer any need for special “mappings”. “Mapping” information can be generated at any time as it is maintained as part of normal practice.
A variety of additional outcomes.
Embedding alignment considerations as an everyday practice is the foundation of the project. The availability of “mapping” information and the on-going consideration of alignment will generate a range of additional outcomes. However, the diversity inherent in universities and higher education, combined with the fundamental change in everyday practice which this project aims to achieve means that it is unlikely we can predict successfully all of these outcomes.

Theoretical foundations

The ALTC place significant emphasis on applications having a sound and obvious theoretical base. The theoretical work that have informed my thinking about this project, and which may influence the project, include the following.

Project intent or outcome

Biggs (2001) presents an argument for a reflective institution that focuses on prospective, rather than retrospective quality. Having just re-read the paper, it seems to provide a good fit for a theory/model for the overall intent of the project.

The model is based on the idea of three aspects of AQ:

Quality Model.
i.e. an espoused theory of teaching, for Biggs this is constructive alignment. For the alignment project this might be alignment a little more broadly.
Quality enhancement.
A “teaching delivery system” that is designed in accordance to the quality model, i.e. one which encourages and enables alignments. In addition, the teaching delivery system should also have built-in mechanisms to continually review and improve current practice.
Quality feasibility.
A process by which impediments to quality teaching are removed from the “teaching delivery system”.

The “quality model” is underpinned by a large, overlapping and diverse collection of literature from various areas including: outcomes-based quality/evaluation, instructional alignment (Cohen, 1987); curriculum alignment; graduate attributes; and, of course constructive alignment (Biggs and Tang, 2007; Biggs, 1999 ).

Understandings of universities and organisations

My personal conceptions of most organisations, but especially those like universities, are informed by Dave Snowden’s Cynefin framework and complex adaptive system.

Understandings of leadership

This is obviously an area which needs more consideration, beyond the summary given above.

Based on my limited reading, I like the description of “new leadership” (related to distributed leadership) attributed to Fullan (2008), which include:

respect of employees, rather than simplistic judgmentalism;
A specific focus of this project is to help teaching staff consider the alignment of their courses. It is not to judge them. Biggs (2001) makes the point that under the type of reflective institution he outlines, the focus is on the teaching, not the teacher.
connecting peers with purpose and ownership;
By embedding indications of alignment into the LMS the aim is to create connections between other teaching staff, academic leaders and also teaching support staff. The clear purpose is around considering alignment.
building employees’ and systems capacity; and
This is the specific aim of the project. Building into the institution the capacity in both the employees and its systems to engage regularly in consideration of alignment.
transparency of practice and results.
At the very least, the aim of this project is to enable teaching staff within the same program (i.e. group of courses/degree etc.) to see each others practice. To show what is aligned, where (or where not). It opens up the teaching practice to colleagues, hopefully in a way that is not judgmental.

Process

Based on some of the above and below, I’m leaning very much towards an action research process. The practice of L&T within a university is a complex-adaptive system. As we introduce change into the system, the system will change around us and unexpected event will happen. The type of change suggested involves a fairly widespread change in the practice of teaching academics, but also the institution. In addition, as far as I’m aware, no-one else has tried and reported on this type of change, hence it is novel. While driving towards a particular goal, we have to aim on learning as much as we can during the process.

In addition, Biggs (2001) offers the following

action research, a methodology designed precisely to generate and evaluate in-context innovations (Elliott 1991). As a result of engaging in action research, teachers change their conceptions of teaching, and teach more effectively (Kember 2000).

This type of approach also fits very closely with what is known about staff development.
i.e. the current recommendations are that staff development should be as contextualised as possible. My conception of how teaching staff would be helped to consider the alignment of their courses contains a very heavy assumption on contextualised staff development. In fact, the presence of alignment information and the transparency between courses is aimed at helping this support and development to be increasingly contextualised to the every day teaching practice of the teaching staff.

Teaching practice

One of the fundamental models/theories underpinning this project for me is the observations embodied in the following.

How academics design their teaching is not described by a rational planning model (Lattuca and Stark 2009). In part, this is because the dominant setting for academics is teaching an existing course, generally one the academic has taught previously. In such a setting, academics spend most of their time fine tuning a course or making minor modifications to material or content (Stark 2000). Academics are usually not often required to engage in the development of new courses or major overhauls of existing courses (Stark and Lowther 1988).

The practice of most academics does not separate planning from implementation, and rather than starting with explicit course objectives, starts with content (Lattuca and Stark 2009). In the absence of formally documented teaching goals, the actual teaching and learning that occurs is more in line with the teacher’s implicit internalised knowledge, than that described in published course descriptions (Levander and Mikkola 2009). Formal descriptions of the curriculum do not necessarily provide much understanding about how teachers put their curriculum ideas into action (Argyris and Schon 1974)

References

Argyris, C., & Schon, D. (1974). Theory in practice: Increasing professional effectiveness. Oxford, England: Jossey-Bass.

Biggs, J. (1999). Teaching for quality learning at university. Buckingham: Open University Press.

Biggs, J. (2001). The Reflective Institution: Assuring and Enhancing the Quality of Teaching and Learning. Higher Education, 41(3), 221-238.

Cohen, S. A. (1987). Instructional alignment: Searching for a magic bullet. Educational Researcher, 16(8), 16-20.

Fullan, M. (2008). The Six secrets of change. San Francisco: Jossey-Bass.

Jameson, J. (2006). Leadership in post-compulsory education: Inspiring leaders of the future. London: David Fulton Publishers.

Kember, D. (2000). Action Learning and Action Research: Improving the Quality of Teaching and Learning. London: Kogan Page.

Lattuca, L., & Stark, J. (2009). Shaping the college curriculum: Academic plans in context. San Francisco: John Wiley & Sons.

Levander, L., & Mikkola, M. (2009). Core curriculum analysis: A tool for educational design. The Journal of Agricultural Education and Extension, 15(3), 275-286.

Marshall, S. (2006). Issues in the development of leadership for learning and teaching in higher education (Occasional paper). Sydney: Carrick Institute of Learning and Teaching in Higher Education.

Stark, J. (2000). Planning introductory college courses: Content, context and form. Instructional Science, 28(5), 413-438.

Stark, J., & Lowther, M. (1988). Strengthening the Ties That Bind: Integrating Undergraduate Liberal and Professional Study. Ann Arbor, MI: Professional Preparation Project.

Southwell, D., & Morgan, W. (2009). Leadership and the impact of academic staff development and leadership development on student learning outcomes in higher education: A review of the literature. Sydney: Australian Learning and Teaching Council.

Another spectrum for using indicators to place course websites

By admin

On March 1, 2009

In addie, blackboardIndicators, Chapter 4, elearning, indicators, thesis, Uncategorized

This post adds another perspective borrowed from Gonzalez (2009) as a framework to report or evaluate findings from Col and Ken’s indicators project. Col added an update on his work recently. Like previous post this one borrows a table of dimensions around conceptions of online learning because it may be helpful.

First the table and then how it might be used.

Dimensions

Dimensions delimiting approaches to online teaching – (Gonzalez, 2009: p311)
	Informative/individual learning focuses	Communicative/Networked learning focused
Intensity of use	Small range on media and tools used to support learnign tasks and activities (mainly sources of information with small opportunities for interaction and communication)	Wide range of media and tools used to support learning tasks and activities (with emphasis on interaction and communication)
Resources	Web pages with information. Lecture notes. Links to websites.	Web pages with information. Lecture notes. Links to web sites. Discussion boards. Chat. Blogs. Spaces for sharing. Animations. Videos. Still images.
Role of the learner	Select and present information	Design spaces for sharing and communication. Support the process.
Role of the students	Study individually information provided	Participate in a process of knowledge building

How might it be used

The above dimensions could be used to develop “analysis routines” that would place courses within these dimensions. Some potential approaches:

Variety and use of tools and media within a course site. (Intensity of use and Resources)
Group the different tools available in the course management system into different types. e.g. those used for information distribution and those for interaction/communication. Count the number of different types of tools present in a course site and the level of usage.
The difficulty here is the increasing use of non-CMS based tools for communication. e.g. I know of an increasing number of staff and students who are using external tools such as Messenger to work around the limitations of CMS services.
Measure student and staff activity (Role of the lecturer/students)
I believe Blackboard, the main CMS at our institution, tracks the activity in some detail of each course site participant. If the type of activity can be categorised into groups (e.g. adding information to the site, using information on the site, posting to a discussion forum, responding to a post in a discussion forum etc.) then analysis could be run against the activity of all participants. This would identify the type of role the main groups are taking on.

What’s the value of this?

I can hear some thinking, “so what!”. What is the value of this sort of thing? A couple of thoughts.

As a framework to help make sense of the data.
From my perspective it appears that the project is “drowning in data” and could use some sort of reviewed framework with which to organise or structure their investigations. These dimensions might provide it.
Enable institutions to get a handle on what is happening.
Most of it ain’t great. The combination of the dimensions and the data potentially enable institutions, that are spending a lot of money on course management systems, to improve the awareness they have of what is actually happening. At the very least some sort of indication of where online courses site within the institution, as imperfect as it will be, sit within the dimensions might start some conversations about online practice that is actually somewhat informed by the reality of what is going on.
As a demonstration of building on the work of others.
It is possible to argue with the value/validity of the knowledge generated by Gonzalez (2009) – but then it’s possible to argue against the validity of just about the knowledge generated by any research project depending on your perspective. However, this work is in a fairly prestigious journal, so it comes with a certain stamp of approval. This will help Col and Ken.
Perfect opportunity for a publication.
Building on the last point, suggests that complimenting the qualitative nature of Gonzalez (2009) with some more quantitative measures from a broad collection of students and courses sounds like a pretty good publication opprotunity (or three).

It’s the potential for discussion within the organisation that is, I believe, of potentially the most beneficial for the most people.

The potential for publication is probably the most interesting to the project participants and frankly by far the easiest.

Further publications

The publication idea would be strengthened if previous work in this area (e.g. the recent ALTC project Learning and teaching performance indicators report either doesn’t do something like this or uses a different set of dimensions.

In addition, Gonzalez interviewed only 7 academics within a single discipline within a single institution. Chances are the results and dimesions identified in the paper are going to exhibit some sort of limitation, potentially caused by the nature of the context. Using a different approach in a different context will at least compliment/reinforce the findings and potentially identify additional dimensions.

References

Gonzalez, C. (2009). “Conceptions of, and approaches to, teaching online: a study of lecturers teaching postgraduate distance courses.” Higher Education 57(3): 299-314.

Data mining of online courses – dominant assumptions and innovation potential

By admin

On February 2, 2009

In addie, elearning, Uncategorized

For almost as long as learning management systems have been around their have been researchers and technologists investigating how the usage logs of these systems can be harnessed to inform and improve learning and teaching. For a little while I was sort of involved in a project that would look at some of this – interest has waned in line with organisational realignments.

A colleague, however, is continuing on with a project in this area. The following is an attempt to reflect about what little I know about the area and see if there is anything of interest to Col. In the following I am trying to identify the dominant assumptions underpinning this sort of work in order to see if there are any holes which might be ripe for exploitation.

Disclaimer: Until recently I haven’t paid much attention to the literature in this area and my recent reading was sparse, interrupted and incomplete. Hence the following could do with the insights of those with more knowledge of the literature to correct oversights, misunderstandings and naive assumptions.

What are we talking about

The common approach to this process uses automated evaluation of system logs and databases (e.g. Zorrila and Alvarez, 2008, Hung & Zhang, 2006; and Heathcote and Dawson, 2005) using some form of data mining or similar technology in order to provide additional information for teaching staff or Universities about the quality of the student experience. The typical form is take the usage logs and databases from a learning management system, put them into some sort of data warehouse system and generate reports for academics and management.

This type of work typically seems to differ on a few characteristics:

The technology/algorithms used to analyse the system logs and present the results.
The type of theory of learning or quality online presence that is used to “measure” the quality of the online course or the student experience.

I guess also there will be different communities within e-learning that take a different slant on this work. For example, I believe the intelligent tutorial system folk probably have a large amount of research and literature focused at analysing the student experience using data mining in order to make informed decisions about what to advise the student to do next.

Dominant assumptions and opportunity for innovation

Col is doing some of this work as part of a research project within his Masters study. So, I’m assuming he is keen to get into some innovation, to do something that hasn’t been done yet. So, in the following I’m trying to identify the dominant assumptions that, at least in my experience, seem to characterise this work. The idea is that over turning these assumptions might reveal an interesting line of work.

The assumptions that I’m thinking about (are there more?) include:

Limit and focus to the LMS logs.
Analysis of LMS logs tells only a very small part of the story about students, staff and their interactions within learning and teaching. Even if the focus remains on the logs of computer systems, most students and staff will be leaving a fairly large trail of usage information through a range of other computer systems that could be combined with those in the LMS.
This is especially important as the type of information gathered by mining LMS logs is limited by the nature and design of the LMS. For example, the Webfuse system (Jones and Gregor, 2006) has a design that by default allows open access to all resources. That is no requirement for users to login. Each system has its own design limits.
Kaplan’s law of instrument continued.
Most of this work is done with similar tools. System logs and databases passed through statistics analysis and/or data mining technology to generate information. While useful, the tendency to focus just on these technologies has the potential to lead to flaws due to Kaplan’s law of instrument. Everything looks like a nail, if all you have is a hammer. Are there alternative tools or approaches that can be used, which may be base on different approaches and hence reveal different insight.
Emphasis on quantitative analysis alone.
Simple log usage figures only tell a very small part of story of the student experience. Much of it is hidden within the words and emotions used and experienced by participants. It’s increasingly widely recognised that a multi-method approach for research is effective through each method covering limitations of other methods. Marrying quantitative analysis with textual analysis (e.g. Leximancer), or qualitative or specific feedback from students (e.g. course barometers; Jones, 2002) might be interesting.
Presenting information for staff and the institution, not the students.
The aim for most seems to be to make the information available to the teaching staff so they can reflect and make improvements. Have seen little work where this information is made pro-actively available to the students so they can use it to guide their learning or student experience. For example, if a student were to see a huge spike in visitations to a particular page that they hadn’t visited, they may consider taking a look.
How is the insight generated used?
Almost all of the research literature I’ve seen so far shows this information solely being used by researchers after the fact. i.e. after the term has finished they’ve analysed the data to see what lessons they can learn. The information may not even be shared with the teaching staff and they rarely seem to talk to academic staff. It would be interesting to see what would happen if you worked with academic staff during a term harnessing a range of different analysis/mining tools to provide information that they could respond to. This would allow tracking of what happens and the generation of insight and requirements within a real context of use, which is likely to be more useful and insightful than requirements gather out of context.

References

Heathcote, Elizabeth and Dawson, Shane (2005) Data Mining for Evaluation, Benchmarking and Reflective Practice in a LMS. In Proceedings E-Learn 2005: World conference on E-learning in corporate, government, healthcare & higher education, Vancouver, Canada.

David Jones, Student feedback, anonymity, observable change and course barometers, World Conference on Educational Multimedia, Hypermedia and Telecommunications, Denver, Colorado, June 2002, pp. 884-889.

Using a blog for course design foult sessions

By admin

On January 30, 2009

In addie, c2d2, Uncategorized

I’ve bitten the bullet and have decided to use WordPress blog to support the 6 hour orientation to course analysis and design I’m supposed to run next week.

It’s probably going to be much more work than I should or planned to put in, but so far it’s been fairly easy. It may be worthwhile.

Good teaching is not innate, it can be "learned" – and what's wrong with academic staff development

By admin

On January 29, 2009

In addie, c2d2, Chapter 2, design theory, elearning, foult, missingPs, thesis, Uncategorized

The title to this post is included in a quote from Kane, Sandretto and Heath (2004)

The research team, comprising two teacher educators and an academic staff developer, embarked upon this research confident in the belief that good teaching is not innate, it can be learned. With this in mind, the project sought to theorise the attributes of excellent tertiary teachers and the relationships among those attributes, with the long-term goal of assisting novice academics in their development as teachers.

This coincides nicely with my current task and also with an idea I came across on the week-end about deliberate practice and the work of Anders Ericsson.

The combination of these “discoveries” is also providing some intellectual structure and support for the REACT idea about how to improve learning and teaching. However, it’s also highlighting some flaws in that idea. Though the flaws aren’t anywhere near as large as what passes for the majority of academic staff development around learning and teaching.

The following introduces these ideas and how these ideas might be used to improve academic staff development.

Excellent tertiary teaching

Kane et al (2004) close the introduction of their paper with

We propose that purposeful reflection on their teaching plays a key role in assisting our participants to integrate the dimensions of subject knowledge, skill, interpersonal relations, research/teaching nexus and personality into recognised teaching excellence. We conclude with a discussion of the implications of our model for staff development efforts.

Their proposition about the role of reflection in contributing to excellent teaching matches with my long held belief and perception that all of the best university teachers I’ve seen have been those that engage in on-going reflection about their teaching, keep looking for new knowledge and keep trying (and evaluating) innovations based on that knowledge in the hope to improve upon their teaching.

The authors summarise a long history of research into excellent teaching that focused on identifying the attributes of excellent teachers (e.g. well prepared, stimulate interest, show high expectations etc.) but they then suggest a very important distinction.

While these, and other studies, contribute to understanding the perceived attributes of excellent teachers, they have had limited influence on improving the practice of less experienced university teachers. Identifying the elements of “good” university teaching has not shed light on how university teachers develop these attributes.

The model the develop is shown below. The suggest

Reflection lies at the hub of our model and we propose that it is the process through which our participants integrate the various dimensions

The authors don’t claim this model to have identified any novel sets of attributes. But they do suggest that

the
way in which the participants think about and understand their own practice through purposeful reflection, that has led to their development of excellence

What’s been said about reflection?

The authors have a few paragraphs summarising what’s been said about reflection in connection to tertiary teaching, for example

Day (1999) wrote “it is generally agreed that reflection in, on and about practice is essential to building, maintaining and further developing the capactities of teachers to think and act professionally over the span of their careers”

They trace reflection back to Dewey and his definition

“an active, persistent, and careful consideration of any belief or supposed form of knowledge in light of the grounds supporting it and future considerations to which it tends

The also mention a framework of reflection outlined by Hatton and Smith (1995) and use it to provide evidence of reflection from their sample of excellent teachers.

Expertise and deliberate practice

Among the many quotes Kane et al (2004) provide supporting the importance of reflection is this one from Stenberg and Horvath (1995)

in the minds of many, the disposition toward reflection is central to expert teaching

Another good quote (Common 1989, p. 385).

“Master teachers are not born; they become. They become primarily by developing a habit of mind, a way of looking critically at the work they do; by developing the courage to recognize faults, and by struggling to improve”

Related to this view is the question “Was Mozart, and other child prodigies, brilliant because of some innate talent?”. This is a question that this blog post takes up. The answer it gives is no. Instead, it’s the amount and quality of practice they engage in which makes the difference. Nurture wins the “nature versus nurture” battle.

The blog post builds on the work of Anders Ericsson and the concept of “deliberate practice”. The abstract for Ericsson et al (1993) is

The theoretical framework presented in this article explains expert performance as the end result of individuals’ prolonged efforts to improve performance while negotiating motivational and external constraints. In most domains of expertise, individuals begin in their childhood a regimen of effortful activities (deliberate practice) designed to optimize improvement. Individual differences, even among elite performers, are closely related to assessed amounts of deliberate practice. Many characteristics once believed to reflect innate talent are actually the result of intense practice extended for a minimum of 10 years. Analysis of expert performance provides unique evidence on the potential and limits of extreme environmental adaptation and learning.

Implications for academic staff development

If reflection or deliberate practice are key to developing mastery or expertise, then how do approaches to academic staff development and associated policies, processes and structures around university learning and teaching help encourage and enable this practice?

Seminars and presentations probably help those that are keen to become aware of new ideas that may aid their deliberate practice. However, attendance at such events are minimal. Much of existing practice seems to provide some level of support to those, the minority, already engaging in deliberate practice around learning and teaching.

The majority seem to be able to get away without engaging like this. Perhaps there’s something here?

References

Common, D.L. (1989). ‘Master teachers in higher education: A matter of settings’, The Review of Higher Education 12(4), 375–387.

Hatton, N. and Smith, D. (1995). ‘Reflection in teacher education: Towards definition and implementation’, Teaching & Teacher Education 11(1), 33–49.

Kane, R., S. Sandretto, et al. (2004). “An investigation into excellent tertiary teaching: Emphasising reflective practice.” Higher Education 47(3): 283-310.

Sternberg, R. and Horvath, J. (1995). ‘A prototype view of expert teaching’, Educational Researcher 24(6), 9–17.

Somethings that are broken with evaluation of university teaching

By admin

On January 25, 2009

In addie, c2d2, elearning, missingPs, Uncategorized

This article from a Training industry magazine raises a number of issues, well known in the research literature, about the significant limitations that exist with the evaluation of university teaching.

Essentially the only type of evaluation done at most universities is what the article refers to as “level 1 smile sheets”. That is student evaluation forms that ask them to rank what they felt they learn, what they felt about the course and the teacher. As Will Thalheimer describes

Smile sheets (the feedback forms we give learners after learning events) are an almost inevitable practice for training programs throughout the workplace learning industry. Residing at Donald Kirkpatrick’s 1st level—the Reaction level—smile sheets offer some benefits and some difficulties.

His post goes on to list some problems, benefits and a potential improvement. Geoff Parkin shares his negative view on them.

The highlight for me from the Training mag article was

In some instances, there is not only a low correlation between Level I and subsequent levels of evaluation, but a negative one.

The emphasis on level 1 evaluation – why

Most interestingly, the article then asks the question, “why do so many training organisations, including universities, continue to rely on level 1 smile sheets?”

The answer it provides is that they are too scared to do them in case of what they find. It’s the ostrich approach of sticking the head in the sand.

What else should be done?

This google book search result offers some background on “level 1” and talks about the other 3 levels. Another resource provides some insights and points to other resources. I’m sure if I dug further there would be a lot more information about alternatives.

Simply spreading the above findings amongst the folk at universities who rely and respond to findings of level 1 smile sheets might be a good start. Probably necessary to start moving beyond the status quo.

What do students find useful?

By admin

On July 12, 2008

In addie, c2d2, courseSites, elearning, Uncategorized

In a growing category of blog posts I’m expanding and attempting to apply my interest in diffusion theory and related theories to increase the use of course websites. A major requirement, as outlined in the previous post, in achieving this requires and understanding of what students find useful?

In this post, I’m trying to bring together some research that I’m aware of which seeks to answer this question by actually asking the students. If you know of any additional research, let me now.

Accessing the student voice

Accessing the student voice is the final report from a project which analysed 280,000 comments on Course Experience Questionnaire’s from 90,000 students in 14 Australian Universities. The final report has 142 pages (and is available from the web page). Obviously the following is a selective synopsis of an aspect of it.

The report summarises (pp. 7 and 8) the 12 sub-domains which attracted the highest percentage of mentions which it equates to those that are important to students. In rank order they are

Course Design: learning methods (14.2% share of the 285,000 hits)
There were 60 different methods identified as the best aspect of their studies which fell into 5 clusters
- 16 face-to-face methods that focused on interactive rather than passive learning
- 7 independent study and negotiated learning methods
- 20 practice-oriented and problem-based learning methods
- 6 methods associated with simulated environments and laboratory methods
- 11 ICT enabled learning methods
Staff: quality and attitude (10.8%)
Staff: accessibility (8.2%)
Course Design: flexibility & responsiveness (8.2%)
Course Design: structure & expectations (6.7%)
Course Design: practical theory links (5.9%)
Course Design: relevance (5.6%)
Staff: teaching skills (5.4%)
Support: social affinity (3.8%)
Outcomes: knowledge/skills (3.8%)
Support: learning resources (3.5%)
Support: infrastructure & learning environment (3.4%)

Going into totals

Course design – 40.6%
Staff – 24.4%
Support – 10.7%
Outcomes: knowledge/skills – 3.8%

Link to the 7 principles

A quote from the report

The analysis revealed that practice-oriented and interactive, face-to-face learning methods attracted by far the largest number of ‘best aspect’ comments.

Of the 7 Principles for Good Practice in Education mentioned in the last post, #3 is “encourages active learning”

What about CQU students

In late 2007 we asked CQU’s distance education students three questions

What did you like or find useful?
What caused you problems?
What would you like to see?

Students were asked to post their answers to an anonymous discussion forum. This means they could see each others posts and respond.

An initial summary of the responses is available and CQU staff can actually view a copy of the discussion forum containing the original student comments.

A simple analysis revealed the following top 10 mentions

106 – Some form of eLecture – video, audio etc.
86 – Quick, effective and polite responses to study queries.
66 – Clear and consistent information about the expectations of the course and assignments e.g. making past assignments and exams available.
55 – Study guides.
53 – Good quality and fast feedback on assignments.
33 – For resources that are essentially print documents to be distributed as print documents.
30 – A responsive discussion board.
27 – Online submission and return of assignments.
25 – More information about exams, including more detailed information on how students went on exams.
21 – Having all material ready by the start of term.

CQU Students – 1996

Back in 1996 CQU staff undertook a range of focus groups of CQU distance education students aim at identifying issues related to improving distance education course quality. This work is described in more detail in a paper (Purnell, Cuskelly and Danaher, 1996) from the Journal of Distance Education.

Arising from this work were six interrelated areas of issues. These issues were used to group the suggested improvements from the DE students, these improvements are explained in detail in the paper and are summarised below.

student contact with lecturers/tutors,
- Easy access to people with relevant expert knowledge and skills (usually the lecturer).
- Flexible hours for such access.
- Some personal contacts through telephone and, where possible, some face-to-face contact.
- Additional learning resources, such as audio- and videotapes to provide more of a personal touch.
assessment tasks,
- Detailed feedback (approximately one written page) on completed assessment tasks indicating how to improve achievement.
- Timely feedback so that students can utilize feedback in future assessment tasks in the unit.
- A one-page criteria and standards sheet showing specific criteria to be used in each assessment task and the standards associated with each criterion (statements of the achievement required for a high grade, etc.).
- Clear advice on assessment tasks in the unit resource materials and in other contacts such as teleconferences.
- Where possible, the provision of exemplar responses to similar assessment tasks be provided in the study materials.
- Lecturers to be mindful of the differences in resources available to rural students compared to those in larger urban areas when setting and marking assessment tasks.
flexibility,
- Non-compulsory residential schools available at various locations of no more than three days’ duration and incorporating use of facilities such as libraries.
- Greater consideration of the complexities of lives of distance education students by encouraging, for example, more self-paced learning.
- Access to accredited study outside traditional semester times.
- Lecturers/tutors to consider more fully the needs of isolated students in rural areas in support provided.
study materials,
- Ensure study materials arrive on time (preferably in the week prior to the commencement of a semester).
- Efficient communications with students-particularly with the written materials provided in addition to the study materials.
- Ensure each unit’s study guide matches other resources used in a unit, such as a textbook.
- Lecturers should be mindful of extra costs for students to complete a unit in which, for example, specialized computer software might be needed; if a textbook must be purchased, it should be used sufficiently to justify its purchase.
- Lecturers should cater to the range of students they have, especially from rural areas, with the study requirements for each unit (many participants reported that self-contained study materials in which there was little or no need to secure other resources to achieve high grades were valued).
mentors, and
- Having access to mentors is desirable but should be optional for students.
- Issues about the role of a mentor need to be clarified.
educational technology.
- Continue to use and make more effective use of technologies familiar to students, such as the telephone and audio- and videocassettes.
- Examine ways of minimizing access costs to the Internet for students, especially in rural areas.
- Provide appropriate technical support for students to be able to access and use the Internet.
- Provide professional development for staff to meet individual needs for using educational technologies involving, for example, interactive television, audio graphics, CD-ROM, e-mail, and the World Wide Web.

The commonalities between this list, from 1996, and the list generated in 2007 are not small.

Creating quality course websites – the pragmatic approach

By admin

On July 12, 2008

In addie, c2d2, courseSites, Uncategorized

In a previous post I laid out some rationale for an organisational approach to increase the usage of course websites. In this post I provide more detail on the rationale behind the pragmatic approach, which was described this way in that previous post.

Pragmatic – ad hoc changes to particular aspects of a course website.
Most academic staff make these changes in an unguided way. I’ll suggest that you are likely to obtain greater success if those ad hoc changes are guided by theories and frameworks such as the Technology Acceptance Model (TAM) and related work (Davis, 1989), Diffusion Theory (Rogers, 1995) and the 7 Principles for Good Practice in Education (Chickering and Gamson, 1987).

I’ll describe each of the three theories that form the foundation of this idea. In a later post, I’ll try and take up the idea of how this could be used in the design of a course website.

The fundamental idea is that these three theories become the basis for guiding the design of a standard course website which becomes the minimal online course presences for an institution. These theories are applied with close attention to the local context and consequently there will be differences between contexts, perhaps even between disciplines or types of courses.

Technology acceptance model

The Technology Acceptance Model (TAM) suggests that there are two main factors which will influence how and when people use an information system (the assumption is that a course website is an information system):

Perceived usefulness.
“The degree to which a person believes that using a particular system would enhance his or her job performance” (Davis 1989).
Perceived ease of use.
“The degree to which a person believes that using a particular system would be free from effort” (Davis 1989).

Some more discussion about the use of TAM within e-learning can be found in (Behrens, Jamieson et al, 2005).

The questions that arise from this idea for the design of a standard course web might include:

What do the students currently find useful?
What additions might they find useful?
The same questions applied to all staff, both teaching and support.
How can these requirements be fulfilled in a way that is easy to use?
How just do you determine that?

Diffusion theory

Diffusion Theory (Rogers 1995) encompasses a number of related theories explaining why people adopt (or don’t) innovations. The best known of these related theories are the perceived attributes.

The idea is that the how a potential adopter perceives an attribute influences whether or not they will adopt. The perceived attributes that have the biggest influence on adoption are:

Relative advantage.
The degree to which an innovation is perceived as better than the idea it supersedes.
Compatibility.
The degree to which an innovation is perceived as being consistent with the existing values, past experiences, and needs of potential adopters.
Complexity.
The degree to which an innovation is perceived as difficult to understand and use.

If you want students to make use of an online course presence then they must perceive the services offered by that course presence to be useful (relative advantage), easy to use (complexity) and something that meets their expectations of a university experience (compatibility).

The questions which arise from this include

What do students expect from their university learning experience?
What are their capabilities when it comes to technology and online learning?
What do they find useful?

This is one theoretical explanation for why you would expect online lectures, especially if implemented with a YouTube like interface, to be seen as a positive thing by students. In particular because most students still see lectures as a core component of a university education. They expect to have lectures.

This prediction is backed up by the findings of the Carrick Project examining the impact of Web-based lecture technologies. You can find particular mention of this in the projects publications.

Jones, Jamieson and Clark (2003) talk more about the use of diffusion theory for choosing potential web-based educational innovations.

This paper moves beyond the perceived attributes component of diffusion theory. These other components of diffusion theory offer a range of insights and potential advice for other aspects of this type of project. For example,

Innovation-decision – is the decision to adopt a particular innovation an optional, collective or authority decision.
Authority decisions enable the fastest adoption, but may be circumvented.
Communication channels – the nature of how information is communicated to people impact on the level of adoption.

The 7 principles

The 7 principles for good practice in undergraduate education were drawn from research on good teaching and learning and were intended to help academics address problems including apathetic students.

The 7 principles are that good practice in education:

encourages contact between students and faculty,
develops reciprocity and cooperation among students,
encourages active learning,
gives prompt feedback,
emphasizes time on task,
communicates high expectations, and
respects diverse talents and ways of learning.

It could be argued that the 7 principles are very specific, research informed advice about how to design activities and resources which students perceive to be useful and provide them with relative advantage. Which has obvious connections with the diffusion theory and TAM.

For example, principle 4 is “gives prompt feedback”. A design feature derived from that might be to return marked assignments to all students within 2 days. Based on my experience with students, they would perceive this as very useful and believe they gain an advantage from it.

This connection suggests that appropriate use of the 7 principles could significantly increase the use of an online course presence.

Implementation considerations – what about the staff?

The insights from diffusion theory and TAM also apply to the teaching staff and even the organisation. Teaching staff are critical to learning and teaching. If they aren’t positively involved it won’t work well. From an organisational perspective, anything that is planned needs to be doable within the resource constraints and also needs to be compatible with the organisation’s current structure.

Creating quality course websites

By admin

On July 12, 2008

In addie, c2d2, courseSites, elearning, Uncategorized

CQUni has an interest in increasing the quality of the course websites, as part of a broader push to improve the quality of learning and teaching. This post is an attempt to engage with some of the issues and develop some suggestions for moving forward.

There are many an argument why this particular focus on course websites might be somewhat misguided. For example, there is a growing argument (under various titles, including Personal Learning Environments) for the need to move away from this sort of institutional focus to a much greater focus on the learning environment of the individual students. While those discussions are important, as an institution we do need to have a pragmatic focus and improve what we do and our students experience.

So, this post and any subsequent project ignores those broader questions. However, that’s not to say that CQU isn’t engaging in those broader questions, we are. For example, the PLEs@CQUni project is aiming to examine some of the questions raised by notions of PLEs and social software and what impact they might/should have on institutional practice.

What is quality and success

Before embarking on any sort of attempt to “improve quality” you should probably seek to define what quality is. When do you know that you have succeeded?

There have been any number of attempts to determine quality standards for course websites. Many of these draw on guidelines from the educational research literature or from the Human Computer Interface, Information Architecture and other web/online related disciplines. I’m from an information systems background so, not surprisingly, I’ll draw on that background.

Within the information systems research literature how to determine success and consequently replicate it has received a great deal of attention. One of the problems this attention has established is that the notion of success is extremely subjective. An IT department will label something successful while a user of the same system may disagree strongly. The finance division may have yet another perspective. For this and other reasons the IS literature has moved onto using system use as a measure of success (Behrens, Jamieson et al. 2005).

i.e. a system or a tool is successful if there is large and sustained usage of it by people. Hopefully those people you intended. As you might imagine there has been subsequent research talking about the quality of that use. However, the level of use has been established as a fairly reliable measure of success.

I’m going to suggest that the level of student and staff use of a course website is a useful benchmark for the success, and even quality, of an online course. You certainly cannot impact on learning outcomes or students’ perceptions and experience through an online course presence, if they don’t make use of it.

If you accept this, then the question becomes what can we do to encourage use, to encourage success.

If you don’t accept it, and many people might, then the rest of this argument becomes almost pointless. If you don’t accept it, I would like to hear the arguments why it doesn’t make sense.

Encouraging use

Drawing on some of the ideas from previous post I’m going to suggest that there are two main approaches you can take to increase the use of a course website

Pragmatic – ad hoc changes to particular aspects of a course website.
Most academic staff make these changes in an unguided way. I’ll suggest that you are likely to obtain greater success if those ad hoc changes are guided by theories and frameworks such as the Technology Acceptance Model (TAM) and related work (Davis, 1989), Diffusion Theory (Rogers, 1995) and the 7 Principles for Good Practice in Education (Chickering and Gamson, 1987).
Re-design – where the entire course (and consequently course website) are re-designed by a project that returns to first principles.
Again, most academic staff I’m familiar with do this type of re-design in a fairly unguided way. I’ll suggest that re-design approaches informed by appropriate educational theories or frameworks are more likely of success. I’ll suggest the approach already being used at CQU, which uses Constructive Alignment (Biggs and Tang, 2007) and the 7 Principles (Chickering and Gamson, ???), work well.

The argument is that a well implemented approach that draws on either of these options has the capability to improve the use of a course website. However, I will propose, that the cost and level of success of each approach is different as shown in the following table.

The following table seeks to suggestion some potential characteristics of the two approaches as applied to an institutional setting. i.e. not an individual course, but a large collection of courses within a program, faculty or university. Obviously these are broad predictions of outcomes abstracted away from a particular context.

Characteristics	Pragmatic	Re-design
Level of use	A significant increase in use is possible	A really significant increase in use is possible
Quality of use/outcomes	Some increase in the quality of use/student outcomes but still largely reliant on the individual student’s capabilities etc rather than the actual course website	Potentially huge increases in the quality of use and student outcomes.
Difficulty/cost of implementation	Somewhat difficult but not all that expensive, large scale change in a university is always difficult.	Extremely difficult and expensive. Typically this will require the academic staff associated with the courses to radically rethink their conceptualisations of learning and teaching. This is not easy.

The suggestion

I’m hoping to expand on this further in subsequent posts as an attempt to outline a project by which an institution like CQU can significantly improve the use of its course websites and subsequent improve the learning experience of its students.

In summary, the proposal involves the following

A broad scale push for pragmatic improvement to course sites
A project to ensure that all course sites are informed by diffusion theory, TAM and the 7 principles in a way that is “automatic” and simple. Essentially, (almost) all course websites should at least illustrate these characteristics.
A process of complete re-design targeting courses with the largest numbers of students.
An on-going process by which the lessons and outcomes of the re-design are fed into the broad scale process of pragmatic improvement.

Understanding approaches to improving a course

By admin

On May 18, 2008

In addie, c2d2, elearning, paperIdeas, Uncategorized

Theoretically the group I work with is charged with helping staff at CQU improve the quality of their learning and teaching. To improve the courses they teach. A task that is not particularly easy for any number of reasons. Including the one identifed by Farhad Saba in this post. i.e. that academics by nature are a fairly solitary bunch and aren’t used to, or particularly like, work with other people to help improve a course. I know this from two perspectives, working where I am now and because I was just such an academic.

Arising from this preference for solitary work is that most academics generally have their own ideas, preferences and methods for improving their courses. This is tied to their conceptualisation of learning and teaching and also their past experience. This variety in how to improve a course raises some interesting problems for those charged with helping them. How do provide assistance to a large number of people who are doing things differently?

One step towards answering that question might be to try and understand the full spectrum of different approaches that people are using. Perhaps if you understand this variety, a way forward might arise.

The following post attempts to formulate a framework to understand the full spectrum of different approaches that I have seen used. The aim is to be descriptive, to understand the different approaches. The intent is not to limit understanding to “proper” instructional design approaches, rather it is to understand all of the different approaches that are used.

The Framework

Within the information systems discipline (and others) two by two frameworks are all rage for this type of thing. So why should I buck the trend. Here’s a first stab.

This framework concentrates on two particular dimensions:

When is the improvement decided upon (the y/vertical axis).
At some stage a decision is made about what improvement(s) will be made to the course. There are two extremes:
1. Emergent – where there is no pre-determined outcome or idea about how to improve the course. What will be improved in the course emerges out of the improvement process.
2. Pre-determined – the particular improvement is known prior to commencing the improvement process.
There is a lot of space between the two extremes.
The type of improvement process
This dimension is concerned with the nature of the particular process used to improve the course. There are two extremes:
1. Pragmatic – the process is aimed at making some form of pragmatic improvement to a particular aspect of the course. Much, even most, of the course will not change.
2. Re-design – the aim of the improvement is a complete is typically aimed at a complete re-examination and re-thinking of the course.

Examples

Many 2×2 frameworks encapsulate 4 categories and seek to place examples into a particular box. This is not that type of framework. In part because this is a very rough, early idea and also because I’m not sure that the area of interest (how do academics seek to improve their courses) can be understood with specific categories. Each of the two dimensions in the framework are meant to represent a spectrum of possibilities. The framework is more a graph.

The placement of the following examples into the framework are meant to be broadly representative and illustrative and not quantitatively calculated, exact placements. I also expect that there might be some disagreement about the placement of some examples.

Constructive Alignment

Constructive alignment is a common type of re-design process and is the preferred method for CDDU.

Constructive alignment seeks to change the fundamental design of a course. It generally questions the conceptualisation of learning and teaching of the teacher. So it’s high on the re-design dimension.

The outcomes are pre-determined to some extent. The constructive part of alignment assumes that learning is best achieved when the student is actively constructing meaning. That “pre-determination” still allows for emergent design, as long as it tends to stay within the constraints of the constructive perspective. The alignment component also provides a bit of pre-determination.

NCAT Program in Course Re-design

The NCAT Program in Course Re-design is also high up the re-design end of the process type dimension. In fact, one of the six characteristics common to these types of projects is “Whole course design” (Twigg, 2003).

However, a number of the other listed characteristics (e.g. active learning, mastery learning, computer-based learning resources) also act to increase the pre-determination.

ADDIE Model

The ADDIE Model is a generic design process (more a design model) and consequently doesn’t pre-determine anything. Knowledge found during the analysis, informs the design which identifies what will be done. However, it might be argued that due to the nature of human beings that such a generic process will be pre-determined due to preferences/experience of the designers involved in the process. But I’ll ignore that for now, this argument could be said to apply to all approaches.

However, ADDIE doesn’t necessarily have to produce a complete re-design. It might just result in a simple and small modification of one aspect. It can range over a broad area of the type of design process dimension.

Problem-based learning

Using problem-based learning as basis for improvement will typically be a re-design process, at least if PBL isn’t already being used.

The outcome is, to some extent, pre-determined as use of PBL limits the possible improvements to those which are consistent with the beliefs/assumptions of PBL. Though, as with constructive alignment, this area of pre-determination is not exactly narrow.

Choosing a new textbook

The purists may not see this one as valid. However, it’s probably the most common approach in certain places.

As a process, it generally involves looking at all the relevant textbooks, choosing one and then modifying the content of the course to fit. Hence this is not likely to be a re-design of the course.

It’s also a fairly high on the pre-determined dimension as this approach typically assumes that the course will not change. The improvement is the new text and the use of its associated resources (lecture slides, quiz questions etc.). There is also not likely to be large change as most textbooks assume pretty much the same pedagogy (lectures, tutorials, assignments, exams).

Strategic approaches

This is what I’m calling the situation where the leadership of an organisational unit (be it program/degree, school, faculty, or entire university) decides to adopt a common approach to all courses. This might include a particular pedagogical approach (e.g. PBL) or delivery model/approach (e.g. all courses will have a website that follows a particular template).

Such an approach is generally pre-determined for all courses offered by the unit. The purpose of a strategic approach is typically to achieve a common goal for all courses.

These type of approaches can run the full spectrum from pragmatic to re-design depending on the nature of the type of improvement (from brand new pedagogical foundation through to adoption of a particular web site template).

Solution driven approaches

These are the type of approaches which can be said, from the most cynical perspective, to be “solutions looking for problems”. These types of approaches often arise from fads. Someone becomes aware of a new idea, approach or technology and decides to use it within their course. The Web, web 2.0, Second Life etc. are just some of the technical examples of this approach. But it’s just not technology. Small tactics or approaches (e.g. minute papers) can also be used this way.

These types of approach are typically pre-determined. It’s been decided to use the particular solution already. It’s just a matter of how.

They are also typically pragmatic. It’s only this particular solution that is being adopted to improve an aspect of the course. Not a complete re-design.

Extreme learning and teaching

The nascent idea of extreme learning and teaching is a one of mine and still somewhat questionable and under development. However, it is a little different and I thought I’d include to see if this might help me think more about where it fits.

Extreme learning and teaching takes it’s basic idea from extreme programming.
In particular the idea that there are some known practices/principles in software engineering that work. So, let’s take those to the extreme. Have a development process that focuses on them.
Extreme L&T draws on Chickering and Gamson’s 7 principles for good practice in education as the principles to maximise.

The idea is to identify how you improve a course you look at the 7 principles and seek strategies to maximise them.

Such an approach is to somewhat pre-determined, it has to maximise at least one of the 7 principles. But that is still a very large area of pre-determination.

It’s also likely to be a fairly pragmatic approach. It’s likely that entire courses wouldn’t be re-designed, but instead small, evolutionary changes. That said, it might be possible to re-design a course (just a bit unlikely).

Futher thoughts

How can this sort of thing help?

A group like CDDU, seeking to help people improve their courses, should be aiming to offer support for the methods that are being used to improve courses. Different areas along both dimensions of this graph/framework require different types of support. If you know where the approaches used by people are located you can target resources to provide the appropriate types of support.

Similarly, different parts of the framework are also going to have different characteristics, strengths and weaknesses. For example,

Approaches with high levels of pre-determination will not be able to adapt to differences in courses and context.
Approaches with high levels of re-design are likely to be much more difficult and time-consuming.
Very pragmatic approaches may not achieve large improvements in learning outcomes, but they’ll be a lot easier to implement.

The purpose of this type of framework is to help improve understanding. Being able to show people where their approach to improving a course fits and explain its likely needs in terms of support, its characteristics

A model for evaluating teaching – a useful lens?

By admin

On April 29, 2007

In addie, asciliteMentor, elearning, Uncategorized

As part of CQU’s moves around improving its evaluation of good teaching I was sent a paper by Keith Trigwell (2001), “Judging university teaching”.

From that paper comes the following “model of teaching (or a framework for judging teaching)”

My interest in this model is the assistance it can provide in thinking about how a group charged with helping academics design, develop and delivery quality learning can understand and go about it’s task. This is a problem I face in my new job.

I think it also offers a useful framework for thinking about literature that may help such a task. Also in framing the research and evaluation tasks which such a group might do to help achieve its goals.

Time to think more deeply about this and whether or not it is really useful, comes later. Time now to write down some of the initial ideas the model has generated, before they’re lost due to the fallibility of an aging memory.

The initial ideas include the following (some expanded below):

Research reinforcing the primacy of the academics’ influence of the student experience
A colleague has already done some surveys around online learning in which one of the questions really reinforced this view. An area for more research and literature review.
It’s more than what the student wants.
Knowing the current state of affairs.
The fine line between manipulation and assistance.
Investigating the model, fleshing it out in more detail.

All of this is framed around trying to understand the problem

How can you effectively and efficiently bring about change for the better in university learning and teaching?

It’s more than what the student wants

While it is important to understand what the student is experiencing and the preferences, desires etc of the student body. This, by itself, is not sufficient to make any meaningful change.

There needs to be an emphasis on how to communicate this knowledge to the teaching staff. Thought needs to be given about how the knowledge of the students’ preferences and experience can be appropriately harnessed/disseminated to change the teaching and learning context and the thinking, planning and strategies of the teaching staff.

If you don’t change these, then the student experience won’t change to any significant level. – there’s a question to be researched.

Knowing the current state of affairs

A group that is attempting to improve (i.e. change) the learning and teaching practices of an organisation and its members (leaving aside, for the moment, questions about the ethics and desirability of such a group thinking it can, or even should, improve/change such things) really should have a good handle on the current state of affairs.

In talking about “good teachers” Trigwell (2001) says that they “recognize the importance of context, and adapt their teaching accordingly; they know how to modify their strategies according to the particular students, subject matter, and learning environment.”

Obviously, at least for me, the same applies for the group trying to improve learning and teaching. It needs to know the current state of affairs with respect to teachers’ strategies, planning, and thinking, as well as the teaching and learning context.

The institution’s teaching and learning context implies both internally and externally to the institution. So knowledge of technology, government and other external influences on the practice of learning and teaching is needed. As is knowledge about the institution. Most importantly, perhaps, is that this needs to be based on the perceptions of the teaching staff. Whether or not senior management believe something may be beside the point in terms of influencing student learning.

The line between manipulation and assistance

It is possible that some of the above can be seen as a framework for understanding how to manipulate academics to achieve the goals of the few. You could certainly use this approach for that. I would hope that’s not what we’ll do, but it is a fine line between descriptive and normative development.

Fleshing out the model

As I look more at the model, a few questions appear.

Why are the arrows to the outer “skins” apparently one way?

Why is there only one arrow to the teaching and learning context? It’s as if the student influences and is influenced by all the four outer skins. But Teacher’s thinking has a much more limited influence.

References

Keith Trigwell, Judging University Teaching, The International Journal for Academic Development, 6(1):65-73

Should the learner always be the focus?

By admin

On February 9, 2007

In addie, elearning, emd, Uncategorized

A discussion of ITForum provided a pointer to an interview with Badrul Khan (who provided the link). In it he suggests that instructional design deals with just 2 dimensions (pedagogy and evaluation) of 8 dimensions required to create learner-focussed learning material.

It includes the observation that the learner must always be the focus of instructional design. I agree, but with a reservation that is linked to my new job in instructional design.

My reservation is based on the following observations

The idea that the learner must always be the focus of instructional design is an innovation for most university faculty.
The diffusion theory literature, along with a lot of other literature, indicates that you will not get adoption of an innovation by simply coming along and saying, “This is better than what you are doing, you should really do it.”.
The person making the adoption decision in instructional design is each individual faculty member. Regardless of whether or not they are being “required” to do it, they still make a decision.

Hence, I tend to think that the focus of instructional design (defined as what we do to help academics design instruction) is on encouraging the faculty member to adopt the innovation that “the learner should be the focus”.

Diffusion theory has been criticised for having a pro-innovation bias that, amongst other effects, can separate members of a social system into the superior innovators group and the inferior recalcitrants group (McMaster and Wastell, 2005). i.e. you are a bad boy if you don’t accept the innovation.

This is not what should be aimed for, instead something like the following

Here we would argue that the innovation succeeded due to internal development, through a participative process involving strong local leadership, engaged staff and the fortuitous occurrence of a series of local crises that aligned all stakeholders around the need for change.

References

McMaster, T., & Wastell, D. (2005). Diffusion – or delusion? Challenging an IS research tradition. Information Technology & People, 18(4), 383-404.