Month: March 2010 Page 1 of 2

The following is the extended presentation abstract I plan to submit to MoodleMoot AU 2010. The idea was to submit a paper, but time has run out. The recent blog posts (starting with this one) about the story of BIM provide some of the early reflection that will form the basis of the presentation. The challenges mentioned in those posts will be abstracted somewhat to generate a series of limitations.

In part, the approach I am taking with this presentation is to respond to the polyannas who complain about me being too negative, and never seeing the positives. It’s always been my argument that what I do is not to ignore the positives, recognising and reusing what’s worked forms an important part of my information systems design theory for e-learning. For example, as far back as 1999 I have three publications (1, 2, 3) where recognising and reusing the positives is a key feature. It has been my argument that the polyannas are so busy focusing on the positives because they don’t want to recognise and engage with what doesn’t work. It’s a case of “don’t mention the war”, the SNAFU principle, confirmation bias and pattern entrainment, defensive routines and the lack of a willingness to question the practices on which ones self esteem is built. For me, it is only through recognising, understanding and addressing the limits that you can encourage innovative learning and teaching. You have to recognise and respond to the context.

Abstract

Open source Learning Management Systems (LMS) such as Moodle are widely recognised as addressing a number of the limitations of proprietary, commercial LMS. Just as those commercial LMS addressed some of the limitations of the “Fred-in-the-shed” era of early web-based e-learning. Early web-based e-learning addressed some problems with text-based Internet e-learning, which addressed limitations of text-based computer-mediated communications…and so it goes on. Rather than being “without limits” this presentation will suggest that e-learning with Moodle, as currently practised, has a number of limits and that progress can be made through the recognition, understanding and removal of those limits.

The presentation will argue and illustrate that these limits place significant barriers in the way of encouraging widespread, simple improvements in learning and teaching. Let alone the barriers these limits create for the development of true pedagogical innovation. The presentation will explain how these limits are not solely, or even primarily, due to the characteristics of Moodle. It will outline how the majority, but not all, of these limits arise from the nature and characteristics of the broader social context, institutions, purpose, processes and people involved with e-learning. It will show how a number of these limitations have been known about and generally ignored for decades, to the detriment of the quality of learning and teaching. The presentation will also seek to identify a variety of approaches or ways of thinking that may help transform the practice of e-learning with Moodle into something that truly is “without limits”.

The presentation’s argument, the identified limitations and the potential solutions all arise from and will be illustrated by drawing on the experience of developing BIM (BAM into Moodle). BIM (https://djon.es/blog/research/bam-blog-aggregation-management/) is a Moodle module released in early 2010 and currently being used at CQUniversity and under consideration by the University of Canberra. BIM allows teaching staff to manage, mark and comment (privately) on individual student blogs that are hosted on the students’ choice of external blog provider. BIM is based on BAM (Blog Aggregation Management) a similar tool integrated into CQUniversity’s home-grown “LMS”. Since 2006, BAM/BIM has been used in 30+ course offerings, by 70+ staff, and with 3000+ students making 20000+ blog posts.

The design of BAM/BIM is intended to remove an inherent limit that underpins the design of all LMS. That is, as an integrated system, the LMS must provide all functionality. It appears that this is a limit that the design of Moodle 2.0 seems focused on removing. However, this presentation will suggest that this is only one of many, often fundamental, limits surrounding the use of Moodle, and that these limits need to be recognised, understood and addressed. The suggestion will be that it is only by doing this that we can aid in the development of truly innovative pedagogy.

Episode 3 of the story of BIM

A descent into Limbo

The initial development and use of BAM was in a single course that finished around November, 2006. This was the last course I was to teach. By early 2007 I had applied for and been selected as the new Head of E-Learning and Materials Development at the same institution. While still an academic role, I would no longer be teaching. Instead I would be the supervisor of a group of 10-20 staff who were responsible for various tasks including curriculum design and desk top publishing. It wasn’t until the middle of 2007 that I became responsible for the support of CQU’s official LMS (Blackboard).

2007 was the start of a significant period of uncertainty. By the time I took over e-learning the Director who attracted me to the position had not had her contract renewed. While I was the designer of Webfuse and very critical of the nature and implementation of traditional LMS (like Blackboard), I found myself in the position of being responsible for Blackboard, but not Webfuse. At around the same time a new Director of information technology was appointed and there was a major re-alignment of IT. i.e. All IT should be the responsibility of the IT division. Various other factors contributed to on-going uncertainty about how to achieve anything related to e-learning. Perhaps the biggest was the perception that the institution was in financial peril and subsequent decisions not to renew contracts.

CHALLENGE #13 A context of increasing uncertainty and reducing funding does not contribute well to the development of innovative pedagogy.

In addition, there were significant shortcomings in the existing processes and outcomes underpinning how materials development and e-learning were currently being done. Not to mention, that after many years absence, it was necessary to figure out how curriculum designers could once again be incorporated into the institution. It was thought that this should be the focus of what I was doing, not on further developments of BAM.

To give some idea of the level of uncertainty. I was employed as the Head of E-learning and Materials Development. But the unit I was in charge of was called the Curriculum Design and Development Unit (CDDU). As stated above, CDDU did not initially include responsibility for e-learning. In addition, CDDU had no official input into the institution’s learning and teaching grants process.

In this context, there were no plans to complete development of BAM. Instead, ad hoc support would be provided to people who wished to use BAM with the understanding that once the situation became more certain decisions could be made about what to do and what not to do. This never really happened.

Subsequent use of BAM, 2006-2009

The following table gives a breakdown of BAM usage by each term. It summarises the number of courses using BAM, the number of staff and students in those courses and the number of blog posts managed by BAM. In summary, during this period 69 staff used BAM to manage 19969 blog posts made by 2789 students.

During this period, there was essentially no publicity for BAM. The staff who chose to use BAM in their courses fell into the following categories:

• They were lumbered with BAM because they were employed to run a course that had already been set up to use BAM.
• Were guided to the use of BAM by someone working within CDDU.
• Based on an earlier experience of BAM, they wanted to use BAM in another course.

The course for which BAM was initially developed was particularly was taught be a different person in each of the following 6 offerings. Only on the 7th subsequent offering was it taught by someone who had taught the course while BAM was being used. In practice, this meant that each of the people teaching the course were using the assessment designed by the previous person. Not assessment they had designed. It was on this 7th offering, that the decision was made to drop BAM from the course.

The limited consistency in responsibility for the course makes it difficult to develop a sense of ownership of the course. This can limit full engagement with the intent of innovative pedagogy. The courses where BAM has been most successful, has been where the coordinator/designer of the course has been instrumental in decision making. “Ownership” of a course and the decisions around its design are increasingly difficult in a context where courses are offered multiple times a year and where, increasingly, teaching staff are casuals. Especially with the application of innovative pedagogies which are challenging the status quo. It often takes time to become familiar with the innovation and the ramifications of the changes it creates within the context of a course.

CHALLENGE #14 A context where there is limited consistency in responsibility for a course, make it difficult to learn about the problems and evolve an innovative pedagogy.

From the start an eventual interest of mine in developing BAM was to explore more social applications of the blog technology. Especially, investigating applications that lean more to co-operation than collaboration as suggested by Stephen Downes. However, once I no longer was responsible for the course, I no longer had the “power” to make that decision about a course. It was up to the coordinators.

CHALLENGE #15 Innovative pedagogy is limited by the conceptions, beliefs, desires and aims of the coordinators of a course.

The hobbled BAM

A common theme underlying the design of BAM was that in its first use I would be the coordinator and I had command line access to the server on which BAM was running. Not to mention the expertise to use it. That meant that it wasn’t necessary to develop a web interface for the coordinator/management tasks for BAM. Examples of these tasks included: configuring a course to use BAM, releasing marked posts to students, running copy detection over student posts, changing students registered blogs and merging BAM results with the institution’s end-of-term results processing system.

Since the limbo period resulted in no changes to BAM, coordinators of the above courses had to make do with BAM as is. There were unable to perform these tasks. Instead they had to ask me to perform them on the command line. This introduced delays.

??What’s the challenge here, there is one, how to phrase it?

Most of the courses that were using BAM during this stage were being taught across most of the institution’s 9 campuses (spread across the eastern seaboard of Australia) and by distance education students. The institution and each of its 9 campuses have an infrastructure and set of processes around offering support for information technology ranging from desktop support through to e-learning and beyond. This infrastructure did not recognise BAM as a centrally supported IT system. It could also be argued that few people within this infrastructure knew anything of the centrally supported IT system on which BAM was based. This lack of knowledge led to some confusion when teaching staff new to BAM asked their local IT folk about problems they were experiencing.

It is difficult to see how an IT infrastructure across multiple campuses, run under different regimes could fully understand and respond to all applications of innovative blended learning. How many courses does an innovation need to run across before it become cost effective for the IT infrastructure to disseminate training about that innovation to its IT support staff? 10, 20, 300, 1000?

CHALLENGE #16 Innovative applications of blended learning often operate at a scale that does not match what is required by an enterprise IT approach to support.

In the case of BAM, this was somewhat increased by the reliance on external blogs and RSS. The majority of staff and students had little familiarity with these technologies and vanishingly small numbers had actually create a blog prior to this course. What about the net generation. Aren’t students already meant to be familiar with the Internet and social media?

CHALLENGE #17 The vaunted net generation doesn’t quite appear to have arrived.

It’s extra work

All of the staff who have used BAM comment on the extra amount of work it involves. This perception arises from a number of different sources, including:

• Blogs, bams and reflective journals are all new to most of these staff and their students. Novelty means more effort to become familiar and find out the strategies and tactics to work around the flaws.
• The design of BAM itself was not always the best and added to the amount of work.
• Over reliance on old mindsets or a misunderstanding of the rationale behind the use of BAM.

There appeared to be a correlation between how negative a staff member was about BAM and how strongly they viewed BAM simply as another assignment to be marked. A significant number of the staff using BAM are employed as casual academic staff. This means that they are given a fixed number of hours to lecture, tutor and mark based on the number of students they are responsible for and the subsequent number of tutes. For example, if 25 students fit in a tute and you have 50 students you are paid for 2 tutes per week. Your payment for those 2 tutes includes payment to do whatever marking of assignments you are expected to do.

Courses using BAM almost always included BAM usage as an assignment that the staff had to mark. In the case of COIS20025 this was an extra assignment on top of the normal two assignments. This mean staff had to mark an extra assignment without any extra pay!

CHALLENGE #18: Fixed methods for the calculation of workload and subsequent payment to causal teaching staff limit the flexibility needed for innovative blended learning pedagogy.

Not surprisingly, these staff saw BAM as yet another assignment, and one they weren’t being marked for.

On the other hand, the coordinators who pushed hard for BAM saw BAM as an instrument to improve learning and teaching. To address fundamental problems with learning in courses. To make teaching more student centered. To make student thinking more visible in order to allow staff to help students be aware of any limitations before submitting assessment that would be marked.

The coordinators choosing to adopt BAM typically did so from a “visionary” or “quality” perspective. Other staff saw it through a more “pragmatic” perspective and consequently there experience was very different. Many, if not most, students also adopted this more “pragmatic” perspective and saw BAM as yet another assignment in which they applied there value logic. i.e. how much is the assignment worth in raw percentage terms, what do I need to do to get the percentage I need.

It was common for similarly pragmatic students to ask why they need to do this on a blog. Couldn’t they just do it on paper or a Word document? Why do I have to learn this new technology? Why all this extra work?

CHALLENGE #19 Innovative pedagogies are perceived in very different ways. The differences in perceptions influence how people engage with them.

This is the second episode in the continuing story of BIM. This episode picks up the story late in the first half of 2006. The initial foray into using individual student blogs to encourage reflection and make student learning more visible was bumbling along. It was now time to think about what I’d be teaching the following term.

Introducing COIS20025, Systems Development Overview

Since 1990 I had been teaching within the information technology program at CQU. For various reasons I was no longer happy teaching within that program, one of the reasons is that through my work in e-learning I’d realised that there was a lot more to technology systems, than technology. You can make a beautiful bit of technology, but if it involves change on the part of the users, they won’t use it they way you intended (if at all). As a result, from the second half of 2006 I was moving into the information systems program. Theoretically, information systems is interested in the combination of software and otherware (i.e. the people, organisations etc.).

My first teaching task within the information systems program was the course COIS20025, Systems Development Overview. It was a Masters level course looking intended to give an overview of how information systems were developed (somewhat surprisingly – though not really – it tended to focus on the technology and methods and very little on the otherware). Traditionally the course was taught across multiple campuses by 10+ teaching staff and 200+ students. I would be the course coordinator responsible for managing all this and in addition teaching the students at the Rockhampton campus and those studying by distance education.

Check what early emails I have about this course. And my move to the information systems school. Give the background here on the course.

Thinking about COIS20025

By early 2006 I was starting to look at and think about what I might have to do with COIS20025. History showed that there were some concerns about the course. For example, a previous coordinator had received a “please explain” about why there was only 1 student getting a HD from 384. The response from the coordinator explained this drawing on reasons that were widely recognised by those at the coal face. i.e. that the majority of the students came from a non-English speaking background (NESB), most had little familiarity with technology, COIS20025 was taken in the first term they were in Australia getting to grips with a foreign culture and these problems were made worse by the fact that 73% of the students in this offering enrolled in the course after term had started. In fact, 21% of the students were enrolled on the last day of week 2 of term. Supposedly the last day students could enroll. A further 9% of students were enrolled after week 2.

Some comments from the previous coordinator

there is nothing in the course itself that makes me believe that it is too difficult for Post-Graduate students to manage…I can not see what else can be done to improve the results in COIS20024 and COIS20025 with the exception of capping the number of students we enrol and haivng a more stringent entry requirement for English Language Skills.

It could also be observed that the methods used in the course differed little from traditional methods from the glory (and probably mythical) days of higher education of motivated students.

This particular tension between teaching staff and the management of teaching staff is not new or specific to this course. My first experience of it was the late 1990s and have seen it in numerous courses. To this day I see no evidence of significant and successful changes to address this problem.

CHALLENGE #6: The increasing complexity of the teaching context within some Australian universities has resulted some significant tension and differences of opinion between management and teaching staff. Many of these remain unresolved.

While the changing nature of the students and the context suggests a need to change toward more innovative methods that engage the students, that address the limited performance of students. Tutty et al (2008) capture the more common approach

The solution to the high failure rate was to change the assessment to satisfy the institutional requirements of satisfied students and reasonable pass rates rather than explore an alternative learning and teaching approach – an effective solution in the current higher education environment that encourages the academic to prioritise other areas, such as research.

CHALLENGE #7: The context within universities is not favourable to exploration of innovative, alternative pedagogies.

Reflective journals – the Word document approach

During 2005, in an attempt to encourage student reflection and hopefully encourage a higher level of learning an assignment was introduced that required students to keep a “reflective” diary. The diary was submitted at the end of term for review by staff. There were two problems observed with this approach: creative writing and plagiarism. Since the diary was not submitted until the end of term, it was felt by teaching staff that many students left the diary until a last minute exercise in creative writing. There was also problems with plagiarism. The second offering of COIS20025 in 2005 had 62 plagiarism incidents.

To address these sorts of problems students were meant to submit their diaries three times during the term to teaching staff for checking. This practice was never widely implemented due to workload and resource problems. In particular, this is a problem for teaching staff at the international campuses who are either casual staff employed for a specific number of students and a particular set of assumptions, or they are full-time staff who operate under a high workload.

CHALLENGE #8: Workload pressures limit the adoption of changes in pedagogy.

The problem

Introduce the problem with the reflective journal assignment.

Back in 2006 I had a problem. I was about to take over a post-graduate Masters course in System development. The course was taught across multiple campuses by 10+ teaching staff and taken by 200+ students. Most of the students were from a non-English speaking background. In order to encourage reflection and higher order learning, the course had one assignment that required the students to keep a reflective journal. Here’s where the problem arises.

Students were told to keep the journal as a Word document and submit it at the end of the term. Consequently, most students did not work on the journal during term, leaving it to the last minute. Teaching staff did not see what was written in the journal and could not respond or comment. The course coordinator – located at a campus some distance from where the majority of the students were located – certainly couldn’t see what was going on. The limited integration between what the student was doing and the marking of the journal probably contributed to significant levels of plagiarism that occurred with the assignment.

How to make it better

Given my interest and experience with blogs, this seemed a perfect fit. In addition, at this stage this course was offered using CQU’s home-grown LMS, Webfuse. The Webfuse assignment submission system was already heavily used in this course, it was part of the way things were done. In addition, I am the main designer of Webfuse and as late of 2004 was the Webfuse team lead. This mean I could see how easy it would be to add a feature to Webfuse that would allow the use of blogs in a large course to be managed in a more efficient way. This feature was to be known as Blog Aggregation Management – BAM.

CHALLENGE #9 It was only through a unique combination of contexts that the idea for BAM was identified. Remove any one of these conditions and BAM probably would not have happened.

After 10 years of working on innovative e-learning through my own interest I was not going to develop BAM on top of the rest of my workload. Especially since there remained large uncertainty about the future of Webfuse (Webfuse eventually was repalced at CQU in 2010). In addition, at this stage my current laptop was getting on a bit. I needed a new one and university computer replacement policy would result in me getting something less than nice. I needed some money to top this up.

The previous coordinator of COIS20025 and I collaborated on a proposal to develop BAM. This was put to the faculty management and included the suggestion that I be given $2000 to develop BAM. The proposal was accepted.

CHALLENGE #10 The development of innovative blended learning pedagogy requires additional funding. Innovation costs initially.

The decision to support the development of BAM was, from my perspective, not a logical thing for the organisation to do. I believe it only happened because a small number of the players within faculty management at this point in time, had positive impressions of Webfuse, myself and the idea of BAM. Had this proposal been considered at another point in time, with different people in those positions, I believe the outcome would have been different.

The initial development and use of BAM

By the 8th of May 2006 I was running a “design workshop” for BAM. An hour long presentation to university staff proposing the idea and describing prototypes of how it might work. The aim was to get the idea out there and get feedback. The video of that presentation can be viewed on Google Video. I need to look through this and the powerpoint slides and find where I was up to with my thinking

On the 11th of September, 2006 a follow up presentation was given to report on the lessons learned so far in the use of BAM in the course. It includes a summary of what has happened as of Week 8 of (a 12 week) term:

• 258 of 278 students have registered blogs;
• Marking of blog posts not done, mostly due to the late arrival of the marking facility within BAM.
• Staff overview of the blogs has been patch.

A significant component of the use of BAM in COIS20025 for this term was spent in designing the questions students were asked to respond to on their blogs. It wasn’t just the technology. The design had moved beyond just using minute papers into some specific questions intended to make visible the students’ preparation for the assignments. The aim was to enable staff to make comments before assignment submission, reduce plagiarism and attempt to increase higher-order learning.

Problems

For various reasons, I do not believe that these aims were achieved on a broad scale. In part, this is because the COIS20025 students were very pragmatic, the majority did what what was required to get the mark they wanted with a minimum of effort, rather engage completely with the intent.

CHALLENGE #11: A significant percentage of students are very pragmatic. They don’t automatically engage with the aims of innovative pedagogy.

Another significant contributor was “change management” issues. Essentially, most staff did not look at student blogs during the term. They were left until marking was done at the end of term. Two main issues created this: the late finish of the interface staff would use to mark the blogs, and workload issues for the staff. While BAM may have reduced the actual work required to mark student journals, the novelty and uncertainty around the system meant this was not how staff perceived the system.

CHALLENGE #12: New or novel pedagogies are always perceived to require more work from participants. This is simply because of the need for participants to become familiar with the novelty. Novelty on top of an overloaded or overly pragmatic set of participants limits engagement.

Successes

One of the more engaged members of the teaching staff made this observation in interviews

Students tend to fall off the wagon…the major reason why the blog was created. So that we can keep track of students working week by week rather than…working the night before the assignment is due…in that respect it seems to have worked well.

The blogs also appeared to confirm Du and Wagner (2005) and provide a useful predictor for learning outcomes for high and low performing students. Of the 9 high performing students in COIS20025, all 9 performed highly on their blog posts. Of the 6 students who did not register a blog (obvious by week 4 of term), 4 failed every course they studied that term and 2 failed two of the three courses they were studying.

Technically the system worked well. Students had a blog without the institution needing to provide a blog service. The management system, once in place, enable marking and tracking of student posts to occur and integrated with institutional management systems. The project was included in the ELI Guide to Blogging (Coghlan, Crawford et al. 2007) released in 2007 which indicated that the marriage of Web 2.0 applications with institutional systems was a compelling part of this project and that it promised to give institutional systems greater efficacy and agility with reduced cost.

More details

More detail about the initial use of BAM in COIS20025 can be found in two publications. One, written in 2006 just after completion, and another written in late 2008 after a period of some reflection.

The following is a start of trying to capture the story of BAM and BIM. The aim is to use this as part of a case study for a paper for MoodleMoot’AU 2010.

As well as trying to re-create the story of BIM from various publications, email archives and presentations the aim of the following is also to identify the challenges faced while developing BIM. This will be the first in a sequence of posts as I attempt to re-create the history.

Origins: Blogs and minute papers

I’d first used blogs in teaching around 2004. Had been aware of them for a while. By the end of 2005 I was back teaching first year programming in courses that were taken by distance education students, traditional on-campus students and international students at capital city, commercial campuses. I was directly responsible for the on-campus and distance education students.

For a long time (going back to the early 90s) I’ve been concerned with the plight of distance education students and the limited support they receive. In an attempt to achieve a couple of aims I wrote a paper titled “Enhancing the learning journey for distance education students in an introductory programming course”. It was intended to try and reinvigorate the REACT process while at the same time giving me the reason to reflect on how I might support the distance education students better and reduce the failure rate in the first year programming courses.

A part of the considerations was the use of blogs. Especially in combination with the idea of minute papers (Murphy & Wolff, 2005). The idea was described as this (Jones, 2005)

The author has used minute papers in face-to-face teaching and found them to be useful. It is thought that asking distance education students to blog a minute paper each time they do some study will provide a minimal level of structure and help the coordinator be aware of how each student is progressing.

Implementation plan

I implemented some of the ideas, including the use of blogs, described in Jones (2005) in the first term of 2006 in a course called Procedural Programming. Before the start of term I emailed all of the distance education students taking the course explaining how and why I wanted to use the blogs to support them. The reward to students for blogging minute papers was that I would be responding directly to their quotes. This included the offer of a pre-submission check of their assignments. i.e. before they submitted, I would look at their assignment and give them feedback.

There were only 17 or so distance education students, so while it was more work, I didn’t think it would be that great. However, I did recognise that it would be additional work, I thought it worthwhile as it would help the students, but there was no way to get recognition for this extra workload. Even though the rationale for doing was sound.

CHALLENGE #1: Organisational processes and policies, especially workload, don’t easily respond to changes in workload balance.

Checking my email archives for the course, I find the first mention of this blog. Apparently, I set up this blog to serve as an example for the students in this course. Here’s the first post reflecting on the early stages of this process.

Those same email archives reveal 3 students sending me the URLs for their blogs.
1 of those blogs is empty, no posts. Another has a couple of posts from the student and some comments from me. I remember this student. We ended up moving discussions to email and those discussions were quite extensive. The last blog had a regular post each week, for most weeks, and also had a comment from me on each one. Is that a 33% success rate? Of the students one failed (no posts), one got a credit (2 posts) and one got a HD (lots of posts). Is that a correlation between engagement on the blog and final result? It’s been since suggested that blogs are a good predictor of student results for high and low achievers.

Another student at this stage commented in an email

I dont know anyone who uses Blogs, and i have never read or used one myself.

CHALLENGE #2 Blogs are novel to many students. Students are conservative, getting them to try something new is a challenge.

Other comments from the same student

If you were offering me presubmission checks in return for a weekly minute sheet/blog i would do it. I can see how a student that was lacking confidence would feel more secure posting a message that only the lecturer would read. The only problem that i see with not having a communal learning environment is that if the lecturer cant perform their duties in a timely manner there is no one else that is aware of the students problems. The onus is fairly heavily on the lecturer to perform.

CHALLENGE #3 A design that relies on the academic to perform, will create a workload and a single point of failure.
CHALLENGE #4 Students are pragmatic. There needs to be reason for them to do something, especially something new.
CHALLENGE #5 The open nature of blogs can create some fear, especially for those who are less confident (this may/could be linked to what is known about shy students posting to discussion forums i.e. they tend not to)

The same student commented

The thing that i found that reduced the “transactional distance” was simply having an email/post/message from someone that called me by my name. The feeling that someone knows your name rather than your student number made a more profound difference to my study experience than i would have thought likely.

One of the points to remember is that under this model the distance education students were receiving a different type of support than the on-campus students.

At the moment, it appears that CQU’s course evaluation system has no record of any evaluations for this course. Will have to dig into that.

Conclusions

On the whole, this use of blogs was not a great success in terms of overall numbers. However, it did provide an opportunity to experiment, to test the waters. Obviously this experiment was largely positive because of what came next.

References

Jones, D. (2005). “Enhancing the learning journey for distance education students in an introductory programming course.” from https://djon.es/blog/?attachment_id=496.

Murphy, L. and D. Wolff (2005). “Take a minute to complete the loop: using electronic Classroom Assessment Techniques in computer science labs.” Journal of Computing Sciences in Colleges 21(1): 150-159.