Assembling the heterogeneous elements for (digital) learning

Month: February 2016

Open educational practice beyond the institution?

The following is a continuation of prior ponderings about about open educational practice (OEP) and open educational resources (OER) in the context of a potential institutional grant application. It’s an attempt to make sense of some of the relevant literature I’ve read and figure out how that might influence the nascent project. In particular, it suggests that some of the OER/OEP literature is limited due to its focus on: OER; the individual; the institution; and, on searching as the means of discovery.

It leads to a nascent idea of a project to address these perceived limitations.  i.e. it’s a project that is not limited to one institution; it’s not focused on the individual actor, but on the connections between them; and, it’s aiming to explore if and with what impacts the OER/OEP can be embedded into existing networks.

Defining some terms – OER and OEP

The OECD (2007) defines OER as

digitised materials offered freely and openly for educators, students, and self-learners to use and reuse for teaching, learning and research (p. 10)

The focus on open resources is perhaps most widely represented in the concern over the cost of textbooks for University students. A problem for which open textbooks are seen as the solution.

This is problematic and is explored more in the next section.

One reaction to these problems has been interest in Open Educational Practices (OEP). OPAL (2011, p. 12) define OEP

as practices which support the (re)use and production of OER through institutional policies, promote innovative pedagogical models, and respect and empower learners as co-producers on their lifelong learning path. OEP address the whole OER governance community: policy makers, managers/ administrators of organisations, educational professionals and learners.

The Capetown Open Education Declaration offers

open education is not limited to just open educational resources. It also draws upon open technologies that facilitate collaborative, flexible learning and the open sharing of teaching practices that empower educators to benefit from the best ideas of their colleagues. It may also grow to include new approaches to assessment, accreditation and collaborative learning

Bossu, Brown and Bull (2014, p. 5) offer this observation on the relationship between OER and OEP

OER movement has evolved from being mainly focused on increasing access to digital educational resources, to being focused on supporting educational practices and promoting quality and innovation in teaching and learning through Open Educational Practices

Beetham et. al. (2012) suggest that OEP encompasses the following

  • Production, management, use and reuse of open educational resources;
  • Developing and applying open/public pedagogies in teaching practice;
  • Open learning and gaining access to open learning opportunities;
  • Practising open scholarship to encompass open access publication, open science and open research;
  • Open sharing of teaching ideas and know-how; and,
  • Using open technologies.

Limitations of the literature and practice

There’s a fairly large and growing literature around OER and more interestingly OEP. What follows are some initial observations drawn from dipping my toe into that literature.

The resource focus

While Bossu et al (2014) suggest that the OER movement has evolved to focus on OEPs it’s hard to kill off the textbook idea. Adam Croom analysed the abstracts from all the OpenEd conferences since 2012 looking for use of the words “OER” and “textbook”. Abstracts with “textbook” rose from under 10% to over 30% from 2012 through 2015.

The reasons behind this are likely to be many and complex. However, an obvious driver is the cost of textbooks. Even Kanye West is concerned about the cost of textbooks

https://twitter.com/kanyewest/status/699686803251585024

This resource focus is also illustrated by the on-going fetish for providing repositories that will hold all of the open content (textbooks) and make it easy for people to find and re-use that content. Even Amazon is getting into the OER repository “business”.

Mike Caufield outlines two reasons why the repository approach “has been a disaster”

  1. “assumes that learning objects are immutable single things, and that the evolution of the object once it leaves the repository is not interesting to us”; and,
  2. “it centalizes resources, and therefore makes the existing ecosystem more fragile”.

He also offers this lesson

products need a marketplace but living things need an ecosystem. Amazon gives us yet another market.

Mike also describes what he sees as a solution in the idea of connected copies.

Bossu et al (2014) concluded that

OEP have potential to lead to more open pedagogical practices and innovative cultures. In other words, a narrow focus on OER per se may not be enough for educational institutions to fundamentally embrace and establish effective open pedagogical practices (p. 16)

It’s just not textbooks where the “product” (resource) focus is evident when it comes to open in education. MOOCs are perhaps the most visible “open” practice from Universities. The “course” (open or not) is yet another product for the institution to market.

The individual focus

A lot of the OER/OEP work has been focused on individuals. Why are they adopting OEP? What are the challenges they face?  It is work that identifies “a need for systematic staff development within institutions, which could address these very practical needs” (Stagg, 2014, p. 155) and other support mechanisms such as (Stagg, 2014, p. 161)

  • an understanding of how to search for, evaluate, and select openly-licenced content for a specific learning context;
  • a working knowledge of Creative Commons and Free Cultural Licences, as well as Public Domain, including knowledge of licence compatibility, and the inherent obligations that each type of licence carries;
  • a working knowledge of the local institutional policies and priorities;
  • the ability to integrate the newly-created OER into the curriculum; and
  • a supporting mechanism (such as a repository) to store the newly created OER, and allow for global discoverability

If OEP requires an ecosystem, then the ecosystem has to involve more than an individual. It’s more than just me using OER in my course(s). Sustainable and effective OEP would appear to require that there be multiple people involved. My OEP relies on the presence of others engaged in OEP.

A major challenge here is that University teaching and the systems that support it are not well known for supporting the sharing of teaching practice. For example, the limited visibility and sharing of resources between courses within the same program.

The institutional  focus

This is a problem that has become another focus within the OER/OEP literature. Some examples include

the absence of mediating artefacts or a supportive institutional environment can inhibit a practitioner’s ability to engage fulsomely with OEP (Stagg, 2014, p. 161)

As Smith and Wang (2007) point out, for an OER initiative to be sustainable in the long term it needs to create value for the host institution. (Bossu et al., 2014, p. 21)

Stagg (2014) uses the Capability Maturity Model (CMM) as inspiration to develop a continuum of OEP. The CMM and this work is based on the assumption that (Stagg, 2014)

An effective organisation seeks to understand repeatable conditions and processes which support projects, and then extend an understanding of the processes to optimise them within an organisational context. There is a realisation that a systematic, defined approach is required, especially when diffusing a new organisation-wide idea. The maturity of process is “the extent to which a specific process is explicitly defined, managed, measured, controlled and effective” (Paulk et al., 1993, p. 21).

A similar evidence-based model could be developed that would provide guidance to open practitioners in a systematic manner, repeatable across contexts (p. 156)

This is not to suggest that there isn’t value in institutions addressing the barriers that prevent adoption of OER/OEP. But the point is that the ecosystem in which OEP occurs involves more than the individual institutions. Beetham et al (2012) argue that the “greatest potential benefits are communal rather than tied to the competitive advantage of individuals or institutions”. The on-going neo-liberal/techno-rationalist trend in the management of universities brings with it a strong focus on institutional advantage, distinction, and efficiency. A focus/mindset that will find it difficult to recognise, let alone value and act for communal benefits. Arguably, this techno-rational mindset is the reason why MOOCs and the most prevalent university engagement in “open”. MOOCs are seen to support institutional distinction and advantage.

The techno-rational mindset is also not likely to deal well with the complexity inherent in an ecosystem. Beetham et al (2012) argue

One effect of openness is to uncouple people in time and space, making connection easier, but complex negotiation of needs, understandings and perceptions more difficult. This is true for learners and teachers, for institutions and (potential) students, for researchers and stakeholders in their research.

Searching and making connections

Masterman and Wild (2013) make a common point that

Searching for potentially suitable resources is arguably the most time-consuming aspect of OER use

Masterman and Wild (2013) identify that “the quantity of items returned by a search for OER remains problematic”. They describe Google searches returning “unmanageable quantity of hits”, but at the same time hiding resources that may reside in “OER collections which are concealed behind registration pages”. OER collections also suffer due to their much “smaller quantity and scope” that reduces the likelihood of successful searches. Mike Caufield has a related, but slightly different take on this problem. There just isn’t enough good quality stuff out there.

https://twitter.com/holden/status/701217345566445568

 

Hence the focus has been on institutions providing better repositories and on helping more people develop more OERs. But is that enough?

What about better connecting OERs into the existing ecosystems of learners and teachers?  If you are teaching (or learning about) X, then there is probably an ecosystem where you and others that teach (or learn about) X “live” (at least occasionally). Would it be easier to find OERs associated with X, if the OERs were connected into that ecosystem?

For example, If you’re a teacher educator in Queensland focused on the digital technologies subject in the new Australian Curriculum, then chances are that your “ecosystem” includes: the QSITE mailing list; the Australian Curriculum website; and, Scootle. If that ecosystem included links etc. to relevant OERs, then you’d probably be more likely to stumble across them and perhaps interact with them?

The Australian Curriculum website currently supports this type of connection by providing direct links to Scootle resources (where available) from content descriptions (e.g. ACTDIK001). This allows school teachers who are tasked with help students demonstrate understanding against this content description to quickly see relevant resources.

At least two interesting questions arise

  1. Has this integration of resources into the ecosystem of teachers helped address the search problem?
  2. Is it possible and beneficial to connect into this ecosystem OERs that are designed to help pre-service (and perhaps in-service) teachers learn about how to teach?

What might be interesting to do?

Let’s start with some assumptions and a question.

OER are living objects that reside within an open ecosystem. OEP is about living within an open ecosystem. The learning and teaching context within Universities tends not to embody these views. How might/can a group of initial teacher educators use these assumptions to engage in OEP?

Context

At the moment, teacher educators from three separate Australian Universities have expressed interest. I believe all are involved in some way with the teaching of courses intended to help prepare pre-service teachers engage with the Technologies learning area.

This includes two “groups” of teacher educators who are from the same institution, who are teaching somewhat related courses, but that are in different programs with different focii. It also includes teacher educators teaching essentially the same course, but at fairly different institutions. Includes courses that have been run for a number of years, and at least one course that has to be designed and taught for the first time by the end of 2016.

The Technologies learning area includes the digital technologies subject. Due to this and their personal backgrounds, each of the teacher educators have some level familiarity with digital technologies. They are all (at least at some level) familiar with the “practitioner ecosystem” (i.e. QSITE, Australian Curriculum website, and Scootle) used by both pre-service and in-service teachers.

Project aim

Encourage the sustainable engagement with OEP amongst teacher educators by exploring and opening up existing practices of the group. Using an approach that rejects some of the standard assumptions around of OER/OEP. i.e. it’s not limited to within one institution; it’s not focused on the individual actor, but the connections between them; and, it’s aiming to explore if and with what impacts the OER/OEP can be embedded into existing networks.

Project activities

An initial set of activities might include:

  1. Map and analyse
    1. Personal beliefs and capabilities to engage in OEP.
    2. institutional beliefs and capabilities to engage in OEP.
    3. the ecosystem in which each operates.
  2. Identify and implement barriers and opportunities to open up individual ecosystems
    What exactly gets done depends on the findings of step 1, but possible examples include:

    1. Sharing existing content via github or similar;
    2. Broaden single course social bookmarking across institutions; and
    3. Exploring how to embed OER/OEP into existing teacher ecosystem..
  3. Analyse, evaluate, and reflect on the experience.

 

References

Beetham, H., Falconer, I., McGill, L., & Littlejohn, A. (2012). Open Practices: a briefing paper. Retrieved from https://oersynth.pbworks.com/w/file/58444186/Open Practices briefing paper.pdf

Bossu, C., Brown, M., & Bull, D. (2014). Adoption, use and management of open educational resources to enhance teaching and learning in Australia. Office of Learning and Teaching. Sydney, Australia.

Masterman, L., & Wild, J. (2013). Reflections on the evolving landscape of OER use. In OER 13: Creating a Virtuous Circle (pp. 1–8). Retrieved from https://www.academia.edu/3361645/Reflections_on_the_evolving_landscape_of_OER_use

OECD. (2007). Giving Knowledge for Free. doi:10.1787/9789264032125-en

OPAL. (2011). Beyond OER: Shifting Focus to Open Educational Practices. Open Education Quality Initiative. Retrieved from https://oerknowledgecloud.org/sites/oerknowledgecloud.org/files/OPAL2011.pdf

Stagg, A. (2014). OER adoption: a continuum for practice. Universities and Knowledge Society Journal, 11(3), 151 – 164. doi:10.7238/rusc.v11i3.2102

Can video be emedded in ePub exported from #moodle book?

Last year my institution installed the Lucimoo epub export tool for the Moodle Book module. I did some of the testing of it. I’m particularly interested (but not overly) in its ability to export a book resource as an ePub file (see I’m innovative, I support the kiddies and their fancy new mobile devices). While the export basically works, the testing identified a number of problems. I’ve fixed most of those (mainly tidying up my hand-crafted HTML and that produced by a few tools I use).

The one remaining problem is

The videos don’t show

The following is an exploration of if and how this problem can be fixed. I haven’t solved the problem. I am no expert. Feel free to point out flaws and recommend improvements.

In summary

  • It is possible to use the epub export tool to produce ePub files that contain videos that will play in some ePub readers on computers.
  • These ePub files don’t appear to work on mobile devices but this needs to be confirmed.
  • For this to work you need to get direct access to the video file and for that file to be in a format that is linked by the HTML 5 video tag and the device/reader being used.
  • To do this on Vimeo, you need a Pro account.

In my context, if I had access to a Vimeo Pro account I could in theory convert most of my “video embeds” over to using the Video tag and help enhance the experience of the ePub user. They would be able to play the video in iBooks on a computer, but not the phone. However, at least on the phone they would see some evidence of a video being present, rather than an empty space.

Beyond the cost of the Vimeo Pro account, that would also require converting all the embedded videos. There’s enough to be annoying.

Initial explanation of the problem

I assumed that the problem was due to my use of Vimeo to host most of my videos. In particular, that ePub readers (iBooks is what I test with) didn’t like the default Vimeo embed HTML.  Possibly because the ePub reader doesn’t have sufficient smarts to parse and do something with the iframe etc.

It can work

This 2010 blog post provides proof that you can play video within an ePub file on a computer (MacOSX/iBooks) and a mobile (iOS/iBooks). Some level of proof given in the image below. If I hit the play button, the video plays quite nicely thank you.

ePub file playing video

Let’s try this with vimeo

Woo hoo! Sadly my quick initial skim (late last week) thought I’d found the solution. The “video” tag. Use that and all is good. (I was missing a few things because I only skimmed the article last week).

First question, can you use the HTML 5 video tag?

An answer to this question on Stack Overflow suggests that it is indeed possible and it provides an example.

In essence, you can find a direct link to an mp4 version of your Vimeo movie and use that with the video tag.

First problem with vimeo and the video tag

As pointed out by one of the comments on the answer to the Stack Overflow question, you have to have a pro Vimeo account to find the direct link to an mp4 version of your Vimeo movie.

A Vimeo pro account costs $USD219 a year.  A little beyond the price I’m prepared to change. Wonder if there’s an institutional solution?

Second problem with vimeo and the video tag

The price is a bit of a bugger, but let’s test this with the Moodle Book module and the Lucimoo epub tool.

So I create a Moodle Book resource that contains three pages (chapters in Moodle Book speak) each of which embeds a video via different means:

  1. the video tag to embed the video from the original tutorial, to demonstrate it working;
  2. use the standard vimeo player to embed one of my videos; and,
  3. the video tag to embed the “direct vimeo video” from the answer to the Stack Overflow question.

Success, it works as expected when viewed on iBooks on the Mac. In addition, when viewing the Book via Moodle all the videos are playable.

The video tag embed from the original tutorial works and I can play the video of the kittens playing.

Video tag embed working

As expected the standard Vimeo player embed did not work. Just an empty page. No indication that there was meant to be a video embedded.

Vimeo embed not working
And very interestingly the embed of a Vimeo video using the video tag did work.

Vimeo embed working (video tag)
But not on the phone

The problem is that it doesn’t work on the phone.

The first page from the original tutorial will display a “play” button indicating that a video is there. But nothing happens when the button is pressed.
IMG_4052

 

The standard Vimeo embed doesn’t work, as expected. But the use of the video tag to embed a Vimeo video shows a “play” button crossed out. Indicating some major problem.

IMG_4053

It works if you embed the video in the ePub file

The ePub file from the original tutorial actually embeds the video file in the ePub file.  It’s local. That’s why it works on the mobile phone.

For this to work with the Book export tool the videos would all have to be local to the book etc. Not likely to happen anytime soon.

Other readers

The original tutorial post is titled “HTML5 Video Works on iBooks on iPad!!”. In addition, I’ve only tested this in iBooks, not other readers. The comments include mention that this approach is not “standard” and that the video playing will not work when viewing the ePub using “Adobe Digital Editions”.

Something that I can confirm. I can view the textual content, but not the video.

Not surprisingly it will work using the Firefox EPUBReader extension.

 

Sentiment analysis of student blog posts

In June last year I started an exploration into the value of sentiment analysis of student blog posts. This morning I’ve actually gotten it to work. There may be some value, but further exploration is required. Here’s the visible representation of what I’ve done.

The following is a screen shot of the modified “know thy student” kludge I’ve implemented for my course. The window shows some details for an individual student from second semester last year (I’ve blurred out identifying elements). The current focus is on the blog posts the student has written.
Sentiment analysis of blog posts

Each row in the above corresponds to an individual blog post. It used to show how long ago the post was written, the post’s title, and provide a link to the blog post. The modified version has the background colour for the cell modified to represent the sentiment of the blog post content. A red background indicates a negative post, a green background indicates a positive post, and a yellow background indicates somewhere in the middle.

The number between 0 and 1 shown next to the post title is the result provided by the Indico sentiment analysis function. The method use to perform the sentiment analysis.

Does this help?

Does this provide any help? Can it be useful?

An initial quick skim of posts from different students seemed to indicate mostly all green. Was the sentiment analysis revealing anything useful? Was it working?

In the following I examine what is revealed by the sentiment analysis by paying close attention to an individual student, the one shown in the image above.

Red blog post – reveal target for intervention?

The “red” blog post from the image above included words like “epic fail”. It tells the story of how the student had problems getting the new software for the course working. It shows as the third post the student made in the semester. The start of this course can be frustrating for students due to technical problems. This particular student didn’t report any of these problems on the course discussion forums.

Given that the course is totally online and there are ~100 students in this offering, there’s little chance for me to have known about these problems otherwise. Had the sentiment analysis been in place during the offering and if it was represented effectively, I might have been able to respond and that response might have been helpful.

Yellow blog post – a problem to address?

The yellow post above is a reflection on the students experience on Professional Experience, in a school, in front of a classroom, actually teaching. It is a reflection on how the student went through an emotional roller coaster on prac (not unusual), how her mentor really helped (also not unusual, but a little less so), but also how the various exemptions she received contributed to her problems.

Very positive blog posts – loved resources?

A number of the posts from this student are as positive as they can get – 1.0. Interestingly, almost all of them are descriptions of useful resources and include phrases like

what a wonderful learning tool …lovely resource…wonderful resource for teachers

What’s next?

Appears that the following are required/might be useful

  1. Explore different representations and analysis
    So far I’ve only looked at the student by student representation. Another forms of analysis/representation would seem potentially useful. Are there differences/patterns across semester, between students that are the same/different on certain characteristics, between different offerings of the course etc.How can and should this representation be made visible to the students?
  2. Set this in place for Semester 1.
    In a couple of weeks the 300+ student version of this course runs. Having the sentiment analysis working live during that semester could be useful.
  3. Explore useful affordances.
    One of the points of the PIRAC framework is that this form of learning analytics is only as useful as the affordances for action that it supports. What functionality can be added to this to help me and the students take action in response?

Reflection

I’ve been thinking about doing this for quite some time. But the business of academic life has contributed to a delay.  Getting this to work actually only required three hours of free time. But perhaps more importantly, it required the breathing space to get it done. That said, I still did the work on a Sunday morning and probably would not have had the time to do it within traditional work time.

 

Minimalism, constructivism and use of Moodle or any other e-learning tool

Ben-Ari (1999) reports an experiment where 10 members of a University department were asked to verbalise their understandings as they completed tasks in Word. The aim was to explore their conceptual understanding of Word and its link with their use of Word. Ben-Ari (1999) writes

Considering the high quality of the subjects, the most surprising result was the low level of use of this very sophisticated software tool.

Another surprising result was

the degree of anthropomorphic volition attributed to the software….”You see that’s what I mean, it behaves strangely

Given the widespread recognition of the limited use of institutional e-learning information systems (e.g. Moodle), I wonder what a similar experiment focused on teacher and student conceptual understanding would reveal?

I’m willing to bet there might be some significant similarities. Especially given my finding last year that branding the LMS can hurt “learning”.

I wonder if this offers some explanation about why a tool like Moodle – designed from a socio-constructivist perspective – is rarely used that way?

I wonder what, if anything, could be fruitfully done to confirm and fix this?

I wonder if there’s any correlation between this and the nature of the type of training provided to teachers and learners? Most of what training I’ve seen seems to rely on what Ben-Ari labels as minimalism

a method for designing manuals for software documentation and for using these manuals in training users of the software. A minimalist manual is short, stresses active learning and considers errors to be opportunities for learning rather than mistakes to be corrected….its insistence that conceptual material not be included in training, or at least that it be deferred until the student is more experienced.

References

Ben-Ari, M. (1999). Bricolage Forever! In Eleventh Workshop on the Psychology of Programming Interest Group (pp. 53–57). Leeds, UK. Retrieved from http://www.ppig.org/papers/11th- benari.pdf

Exploring the relationship between learning design and OEP

Another year and another institutional grant opportunity around openness. This year the focus is on Open Educational Practice and is

designed to raise awareness and understanding of open educational practice (OEP) across USQ and to provide the opportunity for USQ academics to experiment with OEP in courses and programs.

What follows are some early ramblings that have arisen from discussions with various folk about whether or not we might submit an application.

What is Open Educational Practice (OEP)?

I think getting common agreement on an answer to this question will be a major challenge, and not just for us.  Stagg (2014, p. 154) writes

There is evidence to also suggest that OEP is, after ten years, neither widespread, nor well-known (Conole, 2013; Conrad et al., 2013

Wikipedia, amongst much else, offers this on OEP

Open educational practices (OEP) are teaching techniques that draw upon open technologies and high-quality open educational resources (OER) in order to facilitate collaborative and flexible learning.[1][2] They may involve students participating in online, peer production communities [3] within activities intended to support learning [4] or more broadly, any context where access to educational opportunity through freely available online content and services is the norm.[1] Such activities may include (but are not limited to [1]), the creation, use and repurposing of open educational resources and their adaptation to the contextual setting.[4][5][6] OEP can also include the open sharing of teaching practices[1] and aim “to raise the quality of education and training and innovate educational practices on an institutional, professional and individual level”.[7]

What might we do

Initial interest is focused on actually trying to share and re-use open content between different contexts.  Not just making content open (OERs), or using open content (OERs) to produce our own teaching materials, but exploring how, if, and what happens when you try to set up an OEP ecosystem between educators (what about learners? hopefully they’d be included) in different contexts.

So far we’ve identified three possible “contexts” in which we might be able to explore

  1. Between similar courses within a single institution
    One likely participant teaches a course to pre-service teachers based on the Technologies learning area. Another colleague and I have been tasked with developing a course for another program to help pre-service teachers learn about both the Arts and Technologies learning areas. Can we engage in a bit of OEP between these two courses? Not to mention the two cohorts of learners in each course.
  2. Between similar courses between institutions.
    There are other Universities that teach similar courses. Can we engage in a bit of OEP between these courses between universities? Not to mention the different cohorts of learners?
  3. Between universities and teachers.
    In-service teachers may benefit from what’s done in these courses. In-service teachers could definitely help improve what’s done in these courses. Can we engage in a bit of OEP between teacher educators, pre-service teachers, and in-service teachers.

#3 might be a step too far in a year long project, but…

Learning design

When this rough idea was circulated one of those included mentioned some commonality with some earlier work on “Implementing effective learning designs”.

I’ve always been a bit of a learning design skeptic, but off I went to explore this idea. Cameron (2008) was amongst the first papers I came across. The research questions in Cameron (2008, p. 45) certainly resonate with my early thinking about this project (in the following, I’ve replaced “learning designs” with “open educational practices”

What open educational practices can be readily adopted by particular disciplines as templates for best practice?
What pedagogical issues emerge from the implementation of open educational practices in particular contexts?
How can identified barriers to academics’ adoption, adaptation and reuse of open educational practices be overcome?
How can the adoption of effective open educational practices be facilitated by the use of supports and scaffolds, such as, a learning activity planning tool?

The last one is a bit of a stretch, but these still appear to be in the same ballpark.

Learning designs are descriptions of learning and teaching processes that are known to be effective. By abstracting what is known to work into a learning design it is hoped that these designs can be communicated and shared between teaching staff, especially staff who do not have expertise in designing learning. It is hoped that learning designs can act as a pedagogical framework that will help teachers create enhanced learning. This is done by customising the generic learning design in ways appropriate to the context.

If this description is somewhat appropriate, then learning designs are about producing abstractions of good practice and then encouraging others to customise those abstractions to their context. I wonder about the level of tacit knowledge involved in creating those abstractions and the gulf it produces between the creators and users of learning designs.

OEP versus Learning design

As mentioned above the notion of OEP is very much up in the air.  The understanding I’m using here is that OEP is about making the practices I use, and subsequently the artefacts I produce, in my teaching open for others to see, consider, reuse, and re-purpose. Unlike learning designs, it won’t be going through much of a process of abstraction.

What is being shared will still be very contextual. It will be bundled up with the assumptions that I and my environment bring to my teaching. Assumptions that will range from the administrative, technological, pedagogical, etc.

This will make it very difficult for other people to understand what I’ve shared, let alone understand why it is the shape it is, let alone reuse what is shared in their context. This could perhaps all fail.

However, if someone takes the time to engage with that contextual baggage, perhaps they may learn a different way of thinking about a problem. Or better yet, by engaging with my practice they might pass on to me a different way of thinking about a problem.

By sharing the very different models we bring to the act of teaching (and learning) we have to revisit and perhaps remodel our conceptions of teaching. i.e. to learn.

I’m not sure that the use of learning designs require the same level of learning. Since its an abstraction with context removed, does this makes it easier to reuse a learning design than to engage in OEP? Does this also mean that you a likely to learn less by reusing a learning design?

If the contextual difference between those engaging in OEP is too much, does this decrease the likelihood of OEP being adopted and having an impact?

What if you were sharing heavily contextual OEPs within fairly similar contexts, would this impact adoption and impact?

What are the contextual factors that influence adoption and impact?

Design funnels and complexity

Just before I started writing this I read this blog post from Dave Snowden summarising some thinking about Complexity Theory and design thinking. The post suggests that design thinking (emphasis added)

is appropriate in the complicated domain of Cynefin and to some extent as a complex to complicated transition method. But it falls down in the complex domain. A parallel point is that it originates in, and is appropriate for, product creation but starts to have problems in a service environment. The points are linked because service is nearly always complex, product complicated

The post gives an overview of some difficult territory which I need to read and ponder more. But what strikes me is that it can be argued that teaching is a service, not a product. Thus design thinking, if you accept Snowden’s argument, is probably not appropriate for teaching.

Also, at some level the production of learning designs follows a logic similar to design thinking. It aims to understand the complexity of teaching and reduce it to a complicated collection of learning designs that can be reused.

OEP (using the definition above) is about opening up the complexity of teaching so that you can see what others are doing and more easily question, share, and repurpose what they do in your context. i.e. learn.

The problem is that the modern neo-liberal university doesn’t really want to accept and work with complexity. That’s too uncertain and impossible to manage. It wants/needs to reduce complexity to obviousness or complication (using words from the Cynefin framework). Preferably complication because that’s the realm of the expert.

References

Cameron, L. (2008). Implementing effective Learning Designs : An overview of an ALTC Competitive Grants Program project. In L. Cameron & J. Dalziel (Eds.), 3rd International LAMS & Learnign Design Conference (pp. 43–49). Sydney. Retrieved from http://lams2008sydney.lamsfoundation.org/pdfs/04b.pdf

Stagg, A. (2014). OER adoption: a continuum for practice. Universities and Knowledge Society Journal, 11(3), 151 – 164. doi:10.7238/rusc.v11i3.2102

Extending a little thought experiment

David Wiley has posed a little thought experiment that encourages reflection around levels of automation and “personalisation” within a University course. Judging by my Twitter stream it appears to have arisen out of a session or happening from the ELI conference. The experiment describes a particular teacher purpose, outlines four options for fulfilling that purpose, and offers a standard against which to consider those options.

It’s a thought experiment that connects to a practice of mine and the growing status quo around higher education (at least in Australia). It’s also generated some interesting responses.

I’d like to extend that experiment in order to

  1. Reflect on some of the practices I have engaged in.
  2. Highlight some limitations with the current practice of e-learning in Australian higher education.
  3. Point out a potential problem with one perceived future for e-learning (replace the teacher with technology).

First, it would be useful to read Wiley’s original (and short) thought experiment and the responses.

Types of extensions

There are a range of ways in which the original thought experiment could be extended or modified. I’ll be looking at the following variations

  1. Modify the teacher’s purpose. (The support extension)
    In Wiley’s experiment the teacher is seeking to acknowledge success (score 80% or higher on an exam). Does a change in purpose impact your thinking?
  2. Clarify the context. (The inappropriate massification extension)
    Does the nature and complexity of the educational context matter? Does it change your thoughts?
  3. Add or modify an option. (The personalisation extension)
    Wiley gives four options ranging on a scale from manual/bespoke/human to entirely automated. Some of the comments on Wiley’s post offer additional options that vary the relationship between what is automated and what is manual/human. Generally increasing the complexity of the automation to increase it’s level of “personalisation”. At what level does automation of personalisation become a problem? Why?
  4. Question the standard
    The standard Wiley sets is that the students “receive a message ‘from their teacher’ and that students will interpret the messages as such”. In a world of increasingly digitally mediated experiences, does such a standard make sense?
  5. Change the standard each practice is being measured against. (The “connection not the message” extension).

The support extension

In Wiley’s experiment the purpose is stated as the faculty member deciding

that each time a student scores 80% or higher on an exam, she’ll send them an email congratulating them and encouraging them to keep up the good work

What if the purpose was to

Identify all those students who have not submitted an assignment by the due date and don’t already have an extension. Send each of those students an email asking if there’s a problem that she can help with.

This is the purpose for  which I’ve recently developed and used an option similar to Wiley’s option #3.

Changing the purpose doesn’t appear to really change my thoughts about each of the options, if I use the standard from Wiley’s thought experiment

to ensure that students are actually receiving a message “from their teacher” and that students will interpret the messages as such.

With an option #3 like approach, it’s possible that students may not interpret the message as being “from their teacher”/personal. But it’s not sufficient for me to stop (more below)

But it does rule out an automation option suggested by @KateMfD

Email is bad enough, but faux email? Why not make them badges and be done?

A non-submission badge strikes me as problematic.

The inappropriate, massification extension

Does the context within which the course is taught have any impact on your thinking?

The context in which I adopted option #3 was a course with 300+ students. About 160 of those students are online students. That is, they never aren’t expected to attend a campus and the geographic location of most means that they it would be impossible for them to do so. I’m directly responsible for about 220 of those students and responsible for the course overall. There are 2 other staff responsible for two different campus cohorts.

The course is 100% assignment based. All assignments are submitted via a version of the Moodle assignment submission activity that has been modified somewhat by my institution. For the assignment described in this post only 193 of 318 enrolled students had submitted assignments by the due date. Another 78 students had received extensions meaning that 47 students hadn’t submitted by the due date.

The tool being used to manage this process does not provide any method to identify the 47 that haven’t submitted AND don’t have extensions. Someone manually needs to step through the 125 students who haven’t submitted and exclude those that have extensions.

Having done that the teacher is then expected to personally contact 47 different students? Many of whom the teacher will never meet face-to-face? Many of whom chose the online study option due to how well asynchronous learning fits their busy life and part-time study? Even though attempting to personally contact these 47 students is going to consume a significant amount of time?

Another problem is that the system provided by the institution doesn’t provide any other choice than to adopt Wiley’s option #1 (send them each an email). Not only does the system NOT support the easy identification of non-submit, no extension students. It provides no support for sending a bulk email to each student within that category (or any other category).

In order to choose Wiley’s other options a teacher would have to engage in a bit of bricolage just like I did. Which tends not to happen. As an example consider that my course is a 3rd year course. The 300+ students in my course have been studying for at least 3 years in an online mode. Many of them for longer than that because they are studying part-time. They will typically have studied around 16 courses before starting my course. With that in mind here’s what one student wrote in response to me adoption option #3

Thank you for contacting me in regards to the submission. You’re the first staff member to ever do that so I appreciate this a lot.

Does a teaching context that has seen significant massification unaccompanied by appropriate changes in support for both students and teachers make any difference in your thoughts? If the manual options are seen to take time away from supporting other (or all) students? What if the inappropriate massification of higher education means that the teacher doesn’t (and can’t) know enough personal information about (most of the) individual students to craft a meaningful, personal email message?

The personalisation extension

Wiley’s options and some of the responses tend to vary based on the amount of personalisation, and how much of the personalisation is done by a human or is automated.

A human manually checking the gradebook and writing an individual email to each student seems to strike some as more appropriate (more human?). Manually sending an email from a range of pre-written versions also may be ok. But beyond that and people appear to start to stuggle.

What about the option suggested by James DiGioai

scripting the criterion matching step, which informs the teacher which students are above 80%, and pushes her to write bespoke messages for each matching student. She automates the tedious part of the task and let the teacher do the emotional work of connecting with and support her students.

Is it the type of work that is automated that is important?

What about the apparently holy grail of many to automate the teacher out of the learning experience? Are we fearful that technology will replace teachers? Can technology replace teachers?

Or is it the case that technology can and should

replace many of the routine administrative tasks typically handled by teachers, like taking attendance, entering marks into a grading book

Bringing us back to the question of where do you draw this line?

Question the standard

Wiley’s standard is

our faculty member wants to ensure that students are actually receiving a message “from their teacher” and that students will interpret the messages as such.

The assumption being that there is significant value to the student in the teacher sending and being seen to send a message written specifically for the student. A value evident in some of the responses to Wiley’s post.

In this “digital era” does such a standard/value continue to make sense? @KateMfD suggests that in some cases it may not, but in Wiley’s original case it does

But an email of encouragement strikes me as a different kind of thing. It’s intended either to be a personal message, or to masquerade as one. Political campaigning, marketing, all the discourses that structure our lives, and that we justly dismiss as inauthentic, reach for us with the mimicry of personal communication. “Dear Kate” doesn’t make it so.

Is the “is there a problem? can I help?” message that I use in my context one that can be automated? After all, the purpose of the message is that I don’t know enough about the student’s reason for not submitting to personalise the message.

What if the massification of higher education means that the teacher doesn’t (and can’t) know enough about (most of) the students to craft a personal message? Alright to automate?

I have some anecdotal evidence to support this. I have been using options at or around Wiley’s 3rd option for years. An “email merge” facility was a feature we added to a system I designed in the early 2000s. It was one of the most used features, including use by teachers who were using a different system entirely. This facility mirrored the functionality of a “mail merge” facility where you could insert macros in a message that would be replace with information personal to each individual.

One example of how I used was a simple “how’s it going” message that I would send out a key points of the semester. One response I received from a student (which I’m sure I’ve saved somewhere, but can’t find) was along the lines of “I know this is being sent out as a global email, but it still provides a sense of interaction”.

Suggesting that at least for that student there was still value in the message, even though they knew I didn’t hand craft it.

The “connection not the message” extension

Which brings me to my last point. The standard for Wiley’s thought experiment is based on the value of the message being and being seen to be a personal message to the student. That’s not the standard or the value that I see for my practices.

For what it’s worth I think that the “7 Principles of Good Practice for Undergraduate Education” from Chickering and Gamson (1997) are an ok framework for thinking about learning and teaching. The first of their 7 principles is

  1. Encourages Contact Between Students and Faculty
    Frequent student-faculty contact in and out of classes is the most important factor in student motivation and involvement. Faculty concern helps students get through rough times and keep on working

The standard I use is whether or not the practices I use encourage contact between my students and I. Does it create a connection?

Whether or not the students see the message I sent as being personally written for them is not important. It’s about whether or not it encourages them to respond and helps a connection form between us.

In the case of the not submitted, no extension students I’m hoping they’ll respond, explain the reason they haven’t submitted, and provide an opportunity for me to learn a little more about the problems they are having.

While I haven’t done the analysis, anecdotally I know that each time I send out this email I get responses from multiple students. Most, but not all, respond.

For me, this standard is more important than the standard in Wiley’s thought experiment. It’s also a standard that my personal experience suggests that moving further up Wiley’s options is okay.

It’s also a standard which argues against the complete automation of the personalisation process. The reasons why students haven’t submitted their assignment and the interventions that may be needed and appropriate tend to represent the full richness and variety of the human condition. The type of richness and variety for which an automated system can’t (currently?) handle well.

 

Using social bookmarking to enrich the learning environment

Another day and another short presentation showing off a particular aspect of learning and teaching. This time social bookmarking.

http://www.slideshare.net/davidj/using-social-bookmarking-to-enrich-the-learning-environment

Blog for learning. Why? What? How?

Today I’m giving a 10 minutes spiel on why, what, and how I make use of blogs to support student learning in one of my courses. The slides for the talk will will be available below.

 

http://www.slideshare.net/davidj/blog-for-learning-why-what-how

What if our digital technologies were protean? Implications for computational thinking, learning, and teaching

David Jones, Elke Schneider

To be presented at  ACCE’2016 and an extension of Albion et al (2016).

Abstract

Not for the first time, the transformation of global society through digital technologies is driving an increased interest in the use of such technologies in both curriculum and pedagogy. Historically, the translation of such interest into widespread and effective change in learning experiences has been less than successful. This paper explores what might happen to the translation of this interest if the digital technologies within our educational institutions were protean. What if the digital technologies in schools were flexible and adaptable by and to specific learners, teachers, and learning experiences? To provide initial, possible answers to this question, the stories of digital technology modification by a teacher educator and a novice high school teacher are analysed. Analysis reveals that the modification of digital technologies in two very different contexts was driven by the desire to improve learning and/or teaching by: filling holes with the provided digital technologies; modelling to students effective practice with digital technologies; and, to better mirror real world digital technologies. A range of initial implications and questions for practitioners, policy makers, and researchers are drawn from these experiences. It is suggested that recognising and responding to the inherently protean nature of digital technologies may be a key enabler of attempts to harness and integrate digital technologies into both curriculum and pedagogy.

Introduction

Coding or computational thinking is the new black. Reasons given for this increased interest include the need to fill the perceived shortage of ICT-skilled employees, the belief that coding will help students “to understand today’s digitalised society and foster 21st century skills like problem solving, creativity and logical thinking” (Balanskat & Engelhardt, 2015, p. 6), and that computational thinking is “a fundamental skill for everyone” (Wing, 2006, p. 33). Computational thinking is seen as “a universal competence, which should be added to every child’s analytical ability as a vital ingredient of their school learning” (Voogt, Fisser, Good, Mishra, & Yadav, 2015, p. 715). Consequently, there is growing worldwide interest in integrating coding or computational thinking into the school curriculum. One example of this is the Queensland Government’s #codingcounts discussion paper (Department of Education and Training, 2015) which commits the government “to making sure that every student will learn the new digital literacy of coding” (p. 9). It appears that students also recognise the growing importance of coding. The #codingcounts discussion paper (Department of Education and Training, 2015) cites a Microsoft Asia Pacific survey (Microsoft APAC News Centre, 2015) that suggests 75% of students (under 24) in the Asia Pacific “wish that coding could be offered as a core subject in their schools” (n.p.). While not all are convinced of the value of making coding a core part of the curriculum it appears that it is going to happen. Balanskat & Engelhardt (2015) report that 16 of the 21 Ministries of Education surveyed already had coding integrated into the curriculum, and that it was a main priority for 10 of them. Within Australia, the recently approved Technologies learning area of the Australian Curriculum includes a focus on computational thinking combined with design and systems thinking as part of the Digital Technologies subject. This is the subject that is the focus of the Queensland government’s #codingcounts plan and it has been argued that it may also “provide a framework upon which female participation in computing can be addressed” (Zagami, Boden, Keane, Moreton, & Schulz, 2016, p. 13). The question appears to have shifted from if coding or computational thinking should be integrated into the curriculum, toward questions of how and if it can be done effectively in a way that scales for all learners?

These types of questions are especially relevant given the observation that despite extensive efforts over the last 30+ years to eliminate known barriers, the majority of teachers do not yet use digital technologies to enhance learning (Ertmer & Ottenbreit-Leftwich, 2013). It appears that the majority of teachers still do not have the knowledge, skills, resources, and environment in which to effectively use digital technologies to enhance and transform student learning. The introduction of computational thinking – “solving problems, designing systems, and understanding human behaviour, by drawing on the concepts fundamental to computer science” (Wing, 2006, p. 33) – into the curriculum requires teachers to move beyond use of digital technologies into practices that involve the design and modification of digital technologies. In recognition of the difficulty of this move, proponents of integrating computational thinking are planning a range of strategies to aid teachers. One problem, however, is that many of these strategies seem to echo the extensive efforts undertaken to encourage the use of digital technologies for learning and teaching that have yet to prove widely successful. At this early stage, the evaluation and research into the integration of computational thinking into the curriculum remains scarce and with a limited amount of “evidence as to how far teachers really manage to integrate coding effectively into their teaching and the problems they face“ (Balanskat & Engelhardt, 2015, p. 15).

However, attempts to integrate coding or computational thinking into the curriculum are not new. Grover and Pea (2013) identify the long history of computational thinking, tracing it back to recommendations for college students in the 1960s and to Papert’s work with Logo in K12 education in the 1980s. By the mid-1990s, Maddux and Lamont Johnson (1997) write of “a steady waning of interest in student use of the Logo computer language in schools” (p. 2) and examine a range of reasons for this. In the late 1990s, the dotcom boom helped increase interest, but it did not last. By the 2000s the overall participation rate in IT education within Australia declined.  With an even greater decline in enrolments in software development subjects, and especially in female participation (Rowan & Lynch, 2011). The research literature has identified a range of factors for this decline, including the finding that “Students in every participating school joined in a chorus defining the subject as ‘boring’” (Rowan & Lynch, 2011, p. 88). More recently the rise of interest in computational thinking has led to the identification of a range of issues to be confronted, including: “defining what we mean when we speak of computational thinking, to what the core concepts/attributes are and their relationship to programming knowledge; how computational thinking can be integrated into the curriculum; and the kind of research that needs to be done to further the computational thinking agenda in education” (Voogt et al., 2015, p. 716). In this paper, we are interested in exploring the related issue of how and if widespread common perceptions of digital technologies may be hindering attempts to harness and integrate digital technologies into both curriculum and pedagogy.

What if the digital technology environments within education institutions do not mirror the environments in contemporary and future digitalised societies? What if our experience within these limited digital technology environments is negatively impacting our thinking about how to harness and integrate digital technologies into curriculum and pedagogy? What if thinking about digital technology has not effectively understood and responded to the inherent protean nature of digital technologies? What if the digital technologies provided to educators were protean? Might this have an impact on attempts to harness and integrate digital technologies into curriculum and pedagogy? It is these and related questions that this paper seeks to explore.

The paper starts by drawing on a range of literature to explore different conceptions of digital technologies. In particular, it focuses on the 40+ year old idea that digital technologies are the most protean of media. Next, the paper explains how stories of digital technology modification by a high school teacher and a teacher educator were collected and analysed to offer insights into what might happen if our digital technologies were protean. Analysis of these stories is then discussed and used to develop an initial set of implications for practice, policy, and research for attempts to harness and integrate digital technologies into curriculum and pedagogy. The paper suggests that an educational environment that is rich with protean digital technologies appears likely to have a range of positive impacts on attempts to harness and integrate digital technologies into curriculum and pedagogy. However, such an environment requires radically different mindsets than currently used within educational institutions, and is thus likely to be extremely challenging to create and maintain.

Digital technology: A protean meta-medium, or not?

The commonplace notions of digital technologies that underpin both everyday life and research have a tendency to see them “as relatively stable, discrete, independent, and fixed” (Orlikowski & Iacono, 2001, p. 121). Digital technologies are seen as hard technologies, technologies where what can be done is fixed in advance either by embedding it in the technology or “in inflexible human processes, rules and procedures needed for the technology’s operation” (Dron, 2013, p. 35). As noted by Selwyn and Bulfin (2015) “Schools are highly regulated sites of digital technology use” (p. 1) where digital technologies are often seen as a tool that is: used when and where permitted; standardised and preconfigured; conforms to institutional rather than individual needs; and, a directed activity. Rushkoff (2010) argues that one of the problems with this established view of digital technologies is that “instead of optimizing our machines for humanity – or even the benefit of some particular group – we are optimizing humans for machinery” (p. 15). This hard view of digital technologies perhaps also contributes to the problem identified by Selwyn (2016) where in spite of the rhetoric of efficiency and flexibility surrounding digital technologies, “few of these technologies practices serve to advantage the people who are actually doing the work” (p. 5). Digital technologies have not always been perceived as hard technologies.

Seymour Papert in his book Mindstorms (Papert, 1980) describes the computer as “the Proteus of machines” (p. viii) since the essence of a computer is its “universality, its power to simulate. Because it can take on a thousand forms and can serve a thousand functions, it can appeal to a thousand tastes” (p. viii). This is a view echoed by Alan Kay (1984) and his discussion of the “protean nature of the computer” (p. 59) as “the first metamedium, and as such has degrees of freedom and expression never before encountered” (p. 59). In describing the design of the first personal computer, Kay and Goldberg (1977) address the challenge of producing a computer that is useful for everyone. Given the huge diversity of potential users they conclude “any attempt to specifically anticipate their needs in the design of the Dynabook would end in a disastrous feature-laden hodgepodge which would not be really suitable for anyone” (Kay & Goldberg, 1977, p. 40). To address this problem they aimed to provide a foundation technology and sufficient general tools to allow “ordinary users to casually and easily describe their desires for a specific tool” (Kay & Goldberg, 1977, p. 41). They aim to create a digital environment that opens up the ability to create computational tools to every user, including children. For Kay (1984) it is a must that people using digital technologies should be able to tailor those technologies to suit their wants, since “Anything less would be as absurd as requiring essays to be formed out of paragraphs that have already been written” (p. 57). For Stallman (2014) the question is more fundamental, “To make computing democratic, the users must control the software that does their computing!” (n.p.).

This perceived 40-year-old need for individuals to use protean digital technologies to make their own tools in order to fulfil personal desires resonates strongly with the contemporary Maker movement. A movement that is driven by a combination of new technologies that increase the ease of creation, a cultural shift toward do-it-yourself practices, and is seeing people increasingly engaged in creating and customising physical and virtual artefacts. Martinez and Stager (2013) make this link explicit by labelling Seymour Papert as the “Father of the Maker Movement” (n.p.). Similarly, Resnick and Rosenbaum (2013) note the resonance between the Maker movement and a tradition within the field of education that stretches from Dewey’s progressivism to Papert’s constructionism. Resnick and Rosenbaum (2013) see tinkering “as a playful style of designing and making, where you constantly experiment” (p. 165) for which digital technologies – due to their association with logic and precision – may not always appear suitable. A perception reinforced by the evolution of digital technologies after the work of Kay and Goldberg in the 1970s.

The work of Kay, Goldberg, and others at Xerox PARC on Dynabook directly and heavily influenced Apple, Microsoft, and shaped contemporary computing. However, Kay and Goldberg’s conception of computers as a protean medium where tool creation was open to every user did not play a part in that shaping (Wardrip-Fruin & Montfort, 2003). In fact, there’s evidence that digital technologies are getting less modifiable by the end-user. Writing about how our relationship with computers is changing, Turkle (1995) argues that we “have become accustomed to opaque technology” (p. 23). Where early computer systems encouraged, even required, people to understand the mechanism of the computer, the rise of the GUI interface hides the mechanism behind the simulation of a desktop or other metaphor. Limiting users to clicking prepared icons and menus. Desktop personal computers once had an architecture that enabled enhancement and upgrading. While increasingly mobile devices are typically “not designed to be upgraded, serviced or even opened, just used and discarded” (Traxler, 2010, p. 5). The decision by Apple to prevent the creation of executable files on the iPad means “that you can’t make anything that may be used elsewhere. The most powerful form of computing, programming, is verboten” (Stager, 2013, n.p.). But it’s not just the design of technology that hardens digital technologies.

As noted above, Dron (2013) argues that technology can be hardened by embedding it “in inflexible human processes, rules and procedures” (p. 35). Resnick and Rosenabuam (2013) make the point that designing contexts that allow for tinkerability is as important as designing technologies for tinkerability. The affordance of a digital technology to be protean is not solely a feature of the technology. An affordance to be protean arises from the on-going relationship between digital technologies, the people using it, and the environment in which it is used. Being able to code, does not always mean you are able to modify a digital technology. Selwyn and Bulfin’s (2015) positioning of schools as “highly regulated sites of digital technology use” (p. 1) suggest that they are often not a context that are designed for tinkerability through the provision of protean digital technologies.

Even though the context may not provide protean digital technologies, this hasn’t stopped educators modifying digital technologies. Jones, Albion and Heffernan (2016) examine and map stories of digital technology modification by three teacher educators by the traces left in the digital landscape and the levels of modification. Table 1 provides an overview of the levels of digital technology modification used by Jones et. al. (2016). It ranges from simply using a digital technology as is, through changing its operation via configuration options (internal and external), modifying the operation of a digital technology by combining or supplementing it with other digital technologies, and finally to coding. Table 1 suggests that digital technologies can be modified via configuration, combination, and coding.

Table 1: Levels of digital technology modification (Albion et al., 2016)

Type of change Description Example
Use Tool used with no change Add an element to a Moodle site
Internal configuration Change operation of a tool using the configuration options of the tool Change the appearance of the Moodle site by changing Moodle course settings
External configuration Change operations of a tool using means outside of the tool Inject CSS or Javascript into a Moodle site to change its appearance or operation
Customization Change the tool by modifying its code Modify the Moodle source code, or create/install a new plugin
Supplement Use another tool to offer functionality not provided by existing tool Implement course level social bookmarking through Diigo
Replacement Use another tool to replace/enhance functionality provided by existing tool Require students to use external blog engines, rather than the Moodle blog engine

 

Methodology

This paper uses a qualitative case study to describe and explore the potential value, impact, and issues faced by educators when they seek to treat digital technologies as protean. The aim being to offer some initial responses to the question “what if our digital technologies were protean?” As this is an attempt to understand a particular social phenomenon as it occurs in real-life it is well-suited to the case study method (Aaltio & Heilmann, 2010). Data for this case study is drawn from the authors’ own experiences as educators. For Jones this draws on his experiences as a teacher educator at the University of Southern Queensland from commencement in 2012 through 2015. During this time his main teaching responsibility was for a large – over 300 students split evenly between on-campus and online students – 3rd year course within the Bachelor of Education. For Schneider, this draws on her experience as a teacher at secondary schools (neither her current school) within south-east Queensland in 2014 and 2015 teaching grades 7 to 12 in IT and Business subjects.

The authors’ experiences provide a number of advantages for the purpose of exploring the potential impact of protean digital technologies. Both authors have: formal tertiary education in fields related to the development of Information Technology; undertaken professional work within Information Technology; and, later trained as Secondary IPT teachers. Consequently, both authors see digital technologies as more inherently protean than those without an IT background, and have the knowledge and skills necessary to modify existing, somewhat less than protean, digital technologies. While not an activity currently broadly available to all educators, the authors’ experience and knowledge provide an indication of what might be possible if digital technologies available to educators were more protean. At the same time, the authors have different cultural backgrounds (Australia and Canada). The case also explores the impact of protean digital technologies within two very different educational contexts: tertiary and secondary education. The tertiary education context involves a large course with hundreds of students in both on-campus and online modes. This large and diverse student cohort means that there is significant use of digital technologies with online students learning solely via digital technologies. The secondary education context involves a greater number of smaller student cohorts with digital adoption in a state of flux and still primarily delivering teaching and assessing learning with traditional, non-digital means.

The authors engaged in an iterative and cyclical process that involved the gathering, sharing, discussing, and analysing stories of how, why, and what digital technologies they had modified while teaching. Both authors drew on personal records and writings in the form of tweets, blog posts, email archives, and other documents to generate a list of such stories. These stories (Jones: 16, Schneider: 10) were written up using a common format, shared via a Google document, generated on-going discussion, and led to an iterative process of analysis to identify patterns and implications. A major part of the analysis was grouping the stories of digital technology modification via: the purpose (e.g. improve administration, model good practice, teaching, or learning); cause (e.g. inefficient systems, non-existent systems, missing functionality); impact (e.g. save time, improve learning); and, the type of change (as per Table 1). From this analysis a number of evident themes were extracted and are described in the next section.

Themes evident in stories of protean technologies

Upon reading each other’s stories, both authors were immediately struck by the level of commonality between the stories both had told. Not so surprising was that all stories told of attempts to improve learning, teaching, or both. However, even though these stories were taking place in very different types of educational institutions there were three common themes prevalent in stories from both authors. The three themes were: filling holes (14 stories); modelling effective practice (12 stories); and, mirroring the real world (7 stories). There were, however, significant differences in the amount of coding required for these stories and the levels of digital technology modification undertaken.

In terms of coding, eventually none of Schneider’s ten stories involved the use of coding. Two of her stories did initially involve coding (Yahoo Pipes and Java), but she subsequently implemented other modifications that did not require coding. Seven of Jones’ sixteen stories involved coding using Perl, PHP, or jQuery/Javascript. This suggests the digital technologies can be modified without necessarily being able to code. However, it does raise questions about the reasons between the greater prevalence of coding in Jones’ stories. Is it due to the greater reliance on digital technologies within the specific context? Is it his longer work history within higher education? Was Jones less fearful of getting in trouble for wandering away from officially mandated practices? Is it his longer engagement with modifying digital technologies for learning and teaching? Or, are there other factors at play?

Figure 1 describes the level of digital technology modification (as per Table 1) evidence in the stories from each author (some stories involved more than one level of modification). All but one of Schneider’s stories involved supplementing or replacing digital technologies provided by the school. This suggests some significant perceived limitations with the school digital technology environment. Jones’ stories were almost evenly balanced between configuring provided digital technologies, or supplementing/replacing them with different digital technologies.Story Modification.png

Figure 1: Number of stories per author for each level of digital technology modification

Four of Schneider’s stories and ten of Jones’ stories of digital technology modification were designed to fill holes in the functionality provided by institutional technologies. In her very first story (Digital grading using Excel) Schneider outlines her use of Excel spreadsheets to supplement the school’s requirement that teachers update paper-based student profiles located within a dedicated physical folder kept in the head-of-department’s office. Her use of Excel spreadsheets to supplement the required practice provided necessary support for teacher tasks such as maintaining student progress records and discussing progress with individual students. Practices that the school practice did not support – the hole to be filled. In the story “Web scraping to contact not submits” Jones describes a similar hole in an institutionally provided technology. In this story, the University’s online assignment management system provides no mechanism by which students who have not submitted an assignment and have not received an extension can be identified and contacted. Instead, Jones had to use a combination of Perl scripts, regular expressions, manual copying and pasting, and an email client to fill the hole. The value and difficulty in making this particular modification is illustrated by the following quote from a third-year student who was contacted via this modification.

Thank you for contacting me in regards to the submission. You’re the first staff member to ever do that so I appreciate this a lot.

Six of Schneider’s stories and six of Jones’ stories of digital technology modification were intended to improve student learning. These were all driven by a combination of modelling the effective use of digital technologies and/or adopting enhanced pedagogical practices. In “Moviemaker to introduce teacher and topics” Schneider describes how the production by her of a movie trailer for her subject is intended to model the use of digital technologies to visually present information, but also to engage students. In “Course barometers via Google forms” Jones  describes how functionality provided by the University LMS is replaced with Google forms as a way to more effectively gather student feedback, but also model a technology that they may be used by students in their practice. That both authors primarily teach in subjects related to the use of digital technologies would appear to suggest that prevalence of the modelling theme may be reduced for teachers of other areas.

Four of Schneider’s stories and three of Jones’ stories suggest that the institutionally provided digital technologies do not always appropriately mirror the capabilities of real-world technologies and subsequently negatively impact learning and teaching. Both authors share stories about how the visual and content capabilities of institutional learning management systems fail to mirror the diversity, quality, and capabilities of available online technologies, including social networking software. Consequently, both authors tell stories of creating teaching related websites on external blog engines. In “Creating a teaching website with Edublogs” Schneider outlines the visual and functional limitations of the official Learning Management System (LMS) and how use of Edublogs saved teacher time, was more visually appealing, and provided a more authentic experience to students of services they are likely to encounter in the real-world. Schneider also tells stories where computer hardware and network bandwidth provided by the school to students is supplemented through use of personal resources from both students and herself. The story “Encourage student use of phone hot-spots” tells of how the widespread inability of school Internet connections to fulfil learning needs was addressed by encouraging those students with access to use their mobile phone hot spots.

In general, the modification of institutional digital technologies does not come without problems, risks, or costs. Both authors make mention of the additional workload required to implement the changes described, especially when such changes aren’t directly supported or encouraged by the institution.  Such cost can be assuaged through on-going use of the changes and the benefits they generate. However, these types of changes can challenge institutional polices and be frowned upon by management. In “Hacking look and feel” Jones  describes how an institutionally mandated, default look and feel for course websites was modified to avoid a decrease in functionality. A story that also describes how the author had to respond to a “please explain” message from the institutional hierarchy and was for a time seen as “hacking” the institution’s online presence. Similarly, in “Encouraged students to hot-spot with their phones to connect to the web” Schneider describes one digital technology modification that both broke institutional policy, but also enhanced student learning. It is not hard to foresee situations where the outcomes of these stories may well have been considerably more negative for those involved.

What if? Discussion, implications and questions

The perception of digital technologies as protean does not appear widespread within educational institutions. What if our digital technologies were protean? Since designing the context for tinkerability is important (Resnick & Rosenbaum, 2013), what if the context within educational institutions were designed to enable, encourage, and support all teachers and learners in the modification of digital technologies to create the tools they see as necessary to best support their learning and teaching? Understanding and correctly predicting the potential implications and outcomes of such a radical transformation of the complex environment of an education institution is difficult. Hence the following are presented as a tentative exploration of some possible future states and are seen more as questions for exploration and confirmation than firm predictions. The assumption underpinning the following implications and questions is that the experience of the authors described above can be used to generate some indications of what might happen if our digital technologies were protean.

Filling holes – bricolage

One of the reviewers of this paper made the following observation

Some of the tinkerability/evidence of protean behaviour sound rather like the old idea of a kludge – a ‘quick and dirty’ workaround for some computer processes

As noted earlier in the paper, almost 40 years ago, Kay and Goldberg (1977) recognised that any digital technology that attempted to anticipate the needs of a diverse user population would end up as “a disastrous feature-laden hodgepodge which would not be really suitable for anyone” (p. 40). Over recent years the digital technologies used within educational institutions are increasingly enterprise information systems. Systems – such as Learning Management Systems – intended to fulfil the needs of the entire institution and are perhaps more likely to fulfil the prediction of Kay and Goldberg. Jones, Heffernan and Albion (2015) offer a range of additional examples of how institutionally mandated digital technologies are often not suited to specific educational aims and contexts and thus generate the need for ‘digital renovation’.  An example of Koopman’s and Hoffman’s (2003) description of how some “work-arounds are necessary because the computer or software as originally designed simply doesn’t address the problem or task at hand” (p. 72). Koopman and Hoffman (2003) argue that workarounds should not be seen as users departing from officially condoned uses of technology (illustrated above by the increased chance of organisational censure the authors digital renovation risked), but rather as the legitimate practice of adaptive design where the users are helping finish the design of the digital technologies.

A perspective mirrored by Turvey (2012) who argues that the construction of pedagogical tools does not end with production, but instead such tools continue to be refined through “use within a complex ecology of mediating influences, as teachers exercise agency over the development of their professional practice” (p. 114). Further echoed by the argument of Mishra and Koehler (2006) that “there is no single technological solution that applies for every teacher, every course, or every view of teaching” (p. 1029) and that instead quality teaching “requires developing a nuanced understanding of the complex relationships between technology, content, and pedagogy, and using this understanding to develop appropriate context-specific strategies and representations” (p. 1029). Jones, Heffernan and Albion (2015) describe how the protean possibilities of existing digital technologies can be used to engage in ‘digital renovation’ and thus create educational possibilities specific to particular teaching contexts.

Would digital technologies that are protean better support teachers engaging in digital renovation activities that “fill the holes” between those digital technologies and the context-specific requirements of learner and teacher? Would teacher engagement in context-appropriate digital renovation activities lead to improvements in the quality of teaching and learning? If existing digital technologies are largely not protean, what is the nature of the “holes” that are currently experienced by learners and teachers? What impact does an inability to “fill these holes” have on teachers and their workload, sense of agency, their perception of digital technologies, their learners etc.?

Modelling the effective use of digital technologies

The digital technologies subject from the technologies area of the Australian Curriculum defines computational thinking as “A problem solving method that involves various techniques and strategies in order to solve problems that can be implemented by digital systems” (ACARA, 2014). Workarounds, kludges, and digital renovation are examples of the application of computational thinking by users to solve problems that they face. Engaging in digital renovation allowed Schneider to model the application of computational thinking for her secondary computing students. With the incorporation of the Australian Curriculum’s digital technologies subject into the compulsory curricula, the advantages of being able to do this now expand to a majority of teachers. However, as noted above there is the question about whether or not this broader sample of teachers have the experience, knowledge and skills to take advantage of this opportunity. To address this problem a range of professional development opportunities are being made available to teachers.

In the context of ‘technologising literacy education’, Lankshear and Bigum (1999) develop and describe four principles for “guiding further developments in technologizing classrooms” (p. 445) and then show how those principles are seen differently by an ‘insider’ mindset and an ‘outsider-newcomer’ mindset. The first of these principles is ‘Teachers first’. This principle recommends that teachers must first be aided in “making use of new technologies to enhance their personal work before learning to use them in their teaching” (p. 453). The argument is that in order for teachers to be able to make appropriate pedagogical decisions around new technologies “they must first know how to use those technologies for their own purposes (and any benefits of doing so) for their own purposes” (p. 453). Lankshear and Bigum (1999) argue that the intent of this principle is “easy to subvert” (p. 460) by practices “designed to put teachers into classrooms with improved technological skills and understandings, but within the confines of the newcomer-outsider world view” (p. 460). On the other hand, an insider world view focuses both on the importance of addressing teachers’ on-going needs, but also on developing new alliances and articulations around learning, teaching, and the new technologies. Professional development alone is not likely to be sufficient to allow teachers to model computational thinking. Protean digital technologies would seem to be at least a catalyst, if not a pre-requisite, for teachers and others to be able to begin modelling computational thinking in the context of the requirements of the digital technologies subject.

Would the widespread availability of protean digital technologies better enable teachers to develop and model computational thinking? What impact would this have on student learning? Will the absence of protean digital technologies hinder teachers’ ability to develop and refine their computational thinking abilities? Can protean digital technologies help support the creation of new alliances and articulations around learning, teaching, and digital technologies within schools? What other types of support and changes would be required to develop such alliances and articulations? What new alliances and articulations would or should be developed?

Mirror the real world

The introduction of the digital technologies subject into core curricula is being done to ensure that students leave school with the skills necessary to engage in a digital world. It has been suggested that within Australia the introduction of the “compulsory Digital Technologies curriculum may provide a framework upon which female participation in computing can be addressed” (Zagami, Boden, Keane, Moreton, & Schulz, 2016, p. 13). On the other hand, in critiquing school mathematics Lockhart (2009) suggests that “there is surely no more reliable way to kill enthusiasm and interest in a subject than to make it a mandatory part of the school curriculum” (p. 36). A major part of Lockhart’s (2009) critique of school mathematics is a complaint about “the lack of mathematics in our mathematics classes” (p. 29). A problem that arises from a complex set of factors including “that nobody has the faintest idea what it is that mathematicians do” (p. 22) which leads to “forced and contrived” (p. 38) attempts to explain how what happens in mathematics classes as relevant to daily life. Margolis, Estrella, Goode et. al. (2008) found  that classroom practices associated with the teaching of computer science in American schools “can be disconnected from students’ lives, seemingly devoid of real-life relevance” (p. 102). Echoes of the limited relevance problem was found by Rowan & Lynch (2011) in post-compulsory information technology secondary courses in Australia. Margolis et al (2008) argue that it is important that teachers be able to demonstrate to students the relevance and significance of computer science to students’ lived experience, but identify that typically teachers have not received any support in developing approaches that meet this need.

The renewed interest in computational thinking and digital technologies arise from visions of the future, such as that seen by the Queensland Government where digital technologies are “fundamentally transforming the world of work and generating new ways of doing business on a global scale” (Department of Education and Training, 2015, p. 11). A vision of a future real world that is very different from the experience learners and teachers have of digital technologies within schools. As identified by Selwyn and Bulfin (2015), an experience heavy on regulation, standardisation, pre-configuration, directed activity, and on institutional and not individual needs. Suggesting that the prevalent school digital environment is unlikely to help prepare learners and teachers well for the future, fundamentally transformed world. Suggesting also that the teaching of computational thinking within schools may fall into the same trap as the type of school-based mathematics critiqued by Lockhart.

Are current, school-based digital environments suitable for preparing learners and teachers “to understand today’s digitalised society and foster 21st century skills like problem solving, creativity and logical thinking” (Balanskat & Engelhardt, 2015, p. 6)? Would an environment with the widespread availability of protean digital technologies better mirror this future world? What challenges exist in making school-based digital environments better mirror a future world that has been fundamentally transformed by digital

Discussion and Conclusions

This paper has posed the question “What if our digital technologies were protean?” To provide some initial responses to this question the paper has explored what is meant by protean digital technologies and analysed stories of digital technology modification from a high-school teacher and a teacher educator. Analysis of these stories revealed that these educators were driven to modify the available digital technologies while attempting to improve aspects of learning and/or teaching. These attempts at improvement aimed to: fill holes in the functionality provided by the digital technologies; model effective practice with digital technologies; or, better mirror real world digital technologies. Only seven of twenty-six stories of digital technology modification required use of coding. The majority of digital technology modification stories involved the configuration or combination of digital technologies, often to replace digital technologies provided by the organisation. Using this experience as a foundation, the paper has used a range of literature to develop some initial suggestions for what might happen more broadly within education if our digital technologies more protean. Given the complex nature of education and the difficulty of predicting the future, these suggestions are framed as questions for further exploration and confirmation, rather than prediction. However, the authors do suspect that the impact of more protean digital technology within education would be positive for both the teaching of computational thinking, and more broadly for the use of digital technology to enhance learning and teaching.

Actually exploring whether or not this is the case will be quite a challenge. Not the least because the idea of protean digital technologies is diametrically opposed to the existing digital technology environment within most educational institutions, and indeed broader society. Enabling more protean digital technologies within education would need to engage with existing widely held perspectives and practices around difficult issues such as accountability, efficiency, resourcing, risk management, and student safety. This task is made more difficult by the question about whether or not those engaged  with such discussions bring – as identified by Lankshear and Bigum (1999) – an ‘insider’ or ‘outsider-newcomer’ mindset. An ‘outsider-newcomer’ sees “the world as the same, but just more technologised” where the insider sees how pervasive and protean digital technologies means that the world – and subsequently educational institutions – “is radically different” (Lankshear & Bigum, 1999, p. 458). The insider view appears more in line with the espoused rationale behind that rise of computational thinking and coding in schools. However, there remain questions about how much of the rhetoric around digital technology-enabled transformation of society. More pragmatically there is the question of how to provide protean digital technologies within education institutions? A question that might be answered by drawing on research on creating computationally rich environments for learners. Such as Grover and Pea’s  (2013) potential principles including: low floor, high ceiling; support for the “use-modify-create” progression; scaffolding; enable transfer; support equity; and, be systemic and sustainable. Principles that might fruitfully be used to break education out of its traditional norms and structures and allow us to finally explore the question “What IF schools were not encumbered by traditional norms and structures, and technology, social capital and pedagogies were used to their true realisation or potential?”

References

Aaltio, I., & Heilmann, P. (2010). Case Study as a Methodological Approach. In A. J. Mills, G. Durepos, & E. Wiebe (Eds.), Encyclopedia of Case Study Research. (pp. 67–78). Thousand Oaks, CA: Sage Publications.

ACARA. (2014). Computational thinking – Glossary term. Retrieved from 2 July 2016

Balanskat, A., & Engelhardt, K. (2015). Computing our future: Computer programming and coding – Priorities, school curricula and initiatives across Europe. Brussels. Retrieved from http://www.eun.org/c/document_library/get_file?uuid=3596b121-941c-4296-a760-0f4e4795d6fa&groupId=43887

Department of Education and Training. (2015). #codingcounts: A discussion paper on coding and robotics in Queensland schools. Brisbane, Australia. Retrieved from http://advancingeducation.qld.gov.au/SiteCollectionDocuments/Coding-and-robotics-booklet.pdf

Dron, J. (2013). Soft is hard and hard is easy: learning technologies and social media. Form@ Re-Open Journal per La Formazione in Rete, 13(1), 32–43.

Ertmer, P. a., & Ottenbreit-Leftwich, A. (2013). Removing obstacles to the pedagogical changes required by Jonassen’s vision of authentic technology-enabled learning. Computers & Education, 64, 175–182.

Grover, S., & Pea, R. (2013). Computational Thinking in K-12: A Review of the State of the Field. Educational Researcher, 42(1), 38–43.

Jones, D., Albion, P., & Heffernan, A. (2016). Mapping the digital practices of teacher educators: Implications for teacher education in changing digital landscapes. In Proceedings of Society for Information Technology & Teacher Education International Conference 2016 (pp. 2878–2886). Chesapeake, VA: Association for the Advancement of Computing in Education.

Jones, D., Heffernan, A., & Albion, P. (2015). TPACK as Shared Practice: Toward a Research Agenda,. In L. Liu & D. Gibson (Eds.), Research Highlights in Technology and Teacher Education 2015 (pp. 13–20). Waynesville, NC: AACE.

Kay, A. (1984). Computer Software. Scientific American, 251(3), 53–59.

Kay, A., & Goldberg, A. (1977). Personal Dynamic Media. Computer, 10(3), 31–41.

Koopman, P., & Hoffman, R. (2003). Work-arounds, make-work and kludges. Intelligent Systems, IEEE, 18(6), 70–75.

Lankshear, C., & Bigum, C. (1999). Literacies and new technologies in school settings. Pedagogy, Culture & Society, 7(3), 445–465.

Lockhart, P. (2009). A Mathematician’s Lament: How school cheats us out of our most fascinating and imagintive art forms. New York: Bellevue Literary Press.

Maddux, C. D., & Lamont Johnson, D. (1997). Logo: A retrospective. Computers in the Schools, 14(1/2), 1–8.

Margolis, J., Estrella, R., Goode, J., Jullison Holme, J., & Nao, K. (2010). Stuck in the shallow end: Education, race, and computing. Cambridge, MA: MIT Press.

Martinez, S. L., & Stager, G. (2013). Invent to learn: Making, tinkering, and engineering in the classroom. Torrance, CA: Constructing Modern Knowledge Press.

Microsoft APAC News Centre. (2015). Three out of four students in Asia Pacific want coding as a core subject in school, reveals Microsoft study | Asia News Center. Retrieved January 20, 2016, from https://news.microsoft.com/apac/2015/03/23/three-out-of-four-students-in-asia-pacific-want-coding-as-a-core-subject-in-school-reveals-microsoft-study/

Mishra, P., & Koehler, M. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054.

Orlikowski, W., & Iacono, C. S. (2001). Research commentary: desperately seeking the IT in IT research a call to theorizing the IT artifact. Information Systems Research, 12(2), 121–134.

Papert, S. (1980). Mindstorms: children, computers, and powerful ideas. New York: Basic Books.

Resnick, M., & Rosenbaum, E. (2013). Designing for Tinkerability. Design, Make, Play: Growing the next Generation of STEM Innovators, 163–181. doi:Resnick, M.; Rosenbaum, E. (1993). Designing for tinkerability. In Design, Make, Play: Growing the Next Generation of STEM Innovators (pp. 163–181). New York: Routledge.

Rowan, L., & Lynch, J. (2011). The continued underrepresentation of girls in post-compulsory information technology courses: a direct challenge to teacher education. Asia-Pacific Journal of Teacher Education, 39(2), 83–95.

Rushkoff, D. (2010). Program or be programmed: Ten commands for a digital age. New York: OR Books.

Selwyn, N. (2016). The digital labor of digital learning : notes on the technological reconstitution of education work. Retrieved January 25, 2016, from http://newmediaresearch.educ.monash.edu.au/lnm/the-digital-labor-of-digital-learning/

Selwyn, N., & Bulfin, S. (2015). Exploring school regulation of students’ technology use – rules that are made to be broken? Educational Review, 1911(October), 1–17.

Stager, G. (2013). For the love of laptops. Adminstr@tor Magazine. Retrieved January 30, 2016, from http://www.scholastic.com/browse/article.jsp?id=3757848

Stallman, R. (2014). Comment on “We can code IT! Why computer literacy is key to winning the 21st century.” Mother Jones. Retrieved January 26, 2016, from http://www.motherjones.com/media/2014/06/computer-science-programming-code-diversity-sexism-education#comment-1437791881

Traxler, J. (2010). Will student devices deliver innovation, inclusion, and transformation? Journal of the Research Centre for Educational Technology, 6(1), 3–15.

Turkle, S. (1995). Life on the screen: Identity in the age of the Internet. New York: Simon & Schuster.

Turvey, K. (2012). Constructing Narrative Ecologies as a Site for Teachers’ Professional Learning with New Technologies and Media in Primary Education – E-Learning and Digital Media Volume 9 Number 1 (2012). E-Learning and Digital Media, 9(1), 113–126.

Voogt, J., Fisser, P., Good, J., Mishra, P., & Yadav, A. (2015). Computational thinking in compulsory education: Towards an agenda for research and practice. Education and Information Technologies, 20(4), 715–728.

Wardrip-Fruin, N., & Montfort, N. (2003). New Media Reader. Cambridge, MA: MIT Press.

Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35.

Zagami, J., Boden, M., Keane, T., Moreton, B., & Schulz, K. (2016). Female participation in school computing : reversing the trend. Retrieved from http://digitalcareers.edu.au/wp-content/uploads/2015/04/Female-Participation.pdf

Powered by WordPress & Theme by Anders Norén

css.php