Assembling the heterogeneous elements for (digital) learning

Category: indicators Page 1 of 8

Are learning analytics leading us towards a utopian or dystopian future, and what can we as practitioners do to influence this?

What follows will eventually be a summary of my contribution to a the ASCILITE’2017 panel titled Are learning analytics leading us towards a utopian or dystopian future, and what can we as practitioners do to influence this?. Below you’ll find a summary of my prediction and the argument that underpins it, suggestions for more reading, the slides, and references.

Argument

The argument made in the following is that Vision #3 from the LACE Visions for the Future of Learning Analytics captures a likely future for learning analytics in Australian Higher Education. That is,

In 2025, analytics are rarely used in education

I don’t necessarily agree with some of the features of this particular vision. However, it does include predictions of many problems, poor implementation and limited use of learning analytics. Based how Australian higher education currently approaches the use of digital technology in learning and teaching, this future appears likely (to me).

This is because the approach universities take to implementing learning analytics will focus on the development/adoption of an institutional learning analytics tool(s) and encouraging/managing the adoption of that tool within the institution. An approach that tends to what Cavallo (2004) described as an “explicitly topdown and hierarchical, and implicitly view education as a series of depersonalised, decontextualised steps carried out by willing, receptive, non-transforming agents” (p. 96). An approach that assumes that there is the one learning analytics tool that can be scaled across the institution. One size fits all.

It’s an approach that fails to effectively engage with Gasevic et al (2015) describe as a significant tension between course (unit) specific models and general models. A tension that echoes the reusability paradox (Wiley, n.d) in that general models “represent a cost effective &…efficient approach” (Gasevic et al, 2015, p. 83), but at the cost of pedagogical value. In learning and teaching, one size does NOT fit all.

In terms of what can be done, the suggestion is to focus on an approach designed to help find the right size for each. An approach that effectively engages with the significant tension of the reusability paradox and aims to work toward maximising pedagogical value.

In terms of specifics, the following offers some early suggestions. First, avoid taking a deficit model of teachers around both digital technology and learning and teaching. Adopt alternative ontological perspectives as the basis for planning. Focus on creating an environment and digital technology platforms that encourage the co-development of contextually specific, embedded and protean learning analytics interventions. Preferably linked with activities focused on helping provide teaching staff with the opportunity to “experience powerful personal experiences” (Cavallo, 2004, p. 102) around how teaching as design combined with learning analytics can respond to their problems and desires.

In addition, the approach should focus on enabling and encouraging teacher DIY learning analytics. DIY learning analytics involves teachers customising in different ways learning analytics to fit their context. Not only is this a way to increase the pedagogical value of learning analytics but may be the only way to achieve learning analytics at scale, as Gunn et al (2005) write

…only when the…end users of technology add their requirements, experience and professional practice that mainstream integration is achieved (p. 190)

More reading

Some suggestions for more reading include

  • How to organise a child’s birthday party is a YouTube video sharing a learning story examining how different perspectives influence how to organise.
  • Cavallo (2004) makes the case for the limitations of the traditional approach in the context of schooling.
  • Gunn et al (2005) make the case that supporting teachers in repurposing learning objects is essential to ensuring adoption and sustainability of learning objects.
  • Jones and Clark (2014) outline the two different mindsets and illustrate the difference in the context of learning analytics.
  • Jones et al (2017) report on example of teacher DIY learning analytics (originally described in Jones and Clark, 2014) and draw some implications for the institutional implementation of learning analytics.
  • Learning analytics, complex adaptive systems and meso-level practitioners: A way forward offers early plans for using an alternative ontology to address the question of learning analytics within higher education.

Slides

View below or download the Powerpoint slides.

References

Cavallo, D. (2004). Models of growth – Towards fundamental change in learning environments. BT Technology Journal, 22(4), 96–112.

Dede, C. (2008). Theoretical perspectives influencing the use of information technology in teaching and learning. In J. Voogt & G. Knezek (Eds.), International Handbook of Information Technology in Primary and Secondary Education (pp. 43–62). New York: Springer.

Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2015). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicating learning success. The Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002

Gunn, C., Woodgate, S., & O’Grady, W. (2005). Repurposing learning objects: a sustainable alternative? ALT-J, 13(3), 189–200.
Mishra, P., & Koehler, M. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054.

Tyack, D., & Cuban, L. (1995). Tinkering towards utopia: A century of public school reform. Cambridge, MA: Harvard University Press.

Wiley, D. (n.d.). The Reusability Paradox. Retrieved from http://cnx.org/content/m11898/latest/

Yoo, Y., Boland, R. J., Lyytinen, K., & Majchrzak, A. (2012). Organizing for Innovation in the Digitized World. Organization Science, 23(5), 1398–1408.

Teacher DIY learning analytics – implications & questions for institutional learning analytics

The following provides a collection of information and resources associated with a paper and presentation given at ALASI 2017 – the Australian Learning Analytics Summer Institute in Brisbane on 30 November, 2017. Below you’ll find an abstract, a recording of a version of the presentation, the presentation slides and the references.

The paper examines the DIY development and use of a particular application of learning analytics (known as Know thy student) within a single course during 2015 and 2016. The paper argues that given limitations about what is known about the institutional implementation of learning analytics that examining teacher DIY learning analytics can reveal some interesting insights. The paper identifies three implications and three questions.

Three implications

  1. Institutional learning analytics currently falls short of an important goal.

    If the goal of learning analytics is that “of getting key information to a human being who can use it” (Baker, 2016, p. 607) then institutional learning analytics is falling short, and not just at a specific institution.

  2. Embedded, ubiquitous, contextual learning analytics encourages greater use and enables emergent practice.

    This case suggests that learning analytics interventions designed to provide useful contextual data appropriately embedded ubiquitously throughout the learning environment can enable significant levels of usage, including usage that was unplanned, emerged from experience, and changed practice.

    In this case, Know thy student was used by the teacher on 666 different days (~91% of the days that the tool was available) to find out more about ~90% of the enrolled students. Graphical representations below.

  3. Teacher DIY learning analytics is possible.

    Know thy student was implemented by a single academic using a laptop, widely available software (including some coding), and existing institutional data sources.

Three questions

  1. Does institutional learning analytics have an incomplete focus?

    Research and practice around the institutional implementation of learning analytics tends to appear to have a focus on “at scale”. Learning analytics that can be used across multiple courses or an entire institution. That focus appears to be at the expense of course or learning design specific, which appear to be more useful.

  2. Does the institutional implementation of learning analytics have an indefinite postponement problem?

    Aspects of Know thy student are specific to the particular learning design within a single course. The implementation of such a specific requirement would appear unlikely to have ever been undertaken by existing institutional learning analytics implementation. It would have been indefinitely postponed.

  3. If and how do we enable teacher DIY learning analytics?

    This case suggests that teacher DIY learning analytics is possible and potentially overcomes limitations in current institutional implementation of learning analytics. However, it’s also not without its challenges and limitations. Should institutions support teacher DIY learning analytics? How might that be done?

Usage

The following heat map shows the number of times Know thy student was used on each day during 2015 and 2016.

Know thy student usage clicks per day

The following bar graph contains 761 “bars”. Each bar represents a unique student enrolled in this course. The size of the bar shows the number of times Know thy student was used for that particular student. (One student was obviously used for testing purposes during the development of the tool)

Know thy student usage clicks per student

Abstract

The paper on which it is based has the following abstract.

Learning analytics promises to provide insights that can help improve the quality of learning experiences. Since the late 2000s it has inspired significant investments in time and resources by researchers and institutions to identify and implement successful applications of learning analytics. However, there is limited evidence of successful at scale implementation, somewhat limited empirical research investigating the deployment of learning analytics, and subsequently concerns about the insight that guides the institutional implementation of learning analytics. This paper describes and examines the rationale, implementation and use of a single example of teacher do-it-yourself (DIY) learning analytics to add a different perspective. It identifies three implications and three questions about the institutional implementation of learning analytics that appear to generate interesting research questions for further investigation.

Presentation recording

The following is a recording of a talk given at CQUni a couple of weeks after ALASI. It uses the same slides as the original ALASI presentation, however, without a time limit the description is a little expanded.

Slides

Also view and download here.

References

Baker, R. (2016). Stupid Tutoring Systems, Intelligent Humans. International Journal of Artificial Intelligence in Education, 26(2), 600–614. https://doi.org/10.1007/s40593-016-0105-0

Behrens, S. (2009). Shadow systems: the good, the bad and the ugly. Communications of the ACM, 52(2), 124–129.

Colvin, C., Dawson, S., Wade, A., & Gašević, D. (2017). Addressing the Challenges of Institutional Adoption. In C. Lang, G. Siemens, A. F. Wise, & D. Gaševic (Eds.), The Handbook of Learning Analytics (1st ed., pp. 281–289). Alberta, Canada: Society for Learning Analytics Research (SoLAR).

Corrin, L., Kennedy, G., & Mulder, R. (2013). Enhancing learning analytics by understanding the needs of teachers. In Electric Dreams. Proceedings ascilite 2013 (pp. 201–205).

Díaz, O., & Arellano, C. (2015). The Augmented Web: Rationales, Opportunities, and Challenges on Browser-Side Transcoding. ACM Trans. Web, 9(2), 8:1–8:30. https://doi.org/10.1145/2735633

Dron, J. (2014). Ten Principles for Effective Tinkering (pp. 505–513). Presented at the E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, Association for the Advancement of Computing in Education (AACE).

Ferguson, R. (2014). Learning analytics FAQs. Education. Retrieved from https://www.slideshare.net/R3beccaF/learning-analytics-fa-qs

Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2015). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicating learning success. The Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002

Germonprez, M., Hovorka, D., & Collopy, F. (2007). A theory of tailorable technology design. Journal of the Association of Information Systems, 8(6), 351–367.

Grover, S., & Pea, R. (2013). Computational Thinking in K-12: A Review of the State of the Field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051

Hatton, E. (1989). Levi-Strauss’s Bricolage and Theorizing Teachers’ Work. Anthropology and Education Quarterly, 20(2), 74–96.

Johnson, L., Adams Becker, S., Estrada, V., & Freeman, A. (2014). NMC Horizon Report: 2014 Higher Education Edition (No. 9780989733557). Austin, Texas. Retrieved from http://www.nmc.org/publications/2014-horizon-report-higher-ed

Jones, D., & Clark, D. (2014). Breaking BAD to bridge the reality/rhetoric chasm. In B. Hegarty, J. McDonald, & S. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 262–272).

Kay, A., & Goldberg, A. (1977). Personal Dynamic Media. Computer, 10(3), 31–41.

Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., … Wiedenbeck, S. (2011). The State of the Art in End-user Software Engineering. ACM Comput. Surv., 43(3), 21:1–21:44. https://doi.org/10.1145/1922649.1922658

Kruse, A., & Pongsajapan, R. (2012). Student-Centered Learning Analytics (CNDLS Thought Papers). Georgetown University. Retrieved from https://cndls.georgetown.edu/m/documents/thoughtpaper-krusepongsajapan.pdf

Levi-Strauss, C. (1966). The Savage Mind. Weidenfeld and Nicolson.

Liu, D. Y.-T. (2017). What do Academics really want out of Learning Analytics? – ASCILITE TELall Blog. Retrieved August 27, 2017, from http://blog.ascilite.org/what-academics-really-want-out-of-learning-analytics/

Liu, D. Y.-T., Bartimote-Aufflick, K., Pardo, A., & Bridgeman, A. J. (2017). Data-Driven Personalization of Student Learning Support in Higher Education. In A. Peña-Ayala (Ed.), Learning Analytics: Fundaments, Applications, and Trends (pp. 143–169). Springer International Publishing. https://doi.org/10.1007/978-3-319-52977-6_5

Lonn, S., Aguilar, S., & Teasley, S. D. (2013). Issues, Challenges, and Lessons Learned when Scaling Up a Learning Analytics Intervention. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 235–239). New York, NY, USA: ACM. https://doi.org/10.1145/2460296.2460343

MacLean, A., Carter, K., Lövstrand, L., & Moran, T. (1990). User-tailorable Systems: Pressing the Issues with Buttons. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 175–182). New York, NY, USA: ACM. https://doi.org/10.1145/97243.97271

Mishra, P., & Koehler, M. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054.

Norman, D. A. (1993). Things that make us smart: defending human attributes in the age of the machine. Cambridge, Mass: Perseus.

Repenning, A., Webb, D., & Ioannidou, A. (2010). Scalable Game Design and the Development of a Checklist for Getting Computational Thinking into Public Schools. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education (pp. 265–269). New York, NY, USA: ACM. https://doi.org/10.1145/1734263.1734357

Scanlon, E., Sharples, M., Fenton-O’Creevy, M., Fleck, J., Cooban, C., Ferguson, R., … Waterhouse, P. (2013). Beyond prototypes: Enabling innovation in technology‐enhanced learning. London. Retrieved from http://tel.ioe.ac.uk/wpcontent/%0Duploads/2013/11/BeyondPrototypes.pdf

Sinha, R., & Sudhish, P. S. (2016). A principled approach to reproducible research: a comparative review towards scientific integrity in computational research. In 2016 IEEE International Symposium on Ethics in Engineering, Science and Technology (ETHICS) (pp. 1–9). https://doi.org/10.1109/ETHICS.2016.7560050

Wiley, D. (n.d.). The Reusability Paradox. Retrieved from http://cnx.org/content/m11898/latest/

Wiliam, D. (2006). Assessment: Learning communities can use it to engineer a bridge connecting teaching and learning. JSD, 27(1).

Yoo, Y., Boland, R. J., Lyytinen, K., & Majchrzak, A. (2012). Organizing for Innovation in the Digitized World. Organization Science, 23(5), 1398–1408.

Zittrain, J. L. (2006). The Generative Internet. Harvard Law Review, 119(7), 1974–2040.

Improving teacher awareness, action and reflection on learner activity

The following post contains the content from a poster designed for the 2017 USQ Toowoomba L&T celebration event. It provides some rationale for a technology demonstrator at USQ based on the Moodle Activity Viewer.

What is the problem?

Learner engagement is a key to learner success. Most definitions of learner engagement include “actively participating, interacting, and collaborating with students, faculty, course content and members of the community” (Angelino & Natvig, 2009, p. 3).

70% of USQ students study online. By mid-November 2017, 26,754 students had been active in USQ’s Moodle LMS.

In online learning, the absence of visual cues makes teacher awareness of student activity difficult (Govaerts, Verbert, & Duval, 2011).  Richardson (2011) identifies “the role which teaching staff play in inspiring, challenging and engaging students” as “perhaps the most woefully neglected aspect of quality in higher education” (p. 2)

Learning analytics (LA) is the “use of (big) data to provide actionable intelligence for learners and teachers” (Ferguson, 2014). However, current tools provide poor data aggregation, poor visualisation capabilities and have other limitations that inhibit teachers’ ability to: understand student activity; respond appropriately; and, reflect on course design (Dawson & McWilliam, 2008; Corrin et al, 2013; Jones, & Clark, 2014).

How will it be addressed?

Teachers can be supported through tools that help them “analyse, appraise and improve practices in their everyday activity systems” (Knight et al, 2016, p. 337).

This Technology Demonstrator has implemented and will customise and scaffold the use of the Moodle Activity Viewer (MAV) within the USQ activity system.

The MAV provides a useful and easy to use tool that provides representations of student activity from within all Moodle learning spaces. It provides affordances to support teacher intervention and further analysis.

MAV - How many students

MAV’s overlay answering the question how many and what percentage of students have accessed each Moodle activity & resource?

What are the expected outcomes?

The project aims to explore two questions:

  1. If and how does the provision of contextual, useful, and easy to use representations of online learner activity help teachers analyse, appraise and improve their practices?
  2. If and how does this change in teacher activity influence learner activity and learning outcomes?

MAV - How many clicks

MAV’s overlay answering the question how many times have those students clicked on each Moodle activity & resource?

Want to learn more?

Ask for a demostration of MAV during the poster session.

USQ staff can learn more* about and start using MAV from http://tiny.cc/aboutmav and http://tiny.cc/installmav

* (Only from a USQ campus or via the USQ VPN)

MAV - How many students in a forum

MAV’s overlay answering the question how many and what percentage of students have read posts in this introductory activity?

MAV - Who accessed and how to contact them

MAV’s student access dialog providing details of and enabling teacher contact with the students who have accessed the “Fix my class IWB” forum?

References

Angelino, L. M., & Natvig, D. (2009). A Conceptual Model for Engagement of the Online Learner. Journal of Educators Online, 6(1), 1–19.

Corrin, L., Kennedy, G., & Mulder, R. (2013). Enhancing learning analytics by understanding the needs of teachers. In Electric Dreams. Proceedings ascilite 2013 (pp. 201–205).

Dawson, S., & McWilliam, E. (2008). Investigating the application of IT generated data as an indicators of learning and teaching performance. Queensland University of Technology and the University of British Columbia.

Ferguson, R. (2014). Learning analytics FAQs. Education. Retrieved from https://www.slideshare.net/R3beccaF/learning-analytics-fa-qs

Govaerts, S., Verbert, K., & Duval, E. (2011). Evaluating the Student Activity Meter: Two Case Studies. In Advances in Web-Based Learning – ICWL 2011 (pp. 188–197). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-25813-8_20

Jones, D., & Clark, D. (2014). Breaking BAD to bridge the reality/rhetoric chasm. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Proceedings of the 31st Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2014) (pp. 262–272). Sydney, Australia: Macquarie University.

Knight, P., Tait, J., & Yorke, M. (2006). The professional learning of teachers in higher education. Studies in Higher Education, 31(3), 319–339. https://doi.org/10.1080/03075070600680786

Exploring options for teacher DIY learning analytics


Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

A few of us recently submitted a paper to ALASI’2017 that examined a “case study” of a teacher (me) engaging in a bit of DIY learning analytics. The case was used to drawing a few tentative conclusions and questions around the institutional implementation of learning analytics. The main conclusion is that teacher DIY learning analytics is largely ignored at the institutional level and that there appears to be a need and value to support it. The question is how (and then if supported, what happens)?

This post is the start of an exploration of some technologies that combined may offer some of the affordances necessary to supporting teacher DIY learning analytics. The collection of technologies and the approach owes a significant amount of inspiration to Tony Hirst, especially in this post in which he writes

What I care about are some of the features that Docker has, and how I can use those features to make my own life easier, … supporting personal, DIY, BYOA (“bring your own app”) IT that works at an individual level in the form of end-user applications, or personal digital workbenches

The plan/hope here is that Docker combined with some other technologies can provide a platform to enable a useful combination of do-it-with (DIW) and do-it-yourself (DIY) paths for the institutional implementation of learning analytics. The follow is mostly documenting ad hoc exploration of the technologies.

In the end, I’ve been able to get working a Jupyter notebook working as a JSON API and started explorer docker containers. Laid the ground work for the next step which will be to explore how and if some of this can be combined to integrate some of the work Hazel is doing with some of the Indicators work from earlier in the year.

Learning more – Juypter notebook JSON api

Tony provides a description of using Jupyter Notebooks to provide a JSON API. Potentially this provides a way for DIY teachers to create their own MAV-like server.

Tony’s exploration is informed by this from some aspect of IBM that aims to introduce the Jupyter kernel gateway (github repo)

The README.md from github repo mentions serving HTTP requests from “annotated notebook cells”. Suggesting that the method of annotation will be important. The IBM example code that each API call is handled by a particular block starting with an appropriately formatted comment i.e.

single-line comments containing a HTTP verb … followed by a parameterised URL path

Have a simple example working.

Deploying – user experience

The IBM bit then goes about using Docker to to deploy this API. But before I do that. Lets get some experience at the user en with Tony’s example.

  1. Install VirtualBox
    Question: Is this something a standard user can do?
  2. Install vagrant
  3. command line to install a vagrant plugin

    Question: Too much? But can probably be worked around.

  4. Download the repo as a zip file.

    Had to figure out to go back to the repo “home” to get the download option (long time between drinks doing this).

  5. Run the vagrant file

    Ok, it’s downloading the file from the vagrant server (from the ouseful area on Vagrant).

    It’s a 1.66Gb file. That size could potentially be an issue, suggesting the need for a local copy. Especially given the slow download.

    An hour or two later and it is up and running. There’s a GUI linux box running on my Mac.

Don’t know a great deal about the application that is the focus, but it appears to work. It’s a 3D application, so the screen refresh isn’t all that fast. But as a personal server for DIY teacher analytics, it should work fine, at least in terms of speed.

Running it a second time includes a check to see if it’s up to date and then up it pops.

The box appears to have Perl, Python and Juypter installed.

Deploying – developing a docker/container/images

This raises the question of the best option for creating and sharing a docker/container/insert appropriate term – I’ll go with images – that has Jupyter notebooks and the kernel_gateway tool running. At this stage, this purpose seems best served by a headless virtual machine with browser-based communication the method for interacting with Jupyter notebooks.

Tony appears to do exactly this (using OpenRefine) using Kitematic in this post. Later in the post the options appear to include

  • Sharing images publicly via the Dockerhub registry
  • Use a private Dockerhub registry (one with the free plan)
  • On a local computer
  • Run your own image registry
  • And, I assume use an alternative.

Tony sees using the command line a draw back for running your own. Perhaps not the biggest problem in my case. But what is the best approach?

Dockerhub and its ilk do appear to provide extra help (e.g. official repositories you can build upon).

One set of alternatives appear largely focused on supporting central IT, not the end user. Echoing a concern expressed by Tony.

Intro from another alternative suggests that docker is becoming more generic. Time to look and read further afield.

Intro to containers

From Medium

  • Containers abstract the OS etc to make it simple to deploy
  • Containers usually measured in 10s of megabytes
  • Big distinction made between containers and virtual machines, perhaps boils down to “containers virtualise the OS; virtual machines the hardware”

    Though interesting, the one tried above required the downloading of a virtual machine first. Update: That appears to be because I’m running Mac OS X. If I were on a Linux box, I probably wouldn’t have needed that.

  • The following seem to resonate most with the needs of teacher DIY learning analytics
    • Using containers can decrease the time needed for development, testing, and deployment of applications and services.
    • Testing and bug tracking also become less complicated since you there is no difference between running your application locally, on a test server, or in production.
    • Container-based virtualization are a great option for microservices, DevOps, and continuous deployment.
  • Docker is based on Linux and open source, is the big player.
  • Spends some attention on container orchestration – appears to be focused on enterprise IT.

Following offers a creative intro to Kubernetes

Starts with the case for containers (Docker), but then moves onto orchestration and the need for Kubernetes. Puts containers into a pod, perhaps with more than one if tightly coupled. Goes onto to explain the other features provided by Kubernetes.

And intro to Docker

Rolling my own

Possible technology options

Do the following and I have a web server running in Docker that I can access from my Mac OS browser.

AA17-00936:docker david$ docker run -d -p 80:80 --name webserver nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
afeb2bfd31c0: Pull complete 
7ff5d10493db: Pull complete 
d2562f1ae1d0: Pull complete 
Digest: sha256:af32e714a9cc3157157374e68c818b05ebe9e0737aac06b55a09da374209a8f9
Status: Downloaded newer image for nginx:latest
f1f6925acc31f80faf726358f8de5712458ff3649d2c0626bf3bb37f11d1b070
AA17-00936:docker david$

Dig into tutorials and have a play

Docker share a git repo for tutorials and labs. Which are quite good and useful.

Getting set up with some advice above.

Running your first container includes some simple commands. e.g. to show details of installed images. Showing that they can be quite small.

Question: To have folk install Docker, or do the VM route as above?

AA17-00936:docker david$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              latest              2d696327ab2e        11 days ago         122MB
nginx               latest              da5939581ac8        2 weeks ago         108MB
alpine              latest              76da55c8019d        2 weeks ago         3.97MB
hello-world         latest              05a3bd381fc2        2 weeks ago         1.84kB

Web apps with docker, which also starts looking at the process of rolling your own.

This is where discussion of different types of images commence

  • Base (e.g. an OS) and child images which add functionality to a base image
  • Official images – sactioned by docker
  • user images

Process can be summarised as

  • Create the app (example is using a Python web framework – Flask)
  • Add in a Dockerfile – text file of commands for the Docker daemon when creating an image
  • Build the image

    Does require an account on the Docker cloud

    And there it goes getting all the pre-reqs etc. Quite quick.

And successful running.

Docker Swarm running multiple copies, including on the cloud. Given the use case I’m interested in is people running their own…not a priority.

It does provide a look at Docker Compose files and a more complex application – multiple containers and two networks. Given my focus on using Jupyter Notebooks and perhaps the kernel gateway, this may be simplified a bit.

Seems we’re at the stage of actually trying to do something real.

Create a Docker image – TDIY

Jupyter Notebook, kernel gateway and a simple collection of notebooks – perhaps with greasemonkey script

Misc. related stuff

Bit on microservices (microservice architectural style) pointing out the focus on

principles of loose coupling and high cohesion of services

and in turn a number of characteristics

  • Applications are made up of small independent services

    Is TDIY LA about allowing teachers to create applications by combining these services?

  • Services are independently modifiable and (re)deployable

    But by whom?

  • Decentalised data management: each service can have its own database

    What about each user?

Goes on to list a range of advantages, but the disadvantages include

  • inefficiency – remote calls, network latency, potential duplication etc.

    But going local might help address some of this.

  • Developing a user case could need the cooperation of multiple teams

    This is the biggest barrier to implementation within an instituiton. But raises the spectre of shadow systems, kludges etc.

  • complications in debugging, communication

Microservices and containers covers some of the alternatives.

Seems docker is the place — it’s bought Kitematic and apparently not loved it – a risk for basing the DIY approach on it.

Another part of the story is that you can build your own images and either share them publicly via the Dockerhub registry, keep them locally on your own computer, post them to a private Dockerhub repository (you get a single private repository as part of the Dockerhub free plan, or can pay for more…), or run your own image registry.

Dockerhub is probably the option I want to use here because of the focus on being open, of being cross institutional etc.

Learning, learning analytics and multiple levels: The problem of starvation

In which I play with some analytics and use some literature in an attempt to understand why the institutional implementation of learning analytics as a starvation problem (like most institutional attempts to leverage digital technologies). In this context, I’m using the definition of starvation from computer science.

Multiple time scales of human behaviour and appropriate methods

In a section titled “Data from learning on multiple levels: learning is complex”, Reiman (2016) writes

Nathan & Alibali (2010) distinguish between learning in milliseconds and below (biological), seconds (cognitive), minutes to hours (rational), days to months (socio-cultural), and years and beyond (organizational) (p. 134)

Digging into Nathan & Alibali (2010) reveals the following table titled “Time scales of human behaviour and the corresponding areas of study and research methods” (which is in turn adapted from elsewhere).

Time Scales of Human Behavior and the Corresponding Areas of Study and Research Methods

What I find interesting about this is that it places the types of activities a teacher would do at a very different time scale than what an organisation would do. It also suggests that there are very different methods to be used at these very different levels. Suggesting that using a method in the wrong scale will be less than appropriate.

To perhaps draw a long bow, might this suggest that the methods currently being used to support institutional implementation of learning analytics might work fine at the organisational level, but perhaps a little less so at the level of teaching practice? Or perhaps, the limitation arises from the inability to move up and down levels quickly enough.

(Aside: Especially when you consider that the table above doesn’t (for me) capture the full complexity of reality. Capturing the time scales is important, but I think it fails to capture the fact that as you move down the level of study you are increasing the quantity and diversity of units of study. i.e. At a system level you might be talking about the Australian higher education sector and its use of learning analytics. There are ~38 universities in Australia that could be studied at the organisational level. Within those organisations there are likely to be 1000s of courses, at least 100s of teaching staff, and 10000s of students. Each very diverse.)

A case in point

A team I’m a part of has created a online resource that has been used by staff. From the institutionally provided system I can grab the raw logs and perhaps generate the odd image. But I can’t do easily do the sort of analysis I’d like to do. What works at the institutional level, doesn’t work at the individual course/case level.

But I can engage in some DIY. Jupyter notebooks, Python, plotly, a bit of faffing around with CSV files and….

  • Get a list of the people who accessed the resources and how many times they did and turn it into a graph.
  • Do the same with the different parts of the resource.

    Though it’s a bit more difficult given the limited data in the CSV file

  • And plot the usage against days.

None of this is technically difficult. It’s also something that could be useful to others. But they don’t have the knowledge and the institution doesn’t appear to provide the service.

This particular need is unlikely to receive direct attention. It may well get solved when some enterprise bit of kit gets new functionality. But, if that happens, chances are it will go under-used as users aren’t aware of the capability and the technical folk aren’t aware of the need.

A solution?

What if there were methods by which the institution could move up and down the layers? Dip down into the level of hours and days, build some stuff, see it used and then scale up the useful stuff so its “enterprise”?

References

Reimann, P. (2016). Connecting learning analytics with learning research: the role of design-based research. Learning: Research and Practice, 2(2), 130–142. https://doi.org/10.1080/23735082.2016.1210198

Nathan, M. J., & Wagner Alibali, M. (2010). Learning sciences. Wiley Interdisciplinary Reviews: Cognitive Science, 1(3), 329–345. https://doi.org/10.1002/wcs.54

Learning analytics, quality indicators and meso-level practitioners

When it comes to research I’ve been a bit of failure, especially when measured against some of the more recent strategic and managerial expectations. Where are those quartile 1 journal articles? Isn’t your h-index showing a downward trajectory?

The concern generated by these quantitative indicators not only motivated the following ideas for a broad research topic, but also is one of the issues to explore within the topic. The following outlines early attempts to identify a broader research topics that is relevant enough for current sector and institutional concerns; provides sufficient space for interesting research and contribution; aligns nicely (from one perspective) with my day job; and, will likely provide a good platform for a program of collaborative research.

The following:

  1. explains the broad idea for research topic within the literature; and,
  2. describes the work we’ve done so far including two related examples of the initial analytics/indicators we’ve explored.

The aim here is to be generative. We want to do something that generates mutually beneficial collaborations with others. If you’re interested, let us know.

Research topic

As currently defined the research topic is focused around the design and critical evaluation of the use and value of a learning analytics platform to support meso-level practitioners in higher education to engage with quality indicators of learning and teaching.

Amongst the various aims, are an intent to:

  • Figure out how to design and implement an analytics platform that is useful for meso-level practitioners.
  • Develop design principles for that platform informed by the analytics research, but also ideas from reproducible research and other sources.
  • Use and encourage the use by others of the platform to:
    1. explore what value (if any) can be extracted from a range of different quality indicators;
    2. design interventions that can help improve L&T; and,
    3. to enable for a broader range of research – especially critical research – around the use of quality indicators and learning analytics for learning and teaching.

Quality Indicators

The managerial turn in higher education has increased the need for and use of various indicators of quality, especially numeric indicators (e.g. the number of Q1 journal articles published, or not). Kinash et al (2015) state the quantifiable performance indicators are important to universities because they provide “explicit descriptions of evidence against which quality is measured” (p. 410). Chalmers (2008) offers the following synthesized definition of performance indicators

measures which give information and statistics context; permitting comparisons between fields, over time and with commonly accepted standards. They provide information about the degree to which teaching and learning quality objectives are being met within the higher education sector and institutions. (p. 10)

However, the generation and use of these indicators is not without issues.

There is also a problem with a tendency to rely on quantitative indicators. Quantitative indicators provide insight into “how much or how many, but say little about quality” (Chalmers & Gardiner, 2015, p. 84). Ferguson and Clow (2017) – writing in the context of learning analytics – argue the good quality qualitative research needs to support good-quality quantitative research because “we cannot understand the data unless we understand the context”. Similarly, Kustra et al (2014) suggest that examining the quality of teaching requires significant qualitative indicators to “provide deeper interpretation and understanding of the measured variable”. Qualitative indicators are used by Universities to measure performance in terms of processes and outcomes,however, “because they are more difficult to measure and often produce tentative results, are used less frequently” (Chalmers & Gardiner, 2015, p. 84)

Taking a broader perspective there are problems such as Goodhart’s law and performativity. As restated by Strathern (1997), Goodhart’s Law is ‘When a measure becomes a target, it ceases to be a good measure’ (p. 308) Elton (2004) describes Goodhart’s Law as “a special case of Heisenberg’s Uncertainty Principle in Sociology, which states that any observation of a social system affects the system both before and after the observation, and with unintended and often deleterious consequences” (p. 121). When used for control and comparison purposes (e.g league tables) indicators “distort what is measured, influence practice towards what is being measured and cause unmeasured parts to get neglected” (Elton, 2004, p. 121).

And then there’s the perception that quality indicators and potentially this whole research project becomes an unquestioning part of part of performativity and all of the issues that generates. Ball (2003) outlines the issues and influence of the performative turn in institutions. He describes performativity as

a technology, a culture and a mode of regulation that employs judgements, comparisons and displays as means of incentive, control, attrition and change ^ based on rewards and sanctions (both material and symbolic). The performances (of individual subjects or organizations) serve as measures of productivity or output, or displays of ‘quality’, or ‘moments’ of promotion or inspection (Ball, 2003, p. 216)

All of the above (and I expect much more) all point to there being interesting and challenging questions to explore and answer around quality indicators and beyond. I do hope that any research we do around this topic engages with the necessary critical approach to this research. As I re-read this post now I can’t help but see echoes of a previous discussion Leigh and I have had around inside out, outside in, or both. This approach is currently framed as an inside out approach. An approach where those inside the “system” are aware of the constraints and work to address those. The question remains whether this is possible.

Learning analytics

Siemens and Long (2011) define LA as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (p. 34). The dominant application of learning analytics has focused on “predicting student learning success and providing proactive feedback” (Gasevic, Dawson and Siemens, 2015), often driven by an interest in increasing student retention and success. Colvin et al (2016) found two distinct trajectories of activity in learning analytics within Australian higher education. The first were ultimately motivated by measurement and retention implemented specific retention related learning analytics programs. The second saw retention as a consequence of the broader learning and teaching experience and “viewed learning analytics as a process to bring understanding to learning and teaching practices” (Colvin et al, 2016, p. 2).

Personally, I’m a fan of the second trajectory and see supporting that trajectory as a major aim for this project.

Not all that surprisingly, learning analytics has been applied to the question of quality indicators. Dawson and McWilliam (2008) explored the use of “academic analytics” to

address the need for higher education institutions (HEIs) to develop and adopt scalable and automated measures of learning and teaching performance in order to evaluate the student learning experience (p. 1)

Their findings included (emphasis added):

  • “LMS data can be used to identify significant differences in pedagogical approaches adopted at school and faculty levels”
  • “provided key information for senior management for identifying levels of ICT adoption across the institution, ascertaining the extent to which teaching approaches reflect the strategic institutional priorities and thereby prioritise the allocation of staff development resources
  • refining the analysis can identify “further specific exemplars of online teaching” and subsequently identify “‘hotspots’ of student learning engagement”; “provide lead indicators of student online community and satisfaction”; and, identify successful teaching practices “for the purposes of staff development activities and peer mentoring”

Macfadyen and Dawson (2012) provide examples of how learning analytics can reveal data that offer “benchmarks by which the institution can measure its LMS integration both over time, and against comparable organizations” (p. 157). However, the availability of such data does not ensure use in decision making. Macfadyen and Dawson (2012) also report that the availability of patterns generated by learning analytics did not generate critical debate and consideration of the implications of such data by the responsible organisational committee and thus apparently failed to influence institutional decision-making.

A bit more surprising, however, is that in my experience there doesn’t appear to have been a concerted effort to leverage learning analytics for these purposes. Perhaps this is related to findings from Colvin et al (2016) that even with all the attention given to learning analytics there continues to be: a lack of institutional exemplars; limited resources to guide implementation; and perceived challenges in how to effectively scale learning analytics across an institution. There remains little evidence that learning analytics has been helpful in closing the loop between research and practice, and made an impact on university-wide practice (Rogers et al, 2016).

Even if analytics is used, there are other questions such as the role of theory and context. Gasevic et al (2015) argue that while counting clicks may provide indicators of tool use it is unlikely to reveal insights of value for practice or the development of theory. If learning analytics is to achieve an a lasting impact on student learning and teaching practice it will be necessary to draw of appropriate theoretical models (Gasevic et al, 2015). Rogers et al (2016) illustrate how such an approach “supports an ever-deepening ontological engagement that refines our understanding and can inform actionable recommendations that are sensitive to the situated practice of educators” (p. 245). If learning analytics aims to enhance to learning and teaching, it is crucial that it engages with teachers and their dynamic contexts (Sharples et al., 2013). Accounting for course and context specific instructional conditions and learning designs are increasingly seen as imperatives for the use of learning analytics (Gaservic et al, 2015; Lockyer et al, 2013)

There remain many other questions about learning analytics. Many of those questions are shared with the use of quality indicators. There is also the question of how learning analytics can harnessed via means that are sustainable, scale up, and at the same time provide contextually appropriate support. How can the tensions between the need for institutional level quality indicators of learning and teaching, and the inherently contextually specific nature of learning and teaching?

Meso-level practitioners

The limited evidence of impact from learning analytics on learning and teaching practice may simply be a mirror of the broader difficulty that universities have had with other institutional learning technologies. Hannon (2013) explains that when framed as a technology project the implementation of institutional learning technologies “risks achieving technical goals accompanied by social breakdowns or failure, and with minimal effect on teaching and learning practices” (p. 175). Breakdowns that arise, in part, from the established view of enterprise technologies. A view that sees enterprise technologies as unable to be changed, and “instead of optimizing our machines for humanity – or even the benefit of some particular group – we are optimizing humans for machinery” (Rushkoff, 2010, p. 15).

Jones et al (2006) describe the term meso-level to describe the “level that was intermediate between small scale, local interaction, and large-scale policy and institutional procesess” (p. 37). Hannon (2013) describes meso-level practitioners as the “teaching academics, learning technologists, and academic developers” (p. 175) working between the learning and teaching coal-face and the institutional context defined by an institution’s policies and technological systems. These are the people who can see themselves as trying to bridge the gaps between the institutional/technological vision (macro-level) and the practical coal-face realities (micro-level). These are the people who are often required to help “optimise humans for machinery”, but who would generally prefer to do the reverse. Hannon (2013) also observes thateEven though there has been significant growth in the meso-level within contemporary higher education, research has continued to largely focused on the macro or micro levels.

My personal experience suggests that the same can be said about the design and use of learning analytics. Most institutional attempts are focused at either the macro or micro level. The macro level focused largely on large-scale student retention efforts. The micro level focused on the provision of learning analytics dashboards and other tools to teaching staff and students. There has been some stellar work by meso-level practitioners in developing supports for the micro-level (e.g. Liu, Bartimote-Aufflick, Pardo, & Bridgeman, 2017). However, much of this work has been in spite of the affordances and support offered by the macro-level. Not enough of the work, beyond the exceptions already cited, appears to have actively attempted to help optimise the machinery for the humans. In addition, there doesn’t appear to be a great deal of work – beyond the initial work from almost 10 years ago – focused on if and how learning analytics can help meso-level practitioners in the work that they do.

As a result there are sure to be questions to explore about meso-level practitioners, their experience and impact on higher education. Leigh Blackall has recently observed that the growth in meso-level practitioners in the form of “LMS specialists and ed tech support staff” comes with the instruction that they “focus their attentions on a renewed sense of managerial oversight”. Implicating meso-level practitioners in questions related to performativity etc. Leigh also positions these meso-level practitioners as examples of disabling professions. Good pointers to some of the more critical questions to be asked about this type of work.

Can meso-level practitioners break out, or are we doomed to be instruments of performativity? What might it take to break free? How can learning analytics be implemented in a way that allows it to be optimised for the contextually specific needs of the human beings involved, rather than require the humans to be optimised for the machinery? Would such a focus improve the quality of L&T?

What have we done so far?

Initial work has focused on developing an open, traceable, cross-institutional platform for exploring learning analytics. In particular, exploring how recent ideas such as reproducible research and insights from learning analytics might help design a platform that enables meso-level practitioners to break some of the more concerning limitations of current practice.

We’re particularly interested in ideas from Elton (2004) where bottom-up approaches might “be considerably less prone to the undesirable consequences of Goodhart’s Law” (p. 125). A perspective that resonates with our four paths idea for learning analytics. i.e. That it’s more desirable and successful to follow the do-it-with learners and teachers or learner/teacher DIY paths.

The “platform” is seen as an enabler for the rest of the research program. Without a protean technological platform – a platform we’re able to tailor to our requirements – it’s difficult to see how we’d be able to effectively support the deeply contextual nature of learning and teaching or escape broader constraints such as performativity. This also harks back to my disciplinary background as a computer scientist. In particular, the computer scientist as envisioned by Brooks (1996) as a toolsmith whose delight “is to fashion powertools and amplifiers for minds” (p. 64) and who “must partner with those who will use our tools, those whose intelligences we hope to amplify” (p. 64).

First steps

As a first step, we’re revisiting our earlier use of Malikowski, Thompson & Theis (2007) to look at LMS usage (yea, not that exciting, but you have to start somewhere). We’ve developed a set of Python classes that enable the use of the Malikowski et al (2007) LMS research model. That set of classes has been used to develop a collection of Jupyter notebooks that help explore LMS usage in a variety of ways.

The theory is that these technologies (and the use of github to share the code openly) should allow anyone else to perform these same analysis with their LMS/institution. So far, the code is limited to working only with Moodle. However, we have been successful in sharing code between two different installations of Moodle. i.e. one of us can develop some new code, share it via github, and the other can run that code over their data. A small win.

The Malikowski et al (2007) model groups LMS features by the following categories: Content, Communication, Assessment, Evaluation and Computer-Based Instruction. It also suggests that tool use occurs in a certain order and with a certain frequency. The following figure (click on it to see a larger version) is a representation of the Malikowski model.

Malikowski Flow Chart

Looking for engagement?

Dawson and McWilliam (2008) suggested that academic analytics could be used to identify “potential “hotspots” of student learning engagement” (p. 1). Assuming that the number of times students click within an LMS course is a somewhat useful proxy for engagement (a big question), then this platform might allow you to:

  1. Select a collection of courses.

    This might be all the courses in a discipline that scored well (or poorly) on some other performance indicator, all courses in a semester, all large first year courses, all courses in a discipline etc.

  2. Visualise the number of total student clicks within each course clicked on LMS functionality in each of the Malikowski categories.
  3. Visualise the number of clicks per student within each course in each Malikowski category.

These visualisations might then provide a useful indication of something that is (or isn’t) happening. An indication that would not have been visible otherwise and is worthy of further exploration via other means (e.g. qualitative).

The following two graphs were generated by our platform and are included here to provide a concrete example of the above process. Some features of the platform that the following illustrates

  • It generates artefacts (e.g. graphs, figures) that can be easily embedded anywhere on the web (e.g. this blog post). You don’t have to be using out analytics platform to see the artefacts.
  • It can anonymise data for external display. For example, courses in the following artefacts have been randomly given people’s names rather than course codes/names.

Number of total student clicks

The first graph shows a group of 7 courses. It shows the number of students enrolled in each course (e.g. the course Michael has n=451) and the bars represent the total number of clicks by enrolled students on the course website. The clicks are grouped according to the Malikowski categories. If you roll your mouse over one of the bars, then you should see displayed the exact number of clicks for each category.

For example, the course Marilyn with 90 students had

  • 183,000+ clicks on content resources
  • 27,600+ clicks on communication activities
  • 5659 clicks on assessment activities
  • and 0 for evaluation of CBI

Total number of clicks isn’t all that useful for course comparisons. Normalising to clicks per enrolled student might be useful.


 

 

 

 

Clicks per student

The following graph uses the same data as above, however, the number of clicks is now divided by the number of enrolled students. A simple change in analysis that highlights differences between courses.

2000+ clicks on content per student certainly raises some questions about the Marilyn course. Whether that number is good, bad, or meaningless would require further exploration.

 


 

 

 

 

What’s next?

We’ll keep refining the approach, some likely work could include

  • Using different theoretical models to generate indicators.
  • Exploring how to effectively supplement the quantitative with qualitative.
  • Exploring how engaging with this type of visualisation might be a useful as part of professional learning.
  • Exploring if these visualisations can be easily embedded within the LMS, allowing staff and students to see appropriate indicators in the context of use.
  • Exploring various relationships between features quantitatively.

    For example, is there any correlation between results on student evaluation and Malikowski or other indicators? Correlations between disciplines or course design?

  • Combining the Malikowski model with additional analysis to see if it’s possible to identify significant changes in the evolution of LMS usage over time.

    e.g. to measure the impact of organisational policies.

  • Refine the platform itself.

    e.g. can it be modified to support other LMS?

  • Working with a variety of people to explore what different questions they might wish to answer with this platform.
  • Using the platform to enable specific research project.

And a few more.

Want to play? Let me know. The more the merrier.

References

Ball, S. J. (2003). The teacher’s soul and the terrors of performativity. Journal of Education Policy, 18(2), 215–228. https://doi.org/10.1080/0268093022000043065

Brooks, F. (1996). The Computer Scientist as Toolsmith II. Communications of the ACM, 39(3), 61–68.

Chalmers, D. (2008). Indicators of university teaching and learning quality.

Chalmers, D., & Gardiner, D. (2015). An evaluation framework for identifying the effectiveness and impact of academic teacher development programmes. Studies in Educational Evaluation, 46, 81–91. https://doi.org/10.1016/j.stueduc.2015.02.002

Colvin, C., Wade, A., Dawson, S., Gasevic, D., Buckingham Shum, S., Nelson, K., … Fisher, J. (2016). Student retention and learning analytics : A snapshot of Australian practices and a framework for advancement. Canberra, ACT: Australian Government Office for Learning and Teaching. Retrieved from http://he-analytics.com/wp-content/uploads/SP13-3249_-Master17Aug2015-web.pdf

Dawson, S., & McWilliam, E. (2008). Investigating the application of IT generated data as an indicators of learning and teaching performance. Queensland University of Technology and the University of British Columbia.

Elton, L. (2004). Goodhart’s Law and Performance Indicators in Higher Education. Evaluation & Research in Education, 18(1–2), 120–128. https://doi.org/10.1080/09500790408668312

Ferguson, R., & Clow, D. (2017). Where is the Evidence?: A Call to Action for Learning Analytics. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 56–65). New York, NY, USA: ACM. https://doi.org/10.1145/3027385.3027396

Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://doi.org/10.1007/s11528-014-0822-x

Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2015). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicating learning success. The Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002

Hannon, J. (2013). Incommensurate practices: sociomaterial entanglements of learning technology implementation. Journal of Computer Assisted Learning, 29(2), 168–178. https://doi.org/10.1111/j.1365-2729.2012.00480.x

Jones, C., Dirckinck‐Holmfeld, L., & Lindström, B. (2006). A relational, indirect, meso-level approach to CSCL design in the next decade. International Journal of Computer-Supported Collaborative Learning, 1(1), 35–56. https://doi.org/10.1007/s11412-006-6841-7

Kinash, S., Naidu, V., Knight, D., Judd, M.-M., Nair, C. S., Booth, S., … Tulloch, M. (2015). Student feedback: a learning and teaching performance indicator. Quality Assurance in Education, 23(4), 410–428. https://doi.org/10.1108/QAE-10-2013-0042

Liu, D. Y.-T., Bartimote-Aufflick, K., Pardo, A., & Bridgeman, A. J. (2017). Data-Driven Personalization of Student Learning Support in Higher Education. In A. Peña-Ayala (Ed.), Learning Analytics: Fundaments, Applications, and Trends (pp. 143–169). Springer International Publishing.

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing Pedagogical Action: Aligning Learning Analytics With Learning Design. American Behavioral Scientist, 57(10), 1439–1459. https://doi.org/10.1177/0002764213479367

Macfadyen, L., & Dawson, S. (2012). Numbers Are Not Enough. Why e-Learning Analytics Failed to Inform an Institutional Strategic Plan. Educational Technology & Society, 15(3), 149–163.

Malikowski, S., Thompson, M., & Theis, J. (2007). A model for research into course management systems: bridging technology and learning theory. Journal of Educational Computing Research, 36(2), 149-173.

Rogers, Tim, Dawson, Shane, & Gasevic, Dragan. (2016). Learning Analytics and the Imperative for Theory-Driven Research. In The SAGE Handbook of E-learning Research (2nd ed., pp. 232–250).

Rushkoff, D. (2010). Program or be programmed: Ten commands for a digital age. New York: OR Books.

Sharples, M., Mcandrew, P., Weller, M., Ferguson, R., Fitzgerald, E., & Hirst, T. (2013). Innovating Pedagogy 2013: Open University Innovation Report 2 (No. 9781780079370). Milton Keynes: UK. Retrieved from http://www.open.ac.uk/blogs/innovating/

Siemens, G., & Long, P. (2011). Penetrating the Fog: Analytics in Learning and Education. EDUCAUSE Review, 46(5). Retrieved from http://moourl.com/j6a5d

Strathern, M. (1997). “Improving ratings”: audit in the British University system. European Review, 5(3), 305–321. https://doi.org/10.1002/(SICI)1234-981X(199707)5:3<305::AID-EURO184>3.0.CO;2-4

Emedding plotly graphs in WordPress posts


Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Last year I started using with Perl to play with analytics around Moodle Book usage. This year, @beerc and I have been starting to play with Jupyter Notebooks and Python to play with analytics for meso-level practitioners (Hannon, 2013). Plotly provides a fairly useful platform for generating graphs of various types and sharing the data. Works well with a range of languages and Jupyter Notebooks.

Question here is how well it works with WordPress. WordPress has some (understandable) constraints around embedding external HTML in WordPress posts/pages. But there is a large set of community contributed plugins to WordPress that help with this, including a couple that apparently work with Plotly.

  • wp-plotly designed to embed a Plotly hosted graph by providing the plotly URL. Doesn’t appear to work with the latest version of WordPress. No go
  • Plot.wp provides a WordPress shortcode for Plotly (plotly and /plotly with square brackets) into which you place Plotly json data and hey presto graph. Has a github repo and actually works with the latest version of WordPress.

How to produce JSON from Python

I’m a Python newbie. Don’t really grok it the way I did Perl. I assumed it should be possible to auto-generate the json from the Python code, but how.

Looks like this will work in a notebook, though it does appear to need the resulting single quotes converted into double quotes and two sets of double quotes removed to be acceptable JSON.

#.. Python code to produce plotly figure ready to be plotted
import json
jsonData['data'] = json.dumps( fig['data'])
jsonData['layout'] = json.dumps( fig['layout'])
jsonData

For the graph I’m currently playing with, this ends up with

{"layout": {"yaxis": {"range": [0, 100], "title": "% response rate"}, "title": "EDC3100 Semester 2 MyOpinion % Response Rate", "xaxis": {"ticktext": ["2014 (n=106)", "2015 (n=88)nLeaderboard", "2016 (n=100)nLeaderboard"], "title": "Year", "tickvals": ["2014", "2015", "2016"]}}, 
  "data": [{"type": "bar", "name": "EDC3100", "x": ["2014", "2015", "2016"], "y": [34, 48, 49]}, {"type": "scatter", "name": "USQ average", "x": ["2015", "2016"], "y": [26.83, 23.52]}]}

And the matching graph produced by plotly follows. Roll over the graph to see some “tooltips”.

References

Hannon, J. (2013). Incommensurate practices: sociomaterial entanglements of learning technology implementation. Journal of Computer Assisted Learning, 29(2), 168–178. https://doi.org/10.1111/j.1365-2729.2012.00480.x

Understanding systems conditions for sustainable uptake of learning analytics

My current institution is – like most other universities – attempting to make some use of learning analytics. The following uses a model of system conditions for sustainable uptake of learning analytics from Colvin et al (2016) to think about how/if those attempts might be enhanced. This is done by

  1. summarising the model;
  2. explaining how the model is “wrong”; and,
  3. offering some ideas for future work.

My aim here is mainly a personal attempt to make sense of what I might be able to do around learning analytics (LA) given the requirements of my current position. Requirements that include:

  • to better know my “learner”;

    In my current role I’m part of a team responsible for providing professional learning for teaching staff. My belief is that the better we know what the teaching staff (our “learners”) are doing and experiencing, the better we can help. A large part of the learning and teaching within our institution is supported by digital technologies. Meaning that learning analytics (LA) is potentially an important tool.

    How can we adopt LA to better understand teaching staff?

  • to help teaching staff use LA;

    A part of my work also involves helping teaching academics develop the knowledge/skills to modify their practice to improve student learning. A part of that will be developing knowledge/skills around LA.

    How can we better support the adoption of/development of knowledge/skills around LA by teaching staff?

  • increasing and improving research.

    As academics we’re expected to do research. Increasingly, we’re expected to be very pragmatic about how we achieve outcomes. LA is still (at least for now?) a buzz word. Since we have to engage with LA anyway, we may as well do research. Also done a bit in the past, which needs building upon.

    How can we best make a contribution to research around LA?

The model

The following uses work performed by an OLT funded project looking at student retention and learning analytics. A project that took a broader view that resulted in:

Given the questions I asked in the previous section and my current conceptions it appears that much of my work will need to focus on helping encourage the sustainable uptake of LA within my institution. Hence the focus here on that model.

The model looks like this.

Model of system conditions for sustainble uptake of LA (Colvin et al, 2016)

At some level the aim here is to understand what’s required to to encourage educator uptake of learning analytics in a sustainable way. The authors define educator as (Colvin et al, 2016, p. 19)

all those charged with the design and delivery of the ‘products’ of the system, chiefly courses/subjects, encompassing administrative, support and teaching roles

The model identifies two key capabilities that drive “the flow rate that pushes and pulls educators along the educator uptake pipeline from ‘interested‘ to ‘implementing‘”. These are

  1. Strategic capability “that orchestrates the setting for learning analytics”, and
  2. Implementation capability “that integrates actionable data and tools with educator practices”.

There are two additional drivers of the “flow rate”

  1. Tool/data quality – the “tool or combination of tools that manage data inputs and generate outputs in the form of actionable feedback” (Colvin et al, 2016, p. 30).
  2. Research/learning – “the organisational learning capacity to monitor implementations and improve the quality of tools, the identification and extraction of underlying data and the ease of usability of the feedback interface” (Colvin et al, 2016, p. 30)

The overall aim/hope being to create a “reinforcing feedback loop” (Colvin et al, p. 30) between the elements acting in concert that drives uptake. Uptake is accelerated by LA meeting “the real needs of learners and educators”.

How the model is “wrong”

All models are wrong, but some are useful (one explanation for why there are so many frameworks and models within education research). At the moment, I see the above model as useful for framing my thinking, but it’s also a little wrong, but that’s to be expected.

After all, Box (1979) thought

it would be very remarkable if any system existing in the real world could be exactly represented by any simple model. (p. 202)

Consequently, given that Colvin et al (2016) identify the implementation of LA as complex phenomenon “shaped by multiple interrelated dimensions traversing conceptual, operational and temporal domains…as a non-linear, recursive, and dynamic process..” (p. 22), it’s no great surprise that there are complexities not captured by the model (or my understanding and representation of it in this post).

The aim here is not to argue that (or how) the model is wrong. The aim is not to suggest places where the model should be expanded. Rather the aim is to identify the complexities around implementation that aren’t visible in the model (but which may be in the report) and to use that to identify important/interesting/challenging areas for understanding and action. i.e. for me to think about the areas that interest me the most.

“Complexifying” educator uptake

The primary focus (shown within a green box) of the model appears to be encouraging the sustainable uptake of LA by educators. There are at least two ways to make this representation a bit more complex.

Uptake

Uptake is represented as a two-step process moving from Interested to Implementing. There seems to be scope to explore more broadly than just those two steps.

What about awareness. Arguably, LA is a buzz word and just about everyone may be aware of LA. But are they? If they are aware, what is their conceptualisation of LA. Is it just a predictive tool? Is it even a tool?

Assuming they are aware, how many are actually already in the interested state?
I think @hazelj59 has done some research that might provide some answers about this.

Then there’s the 4 paths work that identifies at least two paths for implementing LA that aren’t captured here. These two paths involve doing it with (DIW) the educator, and enabling educator DIY. Rather than simply implementing LA, these paths see the teacher being involved with the construction of different LA. Moving into the tool/data quality and research/learning elements of the model.

educator

The authors define “educator” to include administrative, support and teaching roles. Yet the above model includes all educators in the one uptake process. The requirements/foci/capabilities of these different types of teaching roles are going to be very different. Some of these types of educators are largely invisible in discussions around LA. e.g. there are currently no moves to provide the type of LA that would be useful to my team.

And of course, this doesn’t even mention the question of the learner. The report does explicitly mention a focus on Supporting student empowerment with a focus on a conception of learners that includes their need to develop agency where LA’s role is to help students take responsibility for their learning.

Institutional data foundation: enabling ethics, privacy, multiple tools, and rapid innovation

While ethics isn’t mentioned in the model, the report does highlight discussion around ethical considerations as important. Ethical and privacy considerations are important.

When discussing tool/data quality the report mentions “an analytic tool or combination of tools that manage data inputs and generate outputs in the form of actionable feedback”. Given the complexity of LA implementation (see the above discussion) and the current realities of digital learning within higher education, it would seem unlikely that a single tool would ever be sufficient.

The report also suggests (Colvin et al, 2016, p. 22)

that the mature foundations for LA implementations were identified in institutions that adopted a rapid innovation cycle whereby small scale projects are initiated and outcomes quickly assessed within short time frames

Combined with the increasing diversity of data sources within an institution, these factors seem to suggest that having an institutional data foundation is a key enabler. Such a foundation could provide a common source for all relevant data to the different tools that are developed as part of a rapid innovation cycle. It might be possible to design the foundation so that it embeds institutional ethical, privacy, and other considerations.

Echoing the model, such a foundation wouldn’t need to be provided by a single tool. It might be a suite of different tools. However, the focus would be on encouraging the provision of a common data foundation used by tools that seek to manipulate that data into actionable insights.

Rapid innovation cycle and responding to context

The report argues that the successful adoption of LA(Colvin et al, 2016, pp. 22-23)

is dependent on an institution’s ability to rapidly recognise and respond to organisational culture and the concerns of all stakeholders

and argues that

the sector can further grow its LA capacity by encouraging institutions to engage in similarly diffuse, small-scale projects with effective evaluation that quickly identifies sites of success and potential impact (p 22)

This appears to be key, but how do you do it? How does an institution create an environment that actively encourages and enables this type of “small-scale projects with effective evaluation”?

My current institution currently has the idea of Technology Demonstrators that appears to resonate somewhat with this idea. However, I’m not sure that this project has currently solved the problem of “effective evaluation” or of how/when to scale beyond the initial project.

Adding in theory/educational research

In discussing LA, Rogers et al (2015, p. 233) argues

that effective interventions rely on data that is sensitive to context, and that the application of a strong theoretical framework is required for contextual interpretation

Where does the “strong theoretical framework” come from, if not educational and related literature/research? How do you include this?

Is this where some one/group needs to take on the role of data wrangler to support this process?

How do you guide/influence uptake?

The report assumes that once the elements in the above model are working in concert to form a reinforcing feedback loop that LA will increasingly meet the real needs of learners and educators. That this will in turn accelerate organisational uptake.

At least for me, this begs the question: How do they know – let alone respond to – the needs of learners and educators?

For me, this harks back to why I perceive that the Technology Acceptance Model (TAM) is useless. TAM views an individual’s intention to adopt a particular digital technology as being most heavily influenced by two factors: perceived usefulness, and perceived ease of use. i.e. if the LA is useful and easy to use, then uptake will happen.

The $64K question is what combination of features of an LA tool will be widely perceived by educators to be useful and easy to use? Islam (2014, p. 25) identifies the problem as

…despite the huge amount of research…not in a position to pinpoint…what attributes…are necessary in order to build a high level of satisfaction and which…generate dissatisfaction

I’ve suggested one possible answer but there are sure to be alternatives and they need to be developed and tested.

The “communities of transformation” approach appears likely to have important elements of a solution. Especially if combined with an emphasis on the DIW and DIY paths for implementing learning analytics.

The type of approach suggested in Mor et al (2015) might also be interesting.

Expanding beyond a single institution

Given that the report focuses on uptake of LA within an institution, the model focuses on factors within the institution. However, no institution is an island.

There are questions around how an institution’s approach to LA can be usefully influenced and influence what is happening within the literature and at other institutions.

Future work

Frame this future work as research questions

  1. How/can you encourage improvement in the strategic capability without holding up uptake?
  2. How can an institution develop a data foundation for LA?
  3. How to support rapid innovation cycles, including effective evaluation, that quickly identifies sites of success and potential impact?
  4. Can the rapid innovation cycles be done in a distributed way across multiple teams?
  5. Can a combination of technology demonstrators and an institutional data foundation provide a way foward?
  6. How to support/encourage DIW and DIY approaches to uptake?
  7. Might an institutional data foundation and rapid innovation cycles be fruitfully leveraged to create an environment that helps combine learning design, student learning, and learning analytics? What impact might this have?

References

Box, G. E. P. (1979). Robustness in the Strategy of Scientific Model Building. In R. Launer & G. Wilkinson (Eds.), Robustness in Statistics (pp. 201–236). Academic Press.

Colvin, C., Wade, A., Dawson, S., Gasevic, D., Buckingham Shum, S., Nelson, K., … Fisher, J. (2016). Student retention and learning analytics : A snapshot of Australian practices and a framework for advancement. Canberra, ACT: Australian Government Office for Learning and Teaching. Retrieved from http://he-analytics.com/wp-content/uploads/SP13-3249_-Master17Aug2015-web.pdf

Adding a custom library and a semi-real application – Indicators notebook(s)


Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

So the indicators notebooks/platform is on github. The one and only bit of analysis is almost completely useless and still requires a fair bit of set up code. The aims in this post are

  1. Add in a custom library for connecting to the data source.
  2. Add an indicator/notebook that does something kind of useful.

Hopefully, this will lay the ground work to start converting old Perl-based stuff into this new environment and start doing something more useful.

Custom library

The aim here is to replace all of the following

import json
 
with open('../config.json') as f:
    conf = json.load(f)
 
from sqlalchemy.engine.url import URL 
from sqlalchemy import create_engine
 
engine = create_engine(URL(**conf))

To something like

import Indicators
 
Indicators.connect();

or something slightly more in the correct Python idiom.

That’s done. Will still need to be refined

  1. Better way to specify the path of the config file.

    Hard coded in a file in git is not a great start.

  2. Does it fit with the Python idiom/approach?

Something a little real

Aim here is to do something a little useful to people (or at least me) and to start playing with the visualisation options.

A need I’ve identified in my new role is to have some overall idea of the number of courses, number of teaching staff, number of students etc at my institution. There doesn’t seem to be any easy way to find out and nobody I talk to knows (with a few exceptions).

Aim here is to develop a notebook that shows the number of courses in Moodle per semester.

Lesson learned: In Python, when doing SQL using like and a wildcard – typically % – you need to use %%. As Python reads % as string formatting i.e.

shortname LIKE 'EDC3100_2015_%%'

Years and terms

The first question is how to capture the individual years and terms that I might want to capture individual data for.

I could hard-code this into the notebook, but it will be different at another insitution – or a different data set. So I’m going to try a kludge, add the data to the JSON config file. Like this

  "allYears" : [ 2012, 2013, 2014, 2015 ],
  "allTerms" : [ "2012_1", "2012_2", "2012_3", 
                 "2013_1", "2013_2", "2013_3",
                 "2014_1", "2014_2", "2014_3",
                 "2015_1", "2015_2", "2015_3" ]

This is ugly and will need to be revised, but I’m in a hurry.

Though this raises the question as to whether or not I can access the data now it’s in the Indicators module.

That exploration leads to an additional function in Indicators module to get this variable. This is probably how the problem with moodle prefixes will get fixed.

Yep done. Able to include a prefix in queries. The value is defined in a new config file lms.conf which look slike

{
  "allYears" : [ 2012, 2013, 2014, 2015 ],
  "allTerms" : [ "2012_1", "2012_2", "2012_3",
                 "2013_1", "2013_2", "2013_3",
                 "2014_1", "2014_2", "2014_3",
                 "2015_1", "2015_2", "2015_3" ],
  "mdl_prefix" : "moodle.mdl_"
}

Using the prefix in code looks like

import Indicators
import pandas as pd
 
engine = Indicators.connect()
configuration = Indicators.config()
prefix = configuration['mdl_prefix']
 
query = "select id,username,firstname,lastname from " + prefix + "user where id<30 "
df = pd.read_sql(query,engine)

Segue – groking data frames

I’m still very new to Python and tend to bring my Perl/PHP frames to programming. Need to spend some time groking “the Python way”. In writing this script it’s become obvious I haven’t yet grokked data frames. Hence reading this on data frames and the following.

Actually, I found this from datacamp.com not at all easily accessible, but there are some nuggets there.

Indexing of data frames has a number of different ways to access elements. iloc is the standard array approach i.e. based on position. indexes can also be more hash like.

Mmmm, more work to do.

The kludgy solution

Have added a Course Offerings notebook that includes code like the following that will produce a simple histogram showing number of courses for each year/term within the database.

This code is the year portion. The term graph is almost identical

yearCounts = {}
 
for year in configuration['allYears']:
    query = "select count(id) from " + prefix + "course where " +\
             " shortname like '%%_" + str(year) + "_%%'"
    df = pd.read_sql( query, engine)
    yearCounts[year] = df.loc[0]
 
counts = pd.DataFrame.from_dict( yearCounts,orient='index')
 
counts.plot(kind='bar')

The code for terms generates output like the following

Course per term

Still quite ugly, there are ways to improve the output. A later task. Along with much more learning about Python etc.

Sharing “indicators platform” via github


Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Following on from the last post the following documents how to share the “indicators platform” for analytics via github. It’s largely intended to help @beerc. I doubt there’s nothing (at the moment) that makes this inherently interesting for anyone else.

End result

The (almost completely useless) end result of this work is this github repository.

Hopefully, this will form a foundation that will help it get much more interesting, quickly.

Jupyter notebooks and github

It’s not straight forward to share Jupyter notebooks via github. At least if you wish to maintain privacy of the data.

For example, take a look at the first version of the first notebook placed in the repository. Can you see all the programing error messages at the bottom of the page?

Had the SQL command just before this worked, it would contain actual data of real people. This would be bad.

This is because the raw json file that is the notebook will include the data. This is a good thing when you’re viewing it on your computer. It’s a bad thing when you’re sharing it in a public space.

That’s fixed in a more recent version.

To achieve this, it was necessary to follow the instructions on this page, which involve

  1. Installing a python script on your computer so it can be run.
  2. Configuring git to use this script as a filter when sending stuff to github.
  3. Configuring the meta-data for the notebook to ensure that the content of data blocks would be removed before going to github.

    Suggesting that this step would need to be repeated for each notebook likely to have personal information show up in the data.

Testing that this works

My suggestion is

  1. Follow the instructions below.
  2. Modify the SQL in the notebook to make sure it generates an error (i.e. not real private data)
  3. Commit and push the error version back to github

If no error data shows up on github, then it’s working.

How to use this repository

Basic process should be

  1. Get your box set up to run Jupyter notebooks.

    It is generally, a fairly simple process

  2. Clone a copy of the github repository into the directory your notebooks are being stored – creating a directory called Indicators

    You might want to fork my repository first. This will give you your own github repository. We can then share good changes via pull requests.

  3. Create a file called config.json in the parent directory for Indicators with the following content (changed to suit your Moodle database)
    {
      "drivername": "postgresql",
      "database": "",
      "username": "",
      "host": "localhost",
      "port": "5432",
      "password": ""
    }
  4. Open up the notebook and run the cells.

Jupyter notebook indicators platform: Baby step #1 – Finding an ORM


Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

The last post documented early explorations of Jupyter notebooks ending with a simple query of a Moodle database. This post takes the first baby steps toward some sort of indicators platform using Jupyter notebooks, Python and github. The focus here is to find some form of ORM or other form of database independent layer.

Problem: the code from the last post was specific to Postgresql. If you’re Moodle database is based on another database that code won’t work. The aim here is to enable some level of sharing of code/analysis/indicators. This means needed a way to keep the code independent of database specifics. This is where object relational mappers (ORMS) enter the picture. See this for an argument why this is a good idea.

Pandas and SQLAlchemy

This book offers some examples using sqlalchemy with pandas. A likely combination. This sqlalchemy cheatsheet offers some useful examples.

Need to install sqlalchemy, it appears. Actually just updated

conda install -c anaconda sqlalchemy

Oh dear, time wasted. Needed to restart the notebook server after that.

Process is

  1. Modify the config stuff to create an sqlalchemy engine.
  2. Read the data

Ends up with the following code

import json
 
# Put the config outside the Indicators directory, keep it out of git
with open('../config.json') as f:
    conf = json.load(f)
 
from sqlalchemy.engine.url import URL 
from sqlalchemy import create_engine
 
engine = create_engine(URL(**conf))
 
df = pd.read_sql('select id,username,firstname,lastname ' +
                 'from moodle.mdl_user where id<100 ',engine)
df

The config.json file looks something like the following. The current plan is that this sits above this directory, as this directory and its contents will eventually end up in github

{
  "drivername": "postgresql",
  "database": "someDbaseName",
  "username": "user", 
  "host": "localhost",
  "port": "5432",
  "password": "myPassword"
}

What’s next?

That works and seems a reasonable. Some ideas for the next step

  • Figure out how to remove/handle the moodle schema that’s in the SQL above, not to mention the mdl_ prefix on the table.

    Linked to allowing the code to be run across different institution’s easily.

  • Move the config code into a library for this work?
  • Set up the github repository, get this code shared and start working with @beerc on this.
  • Experiment with how the assumptions built into the Perl code I was using can be translated appropriately into this environment.

    How to develop the class hierarchy (or if) using sqlalchemy.

    How the perl idioms translate into python, sqlalchemy and pandas. Pandas has some nice features that might eliminate some practices.

Playing with Python and Jupyter notebooks for analytics


Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

This is the third in a series of posts documenting “thinking” and progress around the next step of some bricolage with learning analytics and attempts to make some progress with the Indicators project.

The last post in this series revisited some work I did last year. The aim of this post is to consider and perhaps start playing with what’s next. At this stage, I think that’s Python and Jupyter notebooks, but I could be wrong.

Actually, nope. Jupyter notebooks appear to be the way to go. At least worth of a bit more exploration and experimentation. The following explains and documents the process toward a Jupyter notebook that is actually accessing a reasonable Moodle database on postgresql. The main data source I’ll be working with. It seems possible.

Notes to self:

  • Defining indicators and connection with the literature.

    Remember to revisit and explore further the literature mentioned in section 1.4 of this report on teaching culture indicators. There is literature defining and expanding the conception of indicators

  • How does this nascent approach fit with “learning analytics in Australia”

    This project has a number of frameworks/ways of thinking about learning analytics, especially within institutions. How does some of the early thinking driving this work fit within/contradict those? What can we learn?

    e.g. we obviously fit within cluster 2 – focus on pedagogy, curriculum, learning, but how well does what we’re doing fit within the pattern and what impact will that have on outcomes?

  • What can be borrowed/copied from the Loop tool?

    The Loop Tool is the product from an OLTC funded project. It’s open source, works with Blackboard and Moodle, and I believe uses Python.

    It includes a “data warehouse”, which I assume brings together Blackboard/Moodle data. This could provide a great platform upon which to build this work. Not only leverage work done by smart people, but also provide a cross-platform foundation.

    Only question is what limitations exist in this data because they couldn’t get direct access to the database. Appears it relies on csv and course export files. And some of the code still seems a little less than portable (hard coded paths with specific user home directories). Perhaps indicating a need to look for other cross-platform solutions.

    Beyond code re-use, some of the indicatores/patterns/visualiations generated by that tool might serve as useful things to re-create.

  • How does Data Carpentry link with Software Carpentry and which is applicable to the institution?
  • Four facets of reproducibility as a tool to evaluate approaches to learning analytics.

    e.g. how well does a data warehouse informed approach meet these.

  • Link to intelligence amplification.

    A focus on helping people collaborate around the indicators, rather than simple accept or use.

Options?

When “learning analytics” is mentioned within an institution, it seems people are always looking for an application (usually an existing one) such as a data warehouse or tools such as Open Refine or Trifacta. That seems like a combination of Kaplan’s law of instrument and a tendency for most people to see themselves as users of digital technology. That is, to do something with digital technology there must be an application designed specifically to help them with that task, rather than writing code.

This post seeks to argue that “the three biggest positive contributions to reproducible research are iPython (Jupyter) Notebook, knitr, and Galaxy. It positions Juypter and knitr as fitting with those with scripting (coding) experience. While Galaxy are for those who script not so much. Personally (and perhaps self-servingly), I see significant value in being able to script to provide more flexibility and broaden possibilities. Without scripting you are stuck with the model and capabilities of the chosen tool.

This leaves the question to be whether to use Jupyter notebooks or R/knitr. Tony Hirst has begun thinking about comparing these, but still early days. Given I have no real experience with either, it’s difficult to come to a meaningful decision. Suggesting that I’ll tend toward the option that appears most familiar – Jupyter notebooks/Python.

Longer term, there is a glimmer of a hope that using Python to generate the indicators/patterns, might enable later integration of such indicators into a MAV-like tool. This would enable these indicators/patterns to be visible within the learning environment.

As it happens, the argument around Jupyter versus R might not be an argument after all. Apparently – at least at some stage – it is possible to include R code in iPython notebooks. But apparently, Knitr can also include Python code. The comments of that last post reveal a native R kernel for iPython.

Below you will learn how I now know that Jupyter is actually an acronym of the three core languages that the Jupyter notebook approach was designed to support – JUlia, PYThon and R. (it also supports 40 other programming languages)

Getting set up: Python and Juypter

To know I’ve been relying largely on Tony Hirst’s blog posts about his experiences for information on Juypter notebooks. Time to check out other resources.

Jupyter.org

This appears to be the home for Jupyter notebooks and offers the following definition

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.

It also points to JupyterHub as a mechanism to multiple single user Jupyter notebook servers. This might be interesting for sharing within an institution. For later consideration.

It also offers advice on installing Jupyter. I’d played with anaconda/jupyter previously, so a quick test and I’m ready to go. Nothing to install.

Follow the instructions on running the notebook and I’m away.

Installing python from the course below.

Reproducible research and Jupyter Notebooks

Thanks again to Tony Hirst, I’m aware of the curriculum for a workshop titled “Reproducible Research using Jupyter Notebooks. This seems a perfect resource for us to get started.

At this stage, I think I’ll turn this into a diary of my attempts to work through this curriculum (without the facilitated workshop). There are 5 modules, each of the following link to the git repo that contains the materials. Sadly, it appears that there isn’t a consistent format for each.

  • Introduction

    Reasonably accessible from the github site.

    Can find slides that provide a gentle into to using notebooks.

  • Data and Project Organization

    Different format, more use of Jeckyl and markdown. Not immediately obvious how to use. Based on previous version, which is a bit more usable. Google doc that documents additional thinking moving beyond the previous version. Appears that the final version is incomplete

    Some of it is focused on data coming in files. Section on how to structure notebooks.

  • Data Exploration

    Here’s the notebook with the content.

  • Automation

    Links to brainstorming Google doc and links to lesson notebooks: one, there’s a second but it’s currently basically empty. The first is not much better

    References the software carpentry resource on Programming with Python

  • Publication and Sharing

    Covers a few interesting topics, has some content, but incomplete.

The documentation lesson includes mention of a gallery of notebooks, including

Other resources

Learning about reproducible research with Jupyter notebooks

Starting with this slide-deck, which is linked from the git repository for the first module in the workshop.

A jupyter notebook can be thought of as a lab/field diary.

Architecture

  • front-end
    • Web application to develop notebook documents
    • Notebook document representation of the content, including I/O of computations, explanatory text, images, etc.

      These are JSON files saved with .ipynb extension.

  • back-end
    • Notebook server – communication between the kernal and the web browser
    • Kernel responsible for executing the code, different kernels support different languages.

Jupyter architecture

Espoused benefits: Jupyter notebooks for reproducible research

Documentation/literate programming. Exploration and development. Communication and collaboration. Publishing.

Working with notebooks

This slide deck assumes you have Jupyter notebooks installed.

And I’ve created my first notebook. Hello world, ahh markdown how I love you.

A notebook consists of a sequence of cells. Each cell has a type. The default types are: code, markdown, heading and “Raw NBConvert”. Not sure on the last one, but there others a fairly self-explanatory.

NBConvert mechanism to convert notebook into other formats. Can also be included within the python code to allow output to be downloaded as other formats. Explore this more

Important: If the last line of a code cell produces output, then the output is embedded in the notebook just below the cell. If the last line is a variable, then the value of the variable is displayed, including rich output. As long as the semi-colon is not added to the last line. A semi-colon will prevent output generation.

Add a ! to the start of cell and you have a command-line prompt.

And there a nifty macros/shortcut keys – ESC-L – turn on line numbering

Slight problem with the workshop examples and my install. The example that I thought would be interactive, isn’t. The server, however is generating an error that I’ve just spotted

[IPKernelApp] WARNING | Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"

Mm, nice, the Jupyter notebook itself provides support for checkpoints. Simple, embedded version control? There’s also auto-complete.

Save, run the command suggested, restart and all is fixed and working.

The basic of Jupyter look fine. Doable. Need to upskill on Python, translate existing practices into Python and think about the processes.

Intro to reproducible research with Jupyter

This was meant to be the first of the “sessions” in the workshop. The neat thing that this illustrates is that github will render Jupyter notebooks. Actually, thinking about the above, that’s obvious, but also limited. Notebooks = code + markdown. Github won’t render the code, but it will the markdown, as github uses markdown.

Viewing the raw source of the notebook reveals that it’s a little more complex than that. Remember, notebooks are json. One cell = one json element (whatever it’s called in json). Most of the cells in this notebook are markdown. The markdown is rendered.

Four facets of reproducibility

  1. Documentation
  2. Organisation
  3. Automation
  4. Dissemination

This aren’t a bad set of principles for promoting learning analytics. Also points to some weaknesses in standard organisational practices.

Data exploration

The module’s notebook starts with basics of Python libraries, dataframes. Includes sections on

  • assessing structure and cleanliness of data
  • data cleaning
  • tidy data
  • merging data
  • ..
  • visualisation with matplotlib and seaborn

Given the data we’re working with is largely from a database and is reasonably clean, this isn’t likely to be directly useful. The examples using dataframes could be.

Visualisation with Seaborn could also be useful.

Automation

The lesson covers

  • review of good practices – naming conventions…and basic

Not much else.

The model

An early start at grouping what I’ve derived from the above and apply to how we might work.

Jupyter notebooks are created via a web interface. As such, they are stored in a web server’s file space. It can have folders etc, it can link etc. On a laptop this is local. This could be managed using git.

Could try the following

  1. create new directory – Indicators
  2. create a index.pynb

    This becomes the default notebook. Do some initial testing. It could point off to other pynbs

    Initially, could use very basic python. Eventually evolve.

  3. Create a github repo and save it there.
  4. Others could make use of it and contribute
  5. Could export the notebooks as HTML etc and place online? maybe.

http://stackoverflow.com/questions/18734739/using-ipython-notebooks-under-version-control

Getting started

Accessing postgresql from Python

Possible options

  • pscopg2.

    Only problem is that the connection details are hard-coded into the notebook. Not great security in sharing environment.

    Need: to find a way to support local (secret) database configuration

  • doing it with Jupyter

    This explicitly mentions the config file idea and then connects it with psycopg2 and proceeds to show some good examples specific to pandas

Let’s try it. Mmm, psycogp2 not installed. This works.

conda install -c anaconda psycopg2

A bit of playing with the example and that is working. Success!!!

That’s working. A screen shot of the Jupyter notebook is shown below (data blurred out).

There remain a couple of problems with this in terms of abstraction, but I can see how it might work.

Reflecting on playing with learning analytics


Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /home/djones/public_html/blog/wp-content/plugins/wp-syntax/wp-syntax.php on line 383

This is the 2nd in 3 posts thinking about learning analytics and how we might engage with it better. The first rambled on about reproducible research and hunted for what we might do.

This post is an attempt to reflect on some work I did last year trying to design a collection of analytics to explore engagement. The aim is to remind myself what was done and try to distill what worked and what didn’t. Hopefully to inform what we might do this year.

Background

The work last year was trying to work

with a group of fellow teacher educators here at USQ to explore what is happening around student engagement with our online courses

The aim was to generate a range of different engagement related analytics, show them to the teacher educators, explore what if anything was revealed about our courses and think about what that might mean for the courses, but also the exploration of analytics. The hope was that the cycle would help identify more contextually useful analytics that could actually reveal something interesting.

What was done

Version 1 of process

The aim was to generate and share analytics based on the diagram to the right (click on it the diagram to expand). The aim was to generate a website where those involved could see the different analytics applied to their respective courses (and each others). This was intended to drive discussion.

A website was generated to do this. Access was limited to those involved in the project. The following links illustrate the generated analytics using a course I taught.

You may see some menus with other course codes, there won’t be any data for other courses. The menus – at times – don’t work all that well.

  • click/grade and other basic indicators
  • time/grade – nada
  • networks & paths
    • Replies to posts network – normal Q&A forum or announcements forum or ice-breaker activity

      You can zoom in and out with trackpad. Click on network map to move it around.

      Click on the red (or any) hexagon and move it around to more clearly see the number of connections.

      Should be able to see a significant difference between the different types of forums.

    • content & sentiment analysis – nada
    • learning design – not a chance

    So only the simple ones done.

    The educators involved could navigate through the generated site and compare their course with others. This was scafolded lightly using a blog with links to specific analytics and questions. It was never done well or in a concerted manner.

    How was it done

    At a high level, I had a perl script that would loop through an array that specified which type of analytic to perform (e.g. clickGrades, dhits, dhitsGrades) and for which course offerings and modes. The array looked a bit like this

    EDC3100 => {
      "2015_1" => {
        clickGrades => [ qw/ all mode=Online mode=Springfield mode=Toowoomba 
    mode=Fraser_Coast / ],
        dhits => [ qw/ all mode=Online mode=Springfield mode=Toowoomba mode=F
    raser_Coast staff / ],
        dhitsGrades => [ qw/ all mode=Online mode=Springfield mode=Toowoomba 
    mode=Fraser_Coast staff / ],

    The script would loop through these entries and execute the following code (modified just a little for clarity/space)

    my $model = $factory->getModel( OFFERING => "${course}_$offering",
                                    ANALYTIC => $analytic );
     
    my $view = $factory->getView( MODEL => $model, VIEW => $analytic );
     
    my $string = $view->Display( SUBSET => $subset, COURSE => $course, 
                                 OFFERING => $offering, ANALYTIC => $analytic );
     
    writePage( OFFERING => "${course}_$offering",
               ANALYTIC => $analytic, SUBSET => $subset,
               STRING => $string );

    Analytics classes

    Underpinning the above were a collection of analytics classes that could also be called directly. For example, the following.

    my $model = QILTers::Analytics::dhits->new( OFFERING => "EDC3100_2015_1" );
     
    my $view  = QILTers::Analytics::dhits_View->new( MODEL => $model );
     
    my $string = $view->Display(
                    COURSE => "EDC3100", OFFERING => "2015_1",
                                SUBSET => "mode=Toowoomba,grade=F" );
    print $string;

    This type of MVC architecture was used largely because I had a large code-base (and related experience) from my PhD.

    In this context, the model classes converted institutional specific notions. This included

    • that EDC3100_2015_1 could be used to identify a course
    • access to additional information about users (gpa, grade etc) not stored in Moodle.
    • support the SUBSET mechanism to allow focus on particular campuses, grades etc.

    The view classes were responsible for translating the gathered data into a format that could be handed over to a couple of different client libraries (plotly and cytoscape)

    Reflections and lessons learned

    Misc reflections include

    • It took time to get all this set up, hence the lack of progress.
    • The subset functionality was useful.
    • The production of the web pages was all a bit kludgy
    • Need better support for integrating the analysis into discussions with others.
    • Tightly integrating the navigation with the content was a mistake.
    • The client libraries were quite useful, but required some familiarisation.
    • The code here relies on quite out dated technologies.
    • Once the structure was in place, different analysis could be done fairly quickly (given time)
    • Exploration of different analytics or modes, still required intervention by a coder

    Next step, what are others doing in this area? What might be a better way to do this?

Thinking about (more) reproducible research and learning analytics

There are moves afoot to revisit some of the earlier research arising from Indicators project and hopefully move beyond. To do that well we need to give some thought to updating the methods we use to analyse, share and report what we’re doing. In particular, because we’re now a cross-institutional project. The following captures some early exploration of this problem.

This has actually become the first of 2 or 3 blog posts. This one was getting too large. The sequence of post looks likely to be

  1. (this post) Think a bit more about the requirements: what do we want to share? with whom?
  2. Reflecting on some recent past experience.
  3. Looking at some possibilities

After that, hopefully, we’ll actually do something.

Requirements

So what would we like to be able to do? In a perfect world we might be aiming for reproducible research. But that’s beyond the scope of what we need right now. At the very least we want to develop a research method/environment that helps make the research that we’re doing more reproducible.

What will be more reproducible?

The Reproducibility spectrum provided by Peng (2011) appears to include the following components

  1. data;

    The data from institutional systems that is in put into the analysis

  2. code;

    What actually does that analysis.

  3. results indicators/patterns; and,

    The final correlations, patterns etc produced by the analysis. Back in the day we used the term pattern

  4. interpretations/hypotheses/conclusions.

    We’ll be layering upon the results we’ll be layering our own interpretations/conclusions.

These components could be thought of being circular, especially if learner and/or teacher action in response to interpretations is added. Learners and teachers may take action in response to interpretations, which in turn generates more data to be analysed by code… Researcher action also plays a part, through going back to re-analyse the data with a different lens, or designing experiments to generate new data. Echoes of the Siemens (2013, p. 13) data loop, but with a slightly different focus.

Siemens (2013) Learning Analytics Model

Making this cycle more reproducible would be useful.

For example, the following graph has the following components

  1. data;

    Data from a Blackboard LMS over numerous years showing the number of times students had clicked on discussion forums. The data was limited to courses where staff participation (measured by clicks on the course site) was high. Combined with student final result (HD, D, C, P, F).

  2. code;

    Either some messy PHP code written by @beerc, some better structured but still messy Perl code written by me, or a combination of both. Probably with the addition of Excel spreadsheets for visualisation.

  3. results;

    The graph below indicating that that average # of clicks on a discussion forum tends to increase the better a student’s final grade is.

  4. interpretations/hypotheses/conclusions.

    One interpretation is that students get better grades by being more dilligent/expending more effort. Hence they click more. Of course this is a hypothesis.

Average student hits on course site/discussion forum for high staff participation courses

To whom?

Indicators architecture

Way back in 2009, the presentation for the first indicators project paper included the following image to the right as an early explanation of what we were doing.

This image can be mapped somewhat onto the components of what from the previous section

  • data = the institutional databases and the LMS independent data.

    At this point in time, I’m guessing we’ll be working with our institutional data. The LMS independent data idea will have to wait

    That said, the recently announced Project Inspire from Moodle HQ might provide some assistance here.

  • code = typically the arrows between components.
  • results = useful data/information
  • intepretations = useful data/information

The T&L practice and Research ‘clouds’ in the above image point to the Who question

  • T&L practice
    1. Teaching staff – interested in reproducing the research on their course, students, learning designs.
    2. Teaching support staff/leadership – interested in reproducing the research on groups of courses.

    Largely interested in working with the results/interpretations.

  • Research

    Could divide research into various groups based on research interests, methods etc.

    Would potentially be interested in accessing the data, code and results/interpretations.

The useful data/information captures results and interpretations aspects of the what from the previous section.

Potentially, we’d like aspects of this work to be more reproducible (/accessible) for the following broad groups

  1. researchers within the project;

    Given we’re at different institutions. We wouldn’t be sharing data, at least not until a lot more necessary work is done.

    However, we would like to share code, results (suitably anonymised), and interpretations.

    The sharing here also includes co-construction/modification of these artefacts. Suggesting a need for version control and collaboration.

  2. researchers outside the project;

    If we take a more open approach, probably the same sharing as with ourselves. However, early on some of us might like to share more final conclusions of what we do.

    Sharing code tends to imply we’re using the same technology. We might be able to agree to this within the project, but outside is never going to happen. But we should perhaps allow for the possibility.

  3. teaching academics
  4. misc. support groups

What about the learner?

I’m very aware that the above discussion (and much of what I’ve written previously) suffers from the learner being absent. At some level, this is for pragmatic reasons and linked to the problem that the placing some limits section tries to deal with. That said, however, the above would be enhanced by broadening consideration to the student. Something which this project – The Learning Analytics Report Card appears to be giving a lot of thought to.

I came to that project from this talk by Amy Collier. A talk which offers Lesson 2: Solutionism decomplexifies too much. Much of what I do likely comes across as solutionism (given my closeness to the subject, I’m not a reliable judge, what do you think?). However, my conception of this post and the purpose of this project is to start a process that results in methods, tools and thinking that allow more people to be engaged with the work we do.

The title of the talk – It should be necessary to start: Critical digital pedagogy in troubled political times – arises from a book We Make the Road by Walking and Collier makes this connection

In the early stages of the book, as Freire and Horton are discussing the spoken format of the book, Freire remarks that they will make the road by walking, meaning that the book will emerge from their conversations, and he adds what seems like an offhand remark that “in order to start, it should be necessary to start.”

This quote connects with my views on this work in two ways. First, this is just a start. Second, and perhaps more importantly, the rationale behind being able to more easily share how and what we do is to make it easier for us and others to “make the road by walking”.

Placing some limits

Trying to support all of the above would be bigger than Ben Hur. Picking up on this being a start and we’ll make the road by walking, the first step is to focus on a subset. The subset we’ll start with – I think – should be

  1. Practices that help the researchers two different institutions share code, results and interpretations.

    In theory, this could eventually expand beyond our little project and be used by others. But that will/may come.

  2. Practices that help the researchers share (appropriately anonymous) results and interpretation with others.

    Initially, this is liable to be limited to people within our respective institutions, but may extend more broadly.

    Actually, I’m thinking there’s value institutionally and from a research perspective in setting up some sort of mechanism by which we can engage more people in asking questions and drawing conclusions about what is found. A platform that helps analysis, but promotes sharing and collaboration around interpretation.

So maybe we’ll start there.

Next post should reflect back on what was done last year.

References

Siemens, G. (2013). Learning Analytics: The Emergence of a Discipline. American Behavioral Scientist, (August). doi:10.1177/0002764213498851

Early analysis of Moodle data

A small group of teacher educators that I work with are starting to explore some research ideas around engagement, initial teacher education, and in particular the questions that arise out of the Quality Indicators for Learning and Teaching (QILT)

For that project and others I need to get back into analysing institutional Moodle data. The following is a recording of some initial forays on a longer journey. It’s all basically data wrangling. A necessary first step to something more interesting.

To do

Much of the following is just remembering how I’ve configure my local system (a need the points to bad and interrupted practice)

  1. Do I have the data in a database?
    Yes. Postgres9.1 | Databases | studydesk2015 | moodle | tables
  2. Can I access this via Perl/PHP etc.
    Woohoo!  Yes.  Perl is a go (yes, I’m old).  ~Research/2016/QiLTEers/Analysis/myScripts
  3. Can I run some initial simple analysis
    Time to borrow some of Col’s work

Hits and grades

Let’s try the old standard, the pattern between hits on a course site and grades.

Still using the same tables for same purpose

Col’s work was from a while ago. The Moodle database schema – especially logs – has moved on.  Are his scripts work with the current data I have? Does that data I have even come with the tables his scripts use?

mdl_logstore_standard_log – tick

Ahh, but Moodle has changed its approach to logging.  My data has two tables.  The one mentioned and mdl_log (old version). That led to some wasted time – thanks for cluing me in Randip

Clicks

Experiment with 3 courses (some with 2 offerings) and see if I can get total clicks for that course.

Results are

  • 139,392 – 4th year course, S1
  • 220,362 – Big 1st year course, S1
  • 639,750 – Biggish 3rd year course, S1
  • 308,399 – Big 1st year course, S2
  • 185,675 – Biggish 3rd year course (but smaller offering), S2

Clicks per student

Raw total clicks isn’t that useful. What about the clicks per student average?

The 4th year course was showing too many students. What’s the go there?  Ahh, the query to identify students is returning duplicates.

Course Total Clicks # Students Clicks / Student
4th year course, S1 139,392 175 796.5
Big 1st year course, S1 220,362 215 1024.9
Biggish 3rd year course, S1 639,750 323 1980.7
Big 1st year course, S2 308399 451 683.8
Biggish 3rd year course, S2 185675 90 2063.1

 

Clicks and Grades

Next question is if I can produce the slightly more useful pattern between participation and grade.
Average student hits on course site/discussion forum for high staff participation courses

That seems to be working, and some success with caching.

Here’s the first semester 2015 offering of the biggish 3rd year course from the table above (a course I teach).

EDC3100_2015_1.png

And here’s the second semester 2015 offering of the same course.  The S1 offering has both on-campus and off-campus students.  The S2 offering is online only.

EDC3100_2015_2.png

What’s left to do

A lot.  But doing the above has started building the foundation scripts that will help transform the raw institutional data into something that more people can do more analysis with.

Bigger picture tasks to do are (not necessarily in this order)

  1. Polish and build out the data wrangling foundation.
    1. Identify the formats most useful for the next level of the process
    2. Improve the implementation of the scripts
    3. Build out the functionality of the scripts
  2. Identify the questions we want to explore
    1. Break down by student type
    2. Investigate the impact of timing on students
    3. LMS usage frameworks/course signatures
    4. Explore Moodle Book usage
    5. Include discussion forum participation
    6. Explore the use of links by staff and impact.

And many, many more.

 

Updating "more student details"

“More student details” (see image below) is the most visible part of the additional systems I’ve put in place to make teaching bearable (this presentation talks more about it). The problem I’m dealing with here is that a new semester has started, which means I need to re-configure these feral systems to work with the new semester.

And while I’m at it, I’m hoping I might be able to add a bit of sentiment analysis to it.

MoreStudentDetails

What needs to be done

Tentative process appears to be

  1. Create new course ID (5), group id (53), contextid (575) and course label (EDC3100_2015_S2) for new offering – creating the course in local Moodle.
  2. Create a BIM activity in the new course. – 3 – DONE
  3. run participants/parse.pl – DONE
  4. run the user extras. – DONE
  5. Load the initial activity information.- DONE
  6. Load the current activity completion data. – DONE
  7. Load the BIM data. – DONE
  8. Update the greasemonkey script

The basic aim is to populate a database running on my laptop with the following data

  • Student enrolment data.
    • Source – spreadsheet provided by manual dump from student records
    • Destination – Local moodle tables: mdl_user_extras and mdl_user
    • Method
      • Initialisation – set up the Moodle users data
        • update the course label in extras.pl
        • Play around with the columns in the spreadsheet.
      • Maintenance – participants/extras/extra.pl – DONE
        • Can the CSV file be read appropriately – DONE – but missing some of the data I would have liked to have had.
          Keep on eye out for strange data – e.g. UK post codes.
        • Get the users data for this course.
        • See if a match can be made with local Moodle and Peoplesoft data
        • Any missing students (can’t match)
        • insert into mdl_user_extras table
        • update mdl_user table – to update the phone1 and phone2 values
    • Status
  • Student Moodle user data.
    • Source – CSV file produced
    • Destination – Local moodle tables: mdl_user, mdl_groups_members, mdl_role_assignments
    • Method
      BAM/3100/3100_support/participants/parse.pl

      • Initialisation
        1. Update the library files
        2. Create a new local Moodle course – get id.
        3. ??? do I need to create a group ID???? What else is needed
      • Maintenance – script will do most of this — students aren’t enrolled in the course, but are appearing in BIM.
        1. Parse the HTML file. – DONE
        2. Find any existing users – DONE
        3. Add the new students – DONE
        4. Populate the group data – DONE
        5. Populate role assignments – DONE
    • Status
  • Activity completion data – DONE
    • Source
      • HTML file saved of main Moodle site – DONE
      • CSV file from activity completion
    • Destination – local Moodle tables
    • Method
      • Initialisation – activities/parseActivity.pl – DONE
        1. Have the local Moodle course set up.
        2. Ensure that the script has the configuration for the new course.
        3. Dump the main remote Moodle page to HTML.
        4. Run the parseActivity.pl script
          • Parse the html page – DONE
          • update the activity mapping table – DONE
        5. perhaps have to play with the activity translation stuff especially since Moodle 2.8 seems to have changed their method
      • Maintenance
        • List of activities – Whenever a the activity list to be completed changes on the remote site, save a new HTML file and re-run the script.
        • Activity completion – run the parse completion script
          • get progress data – DONE
            And Moodle has changed the format of their spreadsheet. It appears that the … has been removed. Let’s try that.
          • update the local/remote mapping – DONE
          • check the students – DONE
          • update the database – DONE
    • Status
  • Student blog data.
    • Source – HTML file from BIM on remote site
    • Destination – local Moodle BIM tables
    • Method
      • Initialisation – DONE
        1. Create the course
        2. Create the BIM activity
        3. Run the user scripts
        4. Modify the bim/parseBIM.pl script
      • Maintenance
        1. parse the file
        2. Get the users and check data
        3. insert the data – DONE
    • Status – simply save the updated HTML page and re-run the script when updates required

Greasemonkey script

Well the data is in and the update process should be fairly straight forward – though still somewhat manual. Time to move onto the Greasemonkey script. This is what recognises and updates the Moodle page, communicates with the details server and displays a pop up. Changes will likely need to be made to both the client and server to serve the new course.

Greasemonkey first

  • Recognise the course – 7023
    Add that Moodle course id to an if statement and it should start working? As long as I remember to reload the modified script. Yes. It is recognising it.

Server

  • Map course id to BIM
  • Map remote course id to local id
  • need to abstract out the WEEKS data – at the moment this is hard coded to the weeks, but it will do
  • And the module mapping – DONE
  • Fix up the dates

Still some minor bugs, to do list includes

  • Get the accordion working.
  • Timeago isn’t calculating the time since/till end of weeks.
  • Missing data- GPA, # courses complete
  • Add in the sentiment analysis work.

Page 1 of 8

Powered by WordPress & Theme by Anders Norén

css.php