I came across the discussion underpinning these thoughts via a post by Scott Stevenson. His post was titled “Measuring the design process”. It is his take on a post titled “Goodbye Google” by Doug Bowman. Bowman was the “Visual Design Lead” at Google and has recently moved to Twitter as Creative Director.

My take of the heart of the discussion is the mismatch between the design and engineering cultures. Design is okay with relying on experience and intuition for the basis for a decision. While the engineering culture wants everything measured, tested and backed up by data.

In particular, Bowman suggests that the reason for this data-driven reliance is

a company eventually runs out of reasons for design decisions. With every new design decision, critics cry foul. Without conviction, doubt creeps in. Instincts fail. “Is this the right move?”

The doubt, the lack of a reason, purpose, or vision for a change creates a vacuum that needs to be filled. There needs to be some reason to point to for the decision.

When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board.

Can’t be anything wrong with that? Can there? If you’re rational and have data to back you up then you can’t be blamed. Bowman suggests that there is a problem

And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions.

He goes on to illustrate the point, where the focus goes to small questions – should a border be 3, 4 or 5 pixels wide – while the big questions, the important questions that can make a design distinctive become ignored. This happens because hard problems are hard and almost certainly impossible to gather objective data for.

Stevenson makes this point

Visual design is often the polar opposite of engineering: trading hard edges for subjective decisions based on gut feelings and personal experiences. It’s messy, unpredictable, and notoriously hard to measure.

Learning design, e-learning and university teaching

This same problem arises in universities around learning design, e-learning and university teaching. The design of university teaching and learning has some strong connections with visual design. It involves subjective and contextual decisions, it’s messy, unpredictable and hard to measure.

The inherently subjective and messy nature of university teaching brings it into direct tension with two increasingly important and powerful cultures within the modern university:

  1. Corporate management; and
    Since sometime in the 90s, at least within Australia, corporate manageralism has been on the rise within universities. Newton (2003) has a nice section on some of the external factors that have contributed to this rise, I’ve summarised Newton here. Further underpinning this rise has been what Birnbaum (2000) calls “education’s Second Management Revolution” from around 1960 and which “marks the ascendance of rationality in academic management”.
  2. Information technology.
    With the rise of e-learning and other enterprise systems, the corporate IT culture within universities is increasingly strong. In particular, from my cynical perspective, when they can talk the same “rational” talk as the management culture and back this up with reams of data (regardless of validity) and can always resort of techno-babble to confuse management.

Both these cultures put an emphasis on rationality, on having data to support decisions and on being able to quantify things.

Symptoms of this problem

Just taking the last couple of years, I’ve seen the following symptoms of this:

  • The desire to have a fixed, up-front estimate of how long it takes to re-design a course.
    I want you to re-design 4 courses. How long will it take?
  • The attempt to achieve quality through consistency.
    This is such a fundamentally flawed idea, but it is still around. Sometimes it is proposed by people who should know better. The idea that a single course design, word template or educational theory is suitable for all courses at an institution, let alone all learners, sounds good, but doesn’t work.
  • Reports indicating that the re-design and conversion of courses to a new LMS are XX% complete.
    Heard about this just recently. If you are re-designing a raft of different courses, taught be different people, in different disciplines, using different approaches and then porting them to a new LMS, how can you say it is XX% complete. The variety in courses will mean that you can’t quantify how long it will take. You might have 5 or 10 courses completed, but that doesn’t mean you’re 50% completed. The last 5 courses might take much longer.
  • The use of a checklist to evaluate LMSes.
    This has to be the ultimate, use a check list to reduce the performance of an LMS to a single number!!
  • Designing innovation by going out to ask people what they want.
    For example, let’s go and ask students or staff how they want to use Web 2.0 tools in their learning and teaching. That old “fordist”, the archetypal example of rationalism, Henry Ford even knew better than this

    “If I had asked people what they wanted, they would have said faster horses.”

The scary thing is, because design is messy and hard, the rational folk don’t want to deal with it. Much easier to deal with the data and quantifiable problems.

Of course, the trouble with this is summarised by a sign that used to hang in Einstein’s office at Princeton (apparently)

Not everything that counts can be counted, and not everything that can be counted counts.


This mismatch between rationality and the nature of learning and teaching leads, from my perspective, to most of the problems facing universities around teaching. Task corruption and a reliance on “blame the teacher”/prescription approaches to improving teaching arise from this mismatch.

This mismatch arises, I believe, for much the same reason as Bowman used in his post about Google. The IT and management folk don’t have any convictions or understanding about teaching or, perhaps, about leading academics. Consequently, they fall back onto the age-old (and disproved) management/rational techniques. As they give the appearance of rationality.


Birnbaum, R. (2000). Management Fads in Higher Education: Where They Come From, What They Do, Why They Fail. San Francisco, Jossey-Bass.

3 thoughts on “Measuring the design process – implications for learning design, e-learning and university teaching

Leave a Reply

Your email address will not be published. Required fields are marked *