Design and the limits of automation

One of the central themes of U.S. President Barack Obama’s final State of the Union addresses was the idea that wages are flat because of automation. He argued that automation, and in particular, computing, is something rapidly eliminating jobs, especially those that involve routine, proceduralized, deterministic tasks. And with machine learning, AI, and deep learning, many of the tasks that require judgment and decision-making are also being automated.

I was talking about this—forebodingly—with my 14 year old daughter at dinner the other night, and she had a surprising reaction:

“That’s great! Then we can all be artists, designer, and inventors!”

I probed:

“Why won’t those be automated too?”

“Because computers aren’t creative, and even when they are, they don’t have any taste.”

What an interesting hypothesis! We spoke on this a bit longer, and arrived at an interesting conclusion. Computers may be able to generate a lot of ideas (because of their speed and scalability), but when it comes time to selecting which of those ideas are good, they will always struggle, since notions of what makes an idea good are so subtle, multidimensional, and often subjective. This is especially true in, where emotional response has primacy over functionality. For evidence, look at any review of a movie, album, or exhibit. Could a machine predict the critiques, let alone act upon them to improve the art?

Now, even if a computer were able to leverage humanity to make these judgements (posting its ideas on Mechanical Turk for feedback), and even if it were able to synthesize this feedback into new ideas, would humanity tolerate the scale of critique necessary for computers to independently arrive at good designs and good art? It’s hard to imagine. Furthermore, wouldn’t it still be humanity making the judgements of what is right? We would still need critics to offer feedback and constructive critique. Without us, computers would not know what to choose.

Perhaps the implication of this little thought experiment is that the asymptote of computational automation leads to a society of people who do not create, but do critique, constructively. In some domains, we already see this. For example, in electronic dance music, much of the sonic material comes from other pre-existing recordings. Or in DJing, where much of the art is in selecting what to play. Algorithms may take over the task of generating the new art and designs, but we will be the editors and critics.

The watch

The Apple Watch on my wrist.

The Apple Watch on my wrist.

Yes, I bought the watch.

As an HCI researcher, I couldn’t resist knowing: what’s this thing good for?

I’ve worn it for 24 hours now and found that’s it’s good for many a small thing, but no big things. For example:

  • As someone who often has a day full of meetings at random times and places, the killer app for me is being able to glance at my wrist to see where I’m supposed to be —without awkwardly pulling my phone out in the middle of a meeting. It’s hard to overstate how valuable this is to me. It turns the social meaning of pulling out my phone to check my calendar from “I’m checking email/browsing the internet/texting a friend/and generally disinterested in this conversation” into a brief glance that means “I think have somewhere to be, but I’m listening”.
  • When I’m driving, I frequently have thoughts that I’ll lose if I don’t write them down. This creates a critical dilemma: do I grab my phone and try to have Siri transcribe it, but risk my life and a traffic ticket, or risk losing the thought? With the watch, I can dictate thoughts hands free with a quick flick of my wrist and a “Hey Siri, remind me to…”. This is particularly handy for OmniFocus, where I externalize all of my prospective memory.
  • Text messages are much less disruptive socially. No more loud phone vibrations or accidental sounds to disrupt my coworkers. Instead, the watch tells my wrist and I glance down briefly.
  • This is the best UX for Uber. “Hey Siri, open Uber”, tap the request button, and wait 5 minutes. Yes, you can do it on a phone, where you can get far more information, but I’m usually using Uber in unfamiliar places where I don’t necessarily know how safe it is to pull out a big bright iPhone and tap on the screen for a minute. This makes me feel safer.
  • This one is completely idiosyncratic to me, but I absolutely love the Alaska Airlines glance view, which simply shows a countdown until the number of days until my next flight. I hate flying, and somehow, being able to quickly see how many more days of freedom I have before I climb into a tiny box and suffer sinus pain, dry air, and cranky people, gives me a sense of freedom and appreciation for being a land mammal.

So far, everything else is of little value. I don’t like reading Twitter on the device and certainly don’t want to feel every tweet on my wrist. I’m active enough and haven’t ever wanted any device to support exercise. I get way too much email to want to triage on the device for very long. Most of the third party apps aren’t that useful yet (although as companies learn what information and services are most valuable to their users, I believe they’ll improve in their utility).

As a 1.0 device, it has all of the problems you might expect. It’s slow at times, while it talks to my iPhone. The navigation is model clunky and inconsistent. Sometimes Siri hears me say her name, sometimes she doesn’t. These will probably all be ironed out in a version or two, just as with most devices.

These issues aside, if you look at the list of benefits to me above, they fall into some unexpected categories. I thought the value to me would mostly be getting information faster, but most of the value is actually in reducing friction in social interactions and a sense of safety in various situations. This is not a device to get digital stuff done. It’s a device to get digital stuff out of the way.

Programming languages are the least usable, but most powerful human-computer interfaces ever invented

Really? I often think this when I’m in the middle of deciphering some cryptic error message, debugging somme silent failure, or figuring out the right parameter to send to some poorly documented function. I tweeted this exact phrase last week while banging my head against a completely inscrutable error message in PHP. But is it really the case that programming languages aren’t usable?

Yes and no (as with all declarative statements in tweets). But I do think that only some of these flaws are fundamental. Take, for example, Jakob Nielsen’s classic usability heuristics, one rough characterization of usability. One of the most prominent problems in user interfaces is a lack of visibility of system status: usable interfaces should provide clear, timely feedback about the how user input is interpreted so that users know what state a system is in and decide what to do next. When you write a program, there is often a massive gulf between the instructions one writes and the later effects of those instructions on program output and the world. In fact, even a simple program can branch in so many ways that some execution paths are never even observed by the programmer who wrote the instructions, but only by whoever later executes it. Talk about delayed feedback! There are whole bodies of literature on reproducibility, testing, and debugging that try to bridge this disconnect between command and action, better exposing exactly how a program will and will not behave when executed. At best, these tools provide information that guide programmers toward understanding, but this understanding will always require substantial effort, because of the inherent complexity in program execution that a person must comprehend to take action.

Another popular heuristic is Neilsen’s “match between system and the real world”: the system should use concepts, phrases, and metaphors that are familiar to the user. There’s really nothing more in opposition to this design principle than requiring a programmer to speak only in terms that a computer can reliably and predictably interpret. But need to express ideas in computational terms is really inherent to what programming languages are. There are some ways that this can be improved through good naming of identifiers, choice of language paradigm, and a selection of language constructs that reflect the domain that someone is trying to code against. In fact, you might consider the evolution of programming languages to be a slow but deliberate effort to define semantics that better model the abstractions found in the world. We’ll always, however, be expressing things in computational terms and not the messy, ambiguous terms of human thought.

Programming languages fail to satisfy many other heuristics, but can be made significantly more usable with tools. For example, error prevention and error actionability can often be met through careful language and API design. In fact, some might argue that what programming languages researchers are actually doing when they contribute new abstractions, semantics, and formalisms is trying to minimize errors and maximize error comprehensibility. Static type checking, for example, is fundamentally about providing concrete, actionable feedback sooner rather than later. This is very much a usability goal. Similarly, Nielsen’s “recognition rather than recall” heuristic has been met not through language design, but carefully designed and evolved features like autocomplete, syntax highlighting, source file outlines, class hierarchy views, links to callers and callees in documentation, and so on.

There are other usability heuristics for which programming languages might even surpass the usability of their graphical user interfaces. For example, what user interface better supports undo, redo, and cancel than programming languages? With modern text editors and version control, what change can’t be undone, redone, or canceled, at least during design time? Our best programming languages are also perhaps the most consistent, flexible, minimalist, and beautiful user interfaces that exist. These are design principles that most graphical user interfaces struggle to even approach, as they often have to make sacrifices in these dimensions to achieve a better fit with the messy reality of the world.

So while programming languages might lack usability along some dimensions, but with considerable effort in careful tool design, they can approach the usability of graphical interfaces (partly through the use of graphical user interfaces themselves). In fact, there’s been a resurgence of research inventing precisely these kinds of tools (some by me). These usability improvements can greatly increase the accessibility, learnability, and user efficiency of languages (even if they only ever approach the usability of graphical user interfaces).

Now to the second part of my claim: are programming languages really the most “powerful” user interfaces ever invented? This of course depends on what we mean by power. They are certainly the most expressive user interfaces we have, in that we can create more with them than we can with any other user interface (Photoshop is expressive, but we can’t make Photoshop with Photoshop). They might also be the most powerful interfaces in a political sense: the infrastructure we can create with them can shape the very structure of human communication, government, commerce, and cultural production.

But if by power we mean the ability to directly facilitate a particular human goal, there are many tasks for which programming languages are a terrible tool. Sending emails, having a video chat, playing a game, reading the news, managing a to do list, etc. are activities best supported by applications explicitly designed around these activities and their related concepts, not the lower level abstractions of programming languages (or even APIs). In this way, they are probably the least powerful kind of user interface, since they really only facilitate the creation of directly useful human tools.

If there’s any truth to the title of this post, its the implied idea that programming languages are just another type of human-computer interface and the rich and varied design space of user interface paradigms. This has some fun implications. For example, programmers are users too, and they deserve all of the same careful consideration that we give non-programmers using non-programming interfaces. This also means that programming languages researchers are really studying user interface design, like HCI researchers do. There aren’t two fields we might find more dissimilar in method or culture, but their questions and the phenomena they concern are actually remarkably aligned.

For those of you who know me and my work, none of these should be surprising claims. All of my research begins with the premise that programming languages are user interfaces. It’s why, despite the fact that I principally study code, coding, and coders, I pursued a Ph.D. in HCI and not PL, software engineering, or some other disciplined concerned with these phenomena. Programming languages are and will forever be to me, the most fascinating kind of user interface to study and reinvent.

abstraction appropriation

Abstractions and the ability to create them are what make us human. Our ability to reason abstractly and symbolically, to represent what we see and do and to capture and utilize knowledge is fundamental to all forms of human progress and communication. And when we think carefully about what role abstractions have played in human society, we see that our ability to reduce the incredible complexities of the world to their essential natures is behind nearly everything we do as humans.

When looking back on recent history, however, it is possible that humanity has made a fundamental shift in its use of abstractions. We have always used abstractions to communicate and talk, to coordinate, to understand nature and build technologies, from weapons to printing presses to computers, to conceptualize the essential nature of nature, and bend it to our wills and desires. Abstractions, I would argue, have been the primary mediators between humanity and nature, besides our bodies. The axe or the hammer are not simply wood and metal, they are instantiations of abstract ideas that humanity has carried from generation to generation. It is only through the idea of a hammer that a hammer can exist.

In just the past century, however, our use of abstractions has evolved. We now regularly use abstractions not only to mediate our relationship to nature, but also to mediate relationships with ourselves and others. Take, for example, the notion of IQ tests. These use of these tests is not simply to assess: the takers of these tests consume the results of the test and use such information to change perceptions of themselves. Or, consider any modern communication medium, such as e-mail or text messaging. These abstract forms of face to face communication mediate, constrain and mold our conversations in very specific ways.

This in itself isn’t problematic. After all, abstractions, by definition, eliminate detail in order to facilitate communication and action, so there are bound to be abstraction failures and mismatch; their inherent minimalism is also what makes abstractions useful, by helping us to manage the complexity of the world.

But there is a more nefarious way in which our use of abstractions may change human behavior: in many situations, we view abstractions not as a means to an end, but an end in themselves. We begin to mistake the abstraction for the thing it represents.

There are several cultural memes that highlight this phenomena. “Gaming the system,” for example, is the idea that someone will exploit properties of a system of rules or policies in order to effect results that violate the intent of the rules; Baker et al. documented this behavior in educational tutoring software, where students would learn the conditions in which the software would provide aid or answers, and do precisely the actions necessary to most quickly acquire the aid or answers.

Other examples are not about exploitation, but pragmatism. For example, students in high schools and universities want to acquire knowledge and skills, but perceive that it is scores, grades, and degrees—our abstractions of learning in modern education—that are truly important, and not the learning itself. The danger here is less at the individual level (as an individual student may overcome this through reflection), and more at a societal and cultural level: over time, it is possible that the abstractions representing knowledge become so institutionalized that society forgets what they were intended to represent.

I see these abstraction appropriate every day when I teach. Just yesterday I had an enjoyable, but disheartening discussion with a couple of students near graduation who were disappointed in their final grades for a course I taught last quarter. Their concern was that the grade points they received, which were one or two tenths lower than the grade points they typically receive, would lower their GPA several hundredths. I assured them I understood their concern, but also pointed out to them that there was probably not a single person who would ever look at that grade, nor the tenths place of their grade, ever again in their lives. One of them mentioned graduate school applications and I insisted, if they were above a 3.7, what would really matter were their letters, publications, and experience, since the number doesn’t really mean much of anything.

This was disappointing to them, to say the least. I reassured them that it was the products of their work, and the experience they had gained in the course, that would be the truly lasting parts of their education, and that the numbers meant nothing. They thanked me for my time and walked away slightly confused, unsure about what other strange quantitative incentive structures might be in store for them post graduation.

Every educator knows what I’m talking about. Every middle manager who’s had to quantify or categorize their employees’ performance knows what I mean. And while these abstractions may help us facilitate decision making, we rarely think about their side effects on human behavior and the larger incentive structures we propagate through society.

Where else do you see the abstraction misappropriation? And what are the consequences of embedding these abstractions in the software throughout our communications and infrastructure? Is all of this just a manifestation of Campbell’s law, or does this idea go beyond social planning? And what is it about human cognition that leads this phenomena, if it is as widespread as it seems?

What the iPad is and isn’t

After 4 hours of continuous use, I can confidently say that the iPad rocks in many ways, and fails in only a few. It’s a genius way to browse the web, to write short emails, to listen to music, to watch short videos, to use Facebook and Twitter, to give simple presentations, to read news, and to show photos. Theres literally no better experience out there for most of these activities. It’s also a great sketchpad — not as great as a real sketchpad, mind you, but oh so much easier to share and archive.

It fails in the obvious places. The onscreen keyboard is bearable. I can tye a lot faster than on my iPhone or any cell phone, but I’m not getting my typical 80 wpm. A wireless keyboard would make up for a lot of these limitations, but it sort of defeats the purpose of carrying something slim and lightweight. I’ve typed a lot on is in the past few hours and I feel a bit held back, but not so much that I don’t feel productive.

There are still some ways where multitouch is inherently limited. One out of every 10 times I tap or drag, it doesn’t do what I want. This is no different than on the iPhone, but I’ve noticed myself acclimate to the inaccuracy. The device hasn’t gotten any smarter, I’ve just gotten more tolerant.

The iPhone UI toolkit still breaks many pervasive web conventions. For example, I’m typing this in a text field in a WordPress page, and scrolling up to edit the previous paragraphs is incredibly slow, even with the two fingered scroll interaction, compared to a scroll wheel.

But I already love this thing. For all of the activities I mentioned earlier, the iPad is the clear winner. It’ll sit next to me at my desk and be a constant source of distraction during work. I’m thinking it’ll be a dedicated calendar while I use my laptop. Time to start exploring what else this form excels at! Like multitouch visual programming, hint hint…

my juxtaposition on the ipad

Yeah, I’m a little late to the discussion. But as I’ve contemplated over past weeks the merits of the iPads form and function, trying to imagine what I’d do with it and what others might do with it, I keep coming back to the same problem: the iPad, nor the iPod or iPhone, support juxtaposition. That’s what all of this whining about multitasking is about. So many things we do on computers is compare, contrast, and cross-reference between applications, and yet that’s one of the major things the iPad cannot do.

I wish copy and paste were enough, but it’s not. It’s about writing an email about the news article you have open, or quickly checking the status on some build, or reading a dictionary definition online while you’re writing. You can’t do these things on single-task UIs, because the cost of leaving one app, opening another, and then returning to an app is at least 30 seconds. That, and everything you might want to juxtapose against has to be kept in your head for these 30 seconds. Good luck with that when you’re trying to think.

So maybe Apple decided the device wasn’t for thinking or creating. Maybe it’s just for consuming. But even consumption takes juxtaposition. I find myself on my iPhone all the time, wanting to read a Yelp review and see where a place is on the map at the same time.  Because this isn’t possible, the Yelp app tries to do maps well, and the Maps app will probably try to incorporate reviews, leading to substandard experiences in both apps. Or, another example was when I was doing my taxes online: I was referencing advice in forums about teacher deductions (which I found out I can’t take), while trying to decide how to answer a TurboTax question. On the iPad, I’d have to go back and forth between the two, memorizing all the numbers and exceptions in the forum post in order to act upon them in the tax software.

People are going to realize this soon, too, and Apple’s going to suffer for it. Either Apple is just waiting for the right time to support juxtaposition, or their designers just have no idea how people produce and consume information.

tough T

I just spent a day at Edward Tufte’s course on information design at the Seattle Marriott Waterfront. I’ve always known his work, I’ve talked about it in design classes, I’ve told students to read his books, but not once have I heard him speak. Now I can confidently say that his captions speak louder than words. Snicker.

That’s not to say he wasn’t insightful. The books have always been a nice translation of classic design principles into static visual information design, but most of the course was simply him parroting his own words. What made it unbearable was that he spoke them with the lifeless apathy of a statistics professor. Oh wait, he was one.

Aside from his lack of spark, there were a number of nice things about the day. I got a box full of his books; I got a refresher on visual information design; I had a chance to think more about forms of dissemination for my research (I tire of limiting my influence to academic publications). It was also a nice calm before my early May storm of deadlines.