A com­mon com­plaint of research is that it’s not “sur­pris­ing.” For exam­ple, a reviewer might say, “The study was well done, but the results weren’t really that sur­pris­ing.”, or, “I found the results a bit predictable.”

But what do these state­ments really mean? Do they mean, “Had you asked me the research ques­tion, I could have guessed the results with some degree of con­fi­dence.”? Or, “If you asked your research ques­tion of 100 experts, 95 of their guesses would have been right.”?

Maybe we might intend for them to mean that, but they don’t actu­ally cap­ture what hap­pens when a reviewer reads a paper. What usu­ally hap­pens is the reviewer reads the research ques­tion and thinks, “Hm, I could guess, but I’m not sure.” Then, upon read­ing the results, the reviewer thinks, “Well of course, that’s not sur­pris­ing at all.” The test exe­cuted here is not whether an expert can con­fi­dently pre­dict the answer to a research ques­tion, but whether in hind­sight it seems plau­si­ble that an expert could have guessed the result.

In this sense, what makes a result “sur­pris­ing” has less to do with what we know as sci­en­tists and more to do with what we think we know about what other researchers know.

This social fab­ric that appar­ently under­lies our judge­ments of what is known has other inter­est­ing effects on what is accepted as advanc­ing knowl­edge. For exam­ple, that some find­ing has not been pub­lished, is rarely a sat­is­fac­tory argu­ment for why some­thing should be pub­lished. What under­lies this belief it that it is not our goal as sci­en­tists to doc­u­ment every­thing that we know. Instead, it is our job to doc­u­ment the sub­set of what we know that is inter­est­ing, impor­tant, and surprising.

But aren’t most judge­ments of what is inter­est­ing and impor­tant are grounded in the present? How are we to know what is inter­est­ing or impor­tant in the future? Who are we to judge that the future of human­ity will find no inter­est in the unin­ter­est­ing, unim­por­tant results of today? Take, for exam­ple, a recent review I wrote on a paper about using mul­ti­touch, table­top dis­plays for engi­neer­ing design. I argued that it was unclear what prob­lem was being solved. But what if it solves a prob­lem that doesn’t exist yet? Or what if it solves it in such a way that another prob­lem I hadn’t even thought of becomes triv­ial? On what basis could I really judge whether the work would have future worth?

All of this makes me think I don’t give papers a fair shake. Maybe I’ll adopt a new review­ing pro­to­col: instead of read­ing the paper straight through and record­ing my thoughts, I’ll look at the authors’ research ques­tion and try to answer it myself for five min­utes. Then, I’ll read the paper and if they came up with a dif­fer­ent solu­tion or answer that mine (that is of course reli­able, sound, etc.), whether or not I’m sur­prised, the authors get credit for dis­cov­er­ing or invent­ing some­thing that I didn’t know. Of course, If I guessed their results or solu­tion in a mere five min­utes, what could they pos­si­bly have contributed?

I just received my first ever first-authored con­fer­ence paper rejec­tion from FSE. The pri­mary rea­sons, quoted from the reviews, include:

• “The qual­i­ta­tive nature of the study … is liable to mis­in­ter­pre­ta­tion and bias.”
• “I was expect­ing a quan­ti­ta­tive analy­sis: is there any cor­re­la­tion between some of the char­ac­ter­is­tics and between [the results] and the time a bug takes to resolve and its res­o­lu­tion status?
• “I would have thought that what types of ele­ments to look for in dis­cus­sion should be decided before by the researchers as it should be based on the problem”
• “I was expect­ing con­crete advice on HOW the tools should struc­ture the discussion.”

I was hop­ing the review­ers would have been more epis­te­mo­log­i­cally informed. For exam­ple, the first and sec­ond quotes are quite telling: they imply that some forms of empiri­cism are not sub­ject to mis­in­ter­pre­ta­tion or bias. But quan­ti­ta­tive empir­i­cal mea­sures are just as sub­ject to bias as any other mea­sure. For exam­ple, if I had counted cer­tain kinds of data and run cor­re­la­tions between these counts and other out­come mea­sures, not only would one in twenty of them be “sta­tis­ti­cally sig­nif­i­cant” by chance, but whether there was any real mean­ing in the vari­ables depends on the con­struct valid­ity of the quan­ti­ta­tive mea­sure­ments. For exam­ple, if I had cor­re­lated hyper­boles with bug res­o­lu­tion time, not only would the hyper­bole mea­sure have the same lim­i­ta­tions as it did as a qual­i­ta­tive clas­si­fi­ca­tion, but the bug res­o­lu­tion time would have any num­ber of con­tex­tual fac­tors that could influ­ence its true reflec­tion of the hyperbole’s impact on con­sen­sus. Trans­form­ing empir­i­cal obser­va­tions into num­bers does NOT make them objec­tive, nor does it pre­vent bias and mis­in­ter­pre­ta­tion.

The third quote is ironic: this reviewer seems to believe that the only way to ana­lyze a prob­lem is to make some assump­tion about its nature upfront. The whole point of qual­i­ta­tive research is that the more you make upfront assump­tions, the more you bias your find­ings. What this reviewer is propos­ing would have less­ened the objec­tiv­ity of the results and pre­vented us from uncov­er­ing the trends we did.

The last quote reveals the sys­temic bias in soft­ware engi­neer­ing research (and also some HCI venues): qual­i­ta­tive stud­ies are only valu­able if they explic­itly inform design. What this really reduces to is a view that mate­r­ial goods are real work, but the pro­duc­tion of knowl­edge comes for free. Build­ing a sys­tem or automat­ing some activ­ity, even if the sys­tem and automa­tion are entirely imprac­ti­cal in the real world, is more valu­able than under­stand­ing the real world. The com­ment also reveals the reviewer’s lack of under­stand­ing about design: inno­va­tions don’t come from stud­ies, they come from peo­ple. Stud­ies can sup­port design deci­sions (and the results through­out our rejected sub­mis­sion have been quite valu­able in our cur­rent design efforts), but they can­not gen­er­ate ideas. Peo­ple gen­er­ate ideas.

Had I really wanted the paper in, I would have lit­tered the sub­mis­sion with arbi­trary, but seem­ingly objec­tive quan­tifi­ca­tions and cor­re­la­tions of our data (which is what most quan­tifi­ca­tions are in soft­ware engi­neer­ing papers). This has worked in past papers and is a tried and true workaround for the soft­ware engi­neer­ing community’s lack of expe­ri­ence with qual­i­ta­tive meth­ods. Review­ers would have thought, “I don’t get all of this qual­i­ta­tive stuff, but these num­bers are great.” I decided not to do this on prin­ci­ple, since doing so would have only made the results seem more objec­tive with­out adding any real objectivity.

So much for prin­ci­ple. Time to start cor­re­lat­ing things!

I just spent a day at Edward Tufte’s course on infor­ma­tion design at the Seat­tle Mar­riott Water­front. I’ve always known his work, I’ve talked about it in design classes, I’ve told stu­dents to read his books, but not once have I heard him speak. Now I can con­fi­dently say that his cap­tions speak louder than words. Snicker.

That’s not to say he wasn’t insight­ful. The books have always been a nice trans­la­tion of clas­sic design prin­ci­ples into sta­tic visual infor­ma­tion design, but most of the course was sim­ply him par­rot­ing his own words. What made it unbear­able was that he spoke them with the life­less apa­thy of a sta­tis­tics pro­fes­sor. Oh wait, he was one.

Aside from his lack of spark, there were a num­ber of nice things about the day. I got a box full of his books; I got a refresher on visual infor­ma­tion design; I had a chance to think more about forms of dis­sem­i­na­tion for my research (I tire of lim­it­ing my influ­ence to aca­d­e­mic pub­li­ca­tions). It was also a nice calm before my early May storm of deadlines.

I had a meet­ing today with the Human Inter­ac­tions in Pro­gram­ming group at Microsoft Research today, along with a few other UW peo­ple, to talk about poten­tials for col­lab­o­ra­tion. Lots of oppor­tu­ni­ties came out of our meet­ing, but one thing stuck around in my head today after David Notkin made a com­ment about peo­ples’ mis­per­cep­tions about what soft­ware is and is capa­ble of. I replied to his com­ment this notion that there are all kinds of ver­sions of a soft­ware arti­fact: ver­sions on paper, ver­sions in specs, ver­sions float­ing around in devel­op­ers’ heads, ver­sions in users’ heads. And of course, the real ver­sion that ends up being built. The intrigu­ing thing here is how all of these dif­fer­ent con­cep­tions end up affect­ing the actual soft­ware arti­fact. For exam­ple, think of two devel­op­ers dis­cussing some bro­ken fea­ture and how they want to fix it: if you really lis­ten, the dis­cus­sion is about future ver­sions of the cur­rent ver­sion and all of the qual­ity impli­ca­tions of the change. They fill the dis­cus­sion with descrip­tions of the slightly mod­i­fied sys­tem at a vari­ety of lev­els of abstrac­tion, with the goal of debat­ing the mer­its of the mod­i­fied sys­tem. The same con­ver­sa­tion might occur between a pro­gram man­ager and a mar­keter, dis­cussing ways of describ­ing what it is that is being built.

It seems to me that all of these dif­fer­ent ver­sions of the sys­tem are essen­tially sketches, in the broad­est design sense.  They’re ver­bal sketches, and likely so because pro­grams and soft­ware sys­tems are so tricky to draw. So when two devel­op­ers debate a change, they’re essen­tially draw­ing out the mod­i­fi­ca­tion in the ether, let­ting the ideas linger in the air, cri­tique the ideas in their phono­log­i­cal loops. I helped with a study by Mauro Cheru­bini look­ing at devel­op­ers’ sketches, and the result was quite sim­i­lar: in most cases, devel­op­ers use dia­grams in order to sup­port dis­cus­sions about changes to code.

I’m plan­ning some stud­ies look­ing at soft­ware devel­op­ers’ design dis­cus­sions and this idea of a ver­bal sketch could be quite help­ful in find­ing mean­ing in the data.