Here is part two of Emer­son Murphy-Hill’s inter­view with me. This part cov­ers some of the chal­lenges in pub­lish­ing in HCI venues.

Q: A promi­nent pro­po­nent of empir­i­cal soft­ware engi­neer­ing once told me that that he typ­i­cally spends a full page dis­cussing the threats to valid­ity of his eval­u­a­tions. At the same time, it’s not unusual to find a CHI paper that doesn’t dis­cuss threats. How does one choose which threats to include and exclude, and how to present those threats, to the CHI community?

Most CHI papers clearly dis­cuss threats, just not in a sec­tion titled “threats to valid­ity.” This tra­di­tion comes from CHI’s cog­ni­tive psy­chol­ogy research, where the threats were inher­ent to the study design and dis­cussed through­out the method and dis­cus­sion sec­tions. There never needed to be a sep­a­rate sec­tion because it was expected that dis­cus­sion of the lim­i­ta­tions would appear through­out the arti­cle. As a guide­line, one should always dis­cuss all non-obvious threats to valid­ity. Its a nec­es­sary part of hon­est schol­arly work.

Q: Where do you draw the line about whether a threat is obvious?

Some threats are com­mon to all empir­i­cal research: the sam­ple size was to too small, the study may not gen­er­al­ize, sit­u­a­tions may not have been rep­re­sen­ta­tive. These are stan­dard dis­claimers and its always worth men­tion­ing them briefly. The ones to really spend time on are the def­i­n­i­tions and mea­sures one uses and what like­li­hood they have at actu­ally cap­tur­ing the con­cept of inter­est (the con­struct valid­ity) and whether they have any mean­ing for the real world (eco­log­i­cal validity).

Q: Have you had an HCI reviewer sug­gest that your work is bet­ter suited for a soft­ware engi­neer­ing venue, or vice versa? If so, how did you deal with the sug­ges­tion? If not, how do you think you pre­empted it in the first place?

No, I’ve never had a reviewer sug­gest that. Of course, the work that I pub­lish at HCI venues usu­ally has more to do with the actual work of soft­ware engi­neers, their col­lab­o­ra­tions, or their inter­ac­tions with users, as opposed to con­ven­tional soft­ware engi­neer­ing research on automa­tion. I think one of the main stum­bling blocks that soft­ware engi­neer­ing researchers will have try­ing to pub­lish at HCI venues is demon­strat­ing that the prob­lems they work on are of sig­nif­i­cance. For exam­ple, a com­mon type of soft­ware engi­neer­ing paper will find some spe­cific set of cir­cum­stances that can be exploited to auto­mate bug find­ing or prove cor­rect­ness within a cer­tain set of assump­tions. In gen­eral, HCI researchers aren’t inter­ested in these types nar­row con­tri­bu­tions, unless there’s some good evi­dence that the set of cir­cum­stances exploited is large and gen­er­al­iz­able to some degree.

Q: In an HCI paper, where do you make the argu­ment about gen­er­al­iz­abil­ity? Is there room for speculation?

Andrew: There’s always room for spec­u­la­tion. That’s what dis­cus­sion and lim­i­ta­tion sec­tions are for. The whole point of stud­ies is to use a ker­nel of rig­or­ous and trusted analy­sis in order to make pre­dic­tions about the larger con­text of the world. In fact, I think too many soft­ware engi­neer­ing papers sim­ply report results and ignore what impact a tool design or study might have on our under­stand­ing of soft­ware engi­neer­ing. Tools, after all, are embod­i­ments of the­o­ries about the world, and they have just as much poten­tial to teach us about our sur­round­ings as stud­ies – per­haps more.

Q: As a reviewer for HCI venues, what is the most com­mon mis­take that you see soft­ware researchers making?

Being more fas­ci­nated with tech­nol­ogy itself than what tech­nol­ogy does for peo­ple (whether those peo­ple are tech­nol­ogy users or hard­core soft­ware devel­op­ers). More often than not, I will read soft­ware engi­neer­ing papers pub­lished at HCI venues that try hard to per­suade me that the clever tricks they devised are inter­est­ing enough to over­come the min­i­mal impact the tricks will have on users’ work and expe­ri­ence with a tool.

I also see soft­ware engi­neer­ing researchers try to make knowl­edge con­tri­bu­tions about soft­ware devel­op­ment prac­tice with­out cit­ing the large body of work done at CSCW and other con­fer­ences about group work. HCI researchers tend to view soft­ware devel­op­ment as just one of many exam­ples of col­lab­o­ra­tive work. The argu­ment that its spe­cial and unique usu­ally doesn’t fly with­out evidence.

Q: Although HCI sub­mis­sions are often anony­mous, peo­ple tend to be sus­pi­cious of “out­siders,” and may treat out­siders’ work with some undue hos­til­ity. What can a soft­ware researcher do to avoid iden­ti­fy­ing him­self as an out­sider in the HCI community?

All HCI researchers are out­siders. There’s not enough of a con­cen­tra­tion on any one topic or prob­lem for there to be a com­mon core. The best thing to avoid sound­ing naive is to read as much about a topic out­side of your dis­ci­pline as pos­si­ble. HCI draws from cog­ni­tive sci­ence, psy­chol­ogy, design, com­puter sci­ence, engi­neer­ing, anthro­pol­ogy, social psy­chol­ogy, com­mu­ni­ca­tion, edu­ca­tion, and sev­eral other fields. Chances are, there’s work in all of those fields you should at least be aware of, if not read and cite.

Q: Sup­pose that you attempt to solve a usabil­ity prob­lem for a cer­tain kind of soft­ware tool; HCI researchers may per­ceive that you are solv­ing only a very nar­row prob­lem, and thus your con­tri­bu­tion is small. How do you deal with that?

The typ­i­cal solu­tion to this prob­lem is find­ing a com­mu­nity that thinks your prob­lem is broad instead of nar­row. HCI research tends to have a fairly broad view of the world, since its so applied, so under­stand­ably, many prob­lems will viewed as small (just like any non-academic would view our prob­lems as nar­row). The best one can do is demon­strate what rela­tion the prob­lem has to soci­ety and what impact it might have on the world – not just on the tool users.

Q: Where should soft­ware researchers send their human-centered papers?

Anon: CHI is an obvi­ous choice, but it’s the pre­mier con­fer­ence in HCI, which makes it a dif­fi­cult tar­get even for very expe­ri­enced HCI researchers. Beyond CHI, what would we rec­om­mend? I’ve been invest­ing in VL/HCC, the log­i­cal suc­ces­sor to the sadly defunct Empir­i­cal Stud­ies of Pro­gram­mers (ESP) con­fer­ence. It’s a strong sec­ondary con­fer­ence, with a first-rate com­mu­nity, but with less con­tent about pro­fes­sional SEs than I would like. I’m hard-pressed to rec­om­mend another HCI conference.