Comments on: the semblance of objectivity in numbers http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/ Musings about software and the world's attempt to understand it Sat, 07 May 2011 05:07:27 +0000 hourly 1 http://wordpress.org/?v=3.3.2 By: Bad surveys « Catenary http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/comment-page-1/#comment-27 Bad surveys « Catenary Sun, 15 Nov 2009 17:58:13 +0000 http://andyjko.com/?p=86#comment-27 [...] What gets me is that results from a survey as poorly and dishonestly executed as this one will carry greater weight than any non-quantitative arguments simply because they produce a percentage number in the end. We’re in love with quantitative [...] […] What gets me is that results from a survey as poorly and dishonestly executed as this one will carry greater weight than any non-quantitative arguments simply because they produce a percentage number in the end. We’re in love with quantitative […]

]]> By: Andy http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/comment-page-1/#comment-26 Andy Tue, 28 Jul 2009 16:50:10 +0000 http://andyjko.com/?p=86#comment-26 Thanks for your insights Andreas. I completely agree about the importance of establishing some form of causation. Software engineering, as a design discipline, is about changing and improving practice, and understanding the causal relationships between the variables we see in practice is the most reliable way to impact practice. I fully support this perspective. Unfortunately, I've found the software engineering research community to have a very limited notion causality. First, many of the reviews I've seen seem to think that one can *establish* causality. Empiricism can do no such thing. We can only gain confidence in it. And in fact, as any good epistemologist will tell you, most of our confidence comes from convergent validity: using a variety of methods and measures to study the same phenomenon and finding the same underlying truth through each lens. If the software engineering research community continues to limit publishable methods to quantitative empiricism, we will have a very skewed (and I would argue shallow) understanding of practice. And its not like the quantitative empiricism in software engineering research is that good (but it is getting better). Testing theories and demonstrating causality requires experiments and replications of experiments. These rarely get published because of concerns that there's nothing new being established. A well-designed study with a non-result is often quite important in confirming our understanding of the world. Without these works, we practice weak empiricism. Yet there is larger, more systematic problem in what work the community values. Currently, the community only incentives work that *tests* theories, but it has little interest in (qualitative) work that generates new new theories? The whole point of qualitative research is to rigorously extract new theories from empirical observation. This was the goal of my paper, but the reviewers were of the mind, "These theories are great and we will use them to inform tool and process design, but I don't see any real work here." Apparently, three months of rigorous analysis of over 10,000 bug report comments isn't work. Unless those analyses involved numbers :) I think its perfectly reasonable have work that "goes all the way" dominate, especially work that transforms our understanding of how to improve software engineering. But so much of this work lacks any substantive shift in our view of how software engineering work happens and how it can be improved. These shifts in understanding don't come from narrow experiments or clever automations. They come from deep, rigorous analysis of what makes software engineering challenging. This brings me to a larger critique of PCs and conferences in general. Because our work is archived in conferences, which has limited slots, the goal of a PC is unfortunately to choose the "best" of the submitted work as opposed to work that is above some level of quality. If we were a discipline that was interested in creating generalizable, broad knowledge about software engineering, we would structure our modes of dissemination to ensure that all high quality knowledge is spread to the larger community. Currently, the artificial cutoffs imposed by conference venues means that we only spread the most conservative of high quality work. Work that applies new methods or tests a plausible but exotic theory rarely finds a home. I'm actually not bitter; I predicted the paper would be rejected for the methods it used. I just hope for a day where software engineering research uses a wider variety of methods to understand and impact software engineering work. From my view, it's still in its infancy from a scientific perspective, with most researchers confusing "qualitative" with "subjective". Thanks for your insights Andreas. I completely agree about the importance of establishing some form of causation. Software engineering, as a design discipline, is about changing and improving practice, and understanding the causal relationships between the variables we see in practice is the most reliable way to impact practice. I fully support this perspective.

Unfortunately, I’ve found the software engineering research community to have a very limited notion causality. First, many of the reviews I’ve seen seem to think that one can *establish* causality. Empiricism can do no such thing. We can only gain confidence in it. And in fact, as any good epistemologist will tell you, most of our confidence comes from convergent validity: using a variety of methods and measures to study the same phenomenon and finding the same underlying truth through each lens. If the software engineering research community continues to limit publishable methods to quantitative empiricism, we will have a very skewed (and I would argue shallow) understanding of practice.

And its not like the quantitative empiricism in software engineering research is that good (but it is getting better). Testing theories and demonstrating causality requires experiments and replications of experiments. These rarely get published because of concerns that there’s nothing new being established. A well-designed study with a non-result is often quite important in confirming our understanding of the world. Without these works, we practice weak empiricism.

Yet there is larger, more systematic problem in what work the community values. Currently, the community only incentives work that *tests* theories, but it has little interest in (qualitative) work that generates new new theories? The whole point of qualitative research is to rigorously extract new theories from empirical observation. This was the goal of my paper, but the reviewers were of the mind, “These theories are great and we will use them to inform tool and process design, but I don’t see any real work here.” Apparently, three months of rigorous analysis of over 10,000 bug report comments isn’t work. Unless those analyses involved numbers :)

I think its perfectly reasonable have work that “goes all the way” dominate, especially work that transforms our understanding of how to improve software engineering. But so much of this work lacks any substantive shift in our view of how software engineering work happens and how it can be improved. These shifts in understanding don’t come from narrow experiments or clever automations. They come from deep, rigorous analysis of what makes software engineering challenging.

This brings me to a larger critique of PCs and conferences in general. Because our work is archived in conferences, which has limited slots, the goal of a PC is unfortunately to choose the “best” of the submitted work as opposed to work that is above some level of quality. If we were a discipline that was interested in creating generalizable, broad knowledge about software engineering, we would structure our modes of dissemination to ensure that all high quality knowledge is spread to the larger community. Currently, the artificial cutoffs imposed by conference venues means that we only spread the most conservative of high quality work. Work that applies new methods or tests a plausible but exotic theory rarely finds a home.

I’m actually not bitter; I predicted the paper would be rejected for the methods it used. I just hope for a day where software engineering research uses a wider variety of methods to understand and impact software engineering work. From my view, it’s still in its infancy from a scientific perspective, with most researchers confusing “qualitative” with “subjective”.

]]> By: Andreas Zeller http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/comment-page-1/#comment-25 Andreas Zeller Tue, 28 Jul 2009 11:11:47 +0000 http://andyjko.com/?p=86#comment-25 I have been a member of the PC that so frustrated you; and as future ESEC/FSE program chair, I am concerned about your implications. I cannot confirm a systemic bias against qualitative research (nor against quantitative research, for that matter). What I am looking for in a PC is causation, not correlation; and for causation, you not only need a well-established (quantitative) correlation, but also a (qualitative) theory on why things are as they seem. And the reason we're looking for causation is that indeed, we'd like results to be actionable -- such that people can change the cause in order to influence the effect. (In another sense, these papers carry information rather than just data.) Few papers excel in establishing causality, but they tend to dominate papers that are less actionable. And if a paper provides only the initial steps towards further research, it will be dominated by research that has gone all the way. (I don't have access to your submission or its reviews, as I have a conflict with UW; so none of this need to apply to your paper. But if you send the material to me, I'll be happy to give you some extra advice.) Best -- Andreas I have been a member of the PC that so frustrated you; and as future ESEC/FSE program chair, I am concerned about your implications. I cannot confirm a systemic bias against qualitative research (nor against quantitative research, for that matter). What I am looking for in a PC is causation, not correlation; and for causation, you not only need a well-established (quantitative) correlation, but also a (qualitative) theory on why things are as they seem. And the reason we’re looking for causation is that indeed, we’d like results to be actionable — such that people can change the cause in order to influence the effect. (In another sense, these papers carry information rather than just data.)

Few papers excel in establishing causality, but they tend to dominate papers that are less actionable. And if a paper provides only the initial steps towards further research, it will be dominated by research that has gone all the way.

(I don’t have access to your submission or its reviews, as I have a conflict with UW; so none of this need to apply to your paper. But if you send the material to me, I’ll be happy to give you some extra advice.)

Best — Andreas

]]> By: andyjko http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/comment-page-1/#comment-24 andyjko Mon, 01 Jun 2009 22:47:05 +0000 http://andyjko.com/?p=86#comment-24 That's a good suggestion. We just submitted a revised version to CSCW today, but ESEM would be a good place if that doesn't work out. Thanks! That’s a good suggestion. We just submitted a revised version to CSCW today, but ESEM would be a good place if that doesn’t work out. Thanks!

]]> By: andyjko http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/comment-page-1/#comment-23 andyjko Mon, 01 Jun 2009 17:31:21 +0000 http://andyjko.com/?p=86#comment-23 I think the issue is that the implications of studies can always have implications, but don't always have implications for design. Sometimes the implications are for process, organization, or training. I think the problem is that most SE conferences don't really view these other forms of implications as interesting or relevant. I think the issue is that the implications of studies can always have implications, but don’t always have implications for design. Sometimes the implications are for process, organization, or training. I think the problem is that most SE conferences don’t really view these other forms of implications as interesting or relevant.

]]> By: John Daughtry http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/comment-page-1/#comment-22 John Daughtry Mon, 01 Jun 2009 17:15:05 +0000 http://andyjko.com/?p=86#comment-22 Your third point that has bothering me for the past month: are studies worthwhile if they fail to inform design? My current work has no clear path for informing design (right now). In the long term, it may, but I can't articulate how at the moment. Is it worth pursuing? I think so. But, without articulating a clear motivation in the form of motivating design, then how can I sell it to a committee or funding source? Your third point that has bothering me for the past month: are studies worthwhile if they fail to inform design?

My current work has no clear path for informing design (right now). In the long term, it may, but I can’t articulate how at the moment.

Is it worth pursuing? I think so. But, without articulating a clear motivation in the form of motivating design, then how can I sell it to a committee or funding source?

]]> By: Lorin Hochstein http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/comment-page-1/#comment-21 Lorin Hochstein Mon, 01 Jun 2009 16:13:51 +0000 http://andyjko.com/?p=86#comment-21 I suspect that ESEM would have been more receptive to this work, especially given Carolyn Seaman's involvement in the community. Admittedly, ESEM is not considered as prestigious as FSE, but you get to maintain your principles! I suspect that ESEM would have been more receptive to this work, especially given Carolyn Seaman’s involvement in the community. Admittedly, ESEM is not considered as prestigious as FSE, but you get to maintain your principles!

]]> By: andyjko http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/comment-page-1/#comment-20 andyjko Sun, 31 May 2009 22:00:06 +0000 http://andyjko.com/?p=86#comment-20 That's what I usually do. This time I didn't do it out of principle, since it shouldn't be necessary :) So much for principle, back to pragmatism. That’s what I usually do. This time I didn’t do it out of principle, since it shouldn’t be necessary :) So much for principle, back to pragmatism.

]]> By: Neil Ernst http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/comment-page-1/#comment-19 Neil Ernst Sun, 31 May 2009 19:04:52 +0000 http://andyjko.com/?p=86#comment-19 Would some form of mixed methods research design be a way to satisfy the 'numbers people'? Would some form of mixed methods research design be a way to satisfy the ‘numbers people’?

]]> By: Andy Ko and the semblance of objectivity in numbers « Catenary http://blogs.uw.edu/ajko/2009/05/30/the-semblance-of-objectivity-in-numbers/comment-page-1/#comment-18 Andy Ko and the semblance of objectivity in numbers « Catenary Sun, 31 May 2009 15:01:44 +0000 http://andyjko.com/?p=86#comment-18 [...] Ko and the semblance of objectivity in numbers Jump to Comments Andy Ko blogged yesterday about having a paper rejected at FSE, the Foundations of Software Engineering Conference, because [...] […] Ko and the semblance of objectivity in numbers Jump to Comments Andy Ko blogged yesterday about having a paper rejected at FSE, the Foundations of Software Engineering Conference, because […]

]]>