Code.org has been get­ting some great press, and right­fully so: it’s full of great videos, great stats, and great resources. I also think it has a great mis­sion: there are hun­dreds of thou­sands of busi­nesses who need tal­ented soft­ware devel­op­ers in order to grow and pro­vide value, but these busi­nesses can’t find the engi­neers they need. More­over, peo­ple need jobs and soft­ware devel­op­ment jobs are abun­dant and high qual­ity. Hence the need for more stu­dents, more teach­ers, and more classes in com­put­ing. Win, win, right?

I don’t think so. I do believe in this mis­sion. I do research on this mis­sion. I feel strongly that if we don’t mas­sively increase the num­ber of teach­ers in com­put­ing, we’ll get nowhere. But I don’t think that by sim­ply increas­ing the num­ber of peo­ple who can code, we’ll address this gap. This is because the prob­lem, as code.org frames it, is one of quan­tity, where as the prob­lem is actu­ally about qual­ity.

To put it sim­ply, com­pa­nies don’t need more devel­op­ers, they need bet­ter devel­op­ers. The Googles, Face­books, Apples, and Microsofts of the world get plenty of appli­cants for jobs, they just don’t get appli­cants that are good enough. And the rest of the com­pa­nies in the world, while they can hire, are forced to hire devel­op­ers who often lack the tal­ent to cre­ate great soft­ware, lead­ing to a world of poor qual­ity, bro­ken soft­ware. Sure, just train­ing more devel­op­ers might increase the tiny frac­tion who are great, but that seems like a ter­ri­bly inef­fi­cient way of train­ing more great developers.

This brings us back to teach­ing. We absolutely need more teach­ers, but more impor­tantly we need more excel­lent teach­ers and excel­lent learn­ing oppor­tu­ni­ties. We need the kind of learn­ing infra­struc­ture that empow­ers every 15 year old who’s never seen a line of code to become as good as your typ­i­cal CMU, Stan­ford, Berkely, or UW CS grad, with­out nec­es­sar­ily hav­ing to go to those spe­cific schools. (They don’t have the capac­ity for the kind of growth, nor should they). We need to under­stand what excel­lent soft­ware devel­op­ment is, so we can dis­cover ways to help devel­op­ers achieve it.

This infra­struc­ture is going to be dif­fi­cult to cre­ate. For one, there are going to be a tiny frac­tion of excel­lent devel­op­ers who choose to choose to take a 50% pay cut to teach in a high school or uni­ver­sity, and yet we need those engi­neers to impart their exper­tise some­how. We need to under­stand how to cre­ate excel­lent com­put­ing teach­ers and how to empower them to cre­ate excel­lent devel­op­ers. We need to learn how to make com­put­ing edu­ca­tion effi­cient, so that grad­u­ates in com­put­ing and infor­ma­tion sci­ences have 4 years of actual prac­tice, rather than 4 years of inef­fec­tive lec­tures. We need an aca­d­e­mic cli­mate that respects cur­rent modes of com­put­ing edu­ca­tion as largely bro­ken and inef­fec­tive for all but the best and bright­est self-taught.

Unfor­tu­nately, all of this is going to take sig­nif­i­cant invest­ment. The pub­lic and the most prof­itable of our tech­nol­ogy com­pa­nies must reach deep into their pock­ets to fund this research, this train­ing, and this growth that they and our world so des­per­ately needs. And so kudos to code.org and every other bot­tom up effort to democ­ra­tize com­put­ing, but it’s not enough: we need real resources from the top to cre­ate real change.

No, the new iOS 6 Maps is not as good as Google Maps in sev­eral ways. There’s no end of miss­ing data, mis­placed land­marks, poorly con­structed 3D mod­els, miss­ing tran­sit infor­ma­tion, and because of the sig­nif­i­cant down­grade in infor­ma­tion qual­ity, there’s no end of hate for what many describe as a mas­sive mis­step by Apple. Some even describe it as the begin­ning of the end for the com­pany.

Of course, all of this is a bit overblown. The maps application’s user inter­face itself is much more usable than the pre­vi­ous ver­sion and in many ways, the maps them­selves are more read­able. The tran­sit plug-in fea­ture, while com­pletely use­less at the moment, might actu­ally pro­vide a bet­ter expe­ri­ence in the long term, as local apps might be bet­ter able to account for sub­tle dif­fer­ences in tran­sit infor­ma­tion accu­racy and avail­abil­ity. And while Apple is cer­tainly sev­eral years behind in devel­op­ing com­pre­hen­sive and accu­rate map infor­ma­tion, its com­plete­ness and accu­racy will inevitably improve.

The real story here is how Apple com­mu­ni­cated the change, and how soft­ware com­pa­nies com­mu­ni­cate change more gen­er­ally. If you looked only at Apple’s com­mu­ni­ca­tion, you’d think that the new maps was supe­rior in every way, rather than supe­rior in some ways and tem­porar­ily flawed in oth­ers. But most users prob­a­bly didn’t read any­thing about the change at all. They sim­ply pressed “okay” when their phone asked if they wanted to update and sud­denly their whole map­ping expe­ri­ence was different.

My girl­friend had this exact expe­ri­ence, even though I’d told her it had changed. She didn’t rec­og­nize it as the maps app at all; she thought it was a dif­fer­ent app alto­gether and won­dered where Google Maps had gone. For an exist­ing user, there are dozens of new things to learn to do even basic things and Apple pro­vided vir­tu­ally no guid­ance on what these changes were.

The larger ques­tion here is what soft­ware com­pa­nies should com­mu­ni­cate to avoid dra­matic out­bursts of vit­ri­olic hate every time they make a major change. Are release notes enough? Do appli­ca­tions need a stan­dard model for intro­duc­ing and explain­ing changes to users? To what extent should com­pa­nies be respon­si­ble for com­mu­ni­cat­ing neg­a­tive changes, such as aban­doned fea­tures and poorer accu­racy, and the ratio­nale for them? As soft­ware change becomes more inevitable and more rapid, so will the need for more care­fully explained tran­si­tions to new plat­forms, apps, and functionality.

(On a per­sonal note, I’ve found the new Maps to be quite good around Seat­tle. Yes­ter­day I asked Siri for direc­tions to my daughter’s friend’s house and not only did she find her name in the notes in the con­tact for the friend’s par­ents, but Siri found direc­tions that not only routed me around the SR-520 week­end clo­sure, but explained to me that the bridge was closed. The turn-by-turn direc­tions were fast, clear, and accu­rate and the con­tin­u­ously updat­ing ETA was quite help­ful in decid­ing whether to run the errand we’d planned on doing before we knew about the bridge clo­sure. Over­all, a vast improve­ment over the old maps.)

For the first time in my aca­d­e­mic career this week, I was work­ing on a jour­nal paper and a con­fer­ence paper at the same time. This wasn’t entirely inten­tional; both of these papers were going to be CHI papers, but as the results and writ­ing for one of them mate­ri­al­ized, it became clear that not only was the audi­ence not a fit, but I actu­ally couldn’t fit all of the impor­tant results into 10 page SIGCHI for­mat. This real­iza­tion, and the fact that I was work­ing on both simul­ta­ne­ously, led sev­eral real­iza­tions about how the two kinds of sub­mis­sions differ.

First and fore­most, the lack of a strict length restric­tion on the jour­nal paper was sur­pris­ingly free­ing. While on the CHI paper, every other dis­cus­sion with my stu­dent was what to cut and what to keep, dis­cus­sions about the jour­nal paper were much more about what details and results were miss­ing. Obvi­ously, there are advan­tages to each: with the CHI paper we were prob­a­bly forced to be much more con­cise and selec­tive about the most sig­nif­i­cant results; sim­i­larly, the jour­nal paper was slightly more ver­bose than it needed to be, because I didn’t have the threat of desk rejec­tion to force more care­ful edit­ing. At the same time, there were many inter­est­ing things that we had to leave out of the CHI paper that could have fit into just one addi­tional page. With jour­nal paper, the ques­tion was not “what’s most sig­nif­i­cant?” but “is this complete?”

The length dif­fer­ences also had a sig­nif­i­cant effect on how much space we gave to details nec­es­sary for repro­ducibil­ity. For the jour­nal paper, I felt like our task was to enable other researchers under­stand exactly what we did and how we did it. With the CHI paper, our task was to pro­vide enough detail for review­ers to see the rigor of what we did, but the amount of detail we ended up includ­ing really wasn’t enough to actu­ally repro­duce our study. In the long term, this is not good science.

Although the jour­nal paper didn’t have a dead­line, I did impose one on my lab in order to align with the end of sum­mer, since the under­grad research assis­tants on the paper would have to resume classes (as would I). The dead­line worked well enough to moti­vate us to fin­ish the paper, but it also freed us to take an extra day or two to read through the man­u­script a few extra times, improve some of the fig­ures, and ver­ify some of the results that we felt may have been done too hastily. The CHI paper, in con­trast, was rushed, as most CHI sub­mis­sions are. There was just enough time to edit thor­oughly yes­ter­day and sub­mit today, but there’s an exten­sive list of to do’s that we have if the paper is accepted. Sure, we could do them now, but why not wait until review­ers pro­vide more feed­back? With the jour­nal paper, we sub­mit­ted when we felt it was ready.

Of course, the biggest dif­fer­ence between the two sub­mis­sions has yet to come. In Novem­ber, we’ll get CHI reviews back and likely know with some cer­tainty whether the paper will be accepted or rejected. There will be no major revi­sions, no guid­ance from review­ers about what they’d like to see added or changed, and cer­tainly no oppor­tu­nity for major improve­ments if it is accepted. Instead, the reviews will focus on artic­u­lat­ing a ratio­nale for accep­tance or rejec­tion. With the jour­nal paper, I’ll (hope­fully) get three exten­sive posi­tions in a few months on what is miss­ing or wrong with the paper and what they’d like me to change in a revi­sion. The process will likely take longer, but in trade, I hope the paper will be much bet­ter than orig­i­nal manuscript.

One of these processes is designed for speed, the other is designed for qual­ity. I’ll let you guess which is which. And let me be clear: I’m a big fan of con­fer­ences. Most of my work is pub­lished at major HCI and soft­ware engi­neer­ing venues and not jour­nals and I truly enjoy the fact that nearly every­one in our com­mu­nity ral­lies together at the same time of year to con­tribute our lat­est and great­est for review. But as some­one who has the free­dom to really pub­lish in either, I’m really start­ing to ques­tion whether the aver­age con­fer­ence paper can actu­ally be of com­pa­ra­ble qual­ity to the aver­age jour­nal paper. There might just be inher­ent lim­its to a review process that is opti­mized for select­ing papers for pre­sen­ta­tion rather than improv­ing them.

Of course, this isn’t a nec­es­sary dichotomy. I’ve talked to many peo­ple in my research com­mu­nity about blend­ing the two. For exam­ple, if we sim­ply had jour­nals of infi­nite capac­ity and no con­fer­ence papers, and instead put all of our review­ing effort into our jour­nals, we could eas­ily design an annual con­fer­ence where peo­ple present the best work from recent jour­nal pub­li­ca­tions (and work in progress, as we already do). In fact, CHI already lets ToCHI authors present their recently pub­lished papers, so we’re part way there. With changes like this, we might find a nice bal­ance between a review process designed for improv­ing papers and a con­fer­ence designed for fos­ter­ing dis­cus­sion about them.

I’m not really a hard core gamer any­more, but my fas­ci­na­tion with pro­gram­ming did begin with video games (and specif­i­cally, ren­der­ing algo­rithms). So when I saw John Carmack’s 2012 Quake­Con keynote show up in my feed, I thought I’d lis­ten to a bit of it and learn a bit about the state of game design and development.

What I heard instead was a hacker’s hacker talk about his recent real­iza­tion that soft­ware engi­neer­ing is actu­ally a social sci­ence. Across 10 min­utes, he cov­ers many human aspects of devel­oper mis­takes, pro­gram­ming lan­guage design, sta­tic analy­sis, code reviews, devel­oper train­ing, and cost/benefit analy­ses. The empha­sis through­out is mine (and I also tran­scribed this, so I apol­o­gize for any mistakes).

In try­ing to make the games faster, which has to be our pri­or­ity going for­ward, we’ve made a lot of mis­takes already with Doom 4, a lot of it is water under the bridge, but pri­or­i­tiz­ing that can help us get the games done faster, just has to be where we go. Because we just can’t do this going, you know, six more years, what­ever, between games.

On the soft­ware devel­op­ment side, you know there was an inter­est­ing thing at E3, one of the inter­views I gave, I had men­tioned some­thing about how, you I’ve been learn­ing a whole lot, and I’m a bet­ter pro­gram­mer now than I was a year ago and the inter­viewer expressed a lot of sur­prise at that, you know after 20 years and going through all of this that you’d have it all fig­ured out by now, but I actu­ally have been learn­ing quite a bit about soft­ware devel­op­ment, both on the per­sonal crafts­man level but also pay­ing more atten­tion by what it means on the team dynam­ics side of things. And this is some­thing I prob­a­bly avoided look­ing at squarely for years because, it’s nice to think of myself as a sci­en­tist engi­neer sort, deal­ing in these things that are abstract or prov­able or objec­tive on there and there.

In real­ity in com­puter sci­ence, just about the only thing that’s really sci­ence is when you’re talk­ing about algo­rithms. And opti­miza­tion is an engi­neer­ing. But those don’t actu­ally occupy that much of the total time spent pro­gram­ming. You know, we have a few pro­gram­mers that spend a lot of time on opti­miz­ing and some of the select­ing of algo­rithms on there, but 90% of the pro­gram­mers are doing pro­gram­ming work to make things hap­pen. And when I start to look at what’s really hap­pen­ing in all of these, there really is no sci­ence and engi­neer­ing and objec­tiv­ity to most of these tasks. You know, one of the pro­gram­mers actu­ally says that he does a lot of mon­key programming—you know beat­ing on things and mak­ing stuff hap­pen. And I, you know we like to think that we can be smart engi­neers about this, that there are objec­tive ways to make good soft­ware, but as I’ve been look­ing at this more and more, it’s been strik­ing to me how much that really isn’t the case.

Aside from these that we can mea­sure, that we can mea­sure and repro­duce, which is the essence of sci­ence to be able to mea­sure some­thing, repro­duce it, make an esti­ma­tion and test that, and we get that on opti­miza­tion and algo­rithms there, but every­thing else that we do, really has noth­ing to do with that. It’s about social inter­ac­tions between the pro­gram­mers or even between your­self spread over time. And it’s nice to think where, you know we talk about func­tional pro­gram­ming and lambda cal­cu­lus and mon­ads and this sounds all nice and sci­ency, but it really doesn’t affect what you do in soft­ware engi­neer­ing there, these are all best prac­tices, and these are things that have shown to be help­ful in the past, but really are only help­ful when peo­ple are mak­ing cer­tain classes of mis­takes. Any­thing that I can do in a pure func­tional lan­guage, you know you take your most restric­tive sci­en­tific ori­ented code base on there, in the end of course it all comes down to assem­bly lan­guage, but you could exactly the same thing in BASIC or any other lan­guage that you wanted to.

One of the things that’s also fed into that is my older son’s start­ing to learn how to pro­gram now. I actu­ally tossed around the thought of should I maybe have him try to learn Haskell as a 7 year old or some­thing and I decided not to, that I, you know, I don’t think that I’m a good enough Haskell pro­gram­mer to want to instruct any­body in any­thing, but as I start think­ing about how some­body learns pro­gram­ming from really ground zero, it was open­ing my eyes a lit­tle bit to how much we take for granted in the soft­ware engi­neer­ing com­mu­nity, really is just lay­ers of arti­fice upon top a core fun­da­men­tal thing. Even when you go back to struc­tured pro­gram­ming, whether it’s while loops and for loops and stuff, at the bot­tom when I’m sit­ting think­ing how do you explain pro­gram­ming, what does a com­puter do, it’s really all the way back to flow charts. You do this, if this you do that, if not you do that. And, even try­ing to explain why do you do a for loop or what’s this while loop on here, these are all con­ven­tions that help soft­ware engi­neer­ing in the large when you’re deal­ing with mis­takes that peo­ple make. But they’re not fun­da­men­tal about what the computer’s doing. All of these are things that are just try­ing to help peo­ple not make mis­takes that they’re com­monly making.

One of the things that’s been dri­ven home extremely hard is that pro­gram­mers are mak­ing mis­takes all the time and con­stantly. I talked a lot last year about the work that we’ve done with sta­tic analy­sis and try­ing to run all of our code through sta­tic analy­sis and get it to run squeaky clean through all of these things and it turns up hun­dreds and hun­dreds, even thou­sands of issues. Now its great when you wind up with some­thing that says, now clearly this is a bug, you made a mis­take here, this is a bug, and you can point that out to every­one. And every­one will agree, okay, I won’t do that next time. But the prob­lem is that the best of inten­tions really don’t mat­ter. If some­thing can syn­tac­ti­cally be entered incor­rectly, it even­tu­ally will be. And that’s one of the rea­sons why I’ve got­ten very big on the sta­tic analy­sis, I would like to be able to enable even more restric­tive sub­sets of lan­guages and restrict pro­gram­mers even more because we make mis­takes con­stantly.

One of the things that I started doing rel­a­tively recently is actu­ally doing a daily code review where I look through the check­ins and just try to find some­thing edu­ca­tional to talk about to the team. And I anno­tate a lit­tle bit of code and say, well actu­ally this is a bug dis­cov­ered from code review, but a lot of it is just, favor doing it this way because it’s going to be clearer, it will cause less prob­lems in other cases, and it ruf­fled, there were a few peo­ple that got ruf­fled feath­ers early on about that with the kind of broad­cast nature of it, but I think that every­body is appre­ci­at­ing the process on that now. That’s one of those scal­a­bil­ity issues where there’s clearly no way I can do indi­vid­ual code reviews with every­one all the time, it takes a lot of time to even just scan through what every­one is doing. Being able to point out some­thing that some­body else did and say well, every­body should pay atten­tion to this, that has some real value in it. And as long as the team is agree­able to that, I think that’s been a very pos­i­tive thing.

But what hap­pens in some cases, where you’re argu­ing a point where let’s say we should put const on your func­tion para­me­ters or some­thing, that’s hard to make an objec­tive call on, where lots of stuff we can say, this indi­rec­tion is a cache miss, that’s going to cost us, it’s objec­tive, you can mea­sure it, there’s really no argu­ing with it, but so many of these other things are sort of style issues, where I can say, you know, over the years, I’ve seen this cause a lot prob­lems, but a lot of peo­ple will just say, I’ve never seen that prob­lem. That’s not a prob­lem for me, or I don’t make those mis­takes. So it has been really good to be able to point out com­monly on here, this is the mis­take caused by this.

But as I’ve been doing this more and more and think­ing about it, that sense that this isn’t sci­ence, this is just try­ing to deal with all of our human frail­ties on it, and I wish there were bet­ter ways to do this. You know we all want to become bet­ter devel­op­ers and it will help us make bet­ter prod­ucts, do a bet­ter job with what­ever we’re doing, but the fact that it’s com­ing down to train­ing dozens of peo­ple to do things in a con­sis­tent way, know­ing that we have pro­gram­mer turnover as peo­ple come and go, new peo­ple com­ing and look­ing at the code base and not under­stand­ing the con­ven­tions, and there are clearly bet­ter and worse ways of doing things but it’s frus­trat­ingly dif­fi­cult to quan­tify.

That’s some­thing that I’m spend­ing more and more time look­ing at. I read NASA’s soft­ware engi­neer­ing lab­o­ra­tory reports and I can’t seem to get any real value out of a lot of those things. The things that have been valu­able have been auto­mated things, things that don’t require a human to have some analy­sis, have some eval­u­a­tion of it, but just say, enforced or not enforced. And I think that that’s where really where things need to go as larger and larger soft­ware gets devel­oped. And it is strik­ing the scale of what we’re doing now. If you look back at the NASA reports and the scale of things and they con­sid­ered large code bases to be things with three or four hun­dred thou­sand lines of code. And we have far more than that in our game engines now. It’s kind of fun to think that the game engines, things that we’re play­ing games on, have more sophis­ti­cated soft­ware than cer­tainly the things that launch peo­ple to the moon and back and flew the shut­tle, ran Sky­lab, run the space sta­tion, all of these mas­sive projects on there are really out­done in com­plex­ity by any num­ber of major game engine projects.

And the answer is as far as I can tell really isn’t out there. With the NASA style devel­op­ment process, they can deliver very very low bug rates, but it’s at a very very low pro­duc­tiv­ity rate. And one of the things that you wind up doing in so many cases is cost ben­e­fit analy­ses, where you have to say, well we could be per­fect, but then we’ll have the wrong prod­uct and it will be too late. Or we can be really fast and loose, we can go ahead and just be sloppy but we’ll get some­thing really cool hap­pen­ing soon. And this is one of those areas where there’s clearly right tools for the right job, but what hap­pens is you make some­thing really cool really fast and then you live with it for years and you suf­fer over and over with that. And that’s some­thing that I still don’t think that we do the best job at.

We know our code is liv­ing for, real­is­ti­cally, we’re look­ing at a decade. I tell peo­ple that there’s a good chance that what­ever you’re writ­ing here, if it’s not extremely game spe­cific, may well exist a decade from now and it will have hun­dreds of pro­gram­mers, look­ing at the code, using it, inter­act­ing with it in some way, and that’s quite a bur­den. I do think that it’s just and right to impose pretty severe restric­tions on what we’ll let past analy­sis and what we’ll let into it, but there are large scale issues at the soft­ware API design lev­els and fig­ur­ing out things there, that are artis­tic, that are crafts­man like on there. And I wish that there were more quan­tifi­able things to say about that. And I am spend­ing a lot of time on this as we go forward.

Yes­ter­day while I was walk­ing to cam­pus I was lis­ten­ing to a Fresh Air pod­cast on how Con­gress­man Paul Ryan is Shap­ing the GOP. One of Ryan’s favorite ideas appears to be that of Ayn Rand, that to be truly free, we must avoid depend­ing on oth­ers and find a way to sup­port our­selves. He’s apply­ing these same beliefs to pol­icy on pre­cisely what size the gov­ern­ment should be.

This angered me. I came from a mid­dle income house­hold; my mom was a 5th grade teacher and my dad worked in qual­ity assur­ance for food and lenses, and nei­ther were paid par­tic­u­larly well for their trade. The only way I was going to make it to col­lege was to work my ass off in high school, work a part time job to pay for AP exams, get a lot of schol­ar­ships and bor­row a lot of money. So that’s what I did. And when I made it to col­lege, I worked part time, I accrued mas­sive debt, and I made it into a great Ph.D. pro­gram. I was lucky enough to have cho­sen a field where Ph.D. stu­dents get paid out of pub­lic research funds, but I wasn’t paid much (cer­tainly not enough to sup­port myself, my wife and my new­born daugh­ter). So I bor­rowed more, I earned two pub­lic fel­low­ships from the National Sci­ence Foun­da­tion and the Depart­ment of Defense, and we squeaked by for six years finan­cially. After 23 years of pub­lic edu­ca­tion, I’d used $91,000 in Ore­gon taxpayer’s money to fund my K-12 edu­ca­tion, $36,000 of Ore­gon taxpayer’s money to sub­si­dize my Ore­gon State tuition, $7,000 in Pell grants and free inter­est from U.S. tax­pay­ers, $76,000 from an NSF Fel­low­ship, and another $187,500 from an NDSEG Fel­low­ship. And even with all of this help from pub­lic fund­ing, I still had to work part time dur­ing high school and col­lege and was still left with $50,000 of stu­dent loans to repay.

When you start to look at the cost of edu­cat­ing U.S. citizens—whether some­one like me who goes for a ter­mi­nal degree, or some­one who sim­ply wants a col­lege degree—it becomes imme­di­ately clear that a per­son can work incred­i­bly hard to become a valu­able con­trib­u­tor to soci­ety, fully real­iz­ing Ryan and Rand’s vision, and still depend a great deal of sup­port from tax­pay­ers. This idea that peo­ple are either self-supporting or depen­dent leeches is an entirely false dichotomy.

The real ques­tion we should be ask­ing is whether shar­ing the cost of edu­cat­ing our youth is some­thing worth­while and some­thing to be shared. I know that in my own case that with­out pub­lic fund­ing, I sim­ply could not have gone to col­lege. I’m sure I would have been suc­cess­ful in some other way; I would have taught myself, per­haps going to a com­mu­nity col­lege. Or per­haps my par­ents would accrued their own mas­sive debt to send me any­way. Either way, the 100 mil­lion tax­pay­ers who each gave me less than half a penny of their money prob­a­bly don’t miss it. And I hope that the work I do to edu­cate our chil­dren, advance sci­ence, and invent new tech­nolo­gies that make our lives eas­ier is worth that small invest­ment. After all, after a time, the world we live in is not the one we make, but the one our chil­dren and grand­chil­dren make for us.

For the past three days I’ve been at the 2012 UW MSR Sum­mer Insti­tute, which is an annual retreat on an emerg­ing research topic. This year’s topic was “Crowd­sourc­ing Per­son­al­ized Online Edu­ca­tion”. What this really meant in prac­tice was two things: what is the future of edu­ca­tion and how can we lever­age the con­nect­ed­ness of observ­abil­ity of learn­ing online? The work­shop was mainly talks, but there were an impres­sive num­ber of great speak­ers and atten­dees that kept every­one engaged.

There are a lot of impor­tant things that I observed out of all of these dis­cus­sions and talks:

• The first thing that was appar­ent is just how dif­fer­ent the motives and val­ues are in the dif­fer­ent com­mu­ni­ties that attended. The major­ity of the atten­dees were com­ing from a com­put­ing per­spec­tive, with pri­mary inter­ests in cre­at­ing new, more pow­er­ful, and more effec­tive learn­ing tech­nolo­gies. There were a smaller num­ber of learn­ing sci­en­tists, with inter­ests in explain­ing learn­ing and devis­ing bet­ter mea­sure­ments of learn­ing, much more rig­or­ously than any of the com­put­ing folks had done. Two rep­re­sen­ta­tions from the Gates Foun­da­tion also came briefly, and it was clear that their pri­mary inter­ests were much less in spe­cific tech­nolo­gies and much more in cre­at­ing edu­ca­tional infra­struc­ture and new, sus­tain­able mar­kets of edu­ca­tional tech­nolo­gies. There were also rep­re­sen­ta­tives from Khan Acad­emy and Cours­era, who were broadly inter­ested in pro­vid­ing access to con­tent, and mech­a­nisms to enable experts to share con­tent. My view on what’s really new behind all of this press on online learn­ing is that com­put­ing researchers are newly inter­ested in learn­ing and edu­ca­tion: almost every­thing else, except for the scale of access, has been done in online learn­ing before.
• Jen­nifer Widom, Andrew Ng, and Scott Klem­mer (all at Stan­ford), talked about their expe­ri­ences cre­at­ing MOOCs for their courses. The key take away mes­sage is that it is very time con­sum­ing to cre­ate the course, with each spend­ing count­less hours record­ing high qual­ity lec­tures before the hours, nego­ti­at­ing rights for copy­righted mate­r­ial, and work­ing out bugs in the plat­form. All of them implied that run­ning the course the first time was more than a full time job. On the other hand, many were con­fi­dent it would take much less time for later offer­ings and had con­fi­dence that most aspects of the class can scale to be arbi­trar­ily large (even design cri­tiques, in Scott Klemmer’s case, through cal­i­brated peer assess­ment). The one part that doesn’t scale is student-specific con­cerns (for exam­ple, stu­dents get­ting injured and need­ing an exten­sion on an assign­ment). Scott also sug­gested that every order of mag­ni­tude increase in the num­ber of stu­dents demands an order of mag­ni­tude increase in the per­fec­tion of the mate­ri­als (because there are so many more eyes on the mate­r­ial), but again, this is a decay­ing cost, assum­ing the mate­ri­als don’t change frequently.
• In many of the con­ver­sa­tions I had around how MOOCs might change edu­ca­tion, many fac­ulty believed that the sheer avail­abil­ity and acces­si­bil­ity of instruc­tional con­tent would shift the respon­si­bil­i­ties of instruc­tors. Today, most indi­vid­ual instruc­tors are respon­si­ble for mak­ing their own mate­ri­als, mak­ing them acces­si­ble, and then using them to teach. In a world where great mate­ri­als are avail­able for free, these first two respon­si­bil­i­ties dis­ap­pear. The new job of a higher ed instruc­tor may there­fore much less about design­ing mate­ri­als and pro­vid­ing access to them, but cor­rect­ing mis­con­cep­tions, moti­vat­ing stu­dents, design­ing good mea­sure­ments, and build­ing learn­ing com­mu­ni­ties. One could argue that this is an over­all improve­ment (and also that it actu­ally mir­rors the way that text­books work, which are writ­ten by a small num­ber of experts and used as is by instructors).
• Inter­est­ingly, most of the MOOC teach­ers reported that the social expe­ri­ence of stu­dents online were crit­i­cal, includ­ing forum con­ver­sa­tions, ad hoc study groups in dif­fer­ent cities around the world, and peer assess­ments. This might quell a lot of the con­cerns that higher ed teach­ers had about the loss of inter­ac­tion in online—it might just be that the inter­ac­tion shifts from instructor/student inter­ac­tion to student/student inter­ac­tion and student/intelligent tutor inter­ac­tion. Some of the pre­lim­i­nary data sug­gests that stu­dents actu­ally greatly pre­fer this, since they don’t get that much instruc­tor inter­ac­tion already, but they’re get­ting much more student/student inter­ac­tion than in a tra­di­tional co-located course. This might there­fore be an improve­ment over tra­di­tional lecture-based classes, but not classes in which teach­ers inter­act closely with stu­dents (such as small stu­dio courses).
• No one knows what will hap­pen to the edu­ca­tion mar­ket, includ­ing the peo­ple run­ning Kahn and Cours­era. How­ever, there were some pre­dic­tions. First, these plat­forms are going to make it so easy to share and access con­tent, in the same way that the web has for every­thing else, that find­ing and choos­ing con­tent is going to become a crit­i­cal chal­lenge for stu­dents. There­fore, one new role that instruc­tors might play is in select­ing and curat­ing con­tent in a way that is coher­ent and per­son­al­ized for the pop­u­la­tions that they teach.
• Most of the inter­ests related to crowd­sourc­ing are either in (1) enabling classes to be taught at scale (by find­ing ways to free instruc­tors and TAs to have to grade and assess all of the work), (2) to improve the effec­tive­ness, effi­ciency, and or engage­ment of learn­ing activ­i­ties, or (3) to cre­ate new oppor­tu­ni­ties through infor­mal learn­ing, such as through oDesk or Duolingo. Researchers are think­ing about how to use data to opti­mize the sequence of instruc­tion, give just the right hints to cor­rect mis­con­cep­tions, select a task that is chal­leng­ing but not too chal­leng­ing. In my view, this is lead­ing to a renewed inter­est intel­li­gent tutor­ing systems.
• As usual, most of this new research work suf­fers from a lack of ground­ing in and lever­ag­ing of prior lit­er­a­ture in learn­ing sci­ences and intel­li­gent tutor­ing sys­tems. There is tons of research on all of these chal­lenges that com­put­ing researchers are tack­ling, but I don’t seem them really using any of the work. This hap­pens over and over in com­put­ing research, since the inter­ests are often in cre­at­ing new things and not under­stand­ing the things them­selves. I was impressed, how­ever, how much Andrew Ng had lever­aged find­ings in learn­ing sci­ences to sup­port cer­tain design deci­sions in Coursera.
• There was a big under­cur­rent of data sci­ence at the work­shop. Every­one was excited about big data sets and how they might be lever­aged to improve learn­ing tech­nolo­gies. Most of the meth­ods reported were fairly prim­i­tive (AB test­ing, reten­tion rates), but I’m hop­ing this new energy behind learn­ing will lead to much bet­ter meth­ods and tools for doing edu­ca­tional data mining.

Phew! Sorry for the lack of coher­ence here. We cov­ered a lot of ground in 2.5 days and this is just a sliver of it.

For a while now, there have been two com­pet­ing nar­ra­tives around jobs and com­put­ing. One is that com­put­ing will bring an amaz­ing influx of new jobs by cre­at­ing new oppor­tu­ni­ties, new mar­kets, and new ideas. The other is that com­put­ing, far from being a job source, is actu­ally a job sink, replac­ing man­u­fac­tur­ing and infor­ma­tion ser­vices jobs with machines. Thomas Edsall dis­cusses these two nar­ra­tives in a recent opin­ion piece on the NY Times, bring­ing together sev­eral essays and blog posts on the subject.

The most com­pelling idea from this post (and most of it was com­pelling), was the “lump of labor” fal­lacy, which I hadn’t heard of before. This is the idea that there is a fixed amount of work avail­able in the world and it just gets shifted around between cities, com­pa­nies, and coun­tries. Econ­o­mists appar­ently show lit­tle sup­port for this idea, as his­tory has repeat­edly shown that inno­va­tions typ­i­cally cre­ate more work, rather than less.

Andrew McAfee, argues that infor­ma­tion tech­nol­ogy is dif­fer­ent. All past inno­va­tions, he argues, auto­mated things that humans pri­mar­ily could not do (for exam­ple, reach­ing places we could not reach or lift­ing things we could not lift, trans­port­ing us places we could not reach). In con­trast, infor­ma­tion tech­nol­ogy is begin­ning to be capa­ble of doing many infor­ma­tion related things that humans can do, in addi­tion to all of the infor­ma­tion related things we can’t do. There­fore, the only ratio­nal thing for employ­ers to do as soft­ware becomes func­tional and cheap enough is to replace peo­ple with machines.

Is he right? It’s cer­tainly com­pat­i­ble with a rejec­tion of the lump of labor fal­lacy. Com­put­ing can cre­ate more work just like any other inno­va­tion, but McAfee might argue that the new work can also be done com­put­ers. For exam­ple, the fact that I buy a new iPhone every two years means that there does need to be peo­ple to man­u­fac­ture it and fix it when it breaks. But the very tech­nolo­gies embed­ded in the device are the same ones that enable its man­u­fac­tur­ing to be almost com­pletely auto­mated and allow me to get a sub­stan­tial amount of sup­port from Q&A forums archived on the web, rather than using human tech­ni­cal sup­port. On the other hand, that automa­tion and infor­ma­tion access requires a lot of energy, a lot more man­u­fac­tur­ing, and a great deal of human time to main­tain the Inter­net. It seems pos­si­ble that much of this could be taken over by machines eventually.

Can all of the work really be shifted to non-humans? Let’s do a thought exper­i­ment to see. Con­sider a small remote vil­lage of 100 humans run entirely by robots and pow­ered by an effec­tively infi­nite sup­ply of solar energy. One of the human at any given time is an expert roboti­cist who can main­tain and repair all of the robots inde­pen­dently. This roboti­cist trains one of the vil­lage chil­dren to replace her, so that when that roboti­cist dies, there is another to take over. The roboti­cist has immense power because the other 99 peo­ple depend on her to keep the robotic work force func­tional (includ­ing the robotic work force that keeps the rest of the robotic work­force func­tional). The result is that the 99 peo­ple don’t work (because there’s no work to do), and live a life of leisure. The only rea­son that every­one sur­vives is because of the roboticist’s knowl­edge and benev­o­lence and that nearly all of the work has been shifted to the robots. In fact, the robots may even become intel­li­gent enough to fix and main­tain them­selves one day, mak­ing even the roboti­cist obsolete.

There are most cer­tainly things miss­ing from this lit­tle story that make it implau­si­ble. For exam­ple, the pop­u­la­tion wouldn’t stay at 100 peo­ple, espe­cially with every­one liv­ing such a life of leisure. Assum­ing the robots could repro­duce them­selves and gather their own nat­ural resources, the pop­u­la­tion would con­tinue to grow until Earth was out of resources, as it does now.

The more sig­nif­i­cant miss­ing ele­ment, I think, is bore­dom. In such a life of leisure, wouldn’t peo­ple cre­ate work for them­selves, just to be enter­tained or to find mean­ing? I could imag­ine for exam­ple, one par­tic­u­larly inquis­i­tive vil­lager decid­ing to write a book on the mean­ing of life in a world where there is no human work. She might out­source the edit­ing, print­ing, and bind­ing of her book to the robots, but would she out­source the audi­ence? The crit­i­cal reflec­tions? The impas­sioned rebut­tals? Surely the vil­lages would cre­ate work for them­selves, if only to cre­ate mean­ing­ful social bonds and avoid listlessness.

Per­haps the rea­son that “lump of labor” is a fal­lacy, even for com­put­ing, is that work isn’t a sep­a­rate entity from human­ity that can be shifted to and from human­ity. Human­ity is the source of work. Com­put­ing many elim­i­nate forms of work that we are used to in present day soci­ety, but we will inevitably find ways to occupy our­selves oth­er­wise. Per­haps its just the dis­rup­tive tran­si­tions that are painful, where the mid­dle class strug­gles, starves, and loses, only to be moti­vated by their hunger to cre­ate new work with which to fill their bellies.

I have a young face, espe­cially for a pro­fes­sor. Other fac­ulty assume I’m an under­grad, Ph.D. stu­dents assume I’m an under­grad, even under­grads assume I’m an under­grad. In some ways this is nice. I can be stealth on cam­pus, blend­ing in with the rest of the stu­dents. When I’m teach­ing, I have to earn my author­ity rather than get­ting it sim­ply because I look wise (and I like earn­ing things). And the under­grads I teach prob­a­bly relate to me dif­fer­ently sim­ply because I look their age, even though I’m a decade older than most of them.

As a researcher, how­ever, look­ing young can feel like a dis­ad­van­tage, since the wis­dom and knowl­edge one has typ­i­cally grows with age (at least in acad­e­mia). Some­times I feel like peo­ple dis­count my opin­ions because I look young, per­haps because my face com­mu­ni­cates inex­pe­ri­ence. Some­times I feel like I have to com­pen­sate by being extra­or­di­nar­ily artic­u­late or insight­ful, just to get peo­ple to lis­ten to me. At con­fer­ences, peo­ple always ask me what I’m study­ing, who my advi­sor is, or where I go to school. I sus­pect that when peo­ple who don’t know me see me at a con­fer­ence, they think, “just another stu­dent” instead of “I won­der who that impor­tant researcher is” like I do when I see older researchers at conferences.

Not that this has held me back. If any­thing, it means that any suc­cess I’ve had has been earned, which makes it all the more reward­ing. And it shows that acad­e­mia is still indeed some form of mer­i­toc­racy, where it is the ideas and knowl­edge that one pro­duces that ulti­mately shapes our rep­u­ta­tions. In fact, when I’m 60, I’ll prob­a­bly look like I’m in my 40’s (as my par­ents do), which will help me avoid all of the ageism directed at older pro­fes­sors, so any dis­ad­van­tage I have now might turn into an advan­tage later in life. That should enable me to have a nice long career into my 70’s (assum­ing my brain still works!).

Ulti­mately, I feel lucky that ageism is the only real dis­crim­i­na­tion that I face. There are fac­ulty who face ageism, sex­ism, and racism, which seems like an incred­i­ble amount of bias for one per­son to strug­gle against. Fac­ing a bit of ageism here and there makes me empathize with peo­ple who face even more dis­crim­i­na­tion and makes it eas­ier for me to avoid assum­ing any­thing about a per­son until I talk with them. And it helps me respect their suc­cesses even more, because I have a tiny glimpse into what it took to earn it.

As a pro­fes­sor in higher ed, I’ve been think­ing a lot lately about mas­sive open online courses (MOOCs), and in par­tic­u­lar, how they dif­fer from mod­ern class­rooms in higher ed. For a while now, I’ve been quite cer­tain that there are cru­cial ben­e­fits to teach­ing in phys­i­cal envi­ron­ments, but we just haven’t been able to artic­u­late what they are. After a lot of thought and obser­va­tion, I’ve decided that while there are sig­nif­i­cant dif­fer­ences in the teach­ing affor­dances, in prac­tice, most higher ed class­rooms are nearly iden­ti­cal to MOOCs, except in scale, where MOOCs win hands down.

Why? It all comes down to feed­back. (This shouldn’t be sur­pris­ing come from me, since nearly every­thing I think about as a researcher has to do with feed­back). I believe strongly, as does the rest of learn­ing sci­ences and psy­chol­ogy, that feed­back is a fun­da­men­tal part of learn­ing and under­stand­ing. With­out it, peo­ple are left to make sense of the world in a vac­uum. And the oppor­tu­ni­ties for feed­back in most higher ed class­rooms are sim­ply not that dif­fer­ent from those posed by MOOCs.

Con­sider your stan­dard 150 stu­dent lec­ture course. Let’s take CS1 as an exam­ple and imag­ine the best pos­si­ble deliv­ery of this kind of instruc­tion. Such a course would have a sage on the stage, lec­tur­ing about for loops and vari­ables; you might even have inno­v­a­tive class­room prac­tices such as peer learn­ing or quizzes every 10 or fif­teen min­utes. At best, stu­dents get imme­di­ate feed­back on quiz results and per­haps even auto­mated feed­back on home­work assign­ments. A few stu­dents may even be bold enough to ask ques­tions in class or make it to office hours for addi­tional help. If the course uses peer learn­ing, they’ll get some cor­rec­tive feed­back from their peers, which can help greatly. In this set­ting, the real value is access to these lim­ited instruc­tional resources offered by the instruc­tor and the TAs. Ulti­mately, how­ever, the vast major­ity of these 150 stu­dents receive lit­tle, if any, per­son­al­ized instruc­tional feed­back from experts.

Now con­sider a CS1 MOOC, like those offered by Cours­era and Udac­ity. The cen­tral struc­ture of these courses is nearly iden­ti­cal: a sage on the stage imparts wis­dom in brief chunks; stu­dents get feed­back through auto­mat­i­cally graded quizzes and home­work assign­ments. Self-motived learn­ers use this feed­back, and the guid­ance of their fel­low stu­dents via dis­cus­sion boards (ala peer learn­ing), to under­stand and cor­rect their own mis­con­cep­tions. Like the stu­dents in the phys­i­cal class­room, they are largely on their own, unless they are resource­ful enough to solicit feed­back from peers, instruc­tors, or other online resources.

The big dif­fer­ence with a MOOC are that many more stu­dents can be taught and that auto­matic, imme­di­ate feed­back (even if it’s sub­op­ti­mal to an indi­vid­ual human tutor) is guar­an­teed (whereas it is not in most phys­i­cal CS1 courses). It’s no won­der higher ed instruc­tors are scared. For a course of the same kind, the phys­i­cal ver­sion has almost no advan­tages over the online one!

But this doesn’t have to be. There are a huge num­ber of affor­dances that phys­i­cal class­rooms offer that no online course can match. First and fore­most, the walls and sur­faces in class­rooms, and the through­put of face to face com­mu­ni­ca­tion, offer vastly supe­rior means with which to view and cri­tique stu­dent work. The image above, for exam­ple, shows a design meth­ods class that I teach to Ph.D. stu­dents. We con­stantly use the walls to post work and facil­i­tate peer cri­tique and because I can see all of the dis­cus­sion that are hap­pen­ing, and over­hear how well each is going, I know who needs help and when and can quickly con­nect stu­dents who are strug­gling with sim­i­lar ideas. None of this is pos­si­ble online because the chan­nels of com­mu­ni­ca­tion are too few and to serialized.

Another instruc­tional affor­dance that is still absent online is laugh­ter. Don’t laugh! I’m seri­ous. In my brief expe­ri­ence as a teacher, I’ve come to view laugh­ter as a cru­cial part of moti­vat­ing stu­dents who aren’t already self-motivated. Find­ing ways for stu­dents to inter­act in phys­i­cal space, and giv­ing them oppor­tu­ni­ties to not only dis­cuss and work through ideas, but com­ment on the very dis­cus­sion of them in humor­ous ways, cre­ates moti­va­tional bonds in stu­dents that I’ve yet to see hap­pen in MOOCs. There are sim­ply too many peo­ple and too few iden­tity cues to cre­ate these types of moti­vat­ing relationships.

There is one major caveat to this line of rea­son­ing: just because phys­i­cal class­rooms have these affor­dances doesn’t mean they are use­ful or valu­able to teach­ing all sub­jects. A stu­dent tak­ing a med­ical ter­mi­nol­ogy course, where their sole job is to mem­o­rize and under­stand thou­sands of med­ical terms, is prob­a­bly much bet­ter off with an online drill and prac­tice mode of instruc­tion and skip the phys­i­cal class­room. In the same way, it’s likely that online courses will always be infea­si­ble for a Chem­istry lab: there’s sim­ply too much phys­i­cal infra­struc­ture and safety con­cerns to imag­ine stu­dents either cre­at­ing their own lab or find­ing a local one, in which to per­form experiments.

The real chal­lenge then for us in higher ed is to fig­ure out what forms of instruc­tion really do require or strongly ben­e­fit from phys­i­cal space and focus on those. Tear out the rows of desks and replace them with flex­i­ble walls and sur­faces. Invest in course release so that fac­ulty can learn how to uti­lize these phys­i­cal affor­dances. Give fac­ulty whose exper­tise is in some­thing best deliv­ered online the time and incen­tive to cre­ate out­stand­ing ver­sions of these courses that ben­e­fit both stu­dents and their home uni­ver­si­ties, but also stu­dents any­where in the world. This future isn’t one where uni­ver­si­ties dis­ap­pear, but one in which we fac­ulty finally start tak­ing teach­ing seri­ously. It’s about time we had a lit­tle com­pe­ti­tion to give us the clear feed­back that the sta­tus quo isn’t good enough (except online).

Mar­ket­place has a brief, but intrigu­ing story about how com­put­ing is trans­form­ing man­u­fac­tur­ing in the United States. As they explain, machin­ists used to work with their hands, phys­i­cally manip­u­lat­ing mechan­i­cal machines to shape and shred metal and other mate­ri­als into the basic com­po­nents of all kinds of engi­neered mate­ri­als, from small plas­tic trin­kets to air­plane parts.

Today, how­ever, machin­ing is less about oper­at­ing machines, and more about writ­ing code that oper­ates machines (CNC machines, in par­tic­u­lar, stand­ing for com­puter numer­i­cally con­trolled). To learn the CNC pro­gram­ming lan­guage, work­ers typ­i­cally take an 18-week course before their ready to oper­ate CNC machines, but then they can make a rea­son­able man­u­fac­tur­ing wage with­out get­ting their hands dirty or risk­ing injury. This is a clas­sic exam­ple of end-user pro­gram­ming, where some­one has to write code as a means to an end (a phys­i­cal object).

What’s even more fas­ci­nat­ing is the eco­nomic dis­cus­sion sur­round­ing this jobs. Appar­ently, the prob­lem isn’t train­ing the machin­ists, but find­ing peo­ple who want to be trained. The Man­u­fac­tur­ing Insti­tute found in a sur­vey that there are as many 600,000 man­u­fac­tur­ing jobs going unfilled, the major­ity of which are jobs that require these kinds of tech­ni­cal com­put­ing skills. This is there­fore as much a train­ing prob­lem as it is a recruit­ing problem.