For the past three days I’ve been at the 2012 UW MSR Sum­mer Insti­tute, which is an annual retreat on an emerg­ing research topic. This year’s topic was “Crowd­sourc­ing Per­son­al­ized Online Edu­ca­tion”. What this really meant in prac­tice was two things: what is the future of edu­ca­tion and how can we lever­age the con­nect­ed­ness of observ­abil­ity of learn­ing online? The work­shop was mainly talks, but there were an impres­sive num­ber of great speak­ers and atten­dees that kept every­one engaged.

There are a lot of impor­tant things that I observed out of all of these dis­cus­sions and talks:

• The first thing that was appar­ent is just how dif­fer­ent the motives and val­ues are in the dif­fer­ent com­mu­ni­ties that attended. The major­ity of the atten­dees were com­ing from a com­put­ing per­spec­tive, with pri­mary inter­ests in cre­at­ing new, more pow­er­ful, and more effec­tive learn­ing tech­nolo­gies. There were a smaller num­ber of learn­ing sci­en­tists, with inter­ests in explain­ing learn­ing and devis­ing bet­ter mea­sure­ments of learn­ing, much more rig­or­ously than any of the com­put­ing folks had done. Two rep­re­sen­ta­tions from the Gates Foun­da­tion also came briefly, and it was clear that their pri­mary inter­ests were much less in spe­cific tech­nolo­gies and much more in cre­at­ing edu­ca­tional infra­struc­ture and new, sus­tain­able mar­kets of edu­ca­tional tech­nolo­gies. There were also rep­re­sen­ta­tives from Khan Acad­emy and Cours­era, who were broadly inter­ested in pro­vid­ing access to con­tent, and mech­a­nisms to enable experts to share con­tent. My view on what’s really new behind all of this press on online learn­ing is that com­put­ing researchers are newly inter­ested in learn­ing and edu­ca­tion: almost every­thing else, except for the scale of access, has been done in online learn­ing before.
• Jen­nifer Widom, Andrew Ng, and Scott Klem­mer (all at Stan­ford), talked about their expe­ri­ences cre­at­ing MOOCs for their courses. The key take away mes­sage is that it is very time con­sum­ing to cre­ate the course, with each spend­ing count­less hours record­ing high qual­ity lec­tures before the hours, nego­ti­at­ing rights for copy­righted mate­r­ial, and work­ing out bugs in the plat­form. All of them implied that run­ning the course the first time was more than a full time job. On the other hand, many were con­fi­dent it would take much less time for later offer­ings and had con­fi­dence that most aspects of the class can scale to be arbi­trar­ily large (even design cri­tiques, in Scott Klemmer’s case, through cal­i­brated peer assess­ment). The one part that doesn’t scale is student-specific con­cerns (for exam­ple, stu­dents get­ting injured and need­ing an exten­sion on an assign­ment). Scott also sug­gested that every order of mag­ni­tude increase in the num­ber of stu­dents demands an order of mag­ni­tude increase in the per­fec­tion of the mate­ri­als (because there are so many more eyes on the mate­r­ial), but again, this is a decay­ing cost, assum­ing the mate­ri­als don’t change frequently.
• In many of the con­ver­sa­tions I had around how MOOCs might change edu­ca­tion, many fac­ulty believed that the sheer avail­abil­ity and acces­si­bil­ity of instruc­tional con­tent would shift the respon­si­bil­i­ties of instruc­tors. Today, most indi­vid­ual instruc­tors are respon­si­ble for mak­ing their own mate­ri­als, mak­ing them acces­si­ble, and then using them to teach. In a world where great mate­ri­als are avail­able for free, these first two respon­si­bil­i­ties dis­ap­pear. The new job of a higher ed instruc­tor may there­fore much less about design­ing mate­ri­als and pro­vid­ing access to them, but cor­rect­ing mis­con­cep­tions, moti­vat­ing stu­dents, design­ing good mea­sure­ments, and build­ing learn­ing com­mu­ni­ties. One could argue that this is an over­all improve­ment (and also that it actu­ally mir­rors the way that text­books work, which are writ­ten by a small num­ber of experts and used as is by instructors).
• Inter­est­ingly, most of the MOOC teach­ers reported that the social expe­ri­ence of stu­dents online were crit­i­cal, includ­ing forum con­ver­sa­tions, ad hoc study groups in dif­fer­ent cities around the world, and peer assess­ments. This might quell a lot of the con­cerns that higher ed teach­ers had about the loss of inter­ac­tion in online—it might just be that the inter­ac­tion shifts from instructor/student inter­ac­tion to student/student inter­ac­tion and student/intelligent tutor inter­ac­tion. Some of the pre­lim­i­nary data sug­gests that stu­dents actu­ally greatly pre­fer this, since they don’t get that much instruc­tor inter­ac­tion already, but they’re get­ting much more student/student inter­ac­tion than in a tra­di­tional co-located course. This might there­fore be an improve­ment over tra­di­tional lecture-based classes, but not classes in which teach­ers inter­act closely with stu­dents (such as small stu­dio courses).
• No one knows what will hap­pen to the edu­ca­tion mar­ket, includ­ing the peo­ple run­ning Kahn and Cours­era. How­ever, there were some pre­dic­tions. First, these plat­forms are going to make it so easy to share and access con­tent, in the same way that the web has for every­thing else, that find­ing and choos­ing con­tent is going to become a crit­i­cal chal­lenge for stu­dents. There­fore, one new role that instruc­tors might play is in select­ing and curat­ing con­tent in a way that is coher­ent and per­son­al­ized for the pop­u­la­tions that they teach.
• Most of the inter­ests related to crowd­sourc­ing are either in (1) enabling classes to be taught at scale (by find­ing ways to free instruc­tors and TAs to have to grade and assess all of the work), (2) to improve the effec­tive­ness, effi­ciency, and or engage­ment of learn­ing activ­i­ties, or (3) to cre­ate new oppor­tu­ni­ties through infor­mal learn­ing, such as through oDesk or Duolingo. Researchers are think­ing about how to use data to opti­mize the sequence of instruc­tion, give just the right hints to cor­rect mis­con­cep­tions, select a task that is chal­leng­ing but not too chal­leng­ing. In my view, this is lead­ing to a renewed inter­est intel­li­gent tutor­ing systems.
• As usual, most of this new research work suf­fers from a lack of ground­ing in and lever­ag­ing of prior lit­er­a­ture in learn­ing sci­ences and intel­li­gent tutor­ing sys­tems. There is tons of research on all of these chal­lenges that com­put­ing researchers are tack­ling, but I don’t seem them really using any of the work. This hap­pens over and over in com­put­ing research, since the inter­ests are often in cre­at­ing new things and not under­stand­ing the things them­selves. I was impressed, how­ever, how much Andrew Ng had lever­aged find­ings in learn­ing sci­ences to sup­port cer­tain design deci­sions in Coursera.
• There was a big under­cur­rent of data sci­ence at the work­shop. Every­one was excited about big data sets and how they might be lever­aged to improve learn­ing tech­nolo­gies. Most of the meth­ods reported were fairly prim­i­tive (AB test­ing, reten­tion rates), but I’m hop­ing this new energy behind learn­ing will lead to much bet­ter meth­ods and tools for doing edu­ca­tional data mining.

Phew! Sorry for the lack of coher­ence here. We cov­ered a lot of ground in 2.5 days and this is just a sliver of it.