Final sentence added 3/1/2012.
In this December 2011 report [23 page PDF] Brian Greenberg, Leonard Medlock and Darri Stephens report on the "performance and engagement of low‐performing high school algebra students receiving a mix of traditional teacher-led instruction and self‐guided instruction through the Khan Academy website".
The authors seek to compare the performance of two groups of "summer school" learners, one group taught traditionally and one group teacher-supported in largely individual use of Khan Academy content (thus "blended learning"). Both groups made substantial progress (as judged by the increase in percentage questions answered correctly on the MDTP Algebra II Readiness Exam).
The Khan Academy group is reported - with plenty of caveats, and with the rather disconcerting bar chart above - to have made very slightly more progress.
Along the way, the authors make interesting and useful observations about:
- classroom management under a "blended learning" model;
- the use of Google Chromebooks (cheapish quick-to-boot laptops that require a Wi-Fi connection to function fully) in a classroom context;
- how teachers in a blended learning environment can use data about learner performance to inform how, when, and with which learners to intervene;
- how having a plethora of systems in use at the same time or in the same school would prove problematic from a data-management point of view:
"One obvious challenge is that currently each online course or software uses its own data reporting system. For isolated pilots such as ours, these kinds of “walled gardens” can work fine. The teacher simply needs to learn how Khan Academy reports data. However, as teachers start using multiple products in a class or as schools blend technology into multiple courses, it becomes increasingly hard to make sense of all the data."
A striking paragraph in the report - which I can personally relate to having recently finished a course based on very short (40 second to 300 second) videos - concerns the value of Khan Academy videos:
"A final interesting perspective on Khan involves the value of the site’s videos. Most people are drawn to Khan based on its massive video library and Sal’s own charming and engaging teaching style. Like many, we assumed the videos would be the predominant learning mechanism for students tackling new material. In fact, the students rarely watched the videos. This result is consistent with some of the observations in the Los Altos pilot. The students greatly preferred working through the problem sets to watching the videos. Students turned to their peers, the hint, and the classroom teacher much more often than they did the linked Khan video. One possible reason is that the videos are aligned to the broader concept, but do not link directly to the problem students are struggling with. A second hypothesis is that the videos may be too long at eight to ten minutes. If students have 60‐90 minutes to work through multiple concepts in a class period, an investment of ten minutes for a single video feels like a lot. The badges and stars within Khan may also be a disincentive, as there is no immediate reward for watching videos as there is when completing streaks. Lastly, we wonder how many of us really enjoy watching instructional videos for extended periods of time. We are left curious about whether Khan’s videos need to be even more modular and shorter in duration and also about the value of video based instruction."