MOOCs: A review

Are edX and Coursera the future of university education?

By Ethan A. Solomon Feb. 8, 2013

• Facebook
• Twitter
• Reddit
• Email
• Print

Pretty much everybody’s gotten on board the MOOC bandwagon. MIT says its edX platform for “Massive Open Online Courses,” as they’re called, heralds a “revolution in education.” Stanford professors Andrew Ng SM ’98 and Daphne Koeller, who cofounded edX competitor Coursera, have similar ambitions for their startup — and 33 universities have joined with them so far. Political commentators are excited, too: “Let the revolution begin,” proclaimed Thomas L. Friedman in The New York Times.

Despite the hype, MOOC providers do acknowledge that robust online education is in its infancy, and as The Times and NPR describe it, there are “kinks to be worked out.” Universities that offer online courses through edX or Coursera rightly worry about how exams will be administered, how cheaters will be identified, and how grading will be scaled to hundreds of thousands of students in a single course. A lot of smart people are coming up with clever ways to address all those problems, and more.

But few from these universities or in the media have stopped to seriously and publicly ask: Never mind the “kinks,” how do we know these courses are any good?

I could take Anant Agarwal, edX president and former CSAIL chief, on his word: “We will not water down the courses,” he says. “They will continue to be MIT-hard or Harvard-hard.” Or Coursera, which says on its website that “you will watch lectures taught by world-class professors, learn at your own pace, test your knowledge, and reinforce concepts through interactive exercises.”

But this is a revolution we’re talking about, here — there’s no room for error. The only way to really be sure that this stuff is great teaching is to actually take some courses. So I did.

I had two main goals in doing so. First, a side-by-side of courses from two leading MOOC providers can help prospective students make informed choices about which courses to take. Second, like any college courses, MOOCs benefit from critical, independent, and public evaluation from people who don’t have a stake in their outcome.

That second point is especially critical at a time when universities are starting to fundamentally rethink how they educate. What traditions from centuries of brick-and-mortar teaching should be transferred online, and what should we throw out? What worked well about old teaching models, and what can be improved?

6.00x and Machine Learning

Fully and comprehensively evaluating edX (now offering 13 courses) and Coursera (217 courses) would mean taking hundreds of online classes. Instead, I chose one from each to be examples of the experience: edX’s 6.00x (Introduction to Computer Science and Programming), via MITx, and Coursera’s Machine Learning, via Stanford. It is important to note that, just like traditional college courses, your mileage may vary depending upon your interest in the subject and who’s teaching it.

6.00x is a 14-week introduction to fundamental concepts in programming and computer science, taught in Python. It emphasizes recursion and the divide-and-conquer paradigm, object-oriented programming, simulations, basic statistics and data analysis, and optimization problems, in addition to the nuts and bolts of learning to use Python for the first time. The instructors are MIT computer science professors Eric Grimson PhD ’80, John V. Guttag, and Christopher J. Terman PhD ’78.

Machine Learning (ML), taught by Coursera co-founder Andrew Ng SM ’98, is a broad overview of popular machine learning algorithms such as linear and logistic regression, neural networks, SVMs, and k-means clustering, among others. It is light on theory but heavy on applications, aiming to help students get the most practical use out of machine learning algorithms.

I was impressed with both courses for the same reason I’ve been impressed with some “traditional” courses I took at MIT: I felt like I actually learned something I could take with me. Before I say what about the courses I liked (and later, what I didn’t like), I want to explain, for the uninitiated, what’s involved in taking a MOOC.

• Lectures: The backbone of almost all MOOCs on edX/Coursera is the lecture, just like regular college courses. But instead of one to two hour sessions in a giant hall, MOOC lectures are organized in “sequences” of 5–15 minute videos, usually featuring a professor scribbling on PowerPoint slides and marking up graphs and diagrams. On average, MOOC students can expect two to four hours of lecture time per week, but they’re free to play videos at 1.25x or 1.5x speed to cut that down.

• Finger Exercises: Lecture sequences are often sprinkled with short and simple comprehension questions — edX calls them “finger exercises,” since they usually don’t take much work. If you’ve been paying attention to the video, these questions are not challenging and will count for a small portion of your grade, if any at all. Some might ask for the numerical answer to a computation, and others are multiple-choice.

• Homework: Homework in 6.00x and ML are usually combinations of multiple-choice questions and programming assignments. ML would often ask students to implement a learning algorithm covered in lecture, and an online checker would test your algorithm in a few test cases and award credit for passing. Similarly, 6.00x students complete close replicas of regular 6.00 problem sets — implement a recursive function, define classes and methods to carry out a simulation, etc. — also awarding credit according to what test cases your code passes.

• Exams: Both platforms have courses which require students to take midterm and final exams, but ML was not one of them. 6.00x again reflects 6.00; there are two two-hour midterm exams and a four-hour final, which together account for the bulk of the final grade. The tests are similar to problem sets: implement a function to do this-or-that within certain design parameters.

• Forums: MOOC backers say the discussion forums are a big part of the online learning experience. This is where you go if you’re stuck on a question or don’t understand a concept. Sometimes course staff — even the professors — will reply to questions posted on the boards, but more often expect help to come from other students.

• Certification: Tell whomever cares that you took an online course — you’ll have a PDF to prove it! Many MOOCs, including most courses on edX and Coursera, currently offer certificates of completion if you finish enough of a course. The free versions of 6.00x and ML that I took offer “honor code” certificates, which means that no steps were taken to prove that the real Ethan Solomon took the course, merely that somebody who wrote their name as “Ethan Solomon” took the course, and they may or may not have cheated their way through it. More robust forms of identity verification and certification have recently been announced on both platforms, though they carry a fee.

If you’re a student who’s used to watching lectures online, skipping recitations and only showing up to hand in problem sets, the MOOC experience will be nothing fundamentally new to you.

But don’t get the wrong idea. MOOCs are not like OpenCourseWare classes, which are essentially ordinary class material dumped online. Critically, MOOCs offer assessment (albeit more limited than what you’ll find on-campus), some form of interaction with course staff, and deadlines to complete homework and exams. Also, MOOC videos are usually not pure replays of recorded brick-and-mortar lectures. Videos have either been redone from scratch, or hour-long videos have been broken up into more manageable chunks.

The Good

“If at first you don’t succeed, try, try, try again. Heck, try again as much as you want.” This philosophy was big in both 6.00x and ML, and I liked it. Ordinary homework and problem sets offer students plenty of opportunities to get things wrong. ML students can retry review questions up to 100 times, and 6.00x problem set questions can be submitted up to 30 times. Instead of submitting a homework assignment to a TA once, only to get it back days later with red ink and lose it at the bottom of thier backpacks, students have enough opportunities to grapple with the problem until they get it right.

The instant gratification from a right-or-wrong autograder can be addicting. I found myself surprisingly unwilling to give up on a difficult problem, even though I wasn’t working towards official credit for either of these courses. A pointless little green checkmark, paradoxically, is a tempting reward for giving the problem another shot.

The cost to the “try, try again” system is assessment. With 30 or 100 allowed attempts, problem set scores become much less useful for differentiating student performance, because any determined student could probably get full credit. 6.00x skirts this problem by weighting exam scores much more than problem set scores, and exam problems grade submissions the first time they run without syntax errors. And you’re only given five total attempts to run your code, syntax errors or not. In essence, formal assessment tools don’t follow the “try, try again” paradigm.

(Machine Learning had no exams, so it was indeed possible — even expected — that any determined student earn full points on every assignment. As I’ll discuss soon, this might be a problem for ML as far as differentiating student performance goes, but I found it wasn’t a major qualitative impediment to learning.)

6.00x and ML were more efficient information-delivery systems than many courses I had taken at MIT. Lectures are streamlined and chunked into topic-based segments. There’s no class administrivia to get out of the way at the beginning or the end of a lecture, nor must a student sit through a professor’s explanation of a topic he or she already understands.

I didn’t take notes for either courses, but that was OK — the full lecture videos and lecture slides were made permanently available for later reference. I found that the MOOC design let me sit back and actually listen to what the instructor was saying (at 1.25x speed!). It felt liberating to force the instruction to meet my preferred pace, and not the other way around.

Even the “finger exercises” — short questions sprinkled throughout lecture videos — felt worth it. At first I approached them with deep skepticism, remembering my unsatisfactory experience with “clicker questions” in MIT freshman classes. (More often than not, those questions disrupted the lecture’s flow, much like this parenthetical sentence, or the answer and follow-up explanation was already clear to me.) 6.00x deserves particular commendation here; finger exercises forced me to think about what Profs. Grimson, Guttag, or Terman just said. But if I wanted to move through lectures quickly, I could just save the exercises for later.

The lecture videos themselves weren’t bad, either. Andrew Ng describes most machine learning algorithms with ease, though he is able to cut the more complicated corners by glossing over theory and emphasizing application instead. Grimson, Guttag, and Terman move at an MIT-pace through 6.00x — just as edX promised — so those who have never taken a college course before might find it daunting.

Room for improvement

EdX and Coursera MOOCs began in earnest less than a year ago. As an early-adopter, I expected flaws, and encountered some in both courses. How could edX and Coursera do better?

Feedback is deprived on both platforms. Students in 6.00x and ML could expect a “right or wrong” judgment on their code, but usually nothing much more. Where TAs could offer nuanced feedback as to the efficiency of a piece of code in a traditional college course, edX and Coursera can only (currently) hope to confirm that the code passes a few test cases. Everybody knows that wrong answers taken many shapes and sizes — the perfect MOOC would identify why an answer is wrong and suggest particular concepts to review.

It’s not surprising that substantive feedback is a challenge, and Ng and Grimson readily acknowledge that there’s plenty of work to be done here. Automated grading systems are not yet intelligent enough to offer highly individualized feedback, but both platforms are thinking of ways to improve. Ng says Coursera provides custom error messages for the types of incorrect answers that show up frequently; Grimson suggests code-grading could be crowdsourced.

It’s not clear how much progress needs to be made before MOOCs could provide feedback equivalent to that of a human teaching assistant, nor is it clear how educators could rigorously tell when MOOCs have reached that point. But the fact remains that feedback is critical to a good education. Fortunately, edX and Coursera leadership know this well.

“Remember, this is version one,” said Agarwal. “We are working on machine learning and various forms of peer grading that will enable us to provide a lot more detailed feedback.”

Before the time that automated feedback gets really good, MOOC backers might make two important points. First, many MOOCs are designed with the “blended model” of learning in mind, meaning that colleges expect to offer MOOCs in conjunction with real human instructors on-campus, not as full-fledged course replacements. Second, healthy discussion forums can partially compensate for the lack of formal instructor feedback, because other students, and occasionally a edX/Coursera teaching assistant, can provide comments on code or walk students through a solution. Still not as great as sitting down with a TA, but certainly better than nothing.

My experience with edX and Coursera discussion forums was a mixed bag. A voting system tends to increase the visibility of helpful posts and suppress the bad ones, but MOOC-takers should still expect to slog through some whiny/irrelevant/self-congratulatory posts. Welcome to the Internet.

Aside from feedback, organization was by far 6.00x’s greatest challenge. To be sure, the first two-thirds of the course was great. From September through November, problem sets were posted on time, course staff kept the bulletin reasonably up-to-date with news of upcoming exams and new lectures, and the occasional glitch was resolved within a day or two, at most. 6.00x essentially featured the same organizational standards I had come to expect from an average on-campus MIT course — far from perfect, but very manageable.

Unfortunately, things seemed to grind to a crawl, and then a halt, in December. Lectures and problem sets were not posted. The forum buzzed with confused students, some wondering whether the course had ended prematurely, but no edX staff stepped in to clear things up.

On Jan. 4, Grimson posted an apology on the course bulletin, attributing some of the glitches to an “undertaking of this magnitude,” and others to personal crises that led to “a breakdown in process and communication.” Parts of the team did not know some elements of the class were on hold, and Grimson said that the staff would work to ensure that there would be no single points of failure in future versions of the class.

The last problem set was made optional, the last set of lectures were released, and final exam details were posted. The course concluded a couple of weeks later without major problems.

“I apologized to the class because it wasn’t the experience I wanted them to have,” Grimson told me in an interview two weeks ago. “This is part of the growing pains of doing something at this scale.”

Students were overwhelmingly understanding, by the way. “Don’t feel bad at all for the lack of updates. I actually enjoyed a bit of time off ;),” posted a student by the username skyking. “Totally … no need to apologise for your absence. We’re all just grateful this whole thing even exists,” added RHill.

Is it MIT-hard and Stanford-hard?

No. But that answer deserves several important qualifiers.

First, a debate over the “rigor” of an online course is partly semantic. EdX backers say they aspire to make their courses are just as rigorous as an on-campus equivalent, but acknowledge that the courses are still different. Different in the sense that on-campus versions of 6.00x and ML, for example, feature open-ended project- or group-based components — activities which MOOCs can’t yet support.

Depending upon whom you ask, the absence of a rich project experience in an online course categorically makes it less difficult, in that there’s just less work to do. Not to mention the lack of thoughtful feedback on exams and assignments means the bar MOOC students need to pass is a little lower. 6.00 students can’t submit code that “just works;” it has to be good code, too.

However, the claim that edX courses are not “watered-down” is still valid if these courses assess understanding of the same concepts within the limits of online course technology. In other words, nobody’s taking any lecture videos or problem sets and cutting out stuff they think might be too hard for an online audience to understand.

“The content, the material, the flow — it’s pretty much the same rigor,” Agarwal told me. “Of course, the experience is different, but it’s a comparable rigor.”

6.00x isn’t 6.00, but it isn’t watered-down, either. 6.00/6.00x syllabi are virtually identical; problem sets are mostly the same, aside from edX’s inability to give high-level feedback on code. 6.00x exams were substantially similar to 6.00 exams available on OpenCourseWare.

Coursera and edX take different approaches to the rigor question. Ng says that instructors are free to determine course difficulty for themselves, and that Coursera simply provides tools. Some Coursera courses may be tailored to particular audiences — like high school students — and the difficulty modified accordingly. At the same time, faculty from any of Coursera’s 33 partner universities might choose to launch a course if they feel it mirrors on-campus rigor as much as possible.

I’ve never taken an ordinary Stanford course, but Machine Learning was not as challenging as the average MIT undergraduate course, broadly speaking. There were no exams, review questions tended to be simple multiple-choice, and there was a lot of hand-holding through the programming assignments. Most assignments came with a detailed step-by-step guide.

Let me be clear, though. MIT students sometimes equate “more difficult” with “better,” but that’s only true in certain contexts. There’s nothing wrong with a Machine Learning MOOC which can expose a broader audience to really cool supervised and unsupervised learning algorithms. People with full-time jobs or a weaker background in computer science could learn something useful on Coursera that might have been unapproachable in an edX course.

Agarwal says that edX might also experiment with different versions of courses for different audiences, but the goal right now is replicate on-campus rigor as closely as possible.

Learning to teach

At the foundation of any online learning venture is the guess that online tools can make education better. But how can edX and Coursera actually tell that their services improve learning beyond the status quo?

MOOCs are giant experiments. MOOCs collect data about the way students interact with the user interface and draw inferences about how UI changes affect student performance. The platforms also facilitate A/B testing — give one group of students version A of the course, give another version B, and see if there is a significant performance effect.

Last term, edX experimented with two slightly different versions of 6.002x (Circuits and Electronics). Group A saw an extra summary video at the end of each lecture sequence, Group B got nothing special.

“Does [the summary] make it too long? Does it make it make it too painful for students? Does it make it incoherent? We didn’t know,” said Agarwal.

Grimson says MITx will “mine the massive amount of data we have coming from the online courses to answer questions like, ‘How many key concepts are there in this class?,’ ‘Where are the weak points in terms of how we’re teaching it?’ That feeds back into the question of how we teach our own students.”

On the Coursera side, Ng points out that Coursera’s approach is grounded in prior academic studies on online learning and pedagogical research about features like the finger exercises and unlimited retries. Coursera is also examining the “flipped classroom” model — complementing MOOC content with in-person discussion and instruction — in a scientific way (as is edX).

Ultimately, MOOCs might be an opportunity to ask even more fundamental questions about education: What exactly does it mean for a course to be successful? How do we rigorously measure things information retention and concept mastery? And are there any paradigmatically different ways of teaching altogether that MOOCs can uncover?

“We might be evangelizers on the one hand, but as researchers we also have to be skeptics,” said Agarwal. “We have to test and retest all the hypotheses.”