A few weeks back, I attended ICER 2013 at UC San Diego. Afterwards, I went up to San Francisco and had some adventures there (and at Yosemite), and then spent time in Vancouver seeing friends before coming home.

ICER this year was a solid conference, as always. I liked that this year things reverted to having two minute roundtable discussions after every talk, before the Q&A. It makes for a much more engaging conference.

All hail the UCSD Sun God, who benevolently oversaw the conference.

It turns out that many of the same stumbling blocks that people have when learning programming languages are the same as when they learn HTML. Syntax errors, figuring inconsistent syntax, learning that things need to be consistent – even just learning that what you see is not what you get.

In compsci we tend to overlook teaching HTML since it’s a markup language, not a programming language. But what we deal with in compsci is formal languages, and the simplest ones are the markup languages. Playing with a markup language is actually a much simpler place to start than giving novices a fully-fledged, Turing complete language.

Other research talks of note:

• Peter Hubwieser et al presented a paper on categorizing PCK in computer science that I liked; I’d love to see more work on PCK in CS, and look forward to seeing subsequent work using their framework.
• Colleen Lewis et al performed a replication study looking at AP CS exams. I love replication studies, so I may be a bit biased towards it :) In the original paper, they found that the first AP CS exam’s scores were strongly predicted by only a handful of questions – and those questions were ones like:
  int p = 3
  int q = 8
  p = q
  q = p_
  what are the values of p and q?_
  In Colleen’s paper, they found that the newer AP CS exams are much more balanced: things are not predicted by a small number of questions. Good to see!
• Robert McCartney et al revisited the famous McCraken study that found that students can’t program after CS1, and found that students can, but that educators have unrealistic expectations for what students would know by the end of CS1.
• Michelle Friend and Robert Cutler found that grade 8 students, when asked to write and evaluate search algorithms, favour a sentinel search (go every 100 spots, then every 10, then then every 1 spots, etc) over binary search.* Mike Hewner found that CS students pick their CS courses with really no knowledge of what they’ll be learning in the class. It’s one of those findings that’s kind of obvious in retrospect, but we educators really tend to mistake our students as thinking they know what they’re in for. Really, a student about to take a CS2 class doesn’t know what “hash tables” or “graphs” are coming in. Students pick classes more around who the prof is, the class’ reputation, and time of day.

• Finally, Michael Lee et al found that providing assessments improve how many levels people will play in an educational game that teaches programming. It’s a neat paper, but the finding is kind of predictable given the literature on feedback for students.

  I much more enjoyed Michael’s ICER talk from two years ago. He found in the same educational game that the more the compiler was personified, the more people played the game. Having a compiler that gives emoticon-style facial expressions, and uses first person pronouns (I think you missed a semi-colon vs. Missing semicolon) makes a dramatic difference in how much more people engage with learning computing. That’s a fairly ground-breaking discovery and I highly recommend the paper.
  The conference, was of course, not only limited to research talks:

• The doctoral consortium, as always, was a great experience. I got good feedback, about a dozen things to read, as well as awesome conference-buddies!

• The lightning talks were neat. My favourite was Ed Knorr’s talk on teaching fourth-year databases, since third-year and fourth-year CS courses are so often overlooked in the CS ed community. I also liked Kathi Fisler’s talk on using outcome-based grading in CS.
• The discussion talks were interesting! Elena Glassman talked about having students compare solution approaches in programming, which nicely complements the work that I presented this year.* The keynote talk also talked about the importance of comparing-and-contrasting. My takeaway from the keynote, however, was a teaching tip. Scott, the speaker, found that students learnt more from assignments if they were asked upon hand-in to grade themselves on the assignment. (Students would basically get a bonus if their self-assessment agreed with the TAs’ assessment.) It’s such a small thing to add onto the hand-in process, and adds so much to how much students get out of it. I’ll definitely have to try this next time I teach.
  Overall, a great time, and I’m looking forward to ICER 2014 in Glasgow!

At today’s CS Education Reading Group, one of our group members led us through an exercise about group work from “Students’ cooperation in teamwork: binding the individual and the team interests“ by Orit Hazzan and Yael Dubinsky.

It’s an in-class activity to get students thinking about how they work together in software projects. Students are given a scenario: you’ll be on a software team. If the project completes early, the team gets a bonus. How should the bonus be allocated?

1. 100% of the bonus should be shared equally
2. 80% should be for the team to share; 20% should go to the top contributor
3. 50% team, 50% individual
4. 80% team, 20% individual
5. 0% team, 100% to the individual who contributed the mostEverybody in the room got a minute to say which option we’d prefer and to write it down – and then we had a discussion about it. We then went through the rest of Orit’s paper and the variant scenarios.

I was the sole person in the room arguing for 100% team. My reasoning was because individual bonuses are not effective rewards – and often counterproductive.

Large Monetary Awards are Counterproductive

Ariely et al found that larger monetary rewards actually reduce performance on cognitive, intellectual tasks (link).  There’s almost half a century of psychology research arguing this.

And it doesn’t just hold in laboratory studies – though it’s not as simple as in the lab. Pay-for-performance approaches in the white-collar workplace have been repeatedly found to be at least suboptimal.

External motivators generally don’t help with cognitive tasks – internal motivation is really what drives us to do well on cognitive tasks.

Bonuses and Justice

Another problem with bonuses is fairness. Women and a number of other minorities are less likely to get them. They’re less likely to argue they deserve them. Their contributions are more likely to be viewed as less important. And they are perceived as less valuable.

(On that note, tipping in restaurants in the like is known to amplify racial inequalities. The race of the server is a stronger predictor of gratuity size than the quality of the service.)

Student Perceptions of Teamwork

In Orit’s small-sized classes, students opted for 80% team, 20% individual (see her 2003 ITiCSE paper). Why not 100%? One of the things that came up in our discussion is the question “but who is on my team?”

For a lot of our discussants, team composition was the driving factor. Do you have a team you trust? Then 100% for the team, for sure. But what if you don’t know them? Or you don’t trust them?

Katrina Falkner et al did a study on how CS students perceive collaborative activities, which they presented at last year’s SIGCSE. For a lot of students, collaboration stresses them out: they’re not used to it, they’re not experienced at it, and they’re not particularly good at it. But as educators, that’s what we’re here to work on, right?

The biggest source of anxiety for students in Katrina’s study was in who their partners were/would be. Would their partner(s) be smart, hardworking, and reliable?

Team Composition

It turns out randomized groups were the worst for students. Students felt powerless over their performance. We know from other literature that randomized groups is suboptimal for student performance. A much better way to form groups for performance is to group students by ability – strong students with fellow strong students.

On that note – it can be disastrous to pair strong students with weak students on the idea that poor students learn from the weak ones. It seeds/reinforces a lot of student fears about group work: strong students dislike it as they have to do a disproportionate share of the work; weak students learn less as their partner is doing the work for them.

Moving on: the best way to form groups in terms of reducing student anxiety is often to let students pick their groups. I say “often” because for a student who feels like an odd-one-out in the class, or doesn’t know anybody, this can be just as stressful.

Managing Short Term Stress

Stress is another thing worth talking about. Some people do great under pressure, and work better with the focus it gives them. And some people fall apart under stress, and work best without pressure. (And most of us are somewhere in between.)

The good news is that how we interpret anxiety is fairly malleable, and in a good way:

_The first experiment was at Harvard University with undergraduates who were studying for the Graduate Record Examination. Before taking a practice test, the students read a short note explaining that the study’s purpose was to examine the effects of stress on cognition. Half of the students, however, were also given a statement declaring that recent research suggests “people who feel anxious during a test might actually do better.” Therefore, if the students felt anxious during the practice test, it said, “you shouldn’t feel concerned. . . simply remind yourself that your arousal could be helping you do well.” _

Just reading this statement significantly improved students’ performance. They scored 50 points higher in the quantitative section (out of a possible 800) than the control group on the practice test. 

– Why Can Some Kids Handle Stress While Others Fall Apart?

Getting that message out to students is something we ought to be doing – test anxiety hurts a lot of student, as does anxiety about group work. It doesn’t have to be so bad.

In five days, I’ll be heading to ICER 2013 in San Diego, where I’ll be presenting a paper “Comparing and Contrasting Different Algorithms Leads to Increased Student Learning“ (pdf here).

The findings in a nutshell: if you present two different approaches to solving a compsci problem side-by-side and have students compare them, the students will understand the problem better than if you present those approaches sequentially. And importantly, the students will be better transferring their understanding of the problem to similar problems.

Why is this notable? Because the sequential approach is pretty much ~95% of how we teach algorithms and data structures! Just this past term when teaching CS2 I did things this way: a unit on binary trees, then a unit on BSTs, then a unit on heaps. Yet the evidence we have here is that it’s better to show algorithms/data structure variation in parallel.

Background Knowledge

This study is a replication of two studies in math education –  a study on algebra problems, and a fol]low-up study on estimation problems, both by Bethany Rittle-Johnson and Jon R. Star.

In the original algebra study, students were randomly assigned to one of two groups:

• a control group where students were given workbooks that had students saw a worked example solving a problem using one approach; they answered questions about it; then on the next page they saw a second worked example solving the problem using a different approach, and answered questions about it

• an experimental group where the workbooks presented the two worked examples side-by-side and students worked on all those problems on the same page
  The study has a pretest-intervention-posttest model, where the pretest and posttest are the same test. This allows the researchers to see how much students learnt from the intervention. The tests probed three things:

• procedural knowledge – how to solve a problem* procedural flexibility – being able to solve a problem with multiple approaches; being able to generate, recognize and evaluate different approaches

• conceptual knowledge
  And what did they find?

• the compare & contrast group did better than the sequential group with respect to procedural knowledge and procedural flexibility

• the two groups did the same on conceptual knowledge
  The conceptual knowledge thing was a bit of a surprise to them. So, they did another study! This time, they did a prestest-intervention-posttest-followup study. That’s the same as before, but with a second posttest given some time after the study, to see how much students retain. In this study, the math problems were about estimation.

What did they find?

• Again, the compare and contrast group did better on the posttest and the follow-up with regard to procedural knowledge and procedural flexibility

• But again, the two groups are the same on conceptual knowledge.
  It bothered them some more that the conceptual knowledge was different. Some similar studies in other fields would lead one to predict conceptual knowledge would be the same. So, they looked closer at their data:

• For the students who started off with some conceptual knowledge, the compare & contrast condition lead to more learning.

• For the students who had no conceptual knowledge to begin with, it didn’t matter which condition they were given.They speculated that you need to have some existing knowledge of a problem for comparing and contrasting to really shine.

Our study

We ran our study as a pretest-intervention-posttest-followup study, following the estimation study. In our study, CS2 students compared different techniques for hashing. We ran the study in three different sections of CSC 148 at UToronto, with 241 students participating.

Not that surprisingly, students in the compare-and-contrast group performed better at procedural knowledge and procedural flexibility – and the two groups performed the same on conceptual knowledge.

But we found the opposite of Rittle-Johnson and Star when we looked closer at conceptual knowledge:

• students who started with _no_ conceptual knowledge gained more from the compare-and-contrast condition than from the sequential condition
• students who started with some conceptual knowledge performed the same in both conditions
  What we think is going on here is that the compare-and-contrast approach lets students build a schema. By looking at what is different and similar, they can decipher what the important aspects of a problem are.

For the students who already have such a schema in place, when they see things sequentially, they can relate things back to their schema. They already know what the important aspects of a problem are, and so can compare on the fly.

For an expert like me, or any other instructor, this is the same for us. When I look at a hash function, I already know the aspects of hashing that can be varied to solve problems differently. When I see something new presented, I can relate it back to what I already know.

Another difference comes with yardsticks. Experts work with yardsticks like big O notation and complexity classes – that allow us to scale up our knowledge very easily. For novices, it’s a lot easier to handle “mergesort is faster than selection sort” than “mergesort is O(nlgn)”.

For us experts, it makes sense to present information sequentially – because we can easily process things that way. For our students – the ones learning things completely afresh – that’s a lot harder.