I’m currently TAing for CSC2720H Systems Thinking for Global Problems, a graduate-level course on systems thinking. In class today we talked about soft systems thinking (SSM), an approach which uses systems thinking to tackle what are called “wicked problems“. I thought I’d outline one approach to SSM, as it’s useful to CS education research.

Step 1: Identify the domain of interest

Before you can research something, you should first decide what your domain is. What topic? What system are you studying? For example, “teaching computer science” could be your starting point, as could “climate change”.

Chances are you’re looking at a wicked problem. Conklin’s definition of wicked problems are that:

1. The problem is not understood until after the formulation of a solution.
2. Wicked problems have no stopping rule.
3. Solutions to wicked problems are not right or wrong.
4. Every wicked problem is essentially novel and unique.
5. Every solution to a wicked problem is a ‘one shot operation.’
6. Wicked problems have no given alternative solutions.Because you’re looking at a domain which doesn’t have a clear definition or boundaries, you’ll first want to immerse yourself in the domain. One trick is to draw “rich pictures“, which are essentially visualized streams of consciousness.

You should also think about what perspectives you bring into this domain. What biases and privileges do you have going into this? Why are you interested in this domain? What do you have to gain or lose here?

Step 2: Express the Problem Situation

A good expression of a situation should contain no value judgments. As a heuristic, ask yourself if it’s possible for somebody to not see your statement as a problem.

For example: “Global surface temperatures on Earth have been increasing since the late 19th century” is a statement some people may not even see as a problem, whereas “Climate change will destroy our way of life” presents climate change as a problem, rather than as a situation.

The goal of expressing a situation is so that you can then identify the different ways people would frame this as a problem. If you express the situation as inherently problematic, this will narrow the problem frames that you’ll think of.

It may take you a few drafts to come up with a situation statement you’re happy with. That’s fine; SSM is an iterative process. You may find in later steps you’ll want to come back here (or to earlier steps) and revise your work.

Step 3: Identify Different Problem Frames

How you frame a situation will affect the types of analysis and solutions you’ll come up with. Your next step is to think of the different ways the situation could be framed, before you pick which one to proceed with. I’ll give three examples.

Situation: A large fraction of undergraduate students fail in first-year CS. 
Some problem frames:

1. The context is problematic. Students are overburdened in all their classes, and have a difficult time adjusting to university study.2. The curriculum structure is problematic. CS1 material builds on itself to an extent that other first-year classes don’t. If you fall behind, it can be impossible to catch up.
2. The amount of content is problematic. We pack too much material into CS1 for students to properly absorb and understand.4. The pedagogy is problematic. We just aren’t teaching CS1 effectively.
3. The affordances are problematic. We teach CS1 using programming languages which don’t reflect how non-programmers reason about computation (e.g. while loops vs. until loops) 

4. It isn’t a problem. The failure rates, while high, are consistent with other first-year courses.7. Computer science is inherently difficult for humans to learn.
   Situation: Computer science is not available in every high school in Canada.
   Some problem frames:

5. If the general population doesn’t learn about CS, they won’t be able to properly participate in a 21st century democracy. Issues of privacy and security are poorly understood but necessary for democracies to address.

6. Not enough young Canadians are equipped with the necessary computational skills for the workforce.3. CS is generally only offered at affluent, urban, schools. This causes racial and class disparities in access to CS, and in turn, to lucrative jobs.
7. Fewer girls than boys take high school CS; if more girls could take high school CS it could reduce the gender gap.5. There aren’t enough qualified high school teachers to teach CS in every school.
8. Schools are underfunded and overburdened.

9. This isn’t a problem, schools should be focusing on other topics instead! 
   Situation: The percentage of women completing undergrad CS programmes hasn’t changed since the wide-scale creation of women in computing initiatives.
   Some problem frames:

10. The initiatives are being undermined by the same external forces that created the gender disparity in CS.2. The initiatives are themselves faulty. They reinforce gender norms and the subtyping of “female computer scientist” != “computer scientist”.

11. The initiatives are having a positive impact, but not in ways this measurement can capture. For example, they improve the personal experiences of the women in the field, but don’t change the numbers.
12. Without the initiatives, the percentage would have decreased. Before the initiatives, the percentage was decreasing; it has since flatlined.

Step 4: Study the problem frames and pick one

At this point you’ll want to start doing a literature review. As you review the literature you’ll find different papers take different problem frames (explicitly or implicitly).

Each problem frame will lead to different approaches to studying and “solving” the problem. Let’s return to the CS1 failure example:

Situation: A large fraction of undergraduate students fail in first-year CS.

1. The context is problematic. Students are overburdened in all their classes, and have a difficult time adjusting to university study. 

• Possible solutions: increase financial aid, reduce course load 2. The curriculum structure is problematic. CS1 material builds on itself to an extent that other first-year classes don’t. If you fall behind, it can be impossible to catch up.

• Possible solution: reconfigure the CS1 material to be breadth-first, like they have at Harvey Mudd3. The amount of content is problematic. We pack too much material into CS1 for students to properly absorb and understand. 

• _Possible solutions: _spread it out amongst more classes, get CS1 in all high schools4. The pedagogy is problematic. We just aren’t teaching CS1 effectively.

• Possible solutions: use peer instruction and/or pair programming 5. The affordances are problematic. We teach CS1 using programming languages which don’t reflect how non-programmers reason about computation (e.g. while loops vs. until loops) Ultimately for your research you’ll need to pick a single problem frame to work within. At this point you should choose one and justify why you’re picking that one rather than the other ones. Once you’ve picked your problem frame you’ll want to carefully review and analyse the relevant literature.

Step 5: Arena of Action

Think about your problem frame like you’re framing a photograph. Who or what is in the foreground? In the background? Not in the frame at all?

Or, if you see your situation as an arena: who is in the arena? Who is being affected by the situation? Who has the power to change the situation? Who benefits from this framing, and who loses out?

CATWOE Analysis can come in here: who are the customers/clients? The actors? The transformation process? What’s the world view underlying this? Who “owns” or controls this? What environmental constraints are there?

When I took a class on policy analysis, some questions we asked were: “Why was this policy adopted? On whose terms? Why? On what grounds have these selections been justified? Why? In whose interests? Indeed, how have competing interests been negotiated?” as well as “Why now?” and “What are the consequences?” (from Taylor et al, 1997, “Doing Policy Analysis”)

Step 6: Theorize/model the relevant system

Having identified your problem frame, you’ll want to return to the system at hand. Depending on the problem, you may want to model it and/or theorize about it based on the existing literature.

Once you have a model, compare it to real world evidence. You may have to design and perform empirical studies in order to accomplish this.

Step 7: Identify possible/feasible changes to the situation, and take action

Like it says. Soft systems methodology was created for action research, a style of research intended to create social change. In this paradigm, research is bad if it doesn’t help improve the situation.

Your goal in doing this research is to identify changes to the situation which are both possible and feasible. You should identify who has the power to enact these changes. Challenge yourself: contact these people and share your findings. Advocate for your proposed solution.

A few weeks back, I attended ICER 2013 at UC San Diego. Afterwards, I went up to San Francisco and had some adventures there (and at Yosemite), and then spent time in Vancouver seeing friends before coming home.

ICER this year was a solid conference, as always. I liked that this year things reverted to having two minute roundtable discussions after every talk, before the Q&A. It makes for a much more engaging conference.

All hail the UCSD Sun God, who benevolently oversaw the conference.

It turns out that many of the same stumbling blocks that people have when learning programming languages are the same as when they learn HTML. Syntax errors, figuring inconsistent syntax, learning that things need to be consistent – even just learning that what you see is not what you get.

In compsci we tend to overlook teaching HTML since it’s a markup language, not a programming language. But what we deal with in compsci is formal languages, and the simplest ones are the markup languages. Playing with a markup language is actually a much simpler place to start than giving novices a fully-fledged, Turing complete language.

Other research talks of note:

• Peter Hubwieser et al presented a paper on categorizing PCK in computer science that I liked; I’d love to see more work on PCK in CS, and look forward to seeing subsequent work using their framework.
• Colleen Lewis et al performed a replication study looking at AP CS exams. I love replication studies, so I may be a bit biased towards it :) In the original paper, they found that the first AP CS exam’s scores were strongly predicted by only a handful of questions – and those questions were ones like:
  int p = 3
  int q = 8
  p = q
  q = p_
  what are the values of p and q?_
  In Colleen’s paper, they found that the newer AP CS exams are much more balanced: things are not predicted by a small number of questions. Good to see!
• Robert McCartney et al revisited the famous McCraken study that found that students can’t program after CS1, and found that students can, but that educators have unrealistic expectations for what students would know by the end of CS1.
• Michelle Friend and Robert Cutler found that grade 8 students, when asked to write and evaluate search algorithms, favour a sentinel search (go every 100 spots, then every 10, then then every 1 spots, etc) over binary search.* Mike Hewner found that CS students pick their CS courses with really no knowledge of what they’ll be learning in the class. It’s one of those findings that’s kind of obvious in retrospect, but we educators really tend to mistake our students as thinking they know what they’re in for. Really, a student about to take a CS2 class doesn’t know what “hash tables” or “graphs” are coming in. Students pick classes more around who the prof is, the class’ reputation, and time of day.

• Finally, Michael Lee et al found that providing assessments improve how many levels people will play in an educational game that teaches programming. It’s a neat paper, but the finding is kind of predictable given the literature on feedback for students.

  I much more enjoyed Michael’s ICER talk from two years ago. He found in the same educational game that the more the compiler was personified, the more people played the game. Having a compiler that gives emoticon-style facial expressions, and uses first person pronouns (I think you missed a semi-colon vs. Missing semicolon) makes a dramatic difference in how much more people engage with learning computing. That’s a fairly ground-breaking discovery and I highly recommend the paper.
  The conference, was of course, not only limited to research talks:

• The doctoral consortium, as always, was a great experience. I got good feedback, about a dozen things to read, as well as awesome conference-buddies!

• The lightning talks were neat. My favourite was Ed Knorr’s talk on teaching fourth-year databases, since third-year and fourth-year CS courses are so often overlooked in the CS ed community. I also liked Kathi Fisler’s talk on using outcome-based grading in CS.
• The discussion talks were interesting! Elena Glassman talked about having students compare solution approaches in programming, which nicely complements the work that I presented this year.* The keynote talk also talked about the importance of comparing-and-contrasting. My takeaway from the keynote, however, was a teaching tip. Scott, the speaker, found that students learnt more from assignments if they were asked upon hand-in to grade themselves on the assignment. (Students would basically get a bonus if their self-assessment agreed with the TAs’ assessment.) It’s such a small thing to add onto the hand-in process, and adds so much to how much students get out of it. I’ll definitely have to try this next time I teach.
  Overall, a great time, and I’m looking forward to ICER 2014 in Glasgow!

At today’s CS Education Reading Group, one of our group members led us through an exercise about group work from “Students’ cooperation in teamwork: binding the individual and the team interests“ by Orit Hazzan and Yael Dubinsky.

It’s an in-class activity to get students thinking about how they work together in software projects. Students are given a scenario: you’ll be on a software team. If the project completes early, the team gets a bonus. How should the bonus be allocated?

1. 100% of the bonus should be shared equally
2. 80% should be for the team to share; 20% should go to the top contributor
3. 50% team, 50% individual
4. 80% team, 20% individual
5. 0% team, 100% to the individual who contributed the mostEverybody in the room got a minute to say which option we’d prefer and to write it down – and then we had a discussion about it. We then went through the rest of Orit’s paper and the variant scenarios.

I was the sole person in the room arguing for 100% team. My reasoning was because individual bonuses are not effective rewards – and often counterproductive.

Large Monetary Awards are Counterproductive

Ariely et al found that larger monetary rewards actually reduce performance on cognitive, intellectual tasks (link).  There’s almost half a century of psychology research arguing this.

And it doesn’t just hold in laboratory studies – though it’s not as simple as in the lab. Pay-for-performance approaches in the white-collar workplace have been repeatedly found to be at least suboptimal.

External motivators generally don’t help with cognitive tasks – internal motivation is really what drives us to do well on cognitive tasks.

Bonuses and Justice

Another problem with bonuses is fairness. Women and a number of other minorities are less likely to get them. They’re less likely to argue they deserve them. Their contributions are more likely to be viewed as less important. And they are perceived as less valuable.

(On that note, tipping in restaurants in the like is known to amplify racial inequalities. The race of the server is a stronger predictor of gratuity size than the quality of the service.)

Student Perceptions of Teamwork

In Orit’s small-sized classes, students opted for 80% team, 20% individual (see her 2003 ITiCSE paper). Why not 100%? One of the things that came up in our discussion is the question “but who is on my team?”

For a lot of our discussants, team composition was the driving factor. Do you have a team you trust? Then 100% for the team, for sure. But what if you don’t know them? Or you don’t trust them?

Katrina Falkner et al did a study on how CS students perceive collaborative activities, which they presented at last year’s SIGCSE. For a lot of students, collaboration stresses them out: they’re not used to it, they’re not experienced at it, and they’re not particularly good at it. But as educators, that’s what we’re here to work on, right?

The biggest source of anxiety for students in Katrina’s study was in who their partners were/would be. Would their partner(s) be smart, hardworking, and reliable?

Team Composition

It turns out randomized groups were the worst for students. Students felt powerless over their performance. We know from other literature that randomized groups is suboptimal for student performance. A much better way to form groups for performance is to group students by ability – strong students with fellow strong students.

On that note – it can be disastrous to pair strong students with weak students on the idea that poor students learn from the weak ones. It seeds/reinforces a lot of student fears about group work: strong students dislike it as they have to do a disproportionate share of the work; weak students learn less as their partner is doing the work for them.

Moving on: the best way to form groups in terms of reducing student anxiety is often to let students pick their groups. I say “often” because for a student who feels like an odd-one-out in the class, or doesn’t know anybody, this can be just as stressful.

Managing Short Term Stress

Stress is another thing worth talking about. Some people do great under pressure, and work better with the focus it gives them. And some people fall apart under stress, and work best without pressure. (And most of us are somewhere in between.)

The good news is that how we interpret anxiety is fairly malleable, and in a good way:

_The first experiment was at Harvard University with undergraduates who were studying for the Graduate Record Examination. Before taking a practice test, the students read a short note explaining that the study’s purpose was to examine the effects of stress on cognition. Half of the students, however, were also given a statement declaring that recent research suggests “people who feel anxious during a test might actually do better.” Therefore, if the students felt anxious during the practice test, it said, “you shouldn’t feel concerned. . . simply remind yourself that your arousal could be helping you do well.” _

Just reading this statement significantly improved students’ performance. They scored 50 points higher in the quantitative section (out of a possible 800) than the control group on the practice test. 

– Why Can Some Kids Handle Stress While Others Fall Apart?

Getting that message out to students is something we ought to be doing – test anxiety hurts a lot of student, as does anxiety about group work. It doesn’t have to be so bad.

In five days, I’ll be heading to ICER 2013 in San Diego, where I’ll be presenting a paper “Comparing and Contrasting Different Algorithms Leads to Increased Student Learning“ (pdf here).

The findings in a nutshell: if you present two different approaches to solving a compsci problem side-by-side and have students compare them, the students will understand the problem better than if you present those approaches sequentially. And importantly, the students will be better transferring their understanding of the problem to similar problems.

Why is this notable? Because the sequential approach is pretty much ~95% of how we teach algorithms and data structures! Just this past term when teaching CS2 I did things this way: a unit on binary trees, then a unit on BSTs, then a unit on heaps. Yet the evidence we have here is that it’s better to show algorithms/data structure variation in parallel.

Background Knowledge

This study is a replication of two studies in math education –  a study on algebra problems, and a fol]low-up study on estimation problems, both by Bethany Rittle-Johnson and Jon R. Star.

In the original algebra study, students were randomly assigned to one of two groups:

• a control group where students were given workbooks that had students saw a worked example solving a problem using one approach; they answered questions about it; then on the next page they saw a second worked example solving the problem using a different approach, and answered questions about it

• an experimental group where the workbooks presented the two worked examples side-by-side and students worked on all those problems on the same page
  The study has a pretest-intervention-posttest model, where the pretest and posttest are the same test. This allows the researchers to see how much students learnt from the intervention. The tests probed three things:

• procedural knowledge – how to solve a problem* procedural flexibility – being able to solve a problem with multiple approaches; being able to generate, recognize and evaluate different approaches

• conceptual knowledge
  And what did they find?

• the compare & contrast group did better than the sequential group with respect to procedural knowledge and procedural flexibility

• the two groups did the same on conceptual knowledge
  The conceptual knowledge thing was a bit of a surprise to them. So, they did another study! This time, they did a prestest-intervention-posttest-followup study. That’s the same as before, but with a second posttest given some time after the study, to see how much students retain. In this study, the math problems were about estimation.

What did they find?

• Again, the compare and contrast group did better on the posttest and the follow-up with regard to procedural knowledge and procedural flexibility

• But again, the two groups are the same on conceptual knowledge.
  It bothered them some more that the conceptual knowledge was different. Some similar studies in other fields would lead one to predict conceptual knowledge would be the same. So, they looked closer at their data:

• For the students who started off with some conceptual knowledge, the compare & contrast condition lead to more learning.

• For the students who had no conceptual knowledge to begin with, it didn’t matter which condition they were given.They speculated that you need to have some existing knowledge of a problem for comparing and contrasting to really shine.

Our study

We ran our study as a pretest-intervention-posttest-followup study, following the estimation study. In our study, CS2 students compared different techniques for hashing. We ran the study in three different sections of CSC 148 at UToronto, with 241 students participating.

Not that surprisingly, students in the compare-and-contrast group performed better at procedural knowledge and procedural flexibility – and the two groups performed the same on conceptual knowledge.

But we found the opposite of Rittle-Johnson and Star when we looked closer at conceptual knowledge:

• students who started with _no_ conceptual knowledge gained more from the compare-and-contrast condition than from the sequential condition
• students who started with some conceptual knowledge performed the same in both conditions
  What we think is going on here is that the compare-and-contrast approach lets students build a schema. By looking at what is different and similar, they can decipher what the important aspects of a problem are.

For the students who already have such a schema in place, when they see things sequentially, they can relate things back to their schema. They already know what the important aspects of a problem are, and so can compare on the fly.

For an expert like me, or any other instructor, this is the same for us. When I look at a hash function, I already know the aspects of hashing that can be varied to solve problems differently. When I see something new presented, I can relate it back to what I already know.

Another difference comes with yardsticks. Experts work with yardsticks like big O notation and complexity classes – that allow us to scale up our knowledge very easily. For novices, it’s a lot easier to handle “mergesort is faster than selection sort” than “mergesort is O(nlgn)”.

For us experts, it makes sense to present information sequentially – because we can easily process things that way. For our students – the ones learning things completely afresh – that’s a lot harder.