A message from James Gee

James Paul Gee on Working Examples from drew davidson on Vimeo.

Welcome to the bare beginnings of the Worked Examples Project (WEP). We hope this small seed will grow into a forest. So what is a “worked example”? Well, first off, Dan Schwartz at Stanford has suggested that “working examples” is a better term. In fact, it is good to use both terms.

“Worked example” is a term we are borrowing and transforming from science education. In that literature, a worked example is an example of how an expert “works through” (that is, explicates, explains, and models) the solution to a well-known problem. Worked examples are supposed to make the thinking, practices, and values of a discipline overt and public for newcomers.

WEP proposes a new use for worked examples, one applicable far beyond science. When people from different disciplines and domains of expertise are collaborating to develop a new field, they have initially no well defined and widely accepted solutions to core problems. Indeed, until there are such solutions, no one can be sure such an emerging field is possible or necessary.

People from different disciplines and different areas of intervention and policy often take their basic assumptions, values, and methods for granted. It is hard for “outsiders” to enter in and this is a real problem for building new collaborative endeavors. Furthermore, people from different disciplines and domains usually debate theories or perspectives with each other. But these theories and perspectives are hard to assess without concrete and accessible examples of how they apply to specific problems and contexts.

The WEP proposes to have people show examples of how some aspect (big or small) of their ideas, theories, claims, or hypotheses work in terms that people beyond their own disciplines or domains can understand, assess, and appreciate. These examples are “worked” in the sense that they are accessibly explicated in terms of how the author thinks about them, how the author sees them fitting into his or her area of expertise, and how the author thinks they might contribute to an emerging interdisciplinary field or collaboration.

Imagine a worked example as a bid. The author is asking others: Do you think this problem and the proposed solution or approach to it I explicate here shows promise to be an important part of our emerging field or collaboration? If not, show me why not. If so, help me develop my ideas, and join yours to mine, in order to build this new field or collaboration. Worked examples are meant to show how a person’s ideas or theories actually work in concrete and accessible terms. Thus, too, they are also working examples. They are doing the work of trying out new ideas, building the field, crossing disciplinary borders, and inviting collaboration.

Each worked example is a seed. If it is not fruitful, it will die and we will have all learned that some part of the possibility space in the new field is not fruitful. If it does bear fruit, people will add related examples and families of examples and a forest will grow. We will have discovered a fruitful part of the possibility space for further exploration, elaboration, and support. We can also view each worked example as a node in a growing network of related worked examples. Connections among the nodes will grow and die in somewhat Darwinian terms. The best ideas—in the sense that others can connect to, support, and sustain them—will thrive.

Worked examples or working examples can be trial balloons. Their authors can find out if others support their ideas and want to contribute to them. They can invite other people to launch related balloons and see if something bigger, better, and more integrated can emerge. Worked/working examples are a way to map a field before it has fully formed.

WEP is meant to be a platform for starting and sustaining emerging areas of research, intervention, and practice. It is meant to be a test bed that will lead both to illuminating failures and successes. WEP is starting as a support system for the emerging field of digital media and learning, but it is a platform that could be used for any new and developing area of research, intervention, or policy.

WEP is also a potential new form of scholarship. Instead of journal papers written only for “insiders”, worked examples require their authors to show how their claims work so that a broad group of people can make decisions about how and whether they might connect to their own ideas. Micro-communities of specialists can then grow into clusters within larger, better integrated, and more useful fields, families, collaborations, projects, and enterprises with real potential for wide impact on the world.

This new form of scholarship stresses innovation, building new collaborations, and discovering new paths and not just retreading old ones. It stresses as well the search for new and better forms of shared language and methods that can truly facilitate collaboration among diverse areas of expertise to deal with complexity and risk in a fast changing global world.

We are just at the beginning. We want to share early so there is still ample time and space for others to help us improve the WEP. In that sense, this early WEP is itself a worked example or working example. We invite people to help us make our early examples and the format for worked examples better, to add to them, to accept or reject them, and to propose new and different worked examples. We encourage the use of multiple modalities beyond language and the use of rich digital media resources. But we want to stress that the focus is and should always be on clarity and accessibility of ideas rendered concrete and specific enough to engage understanding, debate, collaboration, and eventually implementation.

What are on the WEP platform now are rough sketches of worked examples. They will be improved and new and better formats will be implemented. Nonetheless, let me discuss one way people can think about what is already here.

Take Valerie Shute’s worked example “Evidence Centered Design for Dummies” as an example. Evidence Centered Design (“ECD”) is today one highly influential approach to the design of assessments and to integrating learning and assessment closely. The approach can seem abstract in theory, but it has been implemented in exciting ways in practice. In particular it has been applied fruitfully to the sorts of problem-centered learning to which new digital media have given rise. Valerie’s example seeks to explicate ECD for people who are not centered in psychometrics and for psychometricians who see assessment in much narrower and traditional terms.

The viewer of Valerie’s worked example should ask first: Do I see ECD as a promising part of the emerging field of digital media and learning? Second, the viewer should ask: What other worked examples would I like to see linked to the ECD example to create a “cloud” of examples related to each other in different and interesting ways? We can imagine that people put up examples of other approaches to assessment. We can imagine also that people put up specific examples of ECD applied in actual cases (and, indeed, Valerie has offered just such an example in her worked example on the new MacArthur Quest to Learn School in New York). We can also imagine examples put up that explicate ECD in different ways or that extend or critique Valerie’s worked example. What we will end up with is a “course”, a “cloud”, a “network” that can educate researchers, funders, policy makers and drive the development of the emerging field not just in the WEP, but in a variety of other sites and collaborations, all of which can return to the WEP to extend and deepen it over time.

In another worked example Valerie shows what it means to say a commercial game is good for learning academic like content, using the World of Goo as an example. Many have claimed that some commercial games are good for academic learning, for example in STEM areas, but Valerie makes this claim specific and accessible. Now we can imagine people putting up examples of other such commercial games and examples of non-entertainment games for academic learning, comparing and contrasting approaches to design. We will then have a growing map of such games and design principles.

Dan Schwartz has put up two worked examples arguing for two strikingly novel claims. In one case he is arguing that assessing the choices people make as they learn is a better assessment than traditional assessments of knowledge. Assessments of choices predict assessments of knowledge, as well as assessment of how well prepared learners are for future learning, while assessments of knowledge do not predict good choices or preparation for future learning. We can now imagine Dan and others putting up specific examples of choice-based assessments in various areas, comparing and contrasting them with knowledge assessments. We can also imagine a related family of worked examples of assessments for preparation for future learning related to the family of choice-based assessment examples.

Today there is a great deal of work—spurred by innovations in digital media and learning—on digital artificial tutors, artificial mentors, and artificial agents that can interact with and teach learners. Dan, in a second example, adds a new twist to this work, showing that while artificial agents work well, they work even better when the learner teaches the agent (and we assess the agent) and not the other way round. We can imagine now a family of examples on specific tutors, mentors, and agents, as well as different ways of designing and using them.

David Shaffer has put up a whole set of examples of what he calls “epistemic games”. Epistemic games (which mix virtual and real worlds) teach “content” through problem-solving tools used by professionals (e.g., urban planning, science journalism, engineering) and not under the labels of academic disciplines (e.g., biology, civics, or physics). Shaffer’s approach has been used with success with middle school students who master content, problem solving, and academic language. We can now imagine other people putting up examples of games and other learning systems that take the same sort of approach, but do so in different ways or take it in different directions. We can also imagine linking these worked examples to assessment examples and preparation for future learning examples, since epistemic games integrate assessment and learning in deep ways and appear to be useful devices for preparation for future learning. Indeed, Shaffer has partnered with Bob Mislevy and others to develop new approaches to automated assessment of natural language data and new approaches to artificial mentors.

Sasha Barab has put up a worked example of the concept of worked examples itself. His approach stresses the role of worked examples as a form of “invitational scholarship” and for collaboration. James Gee in New digital media and learning as an emerging area and "worked examples" as one way forward (Cambridge, MA: MIT Press, 2010), a MacArthur Foundation white paper, gives another example of a worked example meant to explicate the concept itself. His approach stresses exposing the structure and assumptions behind one’s argument in lucid and accessible terms. We can imagine now a variety of worked examples of the concept of worked examples itself, explicating other approaches and uses, until we make the concept of worked/working examples itself a central tool for the emerging are of digital media and learning.

James Paul Gee
Mary Lou Fulton Presidential Professor of Literacy Studies
Department of English
Arizona State University