This working example introduces the five general principles that make up Designing for Participation (DFP). Each principles is illustrated with a feature that was introduced and refined in Taiga, one of several virtual worlds in the Quest Atlantis educational videogame.
These principles were used to develop informal and formal assessments and feedback tools in annual cycles of design based refinement. Across five cycles, these led to greater engaged participation, which supported larger and larger gains in enduring understanding and external achievement.
Dan Hickey led the research which
How can we support authentic participation in inquiry that also leads to more understanding and achievement, and evidence of improvement?
This worked example describes how Designing for Participation principles were were used to refine Taiga, one of many worlds in Quest Atlantis
Here is a nice intro to Quest Atlantis.
What are you designing, creating, or researching?
We are now making design principles and articles that can help others learn from our efforts. Designing for Participation (DFP) consists of five general design principles that emerged across a years of research with Taiga and other innovations.
1. REFRAME KNOWLEDGE. THIS MEANS: transform knowledge into tool used in contexts, IN ORDER TO frame learning as using those tools appropriately in specific contexts 2. SCAFFOLD PARTICIPATION. THIS MEANS: embed reflections to foster discourse about using the tools in contexts, IN ORDER TO....help all learners pick up tools initially and start practicing to use them. 3. ASSESS REFLECTIONS. THIS MEAN: have learners reflect on their projects as evidence of engagement, rather than assessing projects directly, IN ORDER TO keep agency with learners, and minimize demands for overly specific examples, rubrics, and feedback 4. CONTROL ACCOUNTABILITY. THIS MEANS: Use individual assessments and rubrics prudently and use achievement tests discretely, IN ORDER TO... keep assessments and tests from undermining participation, while delivering trustworthy evidence 5. ITERATIVELY REFINE: THIS MEANS: continually refine activities, reflections, rubrics and assessments, IN ORDER TO...iteratively improve participation, understanding, and achievement.
This worked example shows how each of these general principles was turned into a more specific principle by designing and refining features in Taiga.
Quest Atlantis is awesome. But many schools need evidence of achievement impact in order to free up computer labs that are increasingly dominated by computer-based test preparation,
Dimension M is a dreary test prep program wrapped in a shallow virtual game. Because of one decent study showing statistically significant achievement impact, it is selling tens of thousands of copies. They have tournaments, scholarships, etc.
Taiga was designed to let 4-6 grade students learn about ecological science and socio-scientific inquiry. SSI is great for learning, because you can reduce complex human problems into concepts and skills to be memorized in an abstract classroom context. Troy Sadler is a SSI scholar who helped design the original version of Taiga
How can we use situative views of learning and assessment to create game features the stealthily enhance participation in the game while indirectly increasing individual understanding and aggregated achievement?
As design-based research, these broader questions are framed as meta-principles. The meta-principle that guides this work comes mostly from applying James Greeno's arguments about situative theories of cognition to assessment. Greeno and others have articulated a theory of cognition that says social activity is primary and that individual cognition and individual behavior are secondary. When applied to assessment, this suggests focusing directly on fostering engaged participation in shared social discourse. Assessments of individual understanding are then used to assess those activities and improve them, rather than to directly advance understanding. Achievement tests are then used to document the impact of those activities on the knowledge of whole groups of learners, and to track any improvement in this regard over time.
In the process of translating the meta-principles into game design features, a coherent set of assessment design principles emerged. In order to yield generally useful knowledge, we framed the design insights that emerged in terms of general DFP principles (that could be generally useful in most learning environments) and specific versions of those principles for educational videogames. Each year, we made principled refinements to Taiga to foster engaged participation in ecological science and socio-scientific inquiry. We looked for evidence that those refinements led students to take up more of the conceptual tools of science in their interaction with non-player characters, and in the five written quests they submitted to the park ranger, who was played by the teacher. We also made sure that those refinements resulting in larger gains in individual understanding and in overall achievement.
What have you done to get ready (planning, research, visioning, etc.)?
An assessment team collaborated with the QA design team from 2005-2010 doing refinements to Taiga to improve participation, understanding, and achievement. We started with several meta-principles about situative views of cognition. These theories treat social activity as the primary form of learning and treat individual cognition and individual behavior as secondary forms of learning. We also had some meta-principles about participatory approaches to assessment. Rather than formative assessments for individual learning, we suggested formative assessment of shared participation, which is then summatively assessed in terms of individual learning.
Most of what we have learned is bound up in the design principles above. More broadly we learned that working to refine design principles across multiple learning environments is a productive way to develop more generally useful principles
We are sharing our work by presenting these five specific principles for designing for participation in educational videogames.
1. REFRAME KNOWLEDGE. . In educational videogames, this means first identifying the knowledge that you want students to be able to use after playing the game. Then ensure players have to use that knowledge to play and succeed in the game. In Taiga, this principle is illustrated by the compelling narrative where players had to practice the concepts and skills of ecological science to help the range save the park.
2. SCAFFOLD PARTICIPATION: For educational videogames, this means helping all players "pick up" the tools and start using them initially, then provide feedback to help them use them appropriately. In Taiga, this meant providing rubrics in 2007 that the ranger/teacher could use for reviewing the reports that students submitted and revised in each of five quests.
In Taiga, these new rubrics also helped direct students to in game resources they could use to better refine their quest submissions
In videogames, it is also important to think about motivation. While the feedback we added certainly scaffolded engagement, many of the students still were not motivated to use it. In 2008, we experimented with the incentives that are common (but controversial) in educational videogames. In two of the classes we emphasized the existing incentives (e.g., special hats, extra powers) that students got for completing the important Quest. Players got special badges for their avatars based on the ranger/teacher's judgement of their quest submissions. In the other two classes we removed the incentives. We found that students in the incentive condition used more of the scientific tools in their quests and used them more appropriately. They also gained significantly more understanding, somewhat more achievement, and reported higher levels of motivation and interest in solving these kinds of problems.
3. EVALUATE REFLECTIONS. . Many educational videogames involve creating some sort of project or artifact. Assessing them directly undermines gameplay and creates huge chores for teachers. Assess reflections on those artifacts instead. In Taiga in 2009, we started having students include a reflection with each report which convinced the ranger that they had learned some important stuff. This enhanced learning and cut back on the demands on the teacher.
4. CONTROL ACCOUNTABILITY. In educational videogames this means you should avoid embedding individual assessments and tests directly into the game. Doing so ruins the game and compromizes there evidence. Use performance assessments before and after the game to refine the game (not to provide feedback to learners). Use simple pre-post achievement tests to prove impact and track improvement over time. In Taiga, we used the Lee River performance assessment which asked players to solve similar problems in a different context. We also used external test items that were aligned to the targeted standards but independent of the curriculum.
5. ITERATIVELY ALIGN. In educational videogames, this means several cycles of iterative refinement. First align the activities with the product or project players create. Then align the project with a performance assessment. And then refine the whole game to maximize achievement. In Taiga, this meant first refining the activities to help students produce good reports in their quests. Then we refined the discourse around the quests to ensure enduring understanding on the Lee River. We used the achievement test to be sure we were increasing learning overall as the Lee River kept getting refined and aligned.