Activity and Assessment Theory in the Design and Understanding of the Packet Tracer Ecosystem

Simulation environments make it possible for students to learn about and interact with complex systems. Putting these capabilities to effective use for learning and for assessing requires a conceptual framework for the knowledge, skills, and ways of thinking that are meant to be developed. It also requires physical and conceptual tools for students to interact with simulated systems, peers, and instructors; for instructors and students to create simulation tasks that are in the students’ zones of proximal development, and get feedback about their actions; and for curriculum designers to effectively design tools and the strategies for using the capabilities effectively. This worked example describes how these challenges were addressed in the context of a global educational program (Cisco Networking Academies) with a simulation tool for learning about computer networks called Packet Tracer. With PacketTracer, the instructors learn to use and create problem-solving scenarios that are at once tuned to the local needs of their student sand consistent with the epistemic frame of “thinking like a network engineer.” We describe how we use the lenses of Activity Theory and Evidence Centered Design, to understand and study the technological,epistemic and social interactions of the educational context and to better serve the students and instructors.

The Cisco Networking Academy (CNA) program is a collaborative effort between Cisco and more than 9,000 educational institutions and partnering organizations in over 160 countries. The goal of the program is to promote the learning of knowledge and skills associated with entry- and mid-level computing and computer networking (Levy and Murnane 2005). To support this effort Cisco provides a number of no-cost tools to schools including online curriculum, online assessment (Behrens, Collison, and DeMark 2005), and Packet Tracer software (Frezzo, Behrens, and Mislevy, in press). Packet Tracer (PT)is a comprehensive simulation, visualization, assessment, and micro-world authoring tool for teaching and learning networking concepts and skills. Packet Tracer is designed as part of an instructional ecosystem grounded on conceptualizations from many fields, including learning sciences, instructional design, assessment design, and cognitive and sociocultural psychology. Following is a worked example of how these theoretical advances have been coordinated in the Packet Tracer ecosystem, with illustrations from real-world usage. The lens of Activity Theory (AT) is used to bring out key relationships at the levels of students, instructors,simulation designers, and instructional developers. Because ongoing feedback on performance in real-world problem-solving is a necessary aspect of the instructional progression, the lens of Evidence Centered Design (ECD) is also introduced as a central conceptual lens for understanding internal technological aspects of the software and pedagogical design. However, it is in the combination of these multiple perspectives that we have found valuable synergy.

The first section provides a brief introduction to the CNA context and how Packet Tracer is used. The next section reviews terminology from activity theory that the discussion draws upon. Attention then turns to the Packet Tracer ecosystem, addressing the interplay among students, tools, and tasks; among instructors, students, tools, and tasks; and among curriculum designers, instructors, students, tools,and tasks. The Evidence Centered Design assessment framework is introduced, and we discuss how the intersection of AT with ECD leads to new insights into activities in the global e-learning system.Throughout the discussion, we note how structuring the activities of instructors and designers within the ecosystem is integral to supporting the students’ learning activities.

Students enter CNA classes to acquire skills that will prepare them to undertake basic computer network design, configuration, and troubleshooting. Surveys completed during the course progressions indicate that approximately one third of students plan to be networking professionals, one third plan to be other types of informational technology (IT) professionals, and one third do not plan IT-related careers. The Cisco Networking Academy program consists of over 9,000 secondary and post-secondary schools of various types including American high schools and community colleges,European and Asian technical universities, and small NGO-led training centers in such locations as prisons or even mobile buses. While most of the courses focus on introductory networking, there is a very popular course called IT Essentials that focuses on computer maintenance and repair.

Data networks provide the mechanisms to move data from one location to another, whether the locations are general computers (e.g.,laptops), phones, video conference systems, or unseen databases providing information on flight status, tax status, or other e-commerce. Data networks range in complexity from single routing devices that might create a wireless local area network (LAN) in one’s home to a large collection of devices that route computer data and voice traffic across a campus, city, or province. To be proficient in this domain, students must have a combination of conceptual understanding of how the systems work (to aid in troubleshooting and other reasoning-based activities), practice with the physical aspects of network connectivity (care and organization of cables, alignment of hardware in physical spaces), and cognitive automaticity for common device configuration commands in the Cisco IOS (originally Internetwork Operating System). As data networks increase in the number and geographic dispersion of devices (perhaps on every floor of every building on a college campus, or spread across a province), the mental models networking professionals need in order to work with these systems become similarly complex. As part of the quality assurance agreement to become a Cisco Networking Academy, every school agrees to have hands-on equipment in a classroom. Hands-on labs of varying complexity are made available with the curriculum to promote the use of this equipment.

While the presence of real equipment in class provides a number of important benefits― including authenticity of interaction with real equipment, sensory feedback when working with cables and plugs, and an ability to experiment with different network configurations―a number of limitations remain in most installations. First, while the equipment is offered at highly discounted rates, most schools can only afford equipment for a few students to use at one time. Because the student-to-network ratio is typically high, students are often rotated through use of equipment, thereby adding logistic difficulties to the classroom environment and possibly causing interruptions during optimal learning sequences. Second, while such an arrangement provides high fidelity with regard to small networks,the behavior and complexity of larger networks cannot be experienced by the students under such circumstances. Third, while real equipment provides strong support for learning field-based procedural skills,these devices purposively hide representations of underlying data structures, processes, and transformations, which are important for individuals to understand “what is going on in the box.”

To address these needs, D. Frezzo and his colleagues have sought to extend the work of Snir, Smith, and Grosslight (1995), who argued in the context of science education that “a new kind of computer simulation is called for, one that allows students to perceive what cannot be directly observed in laboratory experiments.… We will call such simulations ‘conceptually enhanced simulations’” (p.107). Their work likewise took hands-on lab activities as a starting point for understanding what was possible for simulation-based work.They emphasized that “a first crucial factor in designing activities concerns integrating computer models with laboratory experiences” (p. 125).

Packet Tracer can be thought of as providing instructional and assessment services at a number of levels (Frezzo, Behrens, and Mislevy, in press). At the most fundamental level, PT provides a comprehensive network simulation covering a wide variety of networking devices, means of connecting equipment, end devices, and the underlying Cisco IOS. A high degree of proficiency with the Cisco IOS is required for effective planning, configuration, and troubleshooting of networking devices. The behavior of a wide range of protocols is simulated, including the core protocols of the Internet, TCP and IP, the ubiquitous HTTP, and a wide variety of switching and routing protocols. These capabilities provide an extensive space for practice and exploration of networking concepts and procedures. The networking micro-world is simulated with sufficient detail to support both the authenticity required for free-play practice and exploration and the complex visualizations at the next level, called the Interaction Interface level. At this level, interaction is provided with the simulation engine. Pt provides a graphical interface that allows the designing and building of networks by simple drag-and-drop functions, along with visualization interfaces to help learners, in part by giving them control to visualize concepts that are difficult to understand and processes that are hidden and fleeting (cf. Hundhausen 2002;Hundhausen, Douglas, and Stasko 2002). Figure 1 provides a screen-shot of a Packet Tracer session. The simulation layer is hidden but associated with the icons that represent intermediary networking devices (e.g., switches and routers) and end-user devices(e.g., computers and IP phones) on the network. The network represented in the workspace (upper left) is comprehensively simulated. Visualization of data traffic is seen both in the workspace and in the event examination window on the upper right.Progress of network events can be controlled one networking event at a time (capture/forward), through continuous animation (autocapture/play) or allowed to progress in real-time. Additional detail scan be visualized by clicking on events in the event examination area.

Preconfigured micro-worlds can be created, saved, and distributed in a format called an “activity file” that have their own specific extension (“.pka”). This is the method typically used for creating assessments and games. Simpler .PKT files can be used to save and share network configurations without additional pedagogical affordances available at upper layers of the system. In figure 2,the reader can see how a PT activity file appears to the student.Notable features in this example include: the already partially functioning network (green dots indicating functioning network links;red dots indicating links that require student configuration) and the instructions window (with title, objectives, elapsed timer at 36seconds and counting, completion at 0%).

Figure 1 Packet Tracer software in use to visualize control data flow.

Figure 2 Packet Tracer “Activity” File, view of network and instructions as they appear to the student.

How was such a scaffolded activity authored? Authoring control via anActivity Wizard is shown in figure 3: On the left side are its primary functions, such as Variable Manager, Instructions, Answer Network, Initial Network, Password, Test Activity, and Check Activity. At what can conceptually be considered a third layer,beyond the simulation and visualization layers, these authoring features are available to create initial and answer networks, save and lock networks, and provide semantic context-setting information such as simple instructions, stories, or graphics for games,explorations, or assessments. Also in figures 1 and 2, the reader can see network construction palettes provided in the lower left of the screen. As we will discuss below, support in the form of design patterns for task authors and instructors has proven very important in the Packet Tracer ecosystem to help provide students with opportunities to improve their understanding. Figure 4 illustrates in-process feedback provided to students after they have completed part of the exercise that was authored (as shown in figure 3) and presented (as shown in figure 2); this feedback supplements the immediate feedback students receive from the network model as they work through a problem interactively.

Figure 3 Authoring interface, including control of an answer network to choose aspects of the student’s network to be identified and scored for feedback.

Figure 4 Feedback from a partially completed performance showing which aspects of a network have been configured correctly.

These affordances are provided at the fourth and higher conceptual levels to structure scoring and feedback that can likewise serve gaming or assessment functions (Behrens et al. 2007). The Activity Wizard authoring interfaces allow instructors and students to create their own network mini-worlds (in their local natural language) and control the features described above. Three animations are provided in Appendix A to provide additional rich-media examples of the use and features of the PT tool.

The on-line curriculum used in the CNA provides a highly interactive,though highly structured, book-like progression through the course content. As the student progresses through a chapter, Packet Tracer activities are embedded in the curriculum interface. When the student clicks on a page, the .pka file containing the activity definition is launched with the Packet Tracer software and the student is given the opportunity to interact in the micro-world. The activities embedded within the instructional sequence in this manner are tailored to the concepts and skills that are the focus of learning at that point.Packet Tracer also includes an inter-instance communication system that allows users to connect their Packet Tracer micro-worlds with other running instances to support collaborative and competitive instruction, exploration, gaming, and assessment.

Packet Tracer activities fall into a number of broad categories including:

• Exploratory, self-directed learning opportunities

• Demonstrative activities that illustrate important concepts

• Structured practice activities to guide learning

• Assessment opportunities to gauge progress and provide feedback

Design Patterns for Packet Tracer activities are an active area of research for our team. Packet Tracer is thus a user-extensible and configurable micro-world authoring and creation tool that promotes instructional enablement and local customization. This is accomplished by considering cognitive interactions with the domain and promoting design patterns for both creating scenarios and developing authoring tools that promote patterned-based thinking. The patterns, as discussed in following sections, revolve around design,functioning, implementation, and troubleshooting, in the physical,logical, and social domains of computer networking.

Human knowledge and capabilities revolve around interacting with the world and other people, that is, physical and social situations, in useful ways. We are able to do this by becoming attuned to the patterns that characterize situations, and the ways we can act within them and create them. We develop proficiency with these patterns mainly by using them, typically in supported settings―learning elements of them, expanding the range and complexity of our capabilities, combining them in new ways, and seeing how we can use them to achieve our goals. In broad strokes, we develop these capabilities in much the same way when we acquire a new language,become proficient in golf, or learn to troubleshoot computer networks. In each case we see the interplay of individual cognition and interpersonal knowledge, mediated by tools both physical and conceptual, in cycles of acting, reacting, and learning.

Activity Theory (Engeström 1987) provides a unifying framework for anlayzing these interpenetrating elements of human activity. This section provides a brief sketch of the key elements of applied AT, an approach that can be used to analyze many kinds of activities at different scales or of different durations. The topic is too vast to do it justice here in this short presentation, so we focus on ideas that are particularly helpful in thinking about design and learning in the Packet Tracer ecosystem (Frezzo 2009).

We chose activity theory as a conceptual framework because we perceived that it provided an inclusive description of the classroom learning environment for which we were designing software and the activities for its use. Four aspects of activity theory particularly help illuminate the use of Packet Tracer in networking classrooms.Kaptelinin and Nardi (2006, pp. 269-278) used these in the Activity Checklist. In the Preamble (2006, p. 271), they emphasize: (a)“Means/ends (hierarchical structure of activity)” where motive relates to the activity (highest) level, goals relate to the action(middle) level, and conditions relate to the operation (lowest)level. This hierarchy may be used to analyze the flow and breakdowns involved in using software; (b) “Environment(object-orientedness),” which we felt was best described by graphical descriptions (such as Cole and Engeström 1993, p.36) of a subject transforming an object to an outcome through the mediation of artifacts, with rules, community, and division of labor, providing an integrated framework for describing classroom learning environments;(c) “Learning/cognition/articulation (externalization/internalization),” where internalization and externalization processes may be significant in simulation and visualization software activities; and (d) “Development,” where activity theory provides a framework for evolution in practices such as the usage of software tools and hence could provide insight into the evolution of student learning processes. Our intention was to use activity theory as both a lens for description and as a tool for analysis of cases of student interactions involving Packet Tracer software. The Constructing Networks of Action-Relevant Episodes (CN-ARE) methodology (Barab,Hay, and Yamagata-Lynch 2001) helped to operationalize these aspects of activity theory and has been useful in our work.

An important representation in activity theory is built out of so-called mediational triangles. The triangles are formed by bi-directional arrows representing interactions, such as derived from Vygotsky’s concept of mediated activity (Vygotsky 1978, p. 40), and cultural psychology (Cole 1998, p. 119) or as developed further by Engeström (1999, p. 31). Figure 5 is based on the earlier Vygotsky-based representation and also uses terminology from Barab,Hay, and Yamagata-Lynch (2001, p. 75), such as initiator,participant, resource, issue at hand, and practice, which are useful for studies of student use of simulation environments like PacketTracer. Bidirectional arrows between components or elements of the activity system indicate mutual influence, and triangles formed by three elements indicate mediational relationships. For example, in this scheme one might observe a student (the initiator, or subject)doing homework (the practice) on a network modeling problem (the issue at hand, or object, which they are trying to transform into an outcome, such as a working model) using Packet Tracer software (the resource, or mediating artifact).

Using a cultural-historical approach to distributed cognition, Cole and Engeström (1993) describe the basic mediational triangle of Russian activity theory suggested in figure 5, where the subject has two routes to act on the object, a direct phylogenetic route and a second, indirect, artifact-mediated, cultural route. From this basic triangle their explanation takes two directions. First, the intrinsic bidirectionality and codetermination of the legs of the triangle are emphasized. For example, the subject acts on the object, but an image of the object is part of what determines the subject. The dynamic nature of the activity triangle in time is also emphasized: the mutually constitutive, subject-mediator-object relations described by the triangle, far from being a static structure, often lead to a new subject, or more precisely, a new condition of the subject, at a later time.

Figure 5 highlights a number of core ideas in AT. First, while some traditions in psychology may interpret the diagram (and the world) as comprising the exchange between subjects and objects separately, AT argues that the understanding of the subject and object cannot be made separately because it is in the interaction of the two that appropriate meaning is made and held (Kaptelinin and Nardi 2006).Moreover, the subject is defined by historical, intrapersonal and interpersonal forces and needs, leading to an emphasis on the situated understanding of real life interchanges, rather than a focus on theoretical internal trait structures. Another key notion the diagram represents is that subject-object activity is largely mediated by cognitive artifacts. Engeström (1999) wrote:

Mediation by tools and signs is not merely a psychological idea. It is an idea that breaks down the Cartesian walls that isolate the individual mind from the culture and the society… The idea is that humans can control their own behavior―not “from the inside,” on the basis of biological urges, but “from the outside,” using and creating artifacts. This perspective…. is an invitation to serious study of artifacts as integral and inseparable components of human functioning. (p. 29)

In 1990, Engeström (1990) extended the basic AT diagram to include some of the social-historical forces that affect the activity. Figure6, after Engeström (1999, p 31), Barab, Barnett, Yamagata-Lynch,Squire, and Keating (2002, p. 79), and again including terminology from Barab, Hay, and Yamagata-Lynch (2001, p. 75), incorporates these extensions, as adapted to the present discussion, This extended analysis provides a richer framework and one that has been more easily applied in such contexts as human-computer interaction (e.g.Kaptelinin and Nardi 2006). The basic mediational triangle has been extended to include subject, object, mediating artifact, outcome,rules, community, and division of labor, to more fully describe the nature of human activity systems. Cole and Engeström (1993) conclude that the full activity system triangle diagram (along the lines of figure 6) “provides a conceptual map to the major loci among which human cognition is distributed” (p. 8). They continue:

Consequently, activity systems are best viewed as complex formations in which equilibrium is an exception and tensions, disturbances, and local innovations are the rule and the engine of change. When an activity system is followed through time, qualitative overall transformations may also be found. (pp. 8–9)

Jonassen and Rohrer-Murphy (1999, pp. 70-77) advocate activity theory as a framework for designing constructivist learning environments(CLEs) such as PT. They identify six steps for how activity theory may be used to design a CLE. The steps are: “Step One: Clarify Purpose of Activity System” (p. 70); “Step Two: Analyze the Activity System” (p. 71); “Step Three: Analyze the Activity Structure” (p.73); “Step Four: Analyze Tools and Mediators”(p.74); “Step Five: Analyzing the Context” (p. 75); and “Step Six: Analyze Activity System Dynamics” (p. 77).

Jonassen (2000) summarizes the rationale for applying activity theory to learning environments:

Activity theory provides a different lens for analyzing learning processes and outcomes for the purpose of designing instruction.Rather than focusing on knowledge states, it focuses on the activities in which people are engaged, the nature of the tools they use in those activities, the social and contextual relationships amongst the collaborators in those activities, the goals and intentions for those activities, and the objects or outcomes of those activities. (p. 109)

In contrast, traditional cognitive psychology perspectives typically focus more on internal knowledge structures and processes within the subject. Interaction with a computer network directly or with other people in situations involving computer networks, however, highlights the social character of this activity. Individual action is situated in the community, mediated through the physical (hardware and software) tools and, more importantly for the purposes of developing proficiency in the domain, the conceptual tools. What are the concepts and representations that have been developed to make sense of networking? How do they relate to the physical networks? How do people use them to acquire, understand, communicate, and create information?

These tools, as reflected in the top vertex, are targets of learning,but not the only ones; to be useful, they must be brought to bear on problems and goals in situations, of the objects indicated on the right side of the triangle. Purposeful action therefore require the rules shown in the lower right of figure 6, which are the patterns of interacting with people and situations, mediated through the tools―ways of effectively assembling tools and concepts to the evolving situation at hand. This is the AT counterpart of what expertise researchers call “actionable” organizations of knowledge, with respect to recognizing features of unique situations and adapting strategies and procedures to them (Chi, Farr, and Glaser1988).

What are the kinds and sequences of experiences, then, that are optimal for students to develop proficiency in networking? Productive tensions are most likely to occur within the learner’s Zone of Proximal Development (ZPD; Vygotsky’s term for what a learner can do with assistance). At the instructor level, we need to know how to suggest situations and purposes that are in the zone. At the curriculum level, we need to understand progressions of learning,which can be related to a view of progression of what is demanded in situations. At the simulator designer level, we must be able to create such situations and provide feedback, and also provide visualizations that move concepts ahead so as to make emerging knowledge structures actionable, that is, connected with features of situations, affordances of tools, and available actions. As discussed below, curriculum developers in the PT ecosystem create broad design patterns for design under constraints and network troubleshooting to support the community of instructors, as well as more detailed task templates that instructors can revise, extend, or tailor to local needs (Frezzo, Behrens, and Mislevy, in press).

While work on distributed cognition, such as the view of distributed cognition in Hutchins’s (1995) Cognition in the Wild has been of value; the greater methodological specificity found inactivity theory and the call for unification of approaches among a number of writers led to a greater focus on AT as a comprehensive lens. We infer that Cole and Engeström (1993), Engeström (1999),and Jonassen (2000) all argued that one way to map the distribution of cognition is through activity theory.

As Engeström (1999) emphasizes, the core of activity theory is its focus on mediation, namely mediation of subject-object relations by resources (tools), rules, communities, and divisions of labor.Jonassen (2000) emphasizes the utility of activity theory for designing constructivist learning environments. Roth (2004)emphasizes the potential of activity theory for studying classroom interactions. Barab et al. (2002) use activity theory to guide a series of studies of the introduction of simulation software into astronomy classrooms, a research problem very similar to the introduction of PT into networking classrooms. Constantine and Lockwood (1999) described a usage-centered approach to designing usable software that was influential on the Packet Tracer development team; Constantine (2007) extended the usage-centered design approach using activity theory. Sharp, Rogers, and Preece (2007) present activity theory as one approach to both data analysis and field studies in interaction design.

An activity system that generally describes CNA classrooms around the world is depicted in figure 7. A variety of resources are available to mediate the network modeling and problem-solving issues at hand.One common configuration of this classroom activity system is a didactic, teacher-centered teaching style expressed through classroom rules, “traditional” division of labor, instructor-as-initiator,and lecture-as-practice.

Figure 7 Generalized CNA Classroom Activity System.

Research on the introduction of virtual reality simulations into astronomy classrooms (Barab et al., 2002) suggests that tensions and innovations, both within and between activity system components can be useful for describing, analyzing, and improving instruction. For example, they concluded that by identifying classroom tensions, such as student-directedness versus teacher-directedness, instructors could better manage them.

By specifying some of the elements that constitute an activity system and possible mechanisms by which they interact, figure 8 illustrates some of the possible opportunities and tensions in the CNA classroom activity systems, namely the factors impacting a student’s subjective learning experience. Using the CN-ARE methodology described by Barab, Hay, and Yamagata-Lynch (2001), transcribed and chunked “learning episode” data may be summarized through sequences of activity system contextual diagrams (Barab et al.,2002). In figure 8, tensions within activity system elements are indicated by “Vs.” One example is a tension within the issue at hand, where talking about how to use Packet Tracer as a tool for model-building becomes dominant (not desirable) versus making sense of the domain of networking (a more instructionally desirable issue at hand). Contradictions or tensions between activity system elements are indicated by lightning-shaped arrows (Cole and Engeström 1993, p. 36; Engeström 1999, p. 31; Barab et al. 2002, p.103). Shown in figure 8 are tensions between (a) the constructivist pedagogical assumptions of the software as a resource versus a more didactic organization of the class (which could also be seen as residing in the rules component) and (b) the open-endedness of the software’s affordances for novel network building and the lack of support for open-endedness in a given lab assignment (students replay a pre-written assignment).

Figure 8 Possible systemic tensions in CNA classrooms. Tensions within activity system elements indicated as “Vs.” and between activity system components indicated as “a”and “b.”

In approaching the goal of promoting knowledge and skill acquisition, At reminds us that the learners we wish to support are involved in asocially constructed activity (learning) often in highly structured and regulated environments in order to develop new competencies to undertake activities in other socially constructed environments(workplaces) about which they will have even less control and perhaps social foreknowledge. It is our hope that the skills they acquire in our classes provide them an advantage to negotiate the social/interactional complexities ahead of them.

While the activity diagram is useful to help articulate the broader social-symbolic system in which activity occurs, it must be remembered that each individual diagram typically represents the activity for individuals in a single role. In the activity of a day,or a small part of a day, multiple roles interact within and among individuals suggesting that multiple diagrams should be constructed and their interactions examined. For our work, most of the activities need to be considered in terms of the student, the instructor, and the designer. Outcomes from the designer level become inputs to the instructor and student levels. These representations force us to ask: Are the designers sufficiently aware of the rules that guide student and instructor use? Are the mediating artifacts for the designers a help or a hindrance to moving these individuals toward the instructional goals? For example, if the bulk of software, guidelines, training, and rules are oriented toward fixed-response/multiple choice assessment, how will those conceptualizations be brought over and impact the understanding and possibilities offered in the instructional support systems?

To help structure the interactions, the curriculum is designed around a number of activity patterns organized by type of KSA (i.e.,knowledge, skill, or ability) or degree of instructional scaffolding appropriate for that stage of skill development (Frezzo, Behrens, and Mislevy, in press). In terms of KSAs, different design patterns have been established for concept builders, skill builders, design challenges, and troubleshooting. Concept builders focus on addressing conceptual knowledge and may be organized as passive demonstration or require active participation and problem solving. Skill builders focus on practice and development of isolated procedural skills. Design challenges deal with network design issues and are often open ended in their outcome. Troubleshooting tasks are also free-response tasks that require knowledge from a broad range of the curricular scope.

Earlier versions of the curriculum (which did not include embedded PacketTracer activities) were subject to criticism that students were gaining knowledge or skills local to a particular chapter, but not receiving practice on “putting it all together” and thereby didn't generalize their skill from curricular to capstone assessments.Accordingly, activity redesign in the current courses includes 1)embedded activities focusing on the scope of a chapter and providing formative feedback, 2) lab analogs that follow hands-on labs and provide extra practice before or after using hands-on real equipment,and 3) skills integration challenge labs, which provide activities that require skill integration across skills taught in several chapters or the curriculum cumulatively.

These patterns provide a core library for the students and instructors to draw from. In the current implementation, our curriculum authors(instructional designers) distil their knowledge in the form of .pkafiles. Instructors, who mediate all Cisco Network Academy instruction(Cisco does not provide instruction; all courses are offered by local instructors), make decisions to present the coursework, including the sections of the curriculum, for example calling attention to complete a particular .pka activity. The student then interacts, alone,coached, and/or in groups, with the Packet Tracer .pka file and perhaps other people. At least three activity systems,student-centered, instructor-centered, and designer-centered, are interacting. There are many variations of these interacting activity systems under consideration. To cite but two examples, we are interested in designing an adaptive testing and tutoring system base don progressions of Packet Tracer activities for students to use, and we are encouraging a community of practice around instructor-designed locally customized Packet Tracer tasks for students. AT provides us aframework for optimizing the mediational processes at work in the designer, instructor, and student activity systems.

A visualization of this interaction is depicted in figure 9. By considering the alignment and interactions of multiple activity systems graphically, it may suggest connections between outputs atone layer and inputs at another.

Packet Tracer and the other tools provided to instructors and students have been designed with recognition that for many schools,the instructors are acquiring the knowledge, skills, and abilities along with the students, or barely ahead of them. In these situations a traditional pedagogical approach is often observed because the instructor is unlikely to have time to innovate both in their internal knowledge acquisition and in their pedagogical interaction. To accommodate these schools the curriculum is comprehensive and the instructor role between designers and students is likely to provide relatively little transformation in knowledge acquisition. However,even here, instructors often improvise by compensating with field trips, internships, and guest instructors. It must be remembered that even our three interacting activity systems sit within a larger set of activities including real-world internships and other experiences that provide important activities to transition into the space of job activity patterns. At the same time, however, there are many instructors who have the facility to extend and personalize or localize the tools given from the design layer, and for them, the availability of authoring and distribution capabilities of PacketTracer are greatly welcome.

Activity theory has been applied to extend understandings of PacketTracer and its uses in the Networking Academy in a number of ways. First, a series of in-class observation sessions (Frezzo 2009) have been conducted in which the interaction of the student, PacketTracer, the instructors and other artifacts have been understood and communicated in terms of AT. For example, in one observation, students working as an informal team to answer a multiple-choice quiz questions were seen involved in cycles of blockage, exhilaration, and epiphany as they generate theories, find empirical evidence against their theories and develop appropriate re-conceptualizations in light of their interactive dialog with each other and software. These often repeated cycles of failure and success that occur as students work through problems in the micro-world can be conceptualized as “impasse driven learning” (VanLehn 1990) or motivation by failure(Schank 2001). We have heard from students that “PT does not lie” by which we take it they mean their theories are testable against PT. More sophisticated users know that PT does lie as all simulations take short cuts in appropriate areas.

Second, Activity Theory has been used to shape our discourse with partnering educational personnel. For example, in several recent on-line or in person events, instructors from our partnering schools were given a templates of activity theory components and asked to write current state and desired future state scenarios describing student interaction with Packet Tracer an other educational affordances. These transcripts were then used as the basis for additional dialog in the face-to-face context, or as written inputs for subsequent analysis and planning. Participants found this extremely engaging and rated their satisfaction with this activity an average of 4.5 on a five point scale. The structure of the activity theory elements helped the instructors organize their mental model of the nature of the classroom.

A third use of Activity theory has been a valuable resource in focusing dialogs among the different members of the internal development teams. Because of the heterogeneity of the groups with varied backgrounds in education, psychology, assessment, software development and business, an activity theory and corresponding interaction design focus on understanding the educational needs has helped unify and focus discussion. The common language and broad perspective have helped re-center the discussions from single elements of the ecosystem (e.g. interface optimization for short term memory load) to the interaction of the many pieces of the educational ecosystem.

This combination of in-class observation, instructor queries, and team focus has led to significant improvements in the use of PT. This has occurred both by taking the personal experience of the student very seriously at the interface level, as well as broadening the conceptualization of how PT is used in the classroom and what kind of affordances are necessary to support a broad range of possible uses. For example, prior to embedding PT directly into the curricular software, PT was adopted only by instructional innovators. However, by embedding the software directly in the curriculum shell,the “hands on” aspects of PT have wide use. Now the technology innovators focus on customization while the technology followers implement the many pre-constructed activities embedded throughout the curriculum.

Inclusion of assessment tools has always been a hallmark of the CNA(Behrens, Collison, and DeMark 2005). While Packet Tracer is the most flexible and extensive performance-based system available for our assessment at present, a number of experiments and prototypes served as assessment forerunners including the NetPass system (Behrens et al. 2004; Williamson et al. 2004; Williamson, Mislevy, and Bejar2006) and a simulation-based assessment system called the Cisco Network Simulator. From the inception of the program, use of Evidence Centered Design as a theoretical framework was a key determinant of our ability to innovate and expand into performance assessment. ECD provides a descriptive framework that has broad applicability because it breaks the assessment processes down into component parts that are generalizable across a broad range of activities.

Consider,for example, how traditional language of assessment constricts conceptualizations of simulation-based or performance-based assessment. In a multiple-choice task the task is designed (and constrained) to have a question, a response, and an answer. Scoring is almost forgotten by assessment designers as it is built in, and the construct of “correctness” can often hide more complex decisions that need to be made about “correctness of what?” and the purpose of having such correctness in a larger assessment design.However, when faced with a field-based or performance-based task, the traditional language hits roadblocks. If I meet a student and say,“please troubleshoot this network so my computer can interact with the web page,” this seems a natural and appropriate assessment activity. However one is at pains to identify what is the question for the task, what is to be considered the response, what is considered the answer, and what rules would lead to a score.Unfortunately these are constructs that evolved around a particular format of assessment and taken as a general language. ECD starts with the goal of creating a generalized language and applies it to different assessment activities. In this section, we note how representations and affordances of ECD can be viewed through the At lens, as helping instructors and assessment designers develop,deliver, and evaluate tasks in the PT environment that support student learning.

While there are a number of valuable conceptual frames in ECD, the one that is most relevant to the user experience considers the language for assessment delivery, namely the Four Process Model (Almond,Steinberg, and Mislevy 2002). The four processes are depicted in figure 10. They consist of 1) Activity Selection, 2) Presentation, 3)Evidence Identification, and 4) Evidence Accumulation. The social aspects of reporting and communication are not called out in detail but can flow from Evidence Identification for detailed feedback or from Evidence Accumulation for more aggregated feedback and reporting.

The key to the ECD model is the breadth of descriptive use. The notion that all assessments (and not only tests) start with a decision on how to interact with the examinee is represented by the Activity Selection process. The presentation process considers the interaction the examinee has in an assessment context and describes the artifacts left behind as the work product―whether that is the answer to a multiple choice question or the electronic log of the activity a student undertook for network troubleshooting. That is, the outcome of the students’ interaction with a task, the object of their activity, is the object of the “activity of evaluation” at the level of an instructor or an automated scoring routine. Evidence Identification calls out the process of creating rules that identify key parts of the work product and characterize them. The characterization can be in the dimension of correctness (correct or incorrect) or could be more transparent: “This response indicates you use strategy A.” The outputs from this feature identification process are called observables, which are aggregated by whatever evidence accumulation model is used. Observables can exist in many forms. In PT, they consist of a list of binary flags indicating the presence or absence of target features in the work product (e.g.device A could communicate with device B: yes/no), or composite combinations of such primary observables that indicate higher-level properties of a solution, such as connectivity or expandability. The rules for finding and evaluating these features are specified through the Activity Wizard.

The most common evidence accumulation model is “add them up,”though more subtle models are available including factor weights, IRT weights, or Bayesian networks. This cycle is repeated over multiple tasks or task components until the activity selection process ends the assessment. The outcome of the evaluation activity (evidence identification) thus contributes to the object at the next steps in the cycle, namely to update overall impressions and decide how shape further learning or instructional activity of those actors. This could, for example, take the form of an instructor ending the“looking over your shoulder” assessment (activity selection)after observing the student completing a laboratory activity(presentation) in a manner that matched her mental model of proficiency (evidence identification) and combing her impressions with previous impressions (evidence accumulation) to make a new impression of competence. These types of informal assessments occur in real time in classrooms. We have found ECD to be of value because the flexibility of the language captures these in-class dynamics as well as proto-typical “standardized assessment.”

Figure10 Schematic diagram of the Four Process model for assessment delivery systems (after Almond, Steinberg, and Mislevy 2002) .

While ECD has always encouraged users to consider the larger cultural and implementation effects, there has not been a comprehensive framework to establish the needed connection. However, if we consider the delivery elements of ECD as activities, in the AT sense, we open the door to locating those activities in the context of a number of systems in which assessment is embedded. Specifically, the current CNA use of PT for formative assessment may be examined.

Let us consider two ways in which the outcomes of a PT activity exercise are reported and how we might understand this “obvious” or “trivial”assessment activity from an AT standpoint. First, when instructors create PT tasks, they create, using the Activity Wizard, those characteristics of a network in the simulation environment that they want students to interact with, be they in the domain of design,implementation, or troubleshooting. They can use design patterns to create these tasks, to choose the devices, configurations, and requirements that are at an appropriate level for the student or students who will be interacting with them. The Activity Wizard authoring interface builds, behind the scenes, automated scoring routines that will be used to evaluate the students’ work products.This ensures that as the assessment delivery cycle proceeds, student swill receive feedback on the ways in which their solutions were successful or not immediately at the time of activity completion, in a level of detail deemed appropriate by the activity author (which may be a Cisco designer, the local instructor, a distant instructor,or another student). However, it is not merely the feedback from the Pt environment that is thought to suffice for learning. Rather, it is the sense-making activity that this feedback engenders in individuals, among students, or between students and mentors, that promotes further learning.

Now consider the design of a traditional assessment reporting system. In the CNA this is accomplished by an electronic gradebook that exists in both student and instructor versions. To facilitate migration from paper-based gradebooks to computing-based gradebooks, the CNA gradebook looks like a traditional one with the exception that instructors can drill down in the book to look at exam patterns,individual student patterns, and cross-student patterns of response.The text of any question and its scoring rules can be brought up when looking at individual questions. Starting in the fall of 2009, PacketTracer activities are being integrated with the “standardized”gradebook system so Packet Tracer-based examinations can be given on the local machine, and the data sent back to the centralized gradebook for psychometric analysis, research, and individual performance feedback and reporting.

As expressed in figure 11, the implementation is not simply a technical advance;it represents a new kind of activity in the educational ecosystem.What are the norms, rules, and communities that are brought to bear on reporting a student evaluation? In the United States, assessment is considered a personal activity and the sharing of personal performance data by the instructor with other students is considered a crime. Other countries (e.g. Sweden), on the other hand, require high levels of transparency including the publication of student grades. This likely reflects not only differences in views of privacy and the nature of data, but also the corporate versus personal understanding of the educational process.

Figure 11 Activity System Diagram reflecting tensions in reporting simulation-based feedback in an assessment context.

The role of rich performance data and transparency of evidence identification(scoring) rules also provides tensions across communities of instructors. When simulation-based tasks have been used in the past an important tension emerged between the automated scoring of the tool and the instructor’s need for a detailed mental model of the student’s actions. Instructors like the fact that tools such as Pt provide automated scoring, but were concerned that they were “left out” of the process and lost their sense of ownership. Current automated assessment design emphasizes communicating the scoring rules to the instructors in text, as well as providing the original work product (student log files and final configuration files)through the gradebook as well. Activity Theory reminds us that the small “Summary Feedback” process in the ECD diagram is highly contextualized and is mediated by country norms and rules as well as pedagogical frames and instructors’ understanding of the instructional activity.

Recent years have seen rapid progress in fields that are central to learning and assessment, including assessment design and instructional design, simulation capabilities, collaborative spaces for distributed and collaborative learning, and learning sciences as cast broadly to include cognitive, sociocultural, and situative contributions. The challenge is how to put these advances to work practically. The Packet Tracer ecosystem that is developing in the Cisco Networking Academy has provided a real-world laboratory to put the ideas to work. This presentation has described the capabilities of the PT environment, and shown how we have found activity theory as a powerful lens to study and guide its development to the end of improved student learning. Moreover, AT provides synergistic opportunity when the framework is applied with other theoretical lenses such as ECD.

Barab, S. A., M. Barnett, L. C. Yamagata-Lynch, K. Squire, and T.Keating. 2002. Using Activity Theory to understand the systemic tensions characterizing a technology-rich introductory astronomy course. Mind, Culture, and Activity 9 (2): 76–107.

Barab, S. A., K. E. Hay, and L. C. Yamagata-Lynch. 2001. Constructing networks of action-relevant episodes: An in situ research methodology. The Journal of the Learning Sciences 10 (1&2):63–112.

Behrens, J. T., T. A. Collison, and S. F. DeMark. 2005. The seven Cs of comprehensive assessment: Lessons learned from 40 million classroom exams in the Cisco Networking Academy Program. In Online assessment and measurement: Case studies in higher education, K-12and corporate, edited by S. Howell and M. Hricko, 229–245.Hershey, PA.: Information Science Publishing.

Behrens, J. T., D. C. Frezzo, R. J. Mislevy, M. Kroopnick, and D.Wise. 2007. Structural, functional, and semiotic symmetries in simulation-based games and assessments. In Assessment of problem solving using simulations, edited by E. Baker, J. Dickieson, W.Wulfeck, and H. F. O'Neil, 59–80. New York: Earlbaum.

Behrens, J. T., R. J. Mislevy, M. Bauer, D. M. Williamson, and R.Levy. 2004. Introduction to Evidence Centered Design and lessons learned from its application in a Global E-Learning program.International Journal of Testing 4 (4): 295–301.

Chi, M. T. H., M. Farr, and R. Glaser. 1988. Nature of expertise,1st ed.. Mahwah, NJ: Lawrence Erlbaum.

Cole, M. 1998. Cultural psychology: A once and future discipline.Cambridge, MA: Belknap Press.

Cole, M., and Y. Engeström. 1993. A cultural-historical approach to distributed cognition. In Distributed cognitions: Psychological and educational considerations, edited by G. Salomon, 1–46.Cambridge, UK: Cambridge University Press.

Constantine, L. 2007. Activity modeling: Toward a pragmatic integration of Activity Theory with usage-centered design.http://www.foruse.com/articles/activitymodeling.htm, accessed September 16, 2007.

Constantine, L., and L. A. D. Lockwood. 1999. Software for use.New York: Addison-Wesley.

Engeström, Y. 1987. Learning by expanding: An activity theoretical approach to developmental research. Helsinki: OrientaKonsultit.

Engeström, Y. 1990. Learning, working, and imagining: Twelve studies in activity theory. Helsinki: Orienta-Konsulit.

Engeström, Y. 1999. Activity theory and individual and social transformation. In Perspectives on Activity Theory, edited byY. Engeström, R. Miettinen, and R. Punamäki, 19–38.Cambridge, UK: Cambridge University Press.

Frezzo, D. C. 2009. Using Activity Theory to Understand the Role of a Simulation-Based Interactive Learning Environment in a Computer Networking Course. Unpublished doctoral dissertation, University of Hawai‘i, Honolulu, Hawai‘i.

Frezzo, D. C., J. T. Behrens, and R. J. Mislevy. (In press). Design patterns for learning and assessment: Facilitating the introduction of a complex simulation-based learning environment into a community of instructors. The Journal of Science Education and Technology.

Hundhausen, C. 2002. Integrating algorithm visualization technology into an undergraduate algorithms course: Ethnographic studies of asocial constructivist approach. Computers and Education 39(3): 237–260.

Hundhausen, C., S. Douglas, and J. Stasko. 2002. A meta-study of algorithm visualization effectiveness. Journal of Visual Language sand Computing 13 (3): 259–290.

Hutchins, E. 1995. Cognition in the wild. Cambridge, MA: MIT Press.

Jonassen,D. H. 2000. Revisiting activity theory as a framework for designing student centered learning environments. In Theoretical foundations of learning environments,edited by D. H. Jonassen, and S. M. Land, 89–121. Mahwah, NJ:Lawrence Erlbaum.

Jonassen,D. H., and L. Rohrer-Murphy. 1999. Activity Theory as a framework for designing constructivist learning environments. ETR&D47 (1):61–79.

Kaptelinin, V., and Nardi, B. A. 2006. Acting with technology:Activity theory and interaction design, illustrated ed.Cambridge, MA: MIT Press.

Levy, F., and R. J. Murnane. 2005. The new division of labor: How computers are creating the next job market. Princeton: NJ:Princeton University Press.

Roth, W. M. 2004. Activity Theory and Education: An Introduction.Mind, Culture, and Activity, 11(1), 1-8.

Schank, R. C. 2001. Designing World-Class E-Learning : How IBM,GE, Harvard Business School, And Columbia University Are Succeeding At E-Learning (1st ed.). New York: McGraw-Hill.

Sharp, H., Y. Rogers, and J. Preece. 2007. Interaction design.New York: John Wiley & Sons.

Snir, J., C. Smith, and L. Grosslight. 1995. Conceptually enhanced simulations: A computer tool for science teaching. In Software goes to school: Teaching for understanding with new technologies,edited by D. N. Perkins, J. L. Schwartz, M. M. West,and M. S.Wiske,106–129. New York: Oxford University Press.

VanLehn, K. 1990. Mind Bugs: The Origins of Procedural Misconceptions. Cambridge: MIT Press.

Vygotsky, L. S. 1978. Mind in society: Development of higher psychological processes, 14th ed.. Cambridge, MA: HarvardUniversity Press.

Williamson, D. M., M. Bauer, L. S. Steinberg, R. J. Mislevy, J. T.Behrens, and S. F. DeMark. (2004). Design rationale for a complex performance assessment. International Journal of Testing 4(4): 303–332.

Williamson, D. M., R. J. Mislevy, and I. I. Bejar, 2006. Automated scoring of complex tasks in computer-based testing. Mahwah, N.J.:Lawrence Erlbaum.