Tuesday, June 23, 2015

Seeing student learning with visual analytics

Technology allows us to record almost everything happening in the classroom. The fact that students' interactions with learning environments can be logged in every detail raises the interesting question about whether or not there is any significant meaning and value in those data and how we can make use of them to help students and teachers, as pointed out in a report sponsored by the U.S. Department of Education:
New technologies thus bring the potential of transforming education from a data-poor to a data-rich enterprise. Yet while an abundance of data is an advantage, it is not a solution. Data do not interpret themselves and are often confusing — but data can provide evidence for making sound decisions when thoughtfully analyzed.” — Expanding Evidence Approaches for Learning in a Digital World, Office of Educational Technology, U.S. Department of Education, 2013
A radar chart of design space exploration.
A histogram of action intensity.
Here we are not talking about just analyzing students' answers to some multiple-choice questions, or their scores in quizzes and tests, or their frequencies of logging into a learning management system. We are talking about something much more fundamental, something that runs deep in cognition and learning, such as how students conduct a scientific experiment, solve a problem, or design a product. As learning goes deeper in those directions, data produced by students grows bigger. It is by no means an easy task to analyze large volumes of learner data, which contain a lot of noisy elements that cast uncertainty to assessment. The validity of an assessment inference rests on  the strength of evidence. Evidence construction often relies on the search for relations, patterns, and trends in student data.With a lot of data, this mandates some sophisticated computation similar to cognitive computing.

Data gathered from highly open-ended inquiry and design activities, key to authentic science and engineering practices that we want students to learn, are often intensive and “messy.” Without analytic tools that can discern systematic learning from random walk, what is provided to researchers and teachers is nothing but a DRIP (“data rich, information poor”) problem.

A scatter plot of action timeline.
Recognizing the difficulty in analyzing the sheer volume of messy student data, we turned to visual analytics, a whole category of techniques extensively used in cutting-edge business intelligence systems such as software developed by SAS, IBM, and others. We see interactive, visual process analytics key to accelerating the analysis procedures so that researchers can adjust mining rules easily, view results rapidly, and identify patterns clearly. This kind of visual analytics optimally combines the computational power of the computer, the graphical user interface of the software, and the pattern recognition power of the brain to support complex data analyses in data-intensive educational research.

A digraph of action transition.
So far, I have written four interactive graphs and charts that can be used to study four different aspects of the design action data that we collected from our Energy3D CAD software. Recording several weeks of student work on complex engineering design challenges, these datasets are high-dimensional, meaning that it is improper to treat them from a single point of view. For each question we are interested in getting answers from student data, we usually need a different representation to capture the outstanding features specific to the question. In many cases, multiple representations are needed to address a question.

In the long run, our objective is to add as many graphic representations as possible as we move along in answering more and more research questions based on our datasets. Given time, this growing library of visual analytics would develop sufficient power to the point that it may also become useful for teachers to monitor their students' work and thereby conduct formative assessment. To guarantee that our visual analytics runs on all devices, this library is written in JavaScript/HTML/CSS. A number of touch gestures are also supported for users to use the library on a multi-touch screen. A neat feature of this library is that multiple graphs and charts can be grouped together so that when you are interacting with one of them, the linked ones also change at the same time. As the datasets are temporal in nature, you can also animate these graphs to reconstruct and track exactly what students do throughout.

Monday, June 8, 2015

The National Science Foundation funds SmartCAD—an intelligent learning system for engineering design

We are pleased to announce that the National Science Foundation has awarded the Concord Consortium, Purdue University, and the University of Virginia a $3 million, four-year collaborative project to conduct research and development on SmartCAD, an intelligent learning system that informs engineering design of students with automatic feedback generated using computational analysis of their work.

Engineering design is one of the most complex learning processes because it builds on top of multiple layers of inquiry, involves creating products that meet multiple criteria and constraints, and requires the orchestration of mathematical thinking, scientific reasoning, systems thinking, and sometimes, computational thinking. Teaching and learning engineering design becomes important as it is now officially part of the Next Generation Science Standards in the United States. These new standards mandate every student to learn and practice engineering design in every science subject at every level of K-12 education.
Figure 1

In typical engineering projects, students are challenged to construct an artifact that performs specified functions under constraints. What makes engineering design different from other design practices such as art design is that engineering design must be guided by scientific principles and the end products must operate predictably based on science. A common problem observed in students' engineering design activities is that their design work is insufficiently informed by science, resulting in the reduction of engineering design to drawing or crafting. To circumvent this problem, engineering design curricula often encourage students to learn or review the related science concepts and practices before they try to put the design elements together to construct a product. After students create a prototype, they then test and evaluate it using the governing scientific principles, which, in turn, gives them a chance to deepen their understanding of the scientific principles. This common approach of learning is illustrated in the upper image of Figure 1.

There is a problem in the common approach, however. Exploring the form-function relationship is a critical inquiry step to understanding the underlying science. To determine whether a change of form can result in a desired function, students have to build and test a physical prototype or rely on the opinions of an instructor. This creates a delay in getting feedback at the most critical stage of the learning process, slowing down the iterative cycle of design and cutting short the exploration in the design space. As a result of this delay, experimenting and evaluating "micro ideas"--very small stepwise ideas such as those that investigate a design parameter at a time--through building, revising, and testing physical prototypes becomes impractical in many cases. From the perspective of learning, however, it is often at this level of granularity that foundational science and engineering design ultimately meet.

Figure 2
All these problems can be addressed by supporting engineering design with a computer-aided design (CAD) platform that embeds powerful science simulations to provide formative feedback to students in a timely manner. Simulations based on solving fundamental equations in science such as Newton’s Laws model the real world accurately and connect many science concepts coherently. Such simulations can computationally generate objective feedback about a design, allowing students to rapidly test a design idea on a scientific basis. Such simulations also allow the connections between design elements and science concepts to be explicitly established through fine-grained feedback, supporting students to make informed design decisions for each design element one at a time, as illustrated by the lower image of Figure 1. These scientific simulations give the CAD software tremendous disciplinary intelligence and instructional power, transforming it into a SmartCAD system that is capable of guiding student design towards a more scientific end.

Despite these advantages, there are very few developmentally appropriate CAD software available to K-12 students—most CAD software used in industry not only are science “black boxes” to students, but also require a cumbersome tool chaining of pre-processors, solvers, and post-processors, making them extremely challenging to use in secondary education. The SmartCAD project will fill in this gap with key educational features centered on guiding student design with feedback composed from simulations. For example, science simulations can be used to analyze student design artifacts and compute their distances to specific goals to detect whether students are zeroing in towards those goals or going astray. The development of these features will also draw upon decades of research on formative assessments of complex learning.