Research and Publications
Track Xplorer: A System for Visual Analysis of Sensor-based Motor Activity Predictions
Marco Cavallo, Cagatay Demiralp
Detecting motor activities from sensor datasets is becoming increasingly common in a wide range of applications with the rapid commoditization of wearable sensors. To detect activities, data scientists iteratively experiment with different classifiers before deciding on a single model. Evaluating, comparing, and reasoning about prediction results of alternative classifiers is a crucial step in the process of iterative model development. However, standard aggregate performance metrics (such as accuracy score) and textual display of individual event sequences have limited granularity and scalability to effectively perform this critical step. To ameliorate these limitations, we introduce Track Xplorer, an interactive visualization system to query, analyze and compare the classification output of activity detection in multi-sensor data. Track Xplorer visualizes the results of different classifiers as well as the ground truth labels and the video of activities as temporally-aligned linear tracks. Through coordinated track visualizations, Track Xplorer enables users to interactively explore and compare the results of different classifiers, assess their accuracy with respect to the ground truth labels and video. Users can brush arbitrary regions of any classifier track, zoom in and out with ease, and playback the corresponding video segment to contextualize the performance of the classifier within the selected region. Track Xplorer also contributes an algebra over track representations to filter, compose, and compare classification outputs, enabling users to effectively reason about the performance of classifiers. We demonstrate how our tool helps data scientists debug misclassifications and improve the prediction performance in developing activity classifiers for real-world, multi-sensor data gathered from Parkinson's patients.
Exploring Dimensionality Reductions with Forward and Backward Projections
Marco Cavallo, Cagatay Demiralp
Dimensionality reduction is a common method for analyzing and visualizing high-dimensional data across domains. Dimensionality-reduction algorithms involve complex optimizations and the reduced dimensions computed by these algorithms generally lack clear relation to the initial data dimensions. Therefore, interpreting and reasoning about dimensionality reductions can be difficult. In this work, we introduce two interaction techniques, forward projection and backward projection, for reasoning dynamically about scatter plots of dimensionally reduced data. We also contribute two related visualization techniques, prolines and feasibility map to facilitate and enrich the effective use of the proposed interactions, which we integrate in a new tool called Praxis. To evaluate our techniques, we first analyze their time and accuracy performance across varying sample and dimension sizes. We then conduct a user study in which twelve data scientists use Praxis so as to assess the usefulness of the techniques in performing exploratory data analysis tasks. Results suggest that our visual interactions are intuitive and effective for exploring dimensionality reductions and generating hypotheses about the underlying data.
CAVE-AR: A VR User Interface to Interactively Design, Monitor, and Facilitate AR Experiences
Marco Cavallo, Angus Forbes
In this paper we propose CAVE-AR, a novel virtual reality (VR) system for authoring custom augmented reality (AR) experiences and interacting with participating users. We introduce an innovative technique to integrate different representations of the world, mixing geographical information, architectural features, and sensor data, allowing us to understand precisely how users are behaving within the AR experience. By taking advantage of this technique to "mix realities", our VR application provides the designer with tools to create and modify a AR application, even while other people are in the midst of using it. Our VR application further lets the designer track how users are behaving, preview what they are currently seeing, and interact with them through different channels. This enables new possibilities which range from simple debugging and testing to more complex forms of centralized task control, such as placing a virtual avatar in the AR experience to guide a user. In addition to describing details of how we create effective representations of the real-world for enhanced AR experiences and our novel interaction modalities, we introduce two use cases demonstrating the potential of our approach. The first is an AR experience that enables users to discover historical information during an urban tour along the Chicago Riverwalk; the second is a novel scavenger hunt that places virtual objects within a realworld environment to facilitate solving complex multi-user puzzles. In both cases, the ability to develop and test the AR experience remotely greatly enhanced the design process and the novel interaction techniques greatly enhanced overall user experience.
Riverwalk: Incorporating Historical Photographs in Public Outdoor Augmented Reality Experiences
Marco Cavallo, Angus Forbes, Geoffrey Alan Rhodes
We introduce a user-centered Augmented Reality (AR) approach for publishing 2D media archives as interactive content. We discuss the relevant technical considerations for developing an effective application for public outdoor AR experiences that leverage context-specific elements in a challenging, real-world environment. Specifically, we show how a classical marker-less approach can be combined with mobile sensors and geospatial information in order apply our knowledge of the surroundings to the experience itself. Our contributions provide the enabling technology for Chicago 0,0 Riverwalk, a novel app-based AR experience that superimposes historical imagery onto matching views in downtown Chicago, Illinois along an open, pedestrian waterfront located on the bank of the Chicago River. Historical photographs of sites along the river are superimposed onto buildings, bridges, and other architectural features through image-based AR tracking, providing a striking experience of the city's history as rooted in extant locations along the river.
RehabJim: A Third Person Approach To Virtual Reality Biomechanical Rehabilitation
Marco Cavallo, Andrea Rottigni, Elizabeta Marai, James Patton
RehabJim is a Unity3D application developed on behalf of researchers at a major Rehabilitation Institute in order to explore the opportunities that Virtual Reality may offer to biomechanical and neurological rehabilitation. In particular, our work focuses on arm actions peformed while standing by patients post-stroke. We leverage a 3D immersive environment, augmented with Kinect interaction. Our approach follows a third person perspective of the patient's body, and employs two cartoon-style avatars - the patient and their therapist. The user is asked to reach to computer-generated virtual objects with her hands. Our evaluation with users and a domain expert shows that this type of environment leads to an engaging, enjoyable experience that can encourage patients to perform a wide variety of whole-body motions.
DigitalQuest: A Mixed Reality Approach to Scavenger Hunts
Marco Cavallo, Angus Forbes
This paper presents a novel approach for the design of creative location-based mixed reality applications. We introduce a framework called DigitalQuest that simplifies adding geolocated virtual content on top of real-world camera input. Unlike previous work, which relies solely on markers or image pattern recognition, we define a "mirror world" that facilitates interactive mixed reality. DigitalQuest consists of an editor that allows users to easily add their own content as desired and a mobile application that loads content from a server based on the location of the device. Each piece of virtual content can be organized through the editor so that it appears only in certain circumstances, allowing a designer to determine when and where a virtual object is attached to a real-world location. We have used our editor to create a series of futuristic scavenger hunts in which participating teams must solve puzzles in order to access new virtual context appearing in a mixed reality environment via a mobile phone application. In this paper, we introduce our editor and present an example scavenger hunt game, Morimondo, that was built using it. Specifically, we describe our technique to utilize camera and motion sensors on the mobile phone to enable an appropriate level of user engagement within this game. We are able to obtain realistic augmentations with accurate positioning by leveraging sensor fusion and through the use of filters that compensate for sensor noise, using image processing only for error correction or in special situations. The initial success of this project leads us to believe that DigitalQuest could be used to design a wide range of creative multi-user mixed reality applications.
3D City Reconstruction From Google Street View
Despite laser scan 3D point cloud acquisition has greatly improved over the next few years, the process of creating 3D large scale city models is still quite expensive and not straightforward. At the same time, nowadays services such as Google Street View provide a vast amount of geo-registered panoramic imagery, guaranteeing a decent resolution for dense locations at zero cost. Our idea is indeed to leverage this free information provided by Google Street View in order to obtain a cheap and automatizable 3D recontruction of an urban area, by extracting the depth information related to the great number of panoramic images available online.