Current Researchers: Michael Yeganyan, David Schramm, Meg Cramer, Monica Tentori and Gillian Hayes

Past Collaborators: Sen Hirano, Gabriela Marcu, Mohamad Monibi, David Nguyen

Project: Visual schedules and choice boards are tools used in current best practices for helping children with autism and other special needs. These non-verbal kids need help communicating their choices, understanding time and activities, and so on. We are working with Windows SmartPhone, Linux small displays, and large touchscreen-enabled platforms to develop solutions that ease these burdens, provide logging and visualizations of data, and help automate some of the features of using these communication techniques.

We are also investigating how novel recording technologies, like the Microsoft SenseCam can be used to augment communication between non-verbal children with autism and their caregivers – including both parents and teachers.

Watch a talk Gillian gave at Microsoft Research that covers some of the autism and FitBaby work:  http://research.microsoft.com/apps/video/default.aspx?id=121512

This work is supported in part by technical assistance and equipment through a collaboration with Microsoft Reserach Cambdridge.

This work is supported by AutismSpeaks Innovative Technologies For Autism program, a generous equipment donation from Nokia Research Palo Alto, and an NSF CAREER award.

The vSked and SenseCam for Autism projects will be presented at IMFAR 2008 in Chicago.

This work was presented at IDC as a poster at the main conference and at the Design for Children with Special Needs Workshop:

The vSked and SenseCam for Autism projects will be presented at IMFAR this year in Chicago.

This work was presented at IDC as a poster at the main conference and at the Design for Children with Special Needs Workshop:

vSked: Interactive Visual Scheduling Systems
Gillian R. Hayes
David Nguyen
Michael Yeganyan
Sen Hirano
Gabriela Marcu

SenseCam
Gillian R. Hayes
Gabriela Marcu

Mocotos:  Mobile Communications Tools for Children with Special Needs
Mohamad Monibi
Gillian R. Hayes

Interactive and Intelligent Visual Communication Systems
Gillian R. Hayes
Donald J. Patterson
Mohamad Monibi
Sam Kaufman

This work was also presented at the International Meeting for Autism Research as a poster:

Design of Interactive Visual Scheduling Systems
Sam Kaufman
Donald J. Patterson
Gillian R. Hayes

vSked: An Interactive Visual Schedule System for use in Classrooms for Children with Autism
Michael T. Yeganyan, Meg Cramer, Lou Anne Boyd and Gillian R. Hayes

Children with autism often experience substantial challenges in understanding, structuring, and predicting the activities in their daily lives. The use of symbols to represent a series of activities, also known as visual schedules, have been shown to be an effective intervention technique for helping individuals with Autism Spectrum Disorder. In this paper, we describe the design and technical architecture for vSked, an interactive visual scheduling tool that allows group interactivity with content generated through end-user programming. We also outline a scenario that demonstrates how vSked extends the benefits of traditional visual schedules through both interactivity and automatic logging of use. This scenario also indicates the type of experience a user might have during a demonstration session.

Interactive Visual Supports for Children with Autism

Gillian R. Hayes, Sen Hirano, Gabriela Marcu, Mohamad Monibi, David H. Nguyen, and Michael Yeganyan

Interventions to support children with autism often include the use of visual supports, which are cognitive tools to enable learning and the production of language. Although visual supports are effective in helping to diminish many of the challenges of autism, they are difficult and time-consuming to create, distribute, and use. In this paper, we present the results of a qualitative study focused on uncovering design guidelines for interactive visual supports that would address the many challenges inherent to current tools and practices. We present three prototype systems that address these design challenges with the use of large group displays, mobile personal devices, and personal recording technologies. We also describe the interventions associated with these prototypes along with the results from two focus group discussions around the interventions. We present further design guidance for visual supports and discuss tensions inherent to their design.

vSked:  Evaluation of a System to Support Classroom Activities for Children with Autism by Sen Hirano, Michael Yeganyan, Gabriela Marcu, David Nguyen, and Gillian Hayes accepted for CHI 2010.

Visual schedules—the use of symbols to represent a series of activities or steps—have been successfully used by caregivers to help children with autism to understand, structure, and predict activities in their daily lives. We conducted in-depth fieldwork and participatory design sessions, through which we developed vSked, an interactive and collaborative visual scheduling system designed for elementary school classrooms. We evaluated vSked in situ in one autism-specific classroom over three weeks.  In this paper, we present the design principles, technical solution, and results from this successful deployment. Use of vSked resulted in reductions in staff effort required to use visual supports. vSked also resulted in improvements in the perceived quality and quantity of communication and social interactions in the classroom.

Original Source

As a child, Gillian Hayes fainted a lot. Doctors asked her to write down how she felt and what she was doing each time she became woozy.

“I thought it was the most ridiculous thing I’d ever heard. The solution is to create and track a bunch of records?” says Hayes, UC Irvine informatics assistant professor. “There had to be a better way.”

The experience piqued her interest in computerized record-keeping, particularly in the areas of healthcare and education. Today, Hayes designs computerized devices that help teachers work with children who have autism.

Gillian Hayes, UCI informatics assistant professor, designs technology that helps teachers work with children who have autism.

One device, called Selective Archiving, records video of a classroom. If a child with disruptive or dangerous behavioral tendencies acts up, the teacher presses a button to save that section of video, as with TiVo. He or she can then watch the recording after hours, when there are fewer distractions. If the button is never pressed, nothing is saved, which protects privacy.

Teachers of children with autism are required to document and analyze disturbing behavior over weeks or even months to determine trends and monitor progress. “Using our system, teachers at a test school went from a nearly 74 percent undocumented rate to just above 41 percent, which is pretty good when you consider how much happens in a busy classroom,” Hayes says.

Another device, Visual Scheduler (vSked for short), takes children with autism through exercises in which they identify the day of the week, the month, the weather outside and other things a teacher might ask in a typical elementary school classroom. On a handheld touch screen, they press a button to choose an answer. If they pick the wrong one, the correct button shakes and turns colors, eliminating the need for a teacher to physically point to the right answer.

Previously, the children stuck answers to a laminated folder using Velcro – a system that wasn’t able to track trends like always choosing the option on the left. “Those kinds of patterns are really hard to see in the analog, paper-based world, but they’re easy to spot with a computer,” Hayes says.

She’s also interested in record-keeping for people with chronic illnesses or conditions such as asthma and obesity, which can occur episodically throughout an individual’s life.

“We need to start thinking about healthcare over a lifetime and not just the acute moments of having the flu or breaking a leg,” Hayes says. “If we have data when we’re healthy, we’ll know much more quickly when we start to decline. The earlier we’re diagnosed, the better the outcome is going to be.”

Computing, she says, can be applied to most problems.

“Everything has computers – your car, the lights in your house, your microwave,” Hayes says. “By combining the things computers do well with human ingenuity, we can address almost any societal issue, from education to healthcare to the environment.”

— Jennifer Fitzenberger, University Communications

Related Links

• Gillian Hayes faculty profile
• Department of Informatics
• Donald Bren School of Information & Computer Sciences

Congratulations to Gabriela Marcu, David Nguyen, and Gillian Hayes for the acceptance of “Use of a Wearable Recording Device in Therapeutic Interventions for Children with Autism” and congratulations to Michael Yeganyan, Sen Hirano, David Nguyen, and Gillian Hayes for the acceptance of “Interactive and Collaborative Classroom Visual Schedules”

Interactive and Collaborative Classroom Visual Schedules

Michael Yeganyan, Sen Hirano, David H. Nguyen and Gillian R. Hayes
Department of Informatics, University of California, Irvine, Irvine, CA 92697

Background
Visual schedules have been in use as a best practice in schools for many years. These artifacts use words and images to represent activities that will take place (or have taken place) arranged in temporal order to augment understanding of time, events, and places. Research has shown that the use of visual schedules can improve communication with children with autism spectrum disorder (ASD). Often, however, these visual schedules are not interactive and interesting enough to draw and maintain attention; they can be out of date or inaccurate; and they do not support communication and collaboration surrounding activities. Handling these challenges necessitates an immense amount of manual effort by teachers and aides. To offset this burden, we have created innovative computing technologies that we are using to simplify schedule creation and generate useful data that teachers and other caregivers can analyze.

Objectives
Design and implement a visual scheduling system for classrooms to support children with autism and expand or refine current teaching techniques to make use of this new system. Demonstrate these schedules in use sessions by teachers.

Methods
We used paper and simple digital prototypes during sessions in autism classrooms and interviews with teachers and autism specialists. During the sessions, we iteratively developed a system that not only mimics the analog visual schedules currently in use but also provides new features. For example, the system provides students with personal devices that interact collectively with a large shared screen at the front of the room. We are collecting information about system usage and practices of teachers and students during demonstration visits and initial use in classrooms. In particular, we are interested in understanding the potential for social learning as students, teachers, and aides are able to see responses from students who may be struggling on particular activities as well as those who have mastered them.

Results
By spending time in the field with teachers, specialists, and children with autism, we have been able to understand and compile information about the design of interactive visual schedules. These electronic visual schedules can assist teachers in managing their classrooms, in not only setting up exercises but also running them and keeping records that would otherwise be unfeasible. Finally, using both shared large displays for the whole class and smaller networked systems for individual children, we enable new interactions in classrooms, including social and peer learning as well as more efficient and rapid feedback for students and staff about individual progress and abilities.

Conclusions
We have designed and developed an interactive visual scheduling system based on extensive in-school interviews and observations. This system replaces and enhances the features of analog visual schedules in digital form. Through this new system, we also enable new ways of keeping records by automatically logging all interactions with the system and new forms of teaching and learning by dynamically sending and receiving visual information to the students’ networked personal devices.

Use of a Wearable Recording Device in Therapeutic Interventions for Children with Autism

Gabriela Marcu, David H. Nguyen, Gillian R. Hayes
Department of Informatics
University of California, Irvine

Background
A common problem for caregivers of children with autism is facilitating communication, especially when the children are non-verbal. Caregivers use social stories and references to a child’s activities to encourage discussion, but these efforts are challenging and not always effective. Additionally, caregivers are challenged with trying to understand a child’s behavior and reactions without the aid of verbal expression from the child. This project applies the use of SenseCam to therapeutic interventions for children with autism.

SenseCam is a small wearable camera developed at Microsoft Research. It takes photographs automatically at a frequency determined either by a preset interval or in response to sensory input. Photographs can be transferred to a computer and watched at different speeds. SenseCam was designed to augment human memory by providing a first-person account of the wearer’s activities.

Objectives
In this work, we hypothesized that the use of SenseCam by children with autism will enable greater awareness of the child’s perspective during the day and potentially be used in new therapeutic interventions to support communication and understanding. The photographs are captured from the child’s perspective, allowing caregivers to review captured events with a child and encourage them to discuss activities and people appearing in these photographs. The primary objective of this work is to develop processes for inclusion of SenseCam technology into everyday use by children with autism and to modify interfaces to SenseCam media for use in these interventions.

Methods
Children with autism, their daily activities, and their social interactions were observed in the classroom. Interviews with teachers and parents of children with autism provided additional data about communication challenges. Interviews with autism specialists informed the design of use of SenseCam by children with autism.

Results
We have developed a therapeutic intervention using a novel technology–SenseCam from Microsoft Research. In this intervention, children with autism can wear SenseCam during all or part of a typical day. Parents and other caregivers can review photographs taken during school hours while teachers and other school staff review photographs taken during after-school hours. These review sessions can facilitate mutual understanding of activities in these environments and better communication between school and home. Parents, teachers, and other caregivers can also review photographs together with the children to facilitate communication. The photographs can be used to encourage the children to discuss activities, people, social situations, and their feelings. This intervention can improve the children’s expression and augment their understanding of social interactions through open discussion about the photographs.

Conclusions
Working in collaboration with schools and autism specialists, we have developed a technology based therapeutic intervention which can enhance caregiver awareness and improve children’s communication and social understanding. Future work will involve the deployment and evaluation of SenseCam in the classroom and the home among children with autism, their parents, teachers, and other caregivers.

Professor of informatics Gillian Hayes has received an $83,563 award from Autism Speaks for her proposal, “Technology Support for Interactive and Collaborative Visual Schedules”.

This work will focus on developing a digital repository and visual schedule system for use in schools and homes by individuals with autism spectrum disorder (ASD). Using visual schedules, such as words, images and tangible objects to represent activities that will take place (or have taken place) has been shown to reduce the symptoms associated with autism. Using the smart visual schedules system, caregivers can generate reports, share information with one another, and possibly even update an individual schedule at a distance as circumstances change.

Autism Speaks is a New York City-based advocacy organization, founded in February 2005 by Bob Wright, Vice Chairman of General Electric, and his wife Suzanne, to improve public awareness about autism and to promote autism research.

The Wrights founded Autism Speaks to help find a cure for autism spectrum disorders a year after their grandson, Christian, was diagnosed with autism.