NYSCA Mobile Augmented Reality

The Mobile Experiences for Tourism Project was a $106,000 New York State Council on the Arts 2013, Rochester Institute of Technology and Visit Rochester funded project to improve a tourist’s heritage experience through location-based mobile interactions with augmented reality and heritage storytelling. Our main goal was to create fun interactive experiences for school children to inspire their parents to make return visits

Visitor Problems

We focused our research and development on four particular visitor-centered problems when visiting a large heritage destination such as Genesee Country Village & Museum.

  1. Guides, being volunteers, are not always present.
  2. The village is large and easy to not know where to go.
  3. You need more than one day to feel like you have seen the village.
  4. Not easy to figure out what some buildings are or what stories they hold.

Visitor Experience Solutions

  1. Enable the visitors to easily navigate between locations for both utility and serendipity.
  2. Use the relationship between historical buildings and events to create historically relevant scenarios for the visitor to explore and learn.
  3. Deliver stories as information at the right location without impending on the user in his/her overall physical environments.

Heritage Storytelling

A heritage story is a strategy to deliver digital content through video augmented characters (digital agents), relevant to the location where the visitor is standing. The story links the locations together creating continuity from location to location at a historical destination. The heritage stories enable historical practitioners and designers the ability to create historically accurate contexts in which the visitor can learn and explore a single location in a multiple of ways.

Heritage Story: The Mystery of the Missing Child

This heritage story reflects how children were treated in 19th century America. However, the culture and historical information is woven into when and where the visitor moves from location to the location driven by the characters in the story.

The story is about a young girl named Alice who’s brother has gone missing. The visitor explores the Pioneer Settlement while learning 19th century, childhood responsibilities (child labor), building materials (blacksmith trade), cultural norms (teachers could physically hurt children), education standards (girls are not to be educated), medical treatments (herbal medicine for burns) and what a children can do to escape being bullied (they hide).

This slideshow requires JavaScript.

Heritage Augmented Characters

Along the way, the visitor will meet and be lead around the village by the following characters. Alice is the lead character as she engages the visitor to help her find her younger brother guiding the visitor from location to location. Alice introduces the visitor to her sick father, meets Ms Crabapple, the protagonist, and runs into a sympathetic blacksmith. At the end of the story, the mystery is solved and her brother has been under the care of a the village midwife. See below for all characters.Screen Shot 2015-07-07 at 7.48.56 PM

Mobile User Interface

We designed a mobile interface to, not only solve fundamental visitor-experience problems, but also meet the requirements of create a heritage story.

Screen Shot 2015-07-07 at 8.22.07 PM

Performing Augmented Reality


The Performing Augmented Reality project asks the question, what is performing arts in the age of mixed reality when we can place a performer anywhere, at at time and in any context?

This effort was a subset of the Mobile Experiences for Tourism Project and produced by the Blended Interaction Studio. After creating heritage stories at Genesee Country Village Museum (GCVM), a nineteenth-century living history museum.  Note – A heritage story is a cinematic method to deliver digital content through digital agents on mobile devices. We created and directed video shoots for augmented reality production. See video.

The goal of these video shoots was to record a period actor or character. Each character was assigned to a specific geo-location and played when the tourist arrived at the blacksmiths or school house. These videos were shot for augmented reality to create travel continuity from location to location at a historical site. Each period actor below represents a digital character augmented through mobile devices.

Evaluating Augmented Reality Experiences

In early December of 2013, seventeen fifth-grade school students from Allendale Columbia School were invited to experience the New York State Council on the Arts (NYSCA), Mobile Experiences for Tourism Project mobile prototype. Each student was between the ages of 10-11 years old and had been to Genesee Country Village & Museum  in the past. Before the evaluation the 17 students were selected to be in groups averaging 2-3 students. Before the evaluation students were given a brief UI tutorial. During the evaluation, researchers shadowed the student groups. After the evaluation a short survey followed by a short interview was given.

Our evaluation gave us an opportunity uncover new methods of mobile heritage storytelling. After the evaluation, the school students were very eager for more heritage storytelling [1]. School students thought that:

  1.  the heritage story was fun and engaging (100%),
  2. they could easily navigate pioneer settlement with the information provided in the story (82%),
  3. they felt the digital characters added mystery and made the entire field trip more interesting compared to a normal GCVM tour guide (88%) and,
  4. they would be interested in repeat visits if the mobile application gave seasonal narrations during the year (94%).

Our evaluation provided areas for design improvement. School students were mainly natural when taking the time to explore the individual locations e.g. Blacksmith Shop. The majority of students spent too much time looking at the mobile device while moving from location to location. These areas of improvement will help shape the next design iterations as we evaluate our strategy and heritage stories at GCVM.

The results showed that school students would like to create their own heritage story (88%). We collaborated with Allendale Columbia School students to deploy their own heritage stories at GCVM spring 2014. We have conducted workshops to investigate, design and evaluate mobile experiences for heritage storytelling.

This slideshow requires JavaScript.

[1] O’Keefe, B., Benyon, D., Chandwani, G., Menon, M. & Duke II, R. (2014). A Blended Space for Heritage Storytelling. In Proceedings of the 27th International British Computer Society Human Computer Interaction Conference: Sand, Sea & Holiday HCI. (Article 45, 10 pages). Southport, UK, BCS-HCI 14.