Enhancing visitors’ experience on revitalized historic buildings with augmented reality

School of Science and Technology Computing Final Year Project Showcase Enhancing visitors’ experience on revitalized historic buildings with augmented reality

School of Science and Technology 科技學院
Computing Programmes 電腦學系

Enhancing visitors' experience on revitalized historic buildings with augmented reality

CHAN Chun Kit, NG Hoi Ying, IP Long Yin

 
Programme Bachelor of Computing with Honours in Internet Technology
Supervisor Dr. Vanessa Ng
Areas Augmented Reality Applications
Year of Completion 2018
Achievement Top-10 Finalist in Microsoft Imagine Cup HK

Objectives

This project aims to enrich the interactive and visiting experience, which mainly focuses on presenting the whole picture in history and structures of the building.

For public visitors, they can get more fruitful experience by using the application than using existed solution to learn history. They can gain more historical knowledge and interactive experience than using text boards or pictures that commonly used in most of the revitalized historic building in Hong Kong. Through interactive-experience and showing building history by AR technology, visitors can have more ideas of why the government or organisation would preserve the historic building and learn history in an exciting method.

For government or organisation that manages the historic building, by using this application they can provide the interactive experience to visitors. Building itself without wasting spaces for exhibitions and save human resources to hold tour on teaching history and the meaning of historic revitalisation. By adopting different technologies, they can increase the public awareness of the history of the building; and let them have the idea of historic building revitalisation and support this idea.

Video Demonstration

To achieve the project aim, the objective develops an AR application that provides an interactive and fruitful experience to visitors and correctly recognizes the historic building. Here are some sub-objectives:

  • Recognition System. Recognize the historic building and its precise position.
  • 3D model development. Develop 3D models of the historic building that based on building's history in the different period of usage.
  • The integration with object recognition system and 3D model. By integrating object recognition system and 3D model, we can provide an interactive and precise experience to visitors.
  • Interactive environment. By providing interactive methods, like game, text and video to enrich the experiences of the visitors.

Background and Methodology

The solution includes three main functions: augmented reality service, object recognizer and region monitor.

For region monitor, users should stay within the effective coverage area of iBeacons to identify their location in the historic building. As having the signal of iBeacon, the device will call ARkit and the machine learning model of a building.

For object recognizer, as users' device have called the model by iBeacon's ID. The purpose of separate machine learning model is to increase the successful recognition rate. Since there are some spots are having similar characters or difficult to have correct recognition. Separating machine learning model can enhance the success recognition with less similar character's object result.

With successful object recognition, augmented reality services will be provided by using ARkit and SceneKit to render 3D object like buttons, animation and minigame. Users can choose what kind of information should be shown in the application, including local video, video streaming service and website by clicking the button.

Having the above functions, we believe our solution can provide better user experience and interactive method for visiting historic building.

System Design and Implementation

Diagram of system design

The system consists of 4 key components:

1. Region Selector

  • Entry point of the system, user can select their desired region to be visited in the region selector. iBeacon Event Trigger Service will be triggered when the user selected a region, and the region ID will be passed to iBeacon Event Trigger Service.

2. iBeacon Event Trigger Service

  • Consists of 2 sub-components:
  • Region Monitor
  • Keep tracking of user's location and trigger the object recognizer once the user reached the target region.
  • Direction Pointer
  • Provide user location information to guide users to their selected region when users have not reached their selected region.

3. Object Recognizer

  • Provides a mean to connect real-world object with virtual elements. Exhibit ID will be generated when the exhibit is recognized by the object recognizer, and the exhibit ID will be pass to Augmented Reality Service for providing further service.

4. Augmented Reality Service

  • Provides interaction method by ARKit and SceneKit. Object will change or return response as users' action on screen.

User Interface Design

Menu

Region Selector

According to the user interface of Menu, users press the “Explore Now” button to start the application.

Region Monitor (in iBeacon Event Trigger Service)

After the selection of a region, Region Monitor started with Direction Pointer at the same time and keep tracking iBeacon signal. When users have entered the selected region, users' device would notify the users to start the following service by pressing the “Start Now” button.

According to the user interface of Region Selector, users can select the region they would like to go to.

Augmented Reality Service

According to the user and tester interface, different AR service will be provided.In in-app service, animations, in-app video display and mini-game will be shown .

Conclusion and Future Development

As the mention in the introduction section, the problem of enhancing visitor experience is to give an interesting way to provide information to visitors. Therefore, this project aims to develop an AR application to provide interactive and multimedia information to visitors on increase their experience.

There are four project objectives have been defined. First is to build recognition system to recognize the historic building and its position. Recognition models of a historic building or complex in different regions constitute the system in this platform. Users’ device image captured will be pass to the system and return result tags for the nest application action.

Second, we designed and developed 3D models of the historic building in different regions. 3D models design of each region based on building's history in the different period of usage. Each region divide based on their significance on history, architecture and location. Users can select different displayed item with its precise model.

Third, recognition system and 3D model integrated, we can provide an interactive and precise experience to visitors assist by iBeacon.

Last, to provide an interactive environment, by providing interactive methods, like game, text and video to enrich the experiences of the visitors.

By these objectives, we have developed an application using frameworks from Apple Inc and Microsoft Azure. This application can provide an interactive method with AR and precise position with iBeacon.

From the evaluation, we can conclude that our application can increase the visitor experience on visiting historic building. Visitors can use this application to guide them to visit the historic building without the help of map and tour guide. On the other hand, AR in this application can provide a more interactive experience to visitors than using current method of showing information in the exhibit.

AR technology combine with iBeacon position system is changing the way to provide a fruitful experience to visitors. Since these technologies are still an innovative technology, the processing power of mobile device may not be catch up with the requirement. However, we believe that AR technology combine with location service like iBeacon will provide a fruitful visiting experience to visitors.

The limitations of our solution are if the operations will take system resources, such as Bluetooth and camera for AR, to interact with the real environment and frequent update on location by iBeacon will use a lot of battery power and system resources, then cause the heat issue. The reason for this limitation is as most of the devices in the market are not designed for intensive AR, and 3D object interacts with the real environment. These function can be used, but not with the ideal specification.

For future work suggestions in visiting users’ side, first, cross-platform support as iOS is the only supporting system currently. Secord, to develop a method to interact environment objects by visitor's device, such as, controlling light or controlling robotic machine to provide a more in-depth learning experience. Third, to provide visitor flow control and data analytics tool for users. That suggests visitors visit by visitor capacity in the different area in the crowded building to increase overall visitor experience.

In organisers’ side, a management console for the organiser to have a little config with the platform in accordance with organiser’s need.

As this application will not only use in historic building, another usage of the application can also use this application. Such as museum, gallery and exhibition, those can be using multimedia to show the information of the exhibit without changing the looking of it. Moreover, using this application, they can also provide fruitful and informative experience to visitors.

Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course Code Title Credits
  COMP S321F Advanced Database and Data Warehousing 5
  COMP S333F Advanced Programming and AI Algorithms 5
  COMP S351F Software Project Management 5
  COMP S362F Concurrent and Network Programming 5
  COMP S363F Distributed Systems and Parallel Computing 5
  COMP S382F Data Mining and Analytics 5
  COMP S390F Creative Programming for Games 5
  COMP S492F Machine Learning 5
  ELEC S305F Computer Networking 5
  ELEC S348F IOT Security 5
  ELEC S371F Digital Forensics 5
  ELEC S431F Blockchain Technologies 5
  ELEC S425F Computer and Network Security 5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5