School of Science and Technology 科技學院
Computing Programmes 電腦學系

Treating Cockroach Phobia Using Augmented Reality and Graded Exposure with Microsoft Kinect

CHEUNG Stella Wing Nga

ProgrammeBachelor of Science with Honours in Computing
SupervisorDr. Andrew Lui
AreasE-Health and Medical Applications
Year of Completion2013
AwardIEEE Computational Intelligence Chapter Hong Kong FYP Competition 2013 Second Runner-Up

Objectives

This project aims at investigating the help of AR and graded exposure in cockroach phobia therapy with Microsoft Kinect.

In order to let patients handle their fears gradually and to correct their inappropriate cognitions on cockroaches, graded exposure is a good and effective approach. Cockroach phobia sufferers can face their fears step by step. During therapy, patients are guided to reach the levels one by one to relieve their phobia symptoms gently. Once the highest level is met, the treatment is complete and supposed to be successful.

AR can be integrated with graded exposure for better achievement. AR allows a virtual cockroach to be mixed with a real environment, and in doing so to give patients realistic stimuli. Besides its effect on stimuli, AR also helps in managing treatment level. Unlike real cockroaches, reactions of virtual cockroaches are totally under system control. This facilitates the implementation of graded exposure therapy.

The project objectives are described in detail below:

  • Design emotional states of cockroaches. This is an important feature in this project which is essential for characterizing the cockroaches to make them lively and real-like.
  • Design cockroach movement patterns.
  • Design treatment levels for therapists to choose.
  • Design a user interface. Design the interface and the variables for the system including the sizes of cockroaches to be displayed, dimension of the designated area for displaying the cockroaches.
  • Develop a prototyping system based on the above.
  • Evaluation of the prototype system. Cockroach phobia sufferers would be invited to test the system and a survey would be conducted. Data collected in the survey are used for analysis.

Background and Methodology

The system is expected to integrate AR with grade exposure in cockroach phobia therapy. There are two major components for AR implementation, which are virtual cockroaches and depth maps obtained from Microsoft Kinect. A virtual cockroach is developed on top of a 3D cockroach model that is associated with different movement styles for simulations. The depth map is used for tracking changes in the environment such that corresponding responses can be carried out by the virtual cockroach and thus to attain real time interactions. In addition, the system provides a set of treatment levels for graded exposure purpose. Therapists and patients are free to select a particular level at different time according to their agreement.

Virtual Cockroaches are designed to have a set of emotions, namely clam, hungry, stressed and crazy. These emotions support a rich set of behaviors for the virtual cockroaches allow them to move in an organized and realistic way. Since motions and responses performed by the virtual cockroaches directly affect what patients see and how they interact with the cockroaches, these emotions help in producing real-like cockroaches in turn to increase therapy efficacy.

Both calm and hungry are considered as gentle emotions, a cockroach of these emotions would have its antennas and legs moving slowly and it would be walking with slower paces. The difference between the two is that a calm cockroach would go straight and continuously while a hungry cockroach would pause and turn its direction from time to time to search for food. Stressed and crazy are two vigorous emotions. A cockroach of these two emotions would dash instead of walk and its antennas and legs would move faster. A stressed cockroach pauses for a few seconds every time after dashing for a certain distance to detect if any potential danger exists. A crazy cockroach dashes at a speed even higher than that when it is stressed. Besides dashing, it would become more sensitive to changes in the environment and it would also spread its wings attempting to fly away from danger.

Switching between emotions are affected by three factors, which are time, change in environment and randomness. The factors that affect shifting in emotions and the outcomes are listed below:

Emotion FactorResultingEmotion
Calm/HungryRandomnessStressed/Calm to Hungry /Hungry to Calm
StressedChange in EnvironmentStressed
TimeoutCalm
Change in EnvironmentStay Stressed
Near Change in EnvironmentCrazy
CrazyTimeoutStressed
Near Change in EnvironmentStay Crazy

Randomness and changes in the environment are the factors to make a calm or hungy cockroach become stressed. A calm cockroch may become hungry or vice versa under certain percentages. There is a time limit set for both stressed and crazy emotion. If a cockroach is stressed or crazy, whenever there are changes detected, the timer will be reset and the cockroach will stay at the current emotion until changes are over. Only when no more changes are detected over the time limit, the cockroach will calm down, that is from stressed to calm or from crazy to stressed. The whole picture of how each emotion alter to another is shown in the following emotional state transition diagram.

System operation is shown by the following diagram:

This project first designs a set of emotional states for the virtual cockroaches which allows them to have different “feelings” and corresponding reactions at different time and with respect to the stimuli given by patients. Various movement patterns are then designed for supporting the cockroaches to express their emotions. Those patterns are built with a 3D cockroach model under the help of Java3D. The virtual cockroaches are then projected through a projector over a flat area to simulate real cockroaches.

The depth map obtained from the Microsoft Kinect is used for detecting edges, slopes and moving objects in the environment to let the virtual cockroaches carry out different responses.

Edges

Edges are places where discontinuities occur. Besides the table edges, objects having no connections to another such as hands put above the table are also considered as edges. When an edge is found, the differences in height between the neighbouring areas are calculated. For small differences, the virtual cockroaches may climb up or down it. Otherwise, they would turn to a different direction.

Slopes

Slopes refer to places with increases in height continuously. When meeting slopes, virtual cockroaches would either climb or turn to a different direction. Since a depth map is obtained as a top view of the table top, for slopes which are hidden when viewing from above are considered as edges.

For the hardware required in the system, which are the Microsoft Kinect and the projector. The Microsoft Kinect is stuck onto a camera stand and the projector is put on top of it. In doing so, image of the screen is projected onto the wall. However, the whole setup can actually be put horizontally, such as hanging them from the ceiling, to make the image projected on a table top.

The system provides three treatment level options and five size options for cockroach size. When the system is started, the default treatment level is level 1 and a cockroach of size 3 is initialized. Size of the cockroach can be adjusted by the spinner at the bottom of the window. The button labeled with “Add” is used to add a cockroach of the selected size in the spinner. After the “Add” button is clicked, a cockroach will appear at a random position on the screen. To remove the cockroaches, it can be done by clicking the “Remove” button according to the sequence they are added to the environment. There must be at least one cockroach being displayed in the environment. The buttons on the right are used for controlling treatment level.

Evaluation

The evaluation was conducted for investigating the system effectiveness and its performance. Information from both cockroach phobia sufferers and non-sufferers were collected.

17 participants, 11 males and 6 females, were invited to the evaluation session on 25th April, 2013. They were all computing students aged between 20 and 25 inclusively. A lecture room with around 20 seats was used on that day.

The survey used for evaluation was in 5 point likert scale, rating from strongly disagree to strongly agree. When the participants entered the room, they were asked to fill in part I of the survey. They were then put in a queue to take a therapy one by one. During therapy, participants were invited to come to the whiteboard at the front of the room. After the therapy, they were asked to fill in the second part of survey. The overall statistics of the survey is shown below. Open-ended questions are not included in this table.

Participants were asked to fill in part I of the survey before using the system. There were 5 questions. This part is used for gathering their background information to check if they are sufferers and to know more on their level of suffering for facilitating adjustments of the treatment including setting an appropriate starting level for them. After answering the question, participants were invited to take an experiment for evaluating the effectiveness of the system.

At the beginning of the experiment, reaction of the participants will be tested with a cockroach of size 3, which is the size of cockroaches that are common found in household environment. If a participant is too scared of it, cockroaches of size 2 or below will be used. After the participants have adapted to the therapy, three to four cockroaches will be added. The experiment will progress to level 2 if no refusal is received. Participants will then be asked to let the cockroaches climb over their hand or to block their road. If they succeed in doing so, cockroaches of larger sizes will be added to the environment and they will be requested to repeat the above task on the larger ones. After overcoming level 2, the experiment will get to level 3. Participant will be asked to give stimuli to the virtual cockroaches such as moving their hand over them or put their hand beside them. They have to overcome the panicky cockroaches in order to complete the experiment.

The experiment normally starts at level 1, but level 2 or 3 might be chosen as the starting level for non-sufferers or for sufferers with slight symptoms. The experiment might go back to a previous level under requests from participants.

Besides moving speed and reactions carried out by the cockroaches, size was reported as another factor affecting participants’ avoidance. Almost all of the participants thought the cockroaches of size 3 or lower (less than 5cm in length) were acceptable and were able to have interactions with them. When cockroaches of size 4 or 5 (length about 6cm to 8cm) are added, 35% participants said they felt fear and appeared unwilling to interact with those virtual cockroaches. Later at their first time interactions, a hand withdrawal was observed in more than half of the cases. After one to two minutes on average, most of them succeeded at their second or third time. The following shows the agreed percentages about the system effectiveness:

QuestionSufferersNon-sufferersOthers
The system helps in reducing my level of fear of cockroaches67%100%20%
I find cockroaches less scary than before44%67%100%
Multi-level design can facilitate phobia treatment78%67%60%

From the statistics above, it might be strange to see that the percentage of non-sufferers agreeing the system helped in reducing their level of fear and finding cockroaches less scary than before is higher than that reported from suffers and others. According to the observation on participant’s reactions during experiment, all of the non-sufferers actually were happy and thought it was fun to interact with the virtual cockroaches though they might be scared by them once or twice. They probably found the virtual cockroaches more interesting than the real ones which caused the above result. However, the overall percentages in those aspects were not high. The main reason due to the low percentages was the projector was too dim and its resolution was not high enough causing the images to be blurred and making the cockroaches unreal. Other reasons included they have known that the cockroaches were fakes and the stimuli given by 2D images were not strong enough to cause fear. The projector used was a 3M MPro150 with brightness 15lm. If it is replaced by a projector with higher brightness, a better result might be attained.

Conclusion and Future Development

From the evaluation result, the system was successful in relieving cockroach phobia symptoms, which include reducing the anxiety level and uncomfortable feeling. Since the virtual cockroaches are fakes, the effectiveness on the sufferers towards real cockroach has not yet been proven. A follow-up observation has to be carried out for investigating this issue. Nevertheless, this system can serve as a preparation step to get the sufferers ready for in vivo exposure or any further therapy.

Copyright Cheung Wing Nga Stella and Andrew Lui 2013

Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course CodeTitleCredits
 COMP S321FAdvanced Database and Data Warehousing5
 COMP S333FAdvanced Programming and AI Algorithms5
 COMP S351FSoftware Project Management5
 COMP S362FConcurrent and Network Programming5
 COMP S363FDistributed Systems and Parallel Computing5
 COMP S382FData Mining and Analytics5
 COMP S390FCreative Programming for Games5
 COMP S492FMachine Learning5
 ELEC S305FComputer Networking5
 ELEC S348FIOT Security5
 ELEC S371FDigital Forensics5
 ELEC S431FBlockchain Technologies5
 ELEC S425FComputer and Network Security5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5