School of Science and Technology 科技學院
Computing Programmes 電腦學系

VR Stroke Rehabilitation Trainer

Huen Tsz Wa, Choi Chun Kit, Lam Cheuk Hei, Man Yuk Kin

ProgrammeBachelor of Arts with Honours in Computing and Interactive Entertainment
SupervisorDr. William Lai, Dr. Keith Lee
AreasVirtual Reality Applications
Year of Completion2022

Objectives

Project Aim

Based on the present research, existing virtual reality games do not fulfill some of the main core requirements of upper limb training in rehabilitation exercises. For example, the game approach only supports training the motion of pinching or grasping (Sulimanov & Olano, 2019). However, other training activities, such as hand strength training and hand extension training, are not covered in the game. Thus, the game is not able to provide rehabilitation exercises that are covered in training hand motion, extension, and strength.

On the other hand, virtual reality-based stroke rehabilitation games are one of the choices for physiotherapists to attract patients who are feeling bored during rehabilitation treatments. Some virtual reality games are too focused on exercise functions but ignore the element of encouragement. A lack of short-term objectives causes a lack of engagement for patients (Barrett et al., 2016). The game of reward is important because it helps motivate the patient to do exercise continuously by offering visible rewards (Ferreira & Menezes, 2020). Therefore, a reward function is needed for patients when they use virtual reality games for rehabilitation.

Game level difficulties are also an issue with the current virtual reality-based stroke rehabilitation games. Current virtual reality-based stroke rehabilitation games use two common types of approaches to set up the game difficulties. The first one is to set up the difficulties manually before the game is started (Elor et al., 2018). The physiotherapist needs to set up the game difficulties according to the patient’s rehabilitation progress. It is the static approach for setting up game difficulty. In this approach, the physiotherapist is required to adjust the game difficulties when the patient’s rehabilitation has improved. The second one sets up difficulties at different levels (Lin et al., 2020). In this approach, the patient then selects the suitable level for the rehabilitation exercise. However, those game levels have limited the difficulties. It is not possible to dynamically adjust the game difficulties that are matched with the patient’s rehabilitation progress. Therefore, those approaches may not adjust to game difficulties that relate to the patient’s rehabilitation progress more effectively.

To achieve the aim, the objectives of this project include

  • Design of stroke rehabilitation games into a form suitable for use in a VR application.
  • Design user interfaces and interaction styles that are appropriate for stroke patients.
  • Develop a tracking method to identify and report potential stroke rehabilitation game cases.
  • Design and deploy a self-review stroke rehabilitation game application.
  • Evaluate the effectiveness of the application in improving post-stroke syndrome.
  • Analyze the flexibility of hand rehabilitation exercises for different patient requirements.
  • Collect the engagement results of stroke patients from their rehabilitation experiences.
  • Set up a suitable and comfortable rehabilitation experience for stroke patients.

Methodologies and Technologies used

Our solution is based on using the virtual reality system as a stroke rehabilitation method to provide a few different rehabilitation exercises for the hand. Stroke rehabilitation will be provided by using the game. The game content is divided into different kinds of minigames. Each game is related to specific stroke rehabilitation. There are three types of minigames for stroke rehabilitation, such as extension, balance, and strength training. Besides providing games, we will also develop games with dynamically adjustable levels of difficulty that ensure game difficulties are suitable for different patients.

  • VR headset is part of the solution to provide the user movement tracking function and VR application content display.
  • The controller is the input device for the solution, it provides the hand movement tracking function and pressure sensor, including receiving user hand action, user interface interaction, and strength measurement.
  • The database provides storing user training data. User data from the application will be collected for adjusting game difficulties.
  • The VR application provides game content by using the solution. Included using data from the database to adjust game difficulties and transfer user exercise results to the database.

For the supporting technologies, we would develop this project by using the Unity platform with OpenXR Plugin.

VR headset

As we mention that VR headset is part of the solution to provide a host of applications. Many VR headsets are existing in the market including Oculus Quest 2, HTC Vive Pro 2 Sony, PlayStation VR, Valve Index VR Kit, Windows Mixed Reality, etc. Those are fulfilling our requirement of tracking user movement. According to our requirement, the VR headset should be comfortable for the user. However, HTC Vive Pro 2 and PlayStation VR are not flexible, because those VR devices require the base stations for tracking user movement. Therefore, we will use Oculus Quest 2 to be our VR device for the solution, based on its portable, and all-in-one system, as there is no base station needed.

Pressure Sensor

The “PressureSensor” class might be useful while detecting the pressure (Unity, 2020) (OpenXR, KHRONOS). Due to the technology gap, we would face this project. Firstly, the basic VR controllers are not supported with pressure sensors so we might make customized controllers on our own. Next, we need an amount of 3D resources for building a few game scenes and reward decorations, buying assets might be an option to alleviate this technical gap.

Cloud server (EC2)

The game system needs a cloud server to deploy the content management system (CMS), and database and is used to store and transfer user performance data between the CMS and the game system.

Comparison of pressure input devices

After conducting research from the internet, we have found a few possible pressure inputs available in the market:

  1. Ring-con
    • Detect user pull and push strength with the force sensor
    • Price: HKD$648
  2. Valve Index VR controller
    • Check user hand position, finger position, motion, and pressure with 87 sensors
    • Seems to have to be used with the Valve Index VR handset
    • Price: USD$499.90/ USD$1,999.99 (with VR headset) (Not available in Hong Kong)
  3. Grip System
    • Measure, evaluate static and dynamic pressures from grasping objects
    • Study carpal tunnel and repetitive motion syndrome, or analyze the human hold on various tools
    • Price: Contact the company for pricing
  4. Tactile Glove
    • Measure hand movement correctly with 65 individual sensing elements within the glove
    • Full range of motion is supportably
    • Suitable for developing human movements, comfort, and ergonomics through visual feedback of hand interactions
    • Price: Contact the company for pricing

In conclusion,  only Ring Con is a suitable option for doing this project, Valve Index VR controller is too expensive for us, and seems not to support Oculus System, also not sold in Hong Kong.  Grip System and Tactile Glove we have no idea if they are even supported in Unity and VR or not. More importantly, these products can’t be bought at the public market, this is very unfavorable for our market liquidity if we use this device. Ring-con would be our first choice with a reasonable price we can pay with and most commonly, we have found possible ways to let the controller contact with unity which is hopefully can be our best option in using.

Figure 1: Ring-con

Figure 2: Valve Index VR controller

Figure 3: Grip System

Figure 4: Tactile Glove

Technical Gap

Controller for training handgrip strength

Although we have found the driver of Ring-Con (ringrunnermg, 2021), we still have to try and it doesn’t work and the compatibility with Oculus Quest System. If this method fails we might possibly have to retrofit the controller on our own. Honestly, we lack experience with retrofitting hardware devices.

 

            Suitable for stroke patients

We design our rehabilitation game based on the rehabilitation existence suggested by “中風多面睇” which publishing by the Hong Kong Department of Health(Department of Health, p. 38-132, 2019). But these games haven’t been clinically trialed on stroke patients yet, so some of our game designs might need to modify or even need to be rebuilt. Besides that the auto difficulty system needs a lot of tests for adjusting, the auto difficulty has to be suitable with the patient’s state and difficulty can’t be increased or decreased too hard at once.

Design and Creative Idea

As we are going to create stroke rehabilitation mini-games for stroke patients, the art direction and story genre should be more or less simple yet interactive. All posture references come from the stroke rehabilitation exercise guideline called 中風多面睇, published by the Department of Health.

  • For the mini-game “Base Hitter”, the player can train their arm movement in an exciting baseball match. The game focuses on training players’ arms movement.

Our team will create a baseball field scene. In the gameplay, the player has to use his baseball bat to hit the baseball from the pitcher. More information about game ideas in the game document.

Figure 5: Reference for gesturing in the mini-game “Base Hitter”

Figure 6: Screenshot of the mini-game “Base Hitter”

  • For the mini-game “Super Waiter”, the purpose of SuperWaiter is for training body balance.

Figure 7: Reference for gesturing in the mini-game “Super Waiter”

Our team will create a restaurant scene where the player portrays a waiter and takes the plate along the passage. The game simulates the real-life action of being one of the stroke rehabilitation balancing exercises with both hands. More information about game ideas is in the game document.

Figure 8: Screenshot of the mini-game “Super Waiter”

  • For the mini-game “Stop The Fire”, the game focuses on the players’ handgrip strength training. This is a challenging game for the player to retain handgrip strength.

Figure 9: Reference for gesturing in the mini-game “Stop The Fire”

In a virtual house, the player will release the fire extinguisher to put out the fire and refill the water in the extinguisher to use again. More information about game ideas is in the game document.

Figure 10: Screenshot of the mini-game “Stop The Fire”

System interface design

We would have a menu lobby, which is the first view a user would see while using our product. It is a small lobby environment where users can select a game by clicking these three white panels. The left red button would go to a shop menu for buying game items, and the right red button for logout the game.

Figure 11: Screenshot of game lobby UI

In the shop menu, it will display relevant information once the player clicks on the “Shop” button in the menu lobby. This menu shows various items in a list, for example, different color of item for Game01. The player can acquire them by paying for a specific coin.

Figure 12: Screenshot of shop UI

About Game 1, there will display a game instruction at the beginning for guiding the player on how to start the game and how to play the game. The player can interact with the UI by using the VR controller.

Figure 13: Screenshot of game 1 UI

During the game, the big black screen panel is used to show the upcoming time of the ball.

Figure 14: Screenshot of game 1 UI

About Game 2, there are 2 major parts of UI elements including game progress and interactable items. The game progress will be displayed on the top to represent the status of the game progress, and in the top right corner to show the number of player collected items. About interactable items, it is used to display the start game, tutorial, and back to menu UI elements.

Figure 15: Screenshot of game 2 UI

In Game 3, there are also some interactive UI elements, for example, we used the “Push” word on the door to represent the action of starting a game and the “Exit” word on the other side of the door to represent the action of exit the game.

Figure 15: Screenshots of game 3 UI

During the game, the game status will be displayed on the top for showing the score. Also, the percentage of extinguisher storage will be shown on the dashboard.

Figure 16: Screenshots of game 3 UI

Evaluation

Objective of Study

Our objective is to determine whether the three VR stroke rehabilitation games: (1) Base Hitter, (2) Super Waiter, and (3) Stop The Fire, provide beneficial outcomes in playing the VR stroke rehabilitation games, complying with appropriate UI design, interaction environment, and difficulty adjustment, establishing a suitable and comfortable rehabilitation experience for stroke patients, and increasing future engagement in using the VR application.

Methodology of Study

This study’s approach is based on user experiment design. Our team had arranged for a professional physiotherapist to give us post-stroke patients so that we could test our rehabilitation method. In addition to performing the rehabilitation test on the stroke patients, we asked several colleagues of our professional physiotherapist to observe if they could make the proper gesture while playing the rehabilitation game. We presented a questionnaire for his colleagues’ opinions on this rehabilitation approach after they were tested. It assesses these games as appropriate for stroke sufferers, allowing them to measure the system’s success and determine how effective VR stroke rehabilitation games and traditional stroke therapy in a rehabilitation center can improve the mobility of stroke patients.

Details of Study

With the proposed experiment, we will focus on the gesture while playing the rehabilitation game. Our team would ask a professional physiotherapist to watch whether the testers made the proper gestures while playing the rehabilitation game and assess whether these activities were appropriate for stroke sufferers. For the stroke patients, we have discovered that they are mostly aged and have different physical disabilities. Therefore, we had to converse with aged people with different physical disabilities to instruct them on how to play the games. The following shows the experiment for our games.

Significance of Results

Extent of Satisfying Requirements

For the game “Base Hitter”, according to the questionnaire from our physiotherapist’s colleagues, we have received generally positive feedback for satisfying the requirements, with an average score of 6.5. But there is some other feedback our team should be concerned about. In the UI aspect, we have commented that the hot air balloons in the background are significantly appealing while batting the ball, whereas stroke patients may not be able to distinguish between the hitting targets and the balloon and affect the outcome of the result. In the game control aspect, one of our physiotherapist’s colleagues expressed in the questionnaire that it is difficult to target the ball for batting because of the inaccurate distance between the target and the controller. With the conversation and interaction with one of the stroke patients, we have found that in the game difficulty aspect, because of their physical limitation, she is not able to stretch the hand to the hitting targets, therefore, we must hold and direct her hand in order to reach the hitting targets.

For the game “Super Waiter”, according to the questionnaire from our physiotherapist’s colleagues, we have received generally positive feedback for satisfying the requirements, with an average score of 6.5, very similar to the first game. but there are also some other feedbacks our team should also be concerned about. In the game control aspect, we have been told that the game camera is especially unstable in comparison with other games, that the patient may easily get dizzy during and after playing the game, especially for the elderlies or people who cannot withstand the instability. In the game content aspect, one of our physiotherapist’s colleagues has expressed in the questionnaire that the game pace is too quick for post-stroke patients, as the response of the post-stroke patients is not as quick as normal people. From our observation of another stroke patient, we have found that from the difficulty aspect, she is unfamiliar with the strategy of controlling the game, and this leads to a low score. This has shown that the difficulty adjustment is above the level of the stroke patients from the start, and becomes too difficult for them to play.

For the game “Stop the fire”, we have received generally negative feedback for satisfying the requirements. As there is not enough time for review from our physiotherapist’s colleagues, there is no questionnaire feedback provided. But according to our interaction with one of the stroke patients, in the game control aspect, he is unable to stress the Ring-Con to the depth where the system detects the control from the Ring-Con. Moreover, if the stroke patient cannot use his both hands, the rehabilitation test cannot be conducted. In the game design aspect, we have found that the spawn angle for the fire is too broad, as the stroke patient is not able to turn his hand towards the fire target.

Strengths and Weaknesses of Solution

For the effectiveness of the hand extension training from the game “Base Hitter”, by the response of the questionnaire feedback, the colleagues from our physiotherapist generally agree that this game has improved hand extension performance of the stroke patient, and the stroke patient we have tested for do feel tense and tiredness when she stress out her arm after the rehabilitation game is played. This is a good sign, as the game has enhanced the activities of the patient’s have strength, providing evidence of the game design effectiveness.

For the effectiveness of the hand balance training from the game “Super Waiter”, by the response of the questionnaire feedback, our physiotherapist’s colleagues also generally agree that this game does improve stroke patient’s hand balance. Our stroke patient who has been tested for does feel tired after holding her arm at a position after the rehabilitation game is played. This has also shown the hand balancing activities of the patient, providing evidence of the game design effectiveness.

For the effectiveness of the hand strength training from the game “Stop the fire”, by the reaction from our stroke patient, he does feel tired while stressing his both arms after playing the rehabilitation game. Therefore, although the design of the system is not as good as we were expecting, the game does enhance stroke patients hand strength, providing evidence to the effectiveness of the game design.

Other Findings

Besides the findings related to the requirements, there are more findings that would affect the rehabilitation experience:

There are existing technological errors. As the game “Stop the fire” requires the highest technological skills, It does contain technological slips. For example, the sensor of the Ring-Con controller is too narrow, therefore while the stroke patient is not able to extend his/her hand, the system is unable to detect the Ring-Con controller, and the stroke patient is not able to complete the game.

Moreover, Oculus Quest 2 Headset is commented to be too heavy for the player to hold on and play the rehabilitation games, where they may easily get tired after playing the rehabilitation games.

Conclusion

Aims and Objectives

  • The project objective is to develop a VR Rehabilitation Game that can fulfill three aims:
    • To fulfill more kinds of hand training requirements
    • To Encourage the patient
    • A suitable difficulty for the patient

Summary of Achievements

  • After the evaluation, our summary on different games by questionnaire and interview:
    • For game 1, Although the game has reached our goal of improving the hand stretch, some replies said that the ball target is a bit too far from the player, and the hitbox of the bat is confusing. The air balloon in the background would make the patient out of focus.
    • For game 2, the game also achieved our goal of improving the hand balance, but the patient thought that it’s hard to control and a bit too fast.
    • For game 3, the patient thinks that this game is hard to play because he has to use a lot of strength to pull the Ring-Con. He also thinks that it’s hard to turn his body around to save the fire.
  • Besides, we have realized there are a few problems with our solution. Firstly, some of the users feel the headset is heavy while they play for more than around 5 minutes. Another problem is that some of them feel dizzy after using the VR headset.
  • The auto difficulty adjust system is fully operational in 3 games, but some games have to change their difficulty from the level design, which the auto difficulty adjust system cannot help with.
  • The shop system already allows players to buy new skins with coins and apply them to 3 different games.

Limitations of Solutions and Methods

            The limitation of the project is the VR headset itself, because the Oculus Quest 2 weighs only 454g, which is a very light model in VR headsets, but still makes the patient feel heavy. We can’t change the headset weight, but we can change the headset headband to make the head have a better weight balance to make the user feel comfortable.

Figure 17: Oculus Quest 2

Future Development

  • For game 1, the target has to move closer to the player to make sure he can touch it. The air balloon has to scale down a bit or change to a non-eye-catching color to prevent the player from focusing on it. To make the player easier to understand, the hitbox has to cover the whole bat, and six hit points would be reduced to three positions (left, middle, right).
  • For game 2, the balance sensor has to decrease the sensitivity because this is too difficult for the patient. Also try to make it slowly so that the patient can handle it.
  • For game 3, the water pump strength required has to decrease, and the fire spawn point has to stay on the player’s front.
  • For the whole system, we can design a development mode for physiotherapists to set the default difficulty before handing it over to the auto difficulty system to do the rest. The gameplay time has to be controlled around the 20s to 30s to make sure the player won’t easily feel dizzy. It’s better to add a camera reset function so that the helpers don’t need to repair it each time. This is very influential on the user experience.
  • For the encouragement, coins have to be more functional to buy more things such as game effects (e.g., Base Hitter hitting effect) so that there are more alternatives the user can change. On the other hand, the gameplay of these three games has to be improved. A more interesting gameplay can attract the player to play again.
Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course CodeTitleCredits
 COMP S321FAdvanced Database and Data Warehousing5
 COMP S333FAdvanced Programming and AI Algorithms5
 COMP S351FSoftware Project Management5
 COMP S362FConcurrent and Network Programming5
 COMP S363FDistributed Systems and Parallel Computing5
 COMP S382FData Mining and Analytics5
 COMP S390FCreative Programming for Games5
 COMP S492FMachine Learning5
 ELEC S305FComputer Networking5
 ELEC S348FIOT Security5
 ELEC S371FDigital Forensics5
 ELEC S431FBlockchain Technologies5
 ELEC S425FComputer and Network Security5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5