School of Science and Technology 科技學院
Computing Programmes 電腦學系

A Chinese Character Training Mobile Application for Dyslexia Children through Machine Learning and Augmented Reality

Nick Lok Hei HO, Apple Hiu Tung MA, Henry Chun Heng YEE

ProgrammeBachelor of Computing with Honours in Internet Technology
SupervisorProf. Vanessa Ng
AreasAugmented Reality Applications
Year of Completion2020
AwardYPEC 2020 Best Innovation Award (Sub Degree +Undergraduate section)

Objectives

The aim of this project is to encourage dyslexia children to learn chinese characters in a self-directed way through gamification of training exercise and machine learning.

The main objective of the project is to develop a mobile application that can analyze the dyslexia children’s performance when they are using the application and provide a suitable response.

The project has also defined a number of sub-objectives as follows:

  • Design the functionalities of the mobile application
  • Design the user interface of the mobile application
  • Design the content of the AR game
  • Train the machine learning model to improve the precision, recall and accuracy of the response
  • Implement and test each function of the mobile application
  • Evaluate the performance of the machine learning model, the input handwriting may be sent to the existing machine learning apis, since it involves sending data through the internet, it may take up long waiting time to retrieve the response from the api. So testing is needed
  • Organize the response data produced by the machine learning model and present the information back to the user

Video Demonstration

Background and Methodology

Our project is designed to encourage Dyslexia children to learn chinese characters in reading and writing. In order to improve the user experience, we decided to apply the prototype and experimental methodologies.

For prototype methodology, we prefer to build a prototype that can achieve the user needs. We will first develop a basic structure of a mobile app from our previous research on Dyslexia, then we will build a prototype from the structure and improve the app from the user response. After applying this methodology, it can help us rapidly develop the prototype which is suitable for the user. Also, we can detect the bug and reduce part of them before we begin to develop the mobile app.

For experimental methodology, we want to test whether our app can improve user reading and writing on chinese characters.We will separate two groups of users, and one group will use our mobile app and another group will not use our mobile app.After one hour, we will give a test to them, from the test result can show that our mobile app can improve or not. After we get the result, we can get data of what will affect the result and we can improve our game and logic from these datasets.

Techniques and technologies used:

Unity

Game engine for developing our solution

Microsoft Azure cloud platform

For machine learning

Microsoft Azure Custom Vision Service

For machine learning (recognizing normal Chinese character)

Google Cloud Vision API Service

For machine learning (recognizing normal Chinese character)

Vuforia

Game Engine for Augmented Reality development environment

System Architecture

Figure 1: Overview System Design

Our solution involves three major components, an AR Game component, a camera recognition component and a handwriting recognition and training component. The game component on the left hand side involves Vuforia engine and its web service, the camera recognition and handwriting recognition component on the right hand side involves different techniques like cloud service and machine learning algorithm. Note that the game part and camera part require permission in order to access the mobile camera and use it as the input.

System Design and Implementation

Main Screen

There are three buttons in the main screen which allows the dyslexia children access to different components of our application. The game console button, camera button and the book icon represent the augmented reality game component, camera component and training component respectively.

Figure 2: Main Menu

Figure 3: Key Function – Stroke Detection

Training Component

Figure 4: Training Menu

Figure 5: Search Mode

Figure 6: Random Mode

When the user enters the training part, two different modes of training will be allowed for the user to choose, the two modes are shown in Figure 4.

The first mode is search mode (搜尋文字), once it is clicked it will enter the searching mode which allows user to look for specific character as writing exercise.

The second mode is random mode (隨機文字), it will assign a random character to the user to write once it is clicked.

Camera Component

Figure 7: Game Component – Photo Capturing

Figure 8: Game Component – Analyzed Result

Figure 7 shows the photo capturing mode of the camera component. Once the user enters the camera, it will open the mobile camera (permission required). Then the user will be able to capture the handwriting that he/she wanted to recognize.

Figure 8 shows the analyzed result from the two cloud machine learning models. We have organized the output of the result and display the result back to the user.

Note that if the user wrote the character wrong, it will display the correct writing of that specific character. Also, it will indicate whether the components of the character are horizontally swapped or vertically swapped.

Game Component

Figure 9: Game Menu

Figure 10: Unclear level

Figure 11: Clear level

Everytime when the user enters a level, it will start the mobile camera first, then it will look for the Augmented Reality Image Target.

In Figure 10, there are two image targets placed on the table. Once the application recognizes the image targets, it will generate the object (puzzle) and map it to the corresponding image target.

Then the user can play with the image targets by moving it. And the corresponding puzzle will also be moved in the camera.

Once the user built the correct Chinese character with the puzzle. A prefab (object model) will be generated to indicate that the user has completed this stage.

Evaluation

User evaluation

We invited 3 child education experts and 1 special education expert and conducted user evaluation towards them.

The questionnaire result shows the positive point of view toward the implementation of augmented reality and machine learning with special education. According to Figure 5.2, most of the interviewees agreed the usefulness of AR and machine learning toward special education (response answer ranged from neutral to strongly agree).

Also, it makes comparison between the traditional learning style and our solution and it shows a positive result towards our solution.

Figure 12: Featured Question and Response 1

Figure 13: Featured Question and Response 2

Figure 14: Featured Question and Response 3

Conclusion and Future Development

The aim of this project has been achieved partially. We developed a solution which integrates the Augmented Reality and Machine Learning technologies successfully. The training component and game component of the application can assign exercise to the dyslexia children and make responses based on their performance. The camera component can be used as a support tool for the children to check his/her handwriting.

Future work

We suggest contacting different schools, Special Education Needs Association in the future to access different dyslexia children so that we may acquire their handwriting work, which can be used to train the Microsoft model.

We also suggest to perform a model evaluation again on the Microsoft model after new datasets have been used to train the model.

Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course CodeTitleCredits
 COMP S321FAdvanced Database and Data Warehousing5
 COMP S333FAdvanced Programming and AI Algorithms5
 COMP S351FSoftware Project Management5
 COMP S362FConcurrent and Network Programming5
 COMP S363FDistributed Systems and Parallel Computing5
 COMP S382FData Mining and Analytics5
 COMP S390FCreative Programming for Games5
 COMP S492FMachine Learning5
 ELEC S305FComputer Networking5
 ELEC S348FIOT Security5
 ELEC S371FDigital Forensics5
 ELEC S431FBlockchain Technologies5
 ELEC S425FComputer and Network Security5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5