School of Science and Technology 科技學院
Computing Programmes 電腦學系

EYEDENTIFY(便認): A mobile application for improving the self-esteem of dementia patients in their social life with machine learning

Yeung Chin Mong, Leung Tsz Kin, Cheung Man Fai

  
ProgrammeBachelor of Computing with Honours in Internet Technology
SupervisorProf. Vanessa Ng
AreasIntelligent Applications
Year of Completion2021

Objectives

This project aims to develop a mobile application which targets users who are early-stage dementia patients. This application is focusing on their social life. Their self-esteem would be built up through the application. Not only the patients’ self-esteem would be benefited, but also the relationship among people around them can be enhanced.
  1. Design the user interface of the mobile application
  2. Develop the functionalities of the mobile application, it includes
    • Develop a facial recognition function which can display database results
    • Develop an object recognition function which can display pictures stored in the database
    • Develop a location sharing with GPS that will emit GPS signals to the caretaker
    • Develop a memoir that can display photographs with each person in the database individually
  1. Develop a machine learning server to:
    • Recognize and analyze the face of family members through the photos taken by Raspberry Pi
    • Recognize and analyze the photo of family members through uploading photos
    • Recognize and analyze the object of the patient through the photos taken by Raspberry Pi
  1. Train the machine learning model to improve the accuracy of the response
  2. Implement and test each function of the mobile application
  3. Analyzing the performance of the machine learning model

Techniques and technologies used

  • Technologies (Software):
    1. Facial Recognition
    2. Object Recognition
    3. Google Map API, used to locate patients and provide GPS sharing function
    4. Computer Vision Services
    5. Web Framework for facilitating HTTP request
  • Technologies (Hardware):
    1. Raspberry Pi with camera module to capture images
  • Technologies (Core):
    1. Mobile, database, python web server

System Design and Implementation

Dementia patients will use cameras to capture real-world images and transmit them to object recognition and facial recognition functions. Object recognition will retrieve photos from the database to see whether it has been matched and return the result. While the facial recognition function will store facial images into the user database. The memoir function will use the facial recognition function to detect faces inputted by both caretakers and patients and display them to the patients with a grid layout. The location of the dementia patients will send to their relatives through Google Maps API.

Figure 1: Overall system design diagram

Data Flow Diagram:

The database will send photos and names to the game. After users finish the game, they will go to the result screen and show the result. In the meantime, the score will be stored in the database.

Figure 2: Data Flow Diagram of Object recognition

Raspberry Pi will be used as an input device to send video capture to the machine learning server for recognition. At the same time, the database will send all belongings photos to the server in order to match the capture. After successfully matching, the server will send the latest screenshot to the database. Once the user requires the latest position of his/her belonging, the application will retrieve the result from the database through the application. 

Figure 3: Data Flow Diagram of facial recognition

Raspberry Pi will be used as an input device to send video capture to the machine learning server for recognition. At the same time, the database will send all previously matched faces with names to the server in order to match the new one. After successfully matching, the server will send the screenshot with the name of the person identified to the database. Once the user requires the name and relationship of someone, the application will retrieve the result from the database through the application.

Figures 4: Data Flow Diagram of Memoir

When dementia or caretaker user uploads photos to the application. The photos will be sent to the Machine Learning Server. Then the server will request all previously matched faces with names from the database. After matching faces and adding name tags to the photos, the server will update the memoir in the database. Once the Dementia user requires the memoir, the database will send the photos to the application.

Figures 5: Data flow diagram of GPS sharing

Google API will be used to detect the location of dementia patients and send it to their relatives.

Evaluation

Correctness Analysis

In order to test the accuracy and the rate of incorrect detection of facial recognition and object recognition, we have used 40 testing data for facial recognition functions and 30 testing data for the object recognition function to test the functions.

In the test of facial recognition, we have used 4 photos which the people in the photos who are not in the trained data set, and 35 normal photos. The results of detecting the unknown faces are 80%. The program has successfully detected the unknown faces in the 4 photos. Those photos will not show in the application, but they will be saved in the database. There is 1 photo with no tag which means no face in that photo has been detected. The results of detecting the known face (Trained already) are 82.8%. In those 35 photos, the program has returned correct results which means a photo with a correct name tag and relationship has appeared in the application.  After testing, we realized that the people cannot be recognized by the program if they are not facing the camera since the program had returned no faces detected in 3 photos. Also, 3 photos returned the wrong result. Those faces mismatched the name and relationship.

In the test of object recognition, we used 10 photos that did not contain glasses, keys, and wallets and 20 photos that had different positions and groupings of glasses, keys, and wallets. In those 10 photos, there 6 photos identified no glasses, keys, and wallets but 4 photos show either glasses or keys. In recognizing the unknown belonging, the accuracy is 60%. For the other 20 photos, 14 photos successfully detected glasses, keys, or wallets. We discovered that the belonging if were overlapped would fail to detect. Also if the belongings are too far from the camera, the program could not detect the belongings. In recognizing the known belonging, the accuracy is 70%

In conclusion, the accuracy of facial cognition is around 80% – 83.3%. The accuracy of object cognition is around 60% -70%.

Conclusion

To sum up, the implementation of EYEDENTIFY is fruitful. This project has successfully implemented a mobile application which concentrates on the needs of Dementia patients.

The implementation of face recognition and memoir functions has been done smoothly by using Face recognition (DLIB). In the face recognition functions, a real-time face recognition system is realized, the user identifies their family and friends. By using the Raspberry Pi with the camera module, the photo will pass the Face recognition (DLIB) program. After face detection and matching, the results will be shown in the application. Dementia users can remember the identity of the people they are meeting. In the memoir function, both dementia users and caretaker users can upload photos. After matching faces by Face recognition (DLIB), dementia users can see the photo with the name tag of their family and friends and share the same memories with them.

The implementation of the object recognition function has been completed with Tensorflow. A function like lost and found has been created for Dementia users to find their personal belongings e.g. glasses. As Dementia users lose their belonging, they can find the last seen photo of that belonging and locate it. The last seen photo was taken by Raspberry Pi and processed by Tensorflow. Since Dementia users can find their belongings by themselves, fewer controversies with family and friends will happen.

The implementation of the GPS sharing function has been accomplished favorably with Google Map API. Both Dementia users and Caretaker users can see each other’s last seen location which brings an equal relationship.

Future Development

To ensure the application remains competitive and stands the test of time, further development should be done. Both hardware and software can be upgraded to provide a better service.

In hardware development, Raspberry Pi is the core hardware of the whole system. First, the physical size of Raspberry Pi should be reduced to increase portability. The Raspberry Pi must operate with a power bank now. Both items should be combined to reduce the volume. Second, the camera in Raspberry Pi should be upgraded in order to increase the resolution of the photos. A higher resolution can increase the accuracy of the face and object recognition.

In software development, several developments should be completed in the future. First, the existing functions should be enhanced. For the object recognition functions, more items of target objects should be added. There are only four items that can be detected in the current function. Users should be able to add any belonging that they need to find to the function in the future. For the GPS sharing function, a direction feature should be built to show the fastest way to go to the current location of each user. Second, some new features should be created to boost the comprehensiveness of the mobile application. An instant messengers function can be added into the application to close the relationship as users can talk to each other at anytime and anywhere. A ‘GO-HOME’ function can be implemented into the application. As dementia patients will lose their way easily, a function shows the fastest way to go back home and notify their caretaker to ensure their safety.

Future Development

To ensure the application remains competitive and stands the test of time, further development should be done. Both hardware and software can be upgraded to provide a better service.

In hardware development, Raspberry Pi is the core hardware of the whole system. First, the physical size of Raspberry Pi should be reduced to increase portability. The Raspberry Pi must operate with a power bank now. Both items should be combined to reduce the volume. Second, the camera in Raspberry Pi should be upgraded in order to increase the resolution of the photos. A higher resolution can increase the accuracy of the face and object recognition.

In software development, several developments should be completed in the future. First, the existing functions should be enhanced. For the object recognition functions, more items of target objects should be added. There are only four items that can be detected in the current function. Users should be able to add any belonging that they need to find to the function in the future. For the GPS sharing function, a direction feature should be built to show the fastest way to go to the current location of each user. Second, some new features should be created to boost the comprehensiveness of the mobile application. An instant messengers function can be added into the application to close the relationship as users can talk to each other at anytime and anywhere. A ‘GO-HOME’ function can be implemented into the application. As dementia patients will lose their way easily, a function shows the fastest way to go back home and notify their caretaker to ensure their safety.

Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course CodeTitleCredits
 COMP S321FAdvanced Database and Data Warehousing5
 COMP S333FAdvanced Programming and AI Algorithms5
 COMP S351FSoftware Project Management5
 COMP S362FConcurrent and Network Programming5
 COMP S363FDistributed Systems and Parallel Computing5
 COMP S382FData Mining and Analytics5
 COMP S390FCreative Programming for Games5
 COMP S492FMachine Learning5
 ELEC S305FComputer Networking5
 ELEC S348FIOT Security5
 ELEC S371FDigital Forensics5
 ELEC S431FBlockchain Technologies5
 ELEC S425FComputer and Network Security5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5