School of Science and Technology 科技學院
Computing Programmes 電腦學系

Caring Low Vision Patients: A Mobile Application to Assisting the Use of Over the Counter Medicines for Low Vision People

Kwok Chin Hung

  
ProgrammeBachelor of Computing with Honours in Internet Technology
SupervisorDr. Henry LEUNG Man Fai
AreasInnovative Applications
Year of Completion2021

Objectives

The project aim of this project is to develop a mobile application with a medication recognition and information system and reminding system in order to assist people with low vision to take and purchase the over the counter medicine correctly.

A. Medication Recognition and Information System:

The medication information system is to classify the medicine package through the Android mobile phone’s camera. After the user capture the image through the application, the image will be sent to the server for recognition and retrieve the medication information from the database. The obtained information will be sent back to the user’s application. After that, the system will provide the organized medicine information to the user including drug name, dosage, efficacy, caution etc.

B. Reminding System:

The reminding system in this project is a voice-based reminder to remind the user to take the correct medication on time. The user with vision impairment can set the schedule of reminder. Also, the user can register the medicine that they need to take on time by using the medication recognition system. By the time of taking medicine, the system will remind the user to take that registered medication.

In order to achieve the aim, there are a number of objectives in my project as follows:

  1. Collect images of medicine package and train an image classification model.
  2. Develop a medication recognition activity and reminding system in the application.
  3. Develop a server with pre-trained model and OCR function to compare the result of pre-trained model and OCR function.
  4. Develop a database to store and provide the medicine information.
  5. Design the user interface to increase the usability for visually impaired.
  6. Evaluate the performance of recognizing the medicine package.
  7. Evaluate the usability of the application.

Methodologies and Technologies used

A. Convolutional Neural Network:

Convolutional Neural Network (CNN) is a technology that provide high performance for image classification. It is applied in different areas such as handwritten character classification, face recognition and diagnosis disease from X-ray images, etc. Accounting to O’Shea and Nash [13], Convolutional Neural Network is composed by these three kinds of layer: convolutional layers, pooling layers, and fully-connected layers. Albawi, Mohammed and Al-Zawi [14] stated that the performance of the CNN model is affected by the number of layers in the model and the number of layers increase the training time of the model. 

Transfer learning is introduced to address this problem. Hussain, Bird and Faria [15] mentioned that transfer learning is a method that repurpose the pre-trained model from one task to other tasks. Shu [16] presents that transfer learning is a time saving method to obtain a better performance of the model with a small dataset.

B. VGG Neural Network:

Simonyan and Zisserman [17] developed the VGG neural network for image classification. VGG16 and VGG19 are commonly used model for transfer learning. Regarding the VGG16 model, there are 13 convolutional layers plus 3 fully-connected layers in the network. Compared with VGG16, there are 16 convolutional layers plus 3 fully-connected layers in the VGG19 model.

C. ResNet Neural Network:

He, Zhang, Ren and Sun [18] proposed the residual learning technique for training a deeper model and developed the ResNet models. It is stated the residual learning technique can overcome the degradation problem during the training of deep neural network. The ResNet50 model is another frequently used model for transfer learning. The architecture of ResNet50 model is shown below.

Key Technologies
TechnologyDescription
Tensorflow, KerasTensorflow and Keras provided a powerful function for transfer learning and image preprocessing. These libraries are used in training an image classification model and the server for preprocessing the received image data.
Optical Character RecognitionGoogle Vision provides an accurate Optical Character Recognition API for the user to extract the text from the image and return a json object with the text extracted from the image.
SequenceMatcherSequenceMatcher from difflab is very useful for comparing the similarity of two string in Python. This will be used in the server.
Flask Web-application FrameworkFlask is an easy to use framework for developing a server in Python for handling incoming HTTP requests.
MongoDBMongoDB Atlas is a NoSQL cloud database for storing JSON document.
AndroidAndroid is an operating system for a mobile phone. The application will be installed in this environment.
SmartphonePeople with low vision can use our application through the Android smartphone. They can capture image for recognition through the camera and interact with the application by tough screen and screen reader application.
Android BroadcastReceiver, AlarmManagerTwo Android library for receive the signals of reboot and time for the reminding.
OkhttpOkhttp is an Android library for sending the HTTP request to the server.
CameraXOkhttp is an Android library for calling the camera function in the Android devices.

Table 1: Key Technologies

System Design and Implementation

A. Hierarchical Structure Diagram:

Figure 1: Hierarchical Structure Diagram

The whole system in this project is mainly divided into three parts: Android application, Flask server and MongoDB database. There are two sub-system in the Android application which are Reminding system and Recognition system. In the Reminding system, user can set the reminder to remind them to take the medication. The recognition system can provide the medication information after the medicine is identified. Both of these sub-systems require to call the camera intent (The designed and easy to use camera for people who have an issue of vision impairment) to take the photo of the medicine. After taking the photo, the photo will be sent to the server by sending a HTTP request with the Base 64.

Regarding the Flask server, there are three functionalities: handling the incoming HTTP request, getting the medication information from database, recognizing the medication. The server will send the query to the database by using the MongoDB API call. In the recognition system, the image classification model, and Optical Character Recognition (OCR) will be used. The server will call the Google Vision API for the OCR function.

With respect to the MongoDB database, the medicine information will be stored in the cloud database including the drug name, dosage, caution, ingredient, and efficacy. MongoDB is a NoSQL cloud database so that the information is stored in a JSON document.

B. User Interface Design:

Figure 2: Location of help button

Figure 3: Simple camera design

Figure 4: Info table

Figure 5: Reminder list

Figure 6: Reminder setting form

Figure 7: Reminder screen

Evaluation

Performance Evaluation

The aim of the performance evaluation testing is to evaluate the performance of the recognition system. Different factors will be measured in this testing including the accuracy, precision, recall and F1-score.

ClassPrecisionRecallF1-score
Clarityne1.000.960.98
Extra Fast Coltalin1.000.990.99
Cortal (Adults)1.000.970.98
DecaughII1.000.920.96
Mucinex1.000.950.97
Ulance1.000.950.97
Out of distribution (i.e., not recognized)0.661.000.80

Table 2: The evaluation result: Precision, Recall and F1-Score

Usability Evaluation

The aim of the usability testing is to evaluate the usability of the application for low vision user. Office of the Government Chief Information Officer [19] provided a Best Practice Checklist for developers to evaluate the accessibility of the application. The requirements in the checklist are the important features for people with disability including visually impaired to access and use the application. The checklist for Baseline level accessibility application will be used to evaluation the usability of the application.

In order to check whether the application fulfill the Baseline level for visually impaired people to use or not, the procedure of the testing is that the application will be checked with each requirement in the Best Practice Checklist in Appendix A. Some requirements are pre-marked as “Skip”. It means that the requirements are not needed to check. Moreover, the irrelevant requirements can be mark as “N/A”.

Regarding the evaluation result, all related requirements are fulfilled. Therefore, the Baseline level of accessibility application is achieved. Hence, the usability of the application is tested to be enough for low vision user.

Conclusion and Future Development

This project aims to develop a system for assisting the low vision people to take the over the counter medication correctly. The visually impaired people are facing different problems when they need to take the over the counter medication including taking wrong medicine, wrong dosage. By using our system, the low vision user should be able to recognize the over the counter medication, receive the correct medication information and take the medication on time.

There are seven objectives are mentioned. The purpose of these objectives is for achieving the aim which mentioned above. All objectives are satisfied. By evaluating the system, the result shows that the system can provide a good performance for recognizing the over the counter medicine and the application have enough accessibility for low vision user. Therefore, the aim of the project is also achieved.

Suggestions on future work

After obtaining the result of the evaluation process and considering the limitations, some future works can be done to improve the system.

First of all, a classifier can be added to the system to identify whether the image contain two or more medications. If there are two or more medications in the picture, the system should require the user to capture another photo. More research should be done to find or develop the method for the classifier. By adding this classifier, the limitation of prediction error for the image that contain two medications can be improved. Hence, the system will become more reliable for the low vision user to recognize the medication and get the corresponding information.

Secondly, regarding the usability of the Android application, the usability can be further improved to make the application more suitable for low vision user to use. Beside the Baseline level checklist, Office of the Government Chief Information Officer [19] also provided a Best Practice Checklist for Advanced level of accessibility application. That checklist provides some advanced requirements for accessibility application to make the application more convenient for people with disability to use. Therefore, the usability of the application can be further improved in the future.

Finally, more medications should be added to the database and the image classification model. There are many over the counter medications that people can purchase at supermarket or drugstore in Hong Kong. Therefore, we should include more medications in our system to make the low vision user feel more convenient and confidant to take the over the counter medications correctly.

Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course CodeTitleCredits
 COMP S321FAdvanced Database and Data Warehousing5
 COMP S333FAdvanced Programming and AI Algorithms5
 COMP S351FSoftware Project Management5
 COMP S362FConcurrent and Network Programming5
 COMP S363FDistributed Systems and Parallel Computing5
 COMP S382FData Mining and Analytics5
 COMP S390FCreative Programming for Games5
 COMP S492FMachine Learning5
 ELEC S305FComputer Networking5
 ELEC S348FIOT Security5
 ELEC S371FDigital Forensics5
 ELEC S431FBlockchain Technologies5
 ELEC S425FComputer and Network Security5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5