School of Science and Technology 科技學院
Computing Programmes 電腦學系

Banknote Recognition: Using Machine Learning in Assistive Technology for the Visually Impaired

Sin Hang CHUNG, Yuen Yan LEUNG, Hoi Shan PANG

ProgrammeBachelor of Computing with Honours in Internet Technology
SupervisorProf. Vanessa NG
AreasIntelligent Applications
Year of Completion2019

Objectives

 

One of the crucial difficulties that visually impaired people face in daily life is recognizing the value of banknotes. They cannot identify the value of banknotes by eyes but can sometimes identify the value by hands touching the accessibility features on banknotes. However, as time goes by, the accessibility features on banknotes will get damaged and become unreliable. Identifying values on banknotes becomes one of the biggest challenges of them.

The aim of this project is to develop a mobile application to help visually impaired people recognize the value of current Hong Kong banknotes. The application should (1) show the value of banknotes with (2) speech synthesis and (3) cell phone vibration coupled with (4) machine learning.

To achieve the aim, the main objective of the project is to develop a mobile application that can recognize the 2003 series and 2010 series Hong Kong banknotes which are the highest circulation banknotes. The project has also defined a number of sub-objectives as follows:

  • Collect image data for machine learning. The data should contain a wide variety of banknotes’ patterns.
  • Adapt a CNN model for transfer learning. The model will return a correct value of a banknote picture for the visually impaired based on the data collection of different banknotes.
  • Implement the prototype mobile application. Add in the feedback system, which is composed of speech synthesis and vibration.
  • Evaluate the prototype mobile application. We invited visually impaired people to use the mobile application and modify the mobile application functions based on their feedback.

Video Demonstration

Background and Methodology

System architecture

Technique used for training the neural network for predicting more accurate results in banknote prediction

To enhance the accuracy of the neural network in recognizing Hong Kong banknotes, data sampling, data augmentation, and the technique of comparing different models have been applied.

Data sampling

In order for neural network model to extract informative features and learn the patterns of banknotes, high-quality training dataset are required. The training dataset must contain variant forms of banknotes’ images; therefore, discrimination, i.e. can only recognize under certain distance or brightness, will not be occurred in the neural network.

Data augmentation

To avoid overfitting, which is a common problem in training machine learning model, data augmentation has been applied. Data augmentation is a powerful way for image classification with a limited number of data to generate more data from the existed dataset.

Figure 2: Data augmentation

Selecting a model with the best validation accuracy

To get a more accurate result, we had compared different models and eventually chose the best model with high validation accuracy.

System Design and Implementation

The process will begin with a captured image. Then the image starts the pre-processing and feature extraction.

The image will first be passed through to the binary classifier, which is used for determining whether an image is a Hong Kong banknote. If the image is not a Hong Kong banknote, the application will send the non-banknote object feedback. Otherwise, the image will be passed through the multi-class classifier.

When the multi-class classifier has a result, the application will check the confidence of the result. If the confidence of the result is not high enough, the result will be ignored, and the system will require the user to try again. Otherwise, the result will be shown in the application with verbal feedback and vibration.

Figure 3: Flow chart of processing and analyzing images

Figure 4: Recognising the front side of HKD$100 banknote

Figure 5: Recognising the rear side of HKD$100 banknote

Figure 6: Recognising the font side of HKD$20 banknote

Figure 7: Recognising the rear side of HKD$20 banknote

Evaluation

 Time
Test 141ms
Test 277ms
Test 374ms
Test 491ms
  
Min41ms
Max91ms
Avg70.75ms
  

Table 1: Speed of test 1-4

4 recognition tests have been performed. All of them finished under 100ms. The minimum time was 41ms and the maximum time was 91ms. The average time was 70.75ms.

Compared to recognition of banknotes by hands, the time has been improved by 60000%, from 1 minute to 100ms. In contrast with the recognition by Note-measuring Template, the time has been improved by 30000%, from 30 seconds to 100ms.

 

Conclusion and Future Development

The aim of this project has mostly been satisfied. The prototype mobile application can show the value of banknotes with speech synthesis and cell phone vibration coupled with machine learning. As we collected a wide variety of banknotes’ patterns images for machine learning, our trained prototype’s model has returned correct values for banknote pictures, and the feedback system has worked well. The prototypical mobile application has been created with high accuracy and high speed.

In the future, we will improve our work by the development of identifying other countries’ banknotes. We can adapt the model used in the prototype and change the datasets with other countries’ banknotes so that we can help more visually impaired people from other countries to recognize their banknotes.

 
Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course CodeTitleCredits
 COMP S321FAdvanced Database and Data Warehousing5
 COMP S333FAdvanced Programming and AI Algorithms5
 COMP S351FSoftware Project Management5
 COMP S362FConcurrent and Network Programming5
 COMP S363FDistributed Systems and Parallel Computing5
 COMP S382FData Mining and Analytics5
 COMP S390FCreative Programming for Games5
 COMP S492FMachine Learning5
 ELEC S305FComputer Networking5
 ELEC S348FIOT Security5
 ELEC S371FDigital Forensics5
 ELEC S431FBlockchain Technologies5
 ELEC S425FComputer and Network Security5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5