School of Science and Technology 科技學院
Computing Programmes 電腦學系

A hybrid algorithm based on Backpropagation with Magnified Gradient Function guided by Adaptive Particle Swarm Optimization in Neural Network Training

LUI Wing On

Programme Bachelor of Science with Honours in Computing
Supervisor Dr. Vanessa Ng
Areas Neural Networks, Algorithms
Year of Completion 2012
Awards Recieved IEEE HK Section Student Paper Contest 2012 Undergraduate First Runner-Up

Abstract

Neural Network is a useful machine-learning model. Training up a Neural Network is however a complicated task.

Two of the existing Neural Network training algorithms are Backpropagation with Magnified Gradient Function (MGFProp) and Adaptive Particle Swarm Optimization (APSO). MGFProp is good at searching for optima but usually requires a long start up time before it can make any progress. While APSO requires almost no start up time but bad at searching for optima, i.e. usually there are better positions unexplored next to the global best-explored position.

Based on these observations, this project aims to design a hybrid algorithm of MGFProp and APSO and investigate its performance, in terms of learning speed and global optimality. In this project, we use APSO to perform global searches for good starting positions for MGFProp to perform local searches.

The searching agents, known as particles, perform global search by APSO, except one of them perform local search by MGF around the global best position to make sure that the current global best position is at least a local optimum. When the MGFProp is trapped by a local optimum, it restarts the searching at other position based on the heuristic of APSO.

Our performance investigation shows that the hybrid algorithm outperforms both algorithm in terms of the global optimality, and learning rate. This project makes good use of the benefits of both algorithms to provide a new algorithm that has high learning rate, success rate and ease of implementation to save the time and resource for training up a neural network for real-life application. This project also suggests the possibility of applying gradient method in PSO for other purpose besides neural network training.

Copyright Lui Wing On and Vanessa Ng 2012

Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course Code Title Credits
  COMP S321F Advanced Database and Data Warehousing 5
  COMP S333F Advanced Programming and AI Algorithms 5
  COMP S351F Software Project Management 5
  COMP S362F Concurrent and Network Programming 5
  COMP S363F Distributed Systems and Parallel Computing 5
  COMP S382F Data Mining and Analytics 5
  COMP S390F Creative Programming for Games 5
  COMP S492F Machine Learning 5
  ELEC S305F Computer Networking 5
  ELEC S348F IOT Security 5
  ELEC S371F Digital Forensics 5
  ELEC S431F Blockchain Technologies 5
  ELEC S425F Computer and Network Security 5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5