School of Science and Technology 科技學院
Computing Programmes 電腦學系

A Consumer Level 3D Object Scanning Device using Kinect for Web-based C2C Business

Geoffrey POON, YEUNG Yu Yin, CHENG Wing Yeung

  
ProgrammeBachelor of Science with Honours in Web Technologies
SupervisorDr. Raymond Pang
AreasIntelligent Applications
Year of Completion2013
AwardIEEE Macau Student Project Competition 2013 Second Runner-Up

Objectives

The aim of the project is to develop a Kinect-based 3D scanner system to scan object for e-commerce website and analyze the possibility of such a system in e-commerce marketing. Currently, online shopping is popular all in the world. So many companies want to join in the online auction market. But usually most of online shop website only displays the image, for example, a shoes, figure, electronic device and accessories etc. That makes the customer without confident to buy the product because they can’t view the details of product. The 3D scanner will help the sellers to present the visual 3D product to the customers. Then it can increase the attraction of products to the customers. Also, the 3D system will provide simple steps to produce 3D object and output file and then collaborate with the e-commerce website. The website will be built in 3D component system for user to upload the 3D product easily and display the good quality of product.

The project has defined a number of sub-objectives as follows:

  • Investigate an existing software and knowledge for compare the difference technique
  • The design of smart and user-friendly 3D scanner system and website. The system will provide the one-stop service for user from scanning the object that simplifies the process at the scanning. The user will need just a few clicks to scan object without involving complicated techniques.
  • Implement a 3D reconstruction system
  • Implement a e-commerce website
  • Integration of website and 3D reconstruction system
  • Evaluation of the prototype system

Background and Methodology

In the Initial stage, we tested on the ReconstructME SDK for our project but we found that we cannot enhance the sdk easily and lack of the color on the model. As we concerned about the problem of ReconstructME, we are going to develope the whole 3D reconstruction engine on our own. Our approach is different from the ReconstructionMe, in which a triangular mesh model is generated. While, in our 3D reconstruction system, the output model is represented in point cloud. The point cloud contains the color information, therefore we improved that the lack of the color on the model. Moreover, our system is provided with better tracking of the scene and the upload function for uploading the output to the webhost, it helps the user to scan the object and management on their own product more easily. Before the description of the 3D reconstruction system, the procedure of our 3D reconstruction system is shown below:

The both saturations on above will be work by the 3D reconstruction system. In the saturation 1, the Kinect captured the color image of the car and the depth map in the view 1, after that, the Kinect is translated and rotated to the position of View 2 and captured, the point cloud data of these 2 views are different and the 3D reconstruction system will be registry for these point cloud data into the 3D model. The difference of the Saturation 2 is using the rounded table and rotate the car model instead of rotate the Kinect, the point cloud data that captured when rotating the model is different, and the 3D reconstruction system will be align them together. After understanding how our system works, the coming paragraph will showing the main procedure about the 3D reconstruction system as below:

As you see the overview of 3D reconstruction system, there are 3 procedures in the system as below:

Procedure 1: Model Capture

The model capture procedure is used to obtain the point cloud data from the RGB-D data. OpenNI is implemented in our system and RGB-D data are stored in RGBImage[width*height] and Depth[width*height] by default, therefore I can matching them into our own point cloud object easily in next procedure.

Procedure 2: Processing on Point Cloud Data

The RGB-D data is obtained by previous step, and now I am going to mapping them and converting to our point cloud object. Before the conversation on the RGB-D data, the mapped RGB-D data is in projection model, therefore it has to be convert to the world model. The OpenSceneGraph is implemented for the point cloud viewer.

Procedure 3: Point Cloud Registration

There are 5 steps of our point cloud registration and the briefing on each step will be shown below:

  • Down-sampling
  • Normal estimation
  • Feature point extraction
  • Initial alignment
  • ICP refinement

The following shows the details:

For the e-commerce website, our target is to create the C2C Web Store for consumer to do online auction. So we have design the Web store tree map is showed below:

The beta of web store index figure is showed below:

To 3D visualization functional beta versions of the site are created and every aspect of it is tested thoroughly before site launch. Use XB Point Stream and JavaScript to create 3D product rendering page embedded the product detail page. And then provide control feature bar to increase the user interaction and experiment. The product status showed behind the 3D product, product description and other related product under the above content.

The 3D product is showed below:

Evaluation

We have built a prototype system with Intel i5-3.4GHz CPU to realize our idea. The scanning is done by manually moving the Kinect camera around the target object. The system will record every frame that through the ARTookit label setup scan area and its point cloud obtained and display on screen interactively with the real viewfinder as shown below:

There testing products are used in our experiments, they are the bunny, bottle, and Astroboy. The scanned results are list in the table below:

The post-processing time is around 1-3 minutes depending on the size of the model (the total number of points). The process can be further improved by optimizing some of the implements in the current prototype system. The scanned model can be freely rotated, zoom in and out within the web browser environment so as to view the object under different viewpoints as illustrated below:

The accuracy on the matched model may be affected by the noise; therefore we have designed the evaluation on different conditions. The scanned results are listed in a table below:

There are different amount of noises in below three environments. In the similar shape, it included unnecessary unmatched point which reconstructed inaccurate products. Because the limitation on the curvature evaluation by PCA. When under strong light intensity, the senor cannot capture enough depth data to reconstruct the perfect 3D feature. Because infrared sensor will be affected by strong light confuse to the infrared reflection. Compared to these environments, the explicit shape reconstructed better 3D feature and more matched points.

In the web host layer, data transfer speed is very important for user. So the 3D model needs to be good quality and appropriate size to represent the detail. The testing results are listed in a table below:

From left to right side, the first one product size is larger than others because there is only one frame but included unnecessary point cloud as noise and background without match processing. The second product size is filtering out a lot of noise and background but the shape quality is unacceptable. Compared to previous two products, the last one quality and size is better. Because it through matching and optimization processing that size is small is less noise and filter out the background.

Conclusion and Future Development

In this research, we demonstrate that a consumer level 3D object scanning device has significant potential to evolve the current habits of e-commerce because of the portability, affordable cost and user-friendliness for C2C sellers to capture 3D geometry of goods for sale; instead of using traditional 2D images. On the other hand, low-cost 3D reconstruct techniques can be applied to other application areas such as modeling in animation production. In addition, we would like to improve our result from 180 to 360 degree 3D product. This would provide more complete product to the user get more good experience and interaction.

Copyright Cheng Wing Yeung, Geoffrey Poon, Yeung Yu Yin and Raymond Pang 2013

Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course CodeTitleCredits
 COMP S321FAdvanced Database and Data Warehousing5
 COMP S333FAdvanced Programming and AI Algorithms5
 COMP S351FSoftware Project Management5
 COMP S362FConcurrent and Network Programming5
 COMP S363FDistributed Systems and Parallel Computing5
 COMP S382FData Mining and Analytics5
 COMP S390FCreative Programming for Games5
 COMP S492FMachine Learning5
 ELEC S305FComputer Networking5
 ELEC S348FIOT Security5
 ELEC S371FDigital Forensics5
 ELEC S431FBlockchain Technologies5
 ELEC S425FComputer and Network Security5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5