School of Science and Technology 科技學院
Computing Programmes 電腦學系

An Augmented Reality Automatic Furniture Fitting System

CHAN Kwun Kit, LAU Wan Man, TO Kwok Ho

ProgrammeBachelor of Science with Honours in Web Technologies
SupervisorDr. Jeff Tang
AreasIntelligent Applications
Year of Completion2013

Objectives

The aim of this project is to simulate a real room environment in a better way, which allows the users to experience the outcome of possible furniture arrangement and evaluate whether the outcome is acceptable. To achieve the aim, the Augmented Reality technique is applied. The main objective of the project is to measure the dimensions of a real room by a depth camera, which is provided by the Microsoft KINECT sensor. It converts the depth data for our system, and uses those data to estimate the available space in the real room. Afterwards, the users can select the 3D virtual furniture from the system database and put them into the room. Finally, our system enables the user to customize their favorite size or color of furniture and give some suggestion for the user such as whether the space is enough for placing furniture.

The project has defined a number of sub-objectives as follows:

  • Get the required data using Computer Vision techniques
  • Convert the required data into parameters that used for generating a room with virtual furniture.
  • Create and adopt 3D furniture models
  • Enable the user to manual edit the property of the furniture
  • Optimize the furniture arrangement
  • Performance Evaluation

Background and Methodology

The figure below shows how the system flows. When the system start, the system is going to get the depth and the color data by KINECT, and measure a real distance and identify the coordinate for estimation of the wall skirting. The sizes of the floor is estimated from the depth information and the furniture is scaled to real dimensions according to distance with respect to the camera.

And the followings are actually for the user. User can select the furniture what they want from the furniture list and put them into the AR scene. Then, the user can depend on their idea to manual edit the position of furniture. In additional, the system can automatically arrange the selected furniture by prioritizing their shapes. Finally, user can see how the style is after put the furniture in there and save the result for future reference. The use-case diagram is shown below:

In the system, user can through KINECT sensor to get depth data and convert the data to generate AR scène. Application allow user to adjust the floor level when the deviation happened.

Moreover, user can selects different type of 3D furniture and put them into AR scene. When the furniture is selected, system will show the furniture information to user. Also, application allows user using hit detection to control the position of 3D furniture model in an AR scène. Finally, the output AR scene can be saved for the user’s future reference.

The Augmented Reality automatic furniture fitting system is going to be implemented in C# language. The system is equipped with a computer vision module, which is enabled by the KINECT sensor device. According to the literature study, we found most of the related applications that used this device were developed in C#, C++. And by comparing with C# and C++, C# is more simple and stable when writing a GUI program and the project is written in WPF, which defines an interface in XML format so the developer can edit the UI components easily and in an organized way, although C++ is more powerful when writing console applications.

Also, we use KINECT SDK to develop our application. KINECT SDK provides the tools and APIs, both native and managed, that supports applications built with C++, C#, or Visual Basic using Microsoft Visual Studio 2010 or 2012. So we develop KINECT-enabled applications for Microsoft Windows. Another SDK is OpenNI which is an open source used for the development of 3D sensing middleware libraries and applications but it is more complex to identity depth map in our system. So we use KINECT SDK.

Windows Presentation Foundation uses data points to create a mesh. 3D graphics in WPF require many data points for a smooth and clear image. To prevent issues like blurry in 3D model, Foundations of WPF, which the best methodology for programming in 3D with WPF is to use a 3D designer package to create models and meshes, and then import them into WPF. The following shows the details:

For import 3D model into a C# program of WPF, that only support XAML file. We should convert 3D model (.3ds) into XAML. Most of the 3D modeling software has 3D Model to XAML plug-in. Then WPF can import a 3D model is displayed in Windows Form.

About the furniture to a given coordinate, we should set model property of translating and rotate transforms (i.e. transform the model on the floor and spinning it to face the proper direction, respectively). Since we want all of our furniture to be on the floor and not floating in the sky or buried underground, we need concern ourselves with applying the proper x and z translations as well as the proper rotation about the y axis. The following shows the details:

To pick furniture and translate/rotate it in the 3D space, we implemented the “Hit detection” in the system, which enables the user interacting with the selected 3D furniture models with mouse events. The following shows the details:

The scaling of furniture model need match distance changing between KINECT and wall. We use the max detection of KINECT (531.7 x 409cm) to compare the distance between KINECT and skirting. It can the ratio in detected range and use it to scale the ratio of furniture. The following shows the details:

The 3D furniture models are put into 5 levels according to their widths. The longest width is level 5 and starts from it. And put it to back to the wall at the skirting. Then, put the following furniture next to it if there is enough space. Otherwise, put and rotate it at left hand side and checking for other available area to put more another model. The following shows the details:

The Interface of the Main Software simply divided into three parts.

  • Part 1(top-left hand corner) is the AR scene. User can control the 3D furniture model here. And in the next prototype, when connected with the KINECT, the AR scene will show in real view and showing the actual places which can be place for the furniture.
  • Part 2(right hand side) is used for showing the selected furniture information. It will show the name, dimension, color, texture, made in, and the description of the selected furniture. And having two function buttons at the bottom of this part. User can adjust the floor level by pulling up or down of the key. And by press the “Auto Fitting” button for apply the auto fitting algorithm to the furniture model.
  • Part 3(bottom-left hand corner) is the furniture list for user to choose and place the furniture into the room.

The following shows the details:

Evaluation

To evaluate the system, we find some user trying to use the application. And complete a questionnaire for giving us some feedback. In the evaluation, we visited a total of 20 non-technical users. Since the system is need to work with the KINECT device, but not everyone had. So when doing the evaluation, we taken a video of system operation, the respondent may require watching the video to answer a list of question. Also, we setup the system in the school and find other student or teacher from other classes or field to do the experience. This can get a more fair feedback.

In evaluating the following three aspects, those are the system usability, flexibility, accuracy by the user after experience with our system. The following shows the details:

For the results, the overall of evaluation is good and satisfied. The question 6 is highest in evaluation. Absolute majority of interviewer represented that the system is usable in Hong Kong and through the system can helpful to their home design. In the unexpected results, the question 3 and 4 is lowest in evaluation. Some of users think the skirting in the system is not accurate and not flexible. The reason may be between the KINECT and skirting have a noise affect the detection and the KINECT detection is not 100% accuracy. Also the system cannot handle the distance between skirting and KINECT with barrier in present. In addition, the result of question 2 and 7 are pleased. Most of respondents said that the system is easily to use and the results of automatic fitting is satisfied. The following shows the details:

Conclusion of evaluation result, we can find that our system is more helpful furniture buyer and suitable for Hong Kong. Also, most of people like using Augmented Reality (AR) technique to put 3D furniture in the scene of a real room to design their home. This proves that our system has a great potential.

Conclusion and Future Development

In our project, we want to provide an effective and valuable way to do home design in Hong Kong. It is to use KINECT to measure room size to get a real room environment that allows users to examine furniture placement and evaluate the overview. It is our automatic furniture fitting system.

In this system, there are 6 main objectives we satisfy. After all objective was finished, we can distribute two part of this: depth map analysis and auto fitting arrange. Depth map analysis can adjust all model moving and their size contain skirting analysis, furniture model scaling, Horizontal and vertical adjustment of model. The other part is Auto fitting arrange is main function of system which involve almost objective.

Also we find some limitations in our system. First the detection range of KINECT is from 0.8m to 4 m. So we recommend user can start analyses form the corner. The next one is deviations appearing when KINECT scanning ranges increase. We are checking by average points to reduce data fluctuation.

With the future, we propose the system can find the distance between skirting and KINECT with barrier. It is make it more effective and practical to handle more case.

Also, the system can allow user to import furniture in application they want which provide a great diversity of furniture and increase liberty to use the system.

Copyright Chan Kwun Kit, Lau Wan Man, To Kwok Ho and Jeff Tang 2013

Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course CodeTitleCredits
 COMP S321FAdvanced Database and Data Warehousing5
 COMP S333FAdvanced Programming and AI Algorithms5
 COMP S351FSoftware Project Management5
 COMP S362FConcurrent and Network Programming5
 COMP S363FDistributed Systems and Parallel Computing5
 COMP S382FData Mining and Analytics5
 COMP S390FCreative Programming for Games5
 COMP S492FMachine Learning5
 ELEC S305FComputer Networking5
 ELEC S348FIOT Security5
 ELEC S371FDigital Forensics5
 ELEC S431FBlockchain Technologies5
 ELEC S425FComputer and Network Security5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5