School of Science and Technology 科技學院
Computing Programmes 電腦學系

A Study of Executable Specification Frameworks FitNesse versus Concordion

FUNG King Ho

ProgrammeBachelor of Science with Honours in Computing
SupervisorDr. Oliver Au
AreasSoftware Engineering and Cloud Computing
Year of Completion2011

Objectives

The aim of this final year project is to investigate how FIT table enforce readability of requirement specifications and to investigate how programmer choose a specification framework concerning maintainability. This project includes two experiments in which readability experiment compares FIT tables and word document. Secondly, the maintainability experiment compares requirement specification in FitNesse and Concordion.

This project is initiated due to an observation that many software projects fail to meet the customer requirements. Pure word document and oval meeting may not share a common domain language between customers and programmers. Therefore, one aspect of this project is to investigate whether FIT tables, which are in tabular format, can express the business logic in a more readable way by it executable specification.

The objectives of this project have been defined as follows:

  • To prepare instrument for experiment evaluation. An ATM system is used in this project. Database tables and program codes have been redesigned and modified for experiment setup. This system serves as the instrument for executable requirement specifications to work on. The functionalities inside system are used for investigating readability between original document and FIT tables.
  • To evaluate whether FIT tables are more readable than requirements expressed with the word document written in plain text with class diagram. FIT table requirement specifications are built for subjective comparison with the traditional word document. FIT tables, in theory, should express requirement change more accurate than pure text description.
  • To evaluate the maintainability of documents written in FitNesse and Concordion to requirement change situation. To facilitate communication between programmers/customers/testers, a good specification tool is needed. Requirements and information in FitNesse and Concordion are designed to be the same. We would write documents in these two tools for our subject to explore tools difference and limitation.

Background and Methodology

FitNesse is a wiki-based test execution engine that allows users to write, edit, and run FIT tests. It enables developers and customers to express acceptance tests in a tabular format and execute them from a web interface. FitNesse consist of three major parts in its framework. They are System Under Test (SUT), Fixtures and FIT tables. The following shows the relationships between FIT table, fixtures, test runner and system under test:

As stated in Lisa Crispin 2010, FitNesse is unique in using a Wiki for creating and maintaining test cases. The Wiki can be used as a knowledgebase and repository for story and theme requirements. The ease of installation and relative smooth learning curve are the reasons that I choose FitNesse. The original FIT uses html as the test template. The wiki format in FitNesse shows results in tabular format. Its core concept is similar to the following diagram:

The goals of our experiment were to determine whether FIT tables can substitute the role of plain text document and to evaluate which frameworks are more suitable for implementing test cases and writing test cases. A project was simulated to develop an Automatic Transaction Machine (ATM) system. This system allows users to login and perform transaction process.

We adapt ATM system originated from Joshua Chuang. The major reason is that he provide detailed word document which can serve as traditional software document. We consider this is valid for us because we can not write a representative document within a short period of time. We modify ATM system and make this as reference for comparison of frameworks and document formats.

The major tools used in this project include Eclipse plugin com.bandxi.fitnesse_v1.0.2, FitLibrary and Fixture, MYSQL database, the concordion-1.4.1 Junit and the Focuses.

FIT tables are designed for specifying all necessary requirements mentioned in the word document. Different representation formats would be considered in the FIT tables. After consulting with supervisor, the requirement specification which modified to the most readable case would be used to contrast with the original document in the experiments.

ATM system functionality would be preserved to a great extent. Some features such as timeout session would be eliminated for easiness of experiment. New functionalities such as new reward schemes are written for evaluating maintainability of FIT tables in case of requirement change.

Whether test cases are readable and maintainable should be judged by people. Both maintainability and readability would be examined in the experiments. Ideally, subjects of readability experiment would be year two computing students and subjects of maintainability would be year three computing students. Year two students only have to verify which documents is more readable while year three students have to perform coding for new requirement. All subjects in experiments should be freshman in FIT tables formats. The following describes the design of the two experiments:

ContentReadability experimentMaintainability experiment
ComparisonDocumentFramework nature
MaterialFIT tables word documentsFIT tables test cases in Concordion
SubjectsYear 2 computing studentsYear 3 computing students
Sample size6910
Duration1 hour each subject2-4 hours each subject
FocusFIT tables can/ can not substitute document roleWhich framework is better for 1)design test cases 2) implement test cases as code
 ReasonsReasons
Expected ResultValue/limitation of FIT tablesValue/limitation of frameworks Improvement in future

The following shows the differences between the designed frameworks of FitNesse and Concordion:

ContentFitNesseConcordion
Mapping document contents to Java codeHeading rows in tables from implicit mappingsExplicit but hidden instrumentation in the document perform the mappings.
Storage of specificationWikiJava source folder
IntegrationSeparate runnerRun using Junit
PlatformsJava, .net, Python, Ruby, perl, Smalltalk,C++Java, .net, Python, Ruby
RuleTest cases and code naming and format should obey different parsing criteriashould use Junit (assertTrue, assertFalse or assertEqual) to test

The following describes the details of the implementation efforts in the experiements:

To make a trivial system for both frameworks to be functional

  • We need a system act as template for comparison of two frameworks and documents. I adapted Joshua Chuang ATM system. I modify codes in ATMClient.java and Transaction.java. ATMClient is related to ATM graphical user interface. Transaction.java concerns the user operation and interaction with system. I mainly write method in Transaction.java for frameworks and FIT tables to work on.

This task started from October to November.

To design test cases for readability experiment and maintainability experiment

  • We design test cases for functional requirements in readability experiment and maintainability experiments. I design test cases in readability experiment based on Joshua Chuang original document. We embrace all necessary functional requirements in FIT tables. I use wiki to write test cases in FitNesse. Each fixture in FitNesse has specified tabular format. I did use ColumnFixture, DoFixture ,SequenceFixture and SummaryFixture. Therefore, my FIT tables there need to obey the rules stated in above fixtures.
  • In Maintainability experiment, I write test cases for new requirements. Considering FitNesse and Concordion are two different automated acceptance testing framework,

I need to write same documents in two frameworks for comparison. While we use wiki to write test cases in FitNesse, we use HTML to write test cases in Concordion.

To write fixture codes to bridge specification to system under test

  • After having a system under test to work out the functional requirement, I write fixture codes to connect specification to system under test. Both FitNesse and Concordion would have fixture codes. The fixture formats are different. FitNesse involve more syntax care because it has lots of fixture type.

Evaluation

One hundred and twenty two forms were received back from subjects. Out of 122 questionnaires, 69 questionnaires were selected to be valid. To ensure responses from subjects are valid, we filter out questionnaires that subjects did not take this experiment seriously.

All the 69 survey forms were de-identified and individually numbered prior to data entry for result analysis. A 10% random sampling check for accuracy against the data in the original survey forms was conducted after the whole data entry process had been completed. No apparent error was detected by the sampling check on forms numbered 2, 19, 25, 38, 46, 59, 64.

Our readability experiment has six questions. The first two questions are concerned with the document clarity. They are listed as follow:

Question 1: The description of FIT table was clear. (1-5)

Question 2: The description of original document was clear. (1-5)

A 5-point Likert scale was used for ranking the clarity of two documents.

1 ->Strongly disagree, 2 ->Disagree, 3 ->Not Certain, 4 ->Agree and 5 ->Strongly agree

We summarized these results for question 1 and question 2 as follow:

We observe that 39 subjects agree or strongly agree that FIT tables are clear. 18 subjects agree or strong agree that word document is clear. Most subjects concern that FIT tables are clear while they do not sure the word document is clear enough.

For question 3, we ask them to determine which document give them better understanding on system functionalities. From this question, we can see more subjects choosing FIT tables. The result is shown below:

Concerning system functionally, the information in FIT tables and word document version is the same. We are interested in how programmers make their decision on previous question. Therefore, we have follow up question to ask the reasons behind this decision. Since the questions are open ended. We summarized them into following categories:

a. show result /problem clearly with distinguish color

b. can more understandably present test cases with many data values

c. FIT tables are easier to read and clear

d. FIT table provide cases from user perspective/ more focusing

e. FIT is more interactive/repeat many time

f. personal preference to web page format

Subjects respond that FIT tables are more users oriented and understandable. It reflects the

potential usage of FIT tables to simulate functional requirements in system. The result for question 4 is shown below:

Apart from knowing why people choose FIT tables, we also concern why people choose word

document. Their response can reflect the limitation of FIT tables, which help us to make

suggestion. In question 5, there are 21 people that choose word document as better document. Their reasons are summarized as follow:

a: More familiar with reading document than form

b. The document format is clearer with diagram

c. There are troublesome when using FIT system

d. The word document is more detailed/ accurate

e. The word document has different method to present data and classes

f. FIT table require more programming knowledge to understand

g. Not enough time to explore FIT tables benefit

h. Easy to find information in word document

Most subjects think word document include more detailed information. They make this conclusion because they concern charts in word document are valuable. They think test cases in FIT tables are not enough for them to understand system completely. They want to know non functional requirement such as performance/scalability which has been included in word document. The result for question 5 is shown below:

Finally, for question 6, we want to know whether FIT table can substitute the role of traditional word document in requirement definition. This is also the core question we want to answer in this experiment. The result is nearly half and half which is shown below:

From readability experiment, most people think FIT tables are clearer than word documents. Subjects also respond positively that FIT tables give them better understanding on system functionalities. However, there is no strong tendency that subjects believe FIT tables can replace the role of traditional word document. It is interesting that even subjects think FIT tables give them better understanding on system functionalities, they don’t believe FIT tables can replace the role of traditional document completely.

Subjects think FIT tables can present functional requirements well by showing the desired output and actual output. Problem and result can be easily detected with the help of FIT tables. They also recommend that FIT tables work well in programmer to programmer communication.

Another task aims at simulating programmers create test cases in two different frameworks. The questionnaires have stated the purpose as follow:

Case 1: To calculate the score based on amount calculation in different factors

Case 2: To simulate the a group of people combination in room. Certain people can not live with other in the same room.

To allow easier analysis, we focus on four areas including display format of result, easiness of writing specification, easiness of organizing test case and source code readability. We calculate the average among all subjects’ responses. We can see FitNesse only score higher than Concordion in easiness of organizing test cases. Concordion scores higher than FItNesse in other three categories. The result is as follows:

Finally, we want subjects give a satisfaction mark to both frameworks from two perspectives. Again, Concordion scores higher than FitNesse. The result is shown below:

Conclusion and Future Development

We had two experiments in this project. The survey is to find out information to answer some questions that we are interested in. We found table format in FIT tables may not be natural to all subjects. The conventional reading habit and additional programming knowledge are dominant reasons that subjects think FIT tables cannot substitute the role of plain text document. It seems harder to overcome these barriers considering FIT tables must involve certain programming knowledge to handle them.

We can conclude that FIT tables can complement the role of plain text document. Subjects respond positively to this idea. From their responses, FIT tables can present functional requirement and use cases which provide solid examples to word document description. The idea of checking the system functionaries in early stage is also attractive to them.

Related to framework comparisons, we cannot say which framework can substitute other one. They have different user targets. From our experiment, we can see programmers are much familiar with Concordion than FitNesse. They have this preference because test cases are written in HTML format in Concordion. We do not sure if this situation revises if FitNesse improve its format control. We can conclude that programmers should choose Concordion at that moment.

Our experiment design is qualitative. We consider this design is good for exploratory purpose to find limitation of FIT tables and frameworks. In future, we hope there would be quantitative experiment to gain statistics information. We suggest future work can be done on how difference between frameworks affects programmer productivity. From our experiments, we know the differences and subject respond. Our results can be served as foundation of future researches.

Copyright Fung King Ho and Oliver Au 2011

Jonathan Chiu
Marketing Director
3DP Technology Limited

Jonathan handles all external affairs include business development, patents write up and public relations. He is frequently interviewed by media and is considered a pioneer in 3D printing products.

Krutz Cheuk
Biomedical Engineer
Hong Kong Sanatorium & Hospital

After graduating from OUHK, Krutz obtained an M.Sc. in Engineering Management from CityU. He is now completing his second master degree, M.Sc. in Biomedical Engineering, at CUHK. Krutz has a wide range of working experience. He has been with Siemens, VTech, and PCCW.

Hugo Leung
Software and Hardware Engineer
Innovation Team Company Limited

Hugo Leung Wai-yin, who graduated from his four-year programme in 2015, won the Best Paper Award for his ‘intelligent pill-dispenser’ design at the Institute of Electrical and Electronics Engineering’s International Conference on Consumer Electronics – China 2015.

The pill-dispenser alerts patients via sound and LED flashes to pre-set dosage and time intervals. Unlike units currently on the market, Hugo’s design connects to any mobile phone globally. In explaining how it works, he said: ‘There are three layers in the portable pillbox. The lowest level is a controller with various devices which can be connected to mobile phones in remote locations. Patients are alerted by a sound alarm and flashes. Should they fail to follow their prescribed regime, data can be sent via SMS to relatives and friends for follow up.’ The pill-dispenser has four medicine slots, plus a back-up with a LED alert, topped by a 500ml water bottle. It took Hugo three months of research and coding to complete his design, but he feels it was worth all his time and effort.

Hugo’s public examination results were disappointing and he was at a loss about his future before enrolling at the OUHK, which he now realizes was a major turning point in his life. He is grateful for the OUHK’s learning environment, its industry links and the positive guidance and encouragement from his teachers. The University is now exploring the commercial potential of his design with a pharmaceutical company. He hopes that this will benefit the elderly and chronically ill, as well as the society at large.

Soon after completing his studies, Hugo joined an automation technology company as an assistant engineer. He is responsible for the design and development of automation devices. The target is to minimize human labor and increase the quality of products. He is developing products which are used in various sections, including healthcare, manufacturing and consumer electronics.

Course CodeTitleCredits
 COMP S321FAdvanced Database and Data Warehousing5
 COMP S333FAdvanced Programming and AI Algorithms5
 COMP S351FSoftware Project Management5
 COMP S362FConcurrent and Network Programming5
 COMP S363FDistributed Systems and Parallel Computing5
 COMP S382FData Mining and Analytics5
 COMP S390FCreative Programming for Games5
 COMP S492FMachine Learning5
 ELEC S305FComputer Networking5
 ELEC S348FIOT Security5
 ELEC S371FDigital Forensics5
 ELEC S431FBlockchain Technologies5
 ELEC S425FComputer and Network Security5
 Course CodeTitleCredits
 ELEC S201FBasic Electronics5
 IT S290FHuman Computer Interaction & User Experience Design5
 STAT S251FStatistical Data Analysis5
 Course CodeTitleCredits
 COMPS333FAdvanced Programming and AI Algorithms5
 COMPS362FConcurrent and Network Programming5
 COMPS363FDistributed Systems and Parallel Computing5
 COMPS380FWeb Applications: Design and Development5
 COMPS381FServer-side Technologies and Cloud Computing5
 COMPS382FData Mining and Analytics5
 COMPS390FCreative Programming for Games5
 COMPS413FApplication Design and Development for Mobile Devices5
 COMPS492FMachine Learning5
 ELECS305FComputer Networking5
 ELECS363FAdvanced Computer Design5
 ELECS425FComputer and Network Security5