The goals of our experiment were to determine whether FIT tables can substitute the role of plain text document and to evaluate which frameworks are more suitable for implementing test cases and writing test cases. A project was simulated to develop an Automatic Transaction Machine (ATM) system. This system allows users to login and perform transaction process.
We adapt ATM system originated from Joshua Chuang. The major reason is that he provide detailed word document which can serve as traditional software document. We consider this is valid for us because we can not write a representative document within a short period of time. We modify ATM system and make this as reference for comparison of frameworks and document formats.
The major tools used in this project include Eclipse plugin com.bandxi.fitnesse_v1.0.2, FitLibrary and Fixture, MYSQL database, the concordion-1.4.1 Junit and the Focuses.
FIT tables are designed for specifying all necessary requirements mentioned in the word document. Different representation formats would be considered in the FIT tables. After consulting with supervisor, the requirement specification which modified to the most readable case would be used to contrast with the original document in the experiments.
ATM system functionality would be preserved to a great extent. Some features such as timeout session would be eliminated for easiness of experiment. New functionalities such as new reward schemes are written for evaluating maintainability of FIT tables in case of requirement change.
Whether test cases are readable and maintainable should be judged by people. Both maintainability and readability would be examined in the experiments. Ideally, subjects of readability experiment would be year two computing students and subjects of maintainability would be year three computing students. Year two students only have to verify which documents is more readable while year three students have to perform coding for new requirement. All subjects in experiments should be freshman in FIT tables formats. The following describes the design of the two experiments:
|Content||Readability experiment||Maintainability experiment|
|Material||FIT tables word documents||FIT tables test cases in Concordion|
|Subjects||Year 2 computing students||Year 3 computing students|
|Duration||1 hour each subject||2-4 hours each subject|
|Focus||FIT tables can/ can not substitute document role||Which framework is better for 1)design test cases 2) implement test cases as code|
|Expected Result||Value/limitation of FIT tables||Value/limitation of frameworks Improvement in future|
The following shows the differences between the designed frameworks of FitNesse and Concordion:
|Mapping document contents to Java code||Heading rows in tables from implicit mappings||Explicit but hidden instrumentation in the document perform the mappings.|
|Storage of specification||Wiki||Java source folder|
|Integration||Separate runner||Run using Junit|
|Platforms||Java, .net, Python, Ruby, perl, Smalltalk,C++||Java, .net, Python, Ruby|
|Rule||Test cases and code naming and format should obey different parsing criteria||should use Junit (assertTrue, assertFalse or assertEqual) to test|
The following describes the details of the implementation efforts in the experiements:
To make a trivial system for both frameworks to be functional
- We need a system act as template for comparison of two frameworks and documents. I adapted Joshua Chuang ATM system. I modify codes in ATMClient.java and Transaction.java. ATMClient is related to ATM graphical user interface. Transaction.java concerns the user operation and interaction with system. I mainly write method in Transaction.java for frameworks and FIT tables to work on.
This task started from October to November.
To design test cases for readability experiment and maintainability experiment
- We design test cases for functional requirements in readability experiment and maintainability experiments. I design test cases in readability experiment based on Joshua Chuang original document. We embrace all necessary functional requirements in FIT tables. I use wiki to write test cases in FitNesse. Each fixture in FitNesse has specified tabular format. I did use ColumnFixture, DoFixture ,SequenceFixture and SummaryFixture. Therefore, my FIT tables there need to obey the rules stated in above fixtures.
- In Maintainability experiment, I write test cases for new requirements. Considering FitNesse and Concordion are two different automated acceptance testing framework,
I need to write same documents in two frameworks for comparison. While we use wiki to write test cases in FitNesse, we use HTML to write test cases in Concordion.
To write fixture codes to bridge specification to system under test
- After having a system under test to work out the functional requirement, I write fixture codes to connect specification to system under test. Both FitNesse and Concordion would have fixture codes. The fixture formats are different. FitNesse involve more syntax care because it has lots of fixture type.