master teset specification srcp

Master Test Specification - SRCP

Master Test Specificationfor

Simple RailroadCommand Protocol(SRCP)

23 July 2010

Tester, Ankit Singh

High Integrity SystemFachhochschule Frankfurt am Main

University of Applied Sciences

1

Master Test Specification for SRCP

Table of Contents

1. Chapter 1: Introduction............................................................ 4

2. Chapter 2: Test plan................................................................. 5

2.1 Test level....................................................................... 5

2.1.1 Black Box Testing.............................................. 5

2.1.2 White Box Testing............................................. 6

2.2 Test Object and Environment....................................... 7

2.3 Tools.............................................................................. 7

2.4 Schedule........................................................................ 7

2.5 Test areas...................................................................... 8

2.6 Test tuning..................................................................... 9

3. Chapter 3: Test Specification.....................................................10

3.1 Testing techniques......................................................10

3.2 Test Metrics used........................................................10

3.3 Requirement list.........................................................11

4. Chapter 4: Communication........................................................17

5. Chapter 5: Black Box Test Cases …...........................................18

5.1 Mode/Protocol Testing ….............................................18

5.2 Wrong Documentation of Specification....…................21

5.3 Cause-Effect Graph ….................................................23

6. Chapter 6: White Box Testing....................................................26

6.1 White Box Testing Results............................................. 26

6.1.1 Module: ddl-s88.c............................................. 26

6.1.2 Module: srcp-fb.c.…...........................................27

7. Findings …................................................................................30

8. Conclusion …............................................................................31

9. References: ..............................................................................32

2


List of Tables

2.4 Project Schedule …..............................................................8

3.2 Test Metrics ….....................................................................10

3.3.1 General Requirement ….................................................. 13

3.3.2 Bus Allocation................................................................. 16

5.1.1 Valid / correct commands …............................................ 18

5.1.2 Invalid / incorrect commands …...................................... 20

5.2 Wrong Test Specification commands …...........…................ 22

5.3 Handshake Cause-Effect graph Decision ….........................24

List of Figures

2.1.1 Black Box Testing …...........................................................5

2.1.2 White Box Testing …..........................................................6

3.3 General flow of operations................................................12

4 Communication diagram.........................................................18

5.3 Cause-effect Graph...............................................................23

6.1.2.1 General Functionality Relations …...............…............... 27

6.1.2.2 FB generation calls ….....................................................28

6.1.2.3 SET Function of FB …..................................................... 28

6.1.2.4 GET Function of FB …......................................................28

6.1.2.5 INIT Function of FB …......................................................28

List of Charts

5.1 Bar Chart Pass vs Fail ….................................................... 23

5.2 Pie Chart Pass vs Fail ........................................................ 23

3

Chapter1Introduction

Simple Railroad Command Protocol(SRCP), is a middle man platform for various

vendors of model railway. It is an Internet protocol for controlling and programming

(digital) railway systems. The protocol is based on the application layer (layer 7) of

the OSI Reference Model. The major goals of SRCP is to provide a common

operating language among railway models as well as providing multiuser support.

Moreover, scalability issues like safety features are bundled into SRCP.

The aim of this document is to identify layout and investigate test scenarios and test

cases for the software at hand. The first section of the document focus on the myriad

components of test planning in view of SRCP followed by specification of test areas.

The final section examine the test cases identified, defined and be investigated.

4


Chapter 2Test plan

This section details the various components required for the testing of SRCP. It identifies the level of testing, the environment and tools to be used. Additionally, time frame and major test areas are scrutinized.

2.1. Test level

The level of required at this version of testing is unit testing with black box testing method and white box testing.

2.1.1 Black-box Testing

The black-box tests are going to be automatized using PyUnit python test package. Since only the SRCP server application is going to be tested, a client application is going to run unit tests on server. The general black-box testing strategy includes:

1. Running different valid tests on each of commands in combination with each device group; and2. Running different invalid tests on each of commands in combination with each device group.

5

Figure 2.1.1 Black Box Testing

Black-box tests are going to be performed by running client application written inPython programming language. The client application is going to connect to, andcommunicate with SRCP server via TCP protocol.

2.1.2 White-box Testing

Main criteria in white-box testing is going to be code coverage, which means that all branches in control ow graph are going to be covered. The white-box testing is going to be performed on specific modules of SRCP server to ensure the safety of the tested modules. The list of modules to be tested is listed below:

1. srcp-fb.c and2. ddl-s88.c

Figure 2.1.2 White Box Testing

White-box tests are going be performed using VIM or Emacs with help of ctags & etags respectively for moving around the C code of the SRCP Project.

6


2.2. Test Object And Environment

SRCP Server version 2.1.1 is a platform independent model. One SRCP server is

also called Central Unit (CU) and covers the functionality of exactly one CU. The SRCP server is open for communication in two directions. The first side covers the communication between the railroad and the SRCP server over several buses. And The other side of communication is between the SRCP server and the clients. The servers task is to convert commands from the clients to commands for the model railroad and send reports back to the clients. The communication between clients and SRCP server in both directions is done with via SRCP.

We choose Linux 9.10 as a major platform for setting up the test environment.In liaison, windows based platforms will be used if found necessary for clients of SRCP.

2.3 Tools

Various tools for testing will be in action for the testing process. We mainly categorize testing tools into• Documentation tools: Open office suit

• Unit test scripting tools: pyUnit/Python based script

2.4 Schedule

As this is a project for the fulfillment of Advanced Testing Methods module, the duration of the project will be based on the semester duration. The submission of the Project is on 25-7-2010. The Detailed testing project Schedule table is given below.

7


2010 APRIL MAY JUNEWEEK 3 4 1 2 3 4 1 2 3 4

Client Provide Specification Documents for SRCP

Test PlanningMaster Test Specification IMaster Test Specification II

Black Box TestingWhite Box Testing

Test ReportFinal MTS SRCP 0.8.3

Delivered to Client

Table 2.4 Project Schedule

Working Working Parallel

2.5 Test areas

Client – Server Network Communication

Network Communication between a client and a server is the base of software. The client and server communicate in network using TCP protocol. The network establishment should be proper so that client sends requests to server and server acknowledge to the request & sends back appropriate reply to the client. So, Network communication is one of the important area to be tested.

SRCP connection Mode

After a client established a successful connection with SRCP server, a client can switching into different mode given in SRCP. Mainly, there are three modes: I) handshake, II) command, III) info modes. Switching between different modes need to tested properly so that the commands of the individual modes can be used for different type of service given by server.

8

Test areas are defined according to the priorities set out by the documentation of SRCP. We have pointed out the following test areas:1. Level of priority

a. MUST/MUST NOT tagged requirementsb. SHALL/RECOMMEND tagged requirementsc. CAN/OPERATIONAL tagged requirements

2. Valid/Invalid requirements3. General flow of operations requirements

2.6 Test tuning

As this is a very first test document, tuning and improvement are always at the table. Thus, we will employ rudimentary test tuning techniques like that of feedback during presentations, one to one observation, traceability matrix the SRCP main documentation.

9


Chapter 3 Test Specification

This section of the document examine requirements of the system at hand and propose list of requirements for the advancement of test cases.

3.1 Testing techniques

Since the level of testing required at this stage is unit with black box testing techniques, we found that it is advisory to explore and identify best method for accomplishing the object at hand. Equivalence partitioning, boundary value analysis, all pair testing for parameter and traceability matrix specification as an appropriate fit.

3.2 Test Metrics UsedTable 3.2 Test Metrics

Test metric Definition Purpose How to calculateDefect severity The severity level of a

defect indicates the potential business impact for the end user (business impact = effect on the end user x frequency of occurrence).

Provides indicationsabout the quality ofthe product undertest. High-severitydefects means lowproduct quality, andvice versa. At theend of this phase, this information is useful to make the release

Every defect hasseverity levelsattached to it.Broadly, theseare Critical, Se-rious, Mediumand Low.

10

decision based on thenumber of defects and their severity levels.

Test coverage Defined as the extentto which testing covers the products complete functionality.

This metric is an in-dication of the com-pleteness of the testing. It does not indicate anything about the effectiveness of thetesting. This can beused as a criterion to stop testing.

Coverage couldbe with respectto requirements,functional topiclist, businessflows, use cases,etc. It can becalculated basedon the numberof items thatwere covered vs.the total numberof items.

3.3 Requirement list

Before listing out of the details of the SRCP requirements, it is useful to identify the major blocks of the protocol. The following picture depict the overall process of SRCP protocol.

11


Figure 3.3 General flow of operations

Based on the test areas specified in section 2.5, we have the following1. Commands

a. Command modeb. Info mode

2. Reply3. Server state

4. General valid/ invalid requirements

12

Table 3.3.1 General Requirements

Sr.No Item Requirement1 General:

SRCP ServerSRCP server can manage and provide several CUs is valid

2 General: SRCP Server

Separating CUs is made by different bus numbers


SRCP is based on the data transfer technologies of TCP. One port is occupied


SRCP servers have to be prosecuted on the different TCP ports if there are several CUs connected to a computer without a commonSRCP server

5 General: Lexical Units

All character codes are described in their decimal form as # followed by the numeric values of the character

6 General:Lexical Units

Entire communication consists of the characters of the 7 bit ASCII character set in the range from #32 to #127.Also,#9(TAB) for space ,#10 and #13 (CR,LF) for end line are valid

7 General:Lexical Units

Numbers are processed at least as singed 32bit integers.Floating point numbers are NOT used.

8 Command Commands are processed in hand shake and command mode only. They are not valid during info mode and MUST be ignored

9 Command Commands consist of a command word, must followed by a command parameter list separated by white space.

10 Command It is not valid to resume a command in the following line.11 Command If the list consist of several elements they need to be

separated by white space. Elements containing white space are not valid

12 Command The end of line always consists of the character #10 \n .A prefixed #13\r is valid .The character #13 is ignored.

13 Command A single line including the end of the line MUST NOT exceed over a length of 1000 characters

14 Command and Reply

All words are case-sensitive .Commands and replies of the SRCP all always written in uppercase letters

15 Reply A server HAS TO reply to every command of a client16 Reply A reply SHALL be single line. With the sign \(Backslash,

13


#92) before the LF(resp CRLF) it is marked that the reply contains another line

17 Reply IF the reply is an error message the command MUST NOT be executed

18 Reply A reply is complete with the effective end of line .Processing more than one answer in one line is not valid


After the server has sent its reply it processes the next command. That way commands are handled by a strictly two-way communication between the client and the server in a changing order of commands and reactions. The server HAS TO process the commands in the chronological order of receiving them

20 General :SRCP Server

On connection the hand shake mode is activated.Further operation parameters are determined by the client in this mode. The connection changes from the hand shake mode to either the command mode or the info mode according to client’s demand .From all three modes the connection can be closed any time.

21 Command Commands MUST be processed by the server in the order of receiving them

22 Command Unknown or unrecognized commands MUST be replied with the message “410 ERROR unknown command”

23 General: SRCPServer

A client establishes a TCP/IP connection with the server.The server sends a single line welcome string .Then the Communication parameters and the desired operation mode are negotiated between client and server. Theoperation mode starts with the final command GO sent by the client and it has to be confirmed by the server.


The server MUST differ between the client session internally.

25 General:SRCP Server

After establishing connection between client and server,The server sends a text line (The welcome) to client .


Every of these information MUST be seen as a key-value pair. The order of these keys is undetermined .It is also generally possible to specify keys with different values repeatedly .For every key exactly one value is valid.

27 General: During the hand shake phase no other commands than

14

SRCP Server listed are valid.28 Command SET PROTOCOL SRCP<VERSION>,

The server sends as reply either 201 OK PROTOCOL SRCP OR 404ERROR unsupported protocol


SET CONNECTIONMODE SRCP<MODE>,valid are INFO FOR the unidirectional command mode .The server send either 202 OK or 401 ERROR unsupported connection mode


GO by this command the hand shake phase isCompleted and the chosen operation mode is activate. The server send 200 OK<ID> immediately before that the to the client, Otherwise 402 ERROR insufficient data and remains in the hand shake mode .The field<ID> marks the numerical session ID given by the server MUST be unique and MUST NOT be identical to zero.


After completing the hand shake the agreed attributes can not be change and are valid for the entire connection.


Poll of valid messages during handshake 200 OK <ID>,201 OK PROTOCOL SRCP,202 OK CONNECTION OK ,400ERROR unsupported protocol, 401 ERROR unsupported connection mode, 402 ERROR insufficient data,500 ERROR out of resource.

33 Command Valid command types:GET,SET,CHECK,WAIT,INIT,TERM,RESET,VARIFY

34 Command After the INIT the indicated element is in default state. During INFO mode 101 INFO INIT <devicegroup> <addr> is sent for the affected element

35 Command By the command TERM the initialization is cancelled and the effected element is in default state. Before using it again the affected element has to be reinitialized.During INFO mode a message in the form of 102 INFO<devicegroup><addr> is sent for the affected element.

15


The allocation of device groups and commands on the buses

The following overview is supposed to show the allocation of device groups and commands on the buses.

Table 3.3.2 Bus AllocationSET |

CHECKGET WAIT INIT TERM RESET VERIFY

GL 1 1 -- 1 1 1 --SM 1 1 -- 1 1 1 1GA 1 1 -- 1 1 1 --FB 1 1 1 1 1 1 --

TIME 0 0 0 0 0 0 --POWER 1 1 -- 1 1 -- --SERVER -- 0 -- -- 0 0 --SESSION 0 0 -- -- 0 -- --

LOCK 0 0 -- 0 0 -- --DESCRIPTION 0 0 -- -- -- -- --

16

Chapter 4Communication

A client – server communication diagram of SRCPFigure 4 Communication Diagram

17


Chapter 5Black Box Test Cases

This section contains all test cases for Black Box testing of the Simple Railroad Command Protocol.

5.1 Mode/Protocol Testing:-

1. SRCP server Protocol version 0.8.3

Table 5.1.1 VALID / CORRECT COMMANDS

Test Case ID No.

Command Expected Output Behavior Description

1_0 TCP connection 4303 srcpd V2.0.12; SRCP 0.8.3; SRCPOTHER 0.8.4-wip

Indicates server is running & returns the version

1_1 SET PROTOCOL SRCP 0.8

<TimeStamp> 201 OKPROTOCOL SRCP

Client establishes connection with SRCP server

1_2 SET PROTOCOL SRCP 0.8 qwerty

<TimeStamp> 201 OK PROTOCOL SRCP

“qwerty” is ignored and client establishes connection with the server

1_3 SET CONNECTIONMODE SRCP INFO

<TimeStamp> 202 OK SRCP INFO

Client program switches to INFO mode

18

1_4 SET CONNECTIONMODE SRCP INFO 123

<TimeStamp> 202 OK SRCP INFO

“123” is ignored and the client program switches to INFO mode

1_5 SET CONNECTIONMODE SRCP COMMAND

<TimeStamp> 202 OK CONNECTIONMODE

Client program switches to COMMAND mode

1_6 SET CONNECTIONMODE SRCP COMMAND 22

<TimeStamp> 202 OK CONNECTIONMODE

“22” is ignored and client program switches to COMMAND mode

1_7 GO <TimeStamp> 200 OK <numerical session ID>

Client sends above entered commands to server (handshake completes)

********** Command Mode ******* ****************** ***********************1_8 INIT 1 FB 0x1111 200 OK Initialization of bus 1

with address 1_9 GET 1 FB 0x1111 100 INFO 1 FB 0

9846074 Getting information of bus 1

1_10 INIT 0 TIME 1 1 200 OK Initialization of time1_11 SET 0 TIME 364 22 22 22 200 OK Setting time with

22H:22M:22S1_12 GET 0 TIME 100 INFO 0 TIME

364 22 22 28 Getting time of Bus 0

1_13 INIT 1 POWER <TimeStamp> 200 OK

Initialize power device on bus 1

1_14 SET 1 POWER ON <TimeStamp>200 OK

Enable power on bus 1

19


1_15 TERM 1 POWER <TimeStamp>200 OK

Disable power on bus 1

1_16 GET 0 SERVER <TimeStamp>100 INFO0 SERVER RUNNING

Informs the current state of theserver is running

1_17 TERM 0 SERVER <TimeStamp>200 OK

Quits the running server and closes all connections

1_18 GET 1 DESCRIPTION <TimeStamp>100 INFO 1 DESCRIPTION GA GL FB SM POWER LOCK INFO

covers all device groupssupported by the concernedserver on bus 1

1_19 GET 0 DESCRIPTION <TimeStamp>100 INFO 0 DESCRIPTION SESSIONSERVER TIME GM

INFO covers all device groupssupported by the concernedserver on bus 0

1_20 INIT 1 GA 200 N 11

<TimeStamp>200OK

Command initializes a GA at 200address addr in the server.

1_21 SET 1 GA 200 1 1100

200 OK The port 1 of the decoder with theaddress 200 is set on the value 1for 100 ms.

1_22 GET 1 GA 200 1 <TimeStamp>100INFO 1 GA 200 1 0

Server sends all availableinformation about the currentstate of the switch decoderspecified by 200 address

1_23 SET 1 LOCK GA 101 10 <TimeStamp> 200 OK

Sets a lock on GA on bus 1 address 101 for 10 msec

1_24 GET 1 LOCK GA 101 <TimeStamp> 100 INFO 1 LOCK GA

Returns a lock status of GA on bus 1

20

100 <LOCK> address 1001_25 TERM 1 LOCK GA 101 <TimeStamp> 414

ERROR device locked

Terminates lock on device GA on bus 1 address 101

1_26 TERM 0 SERVER <TimeStamp> 200 OK

Terminates Server successfully

Table 5.1.2 INVALID / INCORRECT COMMANDS

Test Case ID No.


2_1 SET PROTOCOL SRCP 0.9

400 ERROR unsupported protocol

Not establishes connection with wrong version

2_2 <TimeStamp> 402 ERROR insufficient data

Pressing enter without any input return Error

2_3 SET <TimeStamp> 410 ERROR unknown command

Returns Error with incomplete command

5.2 Wrong Document Specification

Table 5.2 Wrong Test Specification Commands

Test Case ID No.


3_1 set ConnEctionMode Srcp 410 ERROR Fail : <TimeStamp>

21


Command unknown command 202 OK CONNECTIONMODE

3_2 SET 1 FB 0x111111 0 100 INFO 1 FB 0x111111 0

Fail: <TimeStamp> 422 ERROR unsupported device group

3_3 GET 0 SESSION 2 100 INFO 0 SESSION 2

Fail : <TimeStamp>422ERROR unsupporteddevice group

3_4 RESET 0 SERVER 101 INFO 0 SERVER

Fail : <TimeStamp>423ERROR unsupportedoperation

3_5 TERM 0 TIME 102 INFO 0 TIME Fail : <TimeStamp>422 ERROR unsupported device group

This all above Test cases is been implemented in a python unit testing script 'mod_srcpconn_handler.py'

All the above result can be found by running the python script with default as HOSTID = localhost PORTID = 4303

22


5.3 Cause effect graph

C1: SRCP version set to >= 0.9C2: Mode selection is “COMMAND”C3: Mode selection is “DEFAULT”C4: Mode selection is “INFO”C5: GO command is issued to the server

E1: Version NOT acceptedE2: Acceptable commands can be entered and their respective reply messages can be displayedE3: Commands are ignored, only the info output is sent to the client in a unidirectional mode

Graph 5.3: Cause-Effect Graph

23

Table 5.3: Handshake Cause-Effect graph DecisionDirection

downCauses & Effects

Rule 1 Rule 2 Rule 3 Rule 4

V C1(>=0.9) 1 0 0 0V C2(cmd) 0 1 0 0V C3(def) 0 0 1 0V C4(inf) 0 0 0 1V C5(go) 0 1 1 1V E1(verNOTok) 1 0 0 0V E2(cmdentry) 0 1 1 0V E3(infentry) 0 0 0 1

We ran 32 black box tests cases written in 'mod_srcpconn_handler.py'.

And we found 5 Bugs which was wrongly described in the SRCP specification.

BarChart 5.1: Pass vs Fail Test cases

24

Pass Fail

0

5

10

15

20

25

30 27

5

Test Run32 Tests Run (27 Pass & 5 Failed)

Pass


Pie Chart 5.2: Pass vs Fail Test cases

The simple statistics done on the results of Test cases implemented is given below:

Fail: 15.63% is very high if considered with the safety. We need to reduce this percentage to ensure the accuracy and safety of the product.

It may goes down if implementing more test cases in future or may also goes up. But Developer need to work on the clarity of SRCP specification and functioning of the commands and services in SRCP modules.

25

27

5

Test Run32 Tests Run (27 Pass & 5 Failed)

PassFail

Pass Fail27 5

Percentage 84.38% 15.63%

Chapter 6White Box Test

We performed white box testing on the following modules of the SRCP:1. srcp-fb.c2. ddl-s88.c

Followings are the bugs findings performed by doing statement and logic coverage white-box testing.

6.1 White Box Testing Results

Comments: • Inspections and walk through of the code performed and reviewed • General safe/secure coding guidelines were studied and applied

6.1.1 MODULE: ddl-s88.c

1. function: init_bus_S88: Bugs: result from call to ioperm(S88PORT, 3,1) (which is checking port accessibility)

is not checked. Suggestions: makes sure the return value is OK

2. function: *thr_sendrec_S88: Bugs: usleep not checked for incorrect result (lines 463 and 468 in source) Suggestions: check the result returned by usleep()

26


3. function: *thr_sendrec_S88: Bugs: sleep function result not checked against incorrect values Suggestion: see 2.

4. function: FBSD_ioperm: Bugs: sprintf considered unsafe since the size of data is not checked dynamically,

occurs on line 540 of the source Suggestion: replace sprintf() occurences with snprintf(), see man snprintf

6.1.2 MODULE: srcp-fb.c

• Call graph diagram of main source files and their relation to srcp-fb.c:

Figure 6.1.2.1 General Functionality Relations

27

• FB module generation calls:

Figure 6.1.2.2 FB generation calls• GET & SET function call graphs of FB (Internal calls)

Figure 6.1.2.3 SET functions of FB

Figure 6.1.2.4 GET functions of FB

• Initialization (INIT) function of FB (Internal Calls)

Figure 6.1.2.5 INIT function of FB

1. function: enqueueInfoFB: Bug: msg length HARDCODED and the the msg length is not checked dynamically Suggestions: should be passed any specified message, and calculate size

dynamically

28


2. function: queue_reset_fb: Bug: usleep not checked for incorrect result Suggestions: check the result returned by usleep()

3. function: infoFB: Bug: msg overflow can occur, so message length should be dynamically checked

Suggestion: Check for the overflow

Conclusion:

We managed to find Bugs in the module by doing Inspections and walk through of the code. The suggestions should be implemented to avoid bugs in the modules.

29

Findings• TCP-port 4303 assigned by IANA but not mentioned in specification

• Incomplete data in specification leads to confusion

• Insufficient details in Specification

• In some cases commands are not working on certain device groups

• Differences between Specification & srcpd implementations.

• Proper Return checks are not done in the source codes which can lead to

crashing of the system.

• Overflow check and avoid passing hard-coded message string length. It should

be taken dynamically.

30


Conclusion

• Full code coverage using black-box testing isn’t feasible

• White-box in combination with black-box testing considerably increases code

coverage

• Increased code coverage = Increased code safety

31

References

1) Lecture notes and slides of Dr. Schoenfelder.

2) The Art of Software Testing, Second Edition, Glenford J. Myers

3) http://en.wikipedia.org/wiki/Test_case

32

http://en.wikipedia.org/wiki/Test_case