Testing in Agile Sprint using SuiTest (TestFlo)

Tim and Joe are members of an agile Team. Tim playing a developer role and Joe playing QA role. John is Scrum master. They are working in a Sprint of 2 Weeks. Let us illustrate several situations arises during a Sprint.

Tim: Hi Joe, coming for Sprint Planning meeting?

Joe: Yeah, let’s go.

Tim: Oh! Hectic meeting. But I am sure of what to deliver for this Sprint.

Joe: But tell me more about Story JTP-227.

Tim: Story on JIRA is updated with links where you can find more details. You can come back to me if you need more clarifications.

Joe: Thanks Tim.

Joe: Hi Tim. I have written Test Cases for Story JTP-227, can you review?

Tim: Sure! Where can I see Test Cases and how do I know which one is related to Story?

Joe: Test Cases are listed down under a Story in JIRA

Tim: Really!! I have never seen like that. Are they subtasks for Story?

Joe: No. They are not subtasks. They are Test Cases in JIRA. We are using a plugin called SuiTest. See here how they look

Tim: That’s awesome. Let me review and revert back by EOD.

Tim: Hey Joe, I have review and have put review comments for Test Cases in JIRA.

Joe: Sure I will look into them.

Tim: Hey Joe, just want to check what's the status of Test execution for Story JTP-227.

Joe: 1 sec, let me open JIRA to see Test Execution status for Story … Okay here is the status. Executed 2 Test cases out of 3, 1 Pass, 1 Fail.

Tim: Is JTP-235 failing? What the Bug id?

Joe: JTP-238. You can find Bug ID against failed Test Case.

Tim: Cool! That made it easy for me to understand the steps with which I can reproduce the issue. I can execute this test case to verify my Fix.

John: Hi Joe, what the status of Test execution for Release.

Joe: Hi John. What a pleasant surprise! One sec, I will open JIRA to check Test Execution Status of Test Plan for Release X.

Okay here it is. We have executed 67% and there is one Open Bug JTP-238.

John: That’s a quick answer. Thanks Joe. I am going for lunch. Bye.

John: Hey Joe, are all the test cases written by you ported to JIRA?

Joe: Yeah, we have imported all existing Test Cases using CSV importer

John: One last question, I can see that there are 267 regression tests, do you run all of these and update on JIRA? It will take a lot of time to do this right?

Joe: No John, we don’t update JIRA manually, we use JIRA API to run Automated Tests and update JIRA automatically.

From above illustrations we can understand the following benefits

• Building repository of Test Cases allowing you to create Test Plans
• Snapshots of test cases within Test Plan or Story
• Establish real-time insights  and clear traceability between requirements, test cases, and defects to determine test coverage and ensure tests exists for all requirements.
• JIRA Software integration and Sprint reports
• Import Test Cases from CSV, TestLink and other systems
• Run automated tests on Bamboo/Jenkins by changing the status of your Test Plan and receive updates about the results of tests execution in JIRA using comprehensive REST API
• Plan, track, and proactively manage manual, automated, and API tests in one repository to mitigate risk.
• Use reports to get better understanding of test coverage, defect trending, and sprint status.

Image Courtesy: Images are taken from https://intenso.atlassian.net/wiki/

Traceability Matrix using TestLink/github and Jenkins/Hudson

               At my current work place we were using TestLink (We have moved to SuiTest now) for writing TestCases and my boss asked me to see if all the Stories have Test Cases or not. I have written this simple program and integrated with Jenkins to get a nice graph showing Test Case traceability like below

In the above graph red line shows total Stories in a sprint, green line shows Stories which have at least one Test Case and blue line shows stories which have at least one automated test case.

To achieve this we have followed below rules as a Team.

1. Test Suite in TestLink has same name as Story ID in JIRA
2. Test Suite Name in Java/Script in github has same as Story ID in JIRA

This traceability program takes stories in a Sprint as input and do following things

1. Connects to TestLink and searches for Test Suites corresponding to Stories, also creates statistics 
2. Connects to github and searches for Test Suites corresponding to Stories, also creates statistics

A Jenkins job is created with configuration as follows

The code uses org.eclipse.egit.github.core and testlink-java-api to connect with github and TestLink respectively

 

Github Connector is as follows

TestLink Connector is as follows

Above program will also list down stories which needs test cases.

Now you are ready to trace which stories have Test case and which doesn't have. :)

~Yagnanarayana Dande

Automated Testing of Multi Browser Support using SauceLabs and Jenkins/Hudson

               Its not an easy task to run automated tests on multiple browsers and gather results back to a single location. Its not a good option to have your own setup of all Operating System + Browser combinations in a lab if you don't have a dedicated operations team.

             We can use cloud solutions like Saucelabs for this purpose. Saucelabs provides on demand instance of Operating System + Browser. It uses Remove Webdriver concept to run Selenium Tests on these instances.

             You can use following code to start a remote Run

Now create a Jenkins job with configuration as

Now you are ready for your Multi Browser tests. Start a build and test will execute on Saucelabs. 

Once you launch your tests, logon to saucelabs.com and watch the live steaming on how tests are being executed or wait till end to see TestNG report on Jenkins

~Yagnanarayana Dande

Running Selenium Tests on a headless Jenkins Server

                            QA needs to execute their daily Sanity Tests on Jenkins. To run Selenium Tests we need a machine connected to monitor. And if we don't have such an arrangement then we can follow the steps below to run Selenium Tests on a headless Jenkins Server.

Never run daily runs on tools like Saucelabs.com or browserstack as its a costly affair.

1. Install Xvfb on Jenkins Server

yum install xorg-x11-server-Xvfb

2. Install Firefox on Jenkins Server

yum install firefox

3. Install https://wiki.jenkins-ci.org/display/JENKINS/Xvfb+Plugin on Jenkins

4. In you Job, configure as follows

5. When you run the Job, it will create a screen and execute tests on that screen with GUI

Enjoy your daily runs now :)

~ Yagnanarayana Dande

Integration of TestLink with Jenkins/Hudson to Run Automated Tests

Install the TestNG Plugin on Jenkins

Get API Key from TestLink -> Test Project Management -> API Key

 Go to Jenkins -> Manage Jenkins -> Configure System Add following

Go to Jenkins -> Automation Project -> Build -> Invoke TestLink

Invoke TestLink has 3 Parts

1. Test Link Configuration 

(Using these details Jenkins can get details about Test Cases from TestLink)

 2. Test Execution

(Using this Jenkins execute Test Cases)

3. Result Seeking Strategy

(Using this Jenkins Posts Test Results to TestLink)

Click on Any Build in Jenkins and it shows TestLink Results 

On Right Side Pane of Automation Project, You can see TestLink 

~Yagnanarayana Dande