Lookbusy - A simple application for generating synthetic load on a Linux system

During stress tests it is required to make computational resources like memory, disk space and cpu busy. For this one can do cpu intense operations or memory intense operations or disk intensive operations.

In software testing, a system stress test refers to tests that put a greater emphasis on robustness, availability, and error handling under a heavy load, rather than on what would be considered correct behavior under normal circumstances. In particular, the goals of such tests may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space), unusually high concurrency, or denial of service attacks. - Wikipedia

One can design different tools for this purpose. While browsing last week in my free time I came across this tool on net which is designed exactly for this purpose. It is working perfectly and very easy to use. Thanks to Devin Carraway <lookbusy@devin.com> for providing such a great tool. 


Setup

1. Download Tool which is available at: http://www.devin.com/lookbusy/download/lookbusy-1.3.tar.gz
2. Extract the tar ball in some directory
3. cd into the extracted directory
4. Issue ./configure
5. Issue make
6. Issue 'make install'
7. You are done with the installation and ready to use

Usage

yagna@yagna-desktop:~/Desktop/lookbusy-1.3$ ./lookbusy --help

usage: lookbusy [ -h ] [ options ]
General options:
  -h, --help           Commandline help (you're reading it)
  -v, --verbose        Verbose output (may be repeated)
  -q, --quiet          Be quiet, produce output on errors only

CPU usage options:
  -c, --cpu-util=PCT,  Desired utilization of each CPU, in percent (default
      --cpu-util=RANGE   50%).  If 'curve' CPU usage mode is chosen, a range
                         of the form MIN-MAX should be given.
  -n, --ncpus=NUM      Number of CPUs to keep busy (default: autodetected)
  -r, --cpu-mode=MODE  Utilization mode ('fixed' or 'curve', see lookbusy(1))
  -p, --cpu-curve-peak=TIME
                       Offset of peak utilization within curve period, in
                         seconds (append 'm', 'h', 'd' for other units)
  -P, --cpu-curve-period=TIME
                       Duration of utilization curve period, in seconds (append
               'm', 'h', 'd' for other units)

Memory usage options:
  -m, --mem-util=SIZE   Amount of memory to use (in bytes, followed by KB, MB,

                         or GB for other units; see lookbusy(1))

  -M, --mem-sleep=TIME Time to sleep between iterations, in usec (default 1000)

Disk usage options:
  -d, --disk-util=SIZE Size of files to use for disk churn (in bytes,
                         followed by KB, MB, GB or TB for other units)
  -b, --disk-block-size=SIZE
                       Size of blocks to use for I/O (in bytes, followed
                         by KB, MB or GB)
  -D, --disk-sleep=TIME
                       Time to sleep between iterations, in msec (default 100)
  -f, --disk-path=PATH Path to a file/directory to use as a buffer (default
                         /tmp); specify multiple times for additional paths

Increase Memory Utilization by 1GB

lookbusy -m 1GB

You can verify by using either top or dstat(dstat -m).

Increase CPU Utilization to 70%

lookbusy -c 70

You can verify by using either top or dstat(dstat -cp)

Executing Tests in multiple environments using Selenium Grid

                      Our Business Team came up with a requirement of having support for multiple Browsers like Google Chrome, Internet Explorer 6, Internet Explorer 7 and Internet Explorer 8. Till yesterday I was happily executing my Selenium tests in firefox and now I have to find a way in which I can run my test cases in different browsers. First thought I got is to install all the browsers on my laptopn and test. But unfortunately I realized that I cannot install all the Internet Explorer versions on single machine. So I have started exploring different options and came across Selenium Grid. It is an extension of Selenium Core. The idea is simple there will be a Master and many slaves. Master will give the commands and slaves will be executing these commands.

Process of setting up Selenium Grid.

These steps have to done both on Master and Slave machines. 

1. Download and install the Java JRE. Verify the installation by issuing a command "java -version" on command prompt.
2. Download and extract Apache Ant and put the path in PATH environment variable. Verify the installation by issuing a command "ant -version" on command prompt.
3. Download and extract Selenium Grid.

You are done with the setup. :)

Making a machine master

1. In Command prompt from the folder where you have extracted "Selenium Grid" run "ant launch-hub"
2. To Verify just got to http://localhost:4444/console from your favourite browser
   
3. There will be 3 tables on the Web Page. First shows Environments defined in file named "grid_configuration.yml" located at the root directory of the extracted location of selenium grid. the second column shows what remote controls are available and the third column shows what remote controls are currently running tests.
4. For our purpose let us add some new environments for Internet Explorer 6, Internet Explorer 7, Internet Explorer 8. Copy paste the following lines at the end of "grid_configuration.yml"

       - name:    "*iexplore6"
         browser: "*iexplore"
       - name:    "*iexplore7"
         browser: "*iexplore"
       - name:    "*iexplore8"
         browser: "*iexplore"

You are done with Master Setup. :)

Making a machine which is having Internet Explorer 6 installed as slave

 In Command prompt from the folder where you have extracted "Selenium Grid" run

ant -Denvironment=*iexplore6 launch-remote-control -DhubURL=http://192.168.0.111:4444 -Dhost=192.168.0.112

where 192.168.0.111 is the ipaddress of Master and 192.168.0.112 is the ipaddress of slave. One can use hostname instead of IPAddresses if they are added in DNS.

Making a machine which is having Internet Explorer 7 installed as slave

 In Command prompt from the folder where you have extracted "Selenium Grid" run

ant -Denvironment=*iexplore7 launch-remote-control -DhubURL=http://192.168.0.111:4444 -Dhost=192.168.0.113

where 192.168.0.111 is the ipaddress of Master and 192.168.0.113 is the ipaddress of slave. One can use hostname instead of IPAddresses if they are added in DNS.

Making a machine which is having Internet Explorer 8 installed as slave
    
 In Command prompt from the folder where you have extracted "Selenium Grid" run

ant -Denvironment=*iexplore8 launch-remote-control -DhubURL=http://192.168.0.111:4444 -Dhost=192.168.0.114

where 192.168.0.111 is the ipaddress of Master and 192.168.0.114 is the ipaddress of slave. One can use hostname instead of IPAddresses if they are added in DNS.

Verify the setup at http://localhost:4444/console. You should see something like

You are done with Slaves Setup. :)

Important: You have to use *iexplore6 or *iexplore7 or *iexplore8 in your Test Cases instead of *iexplore.
I am using ROBOT Framework to run my selenium tests. So I can run tests using

pybot.bat -l Run1.html -o Run1out.xml -r Run1Report.html "C:\Users\yagnanarayana dande\Desktop\TestsIE6.html"

Remember Pybot cannot run in parallel. So start different pybots per environment like

pybot.bat -l Run1.html -o Run1out.xml -r Run1Report.html "C:\Users\yagnanarayana dande\Desktop\TestsIE6.html"
pybot.bat -l Run1.html -o Run1out.xml -r Run1Report.html "C:\Users\yagnanarayana dande\Desktop\TestsIE7.html"

pybot.bat -l Run1.html -o Run1out.xml -r Run1Report.html "C:\Users\yagnanarayana dande\Desktop\TestsIE8.html"

To remove any slave from your grid due to reasons like machine died during execution the use

 http://192.168.0.111:4444/registration-manager/unregister?host=192.168.0.113&port=5555&environment=*iexplore7

Unit Tests for JavaScript using JsUnit

Yesterday my boss asked me to test our Product UI for any script errors. As our application is full of JavaScripts I decided it would be wise to write some Unit Tests in JsUnit.

For those who are new to JavaScript let me give you some information. As per Wikipedia 

"JavaScript is primarily used in the form of client-side JavaScript, implemented as part of a web browser in order to provide enhanced user interfaces and dynamic websites.".

Regarding the popularity of Java Script trends.builtwith.com says

708,651 websites using this within the top million sites on the internet and an additional extended total of 4,797,491 websites that are using Javascript.

And why my decision is wise?

1. Runs on “most” browser/platform combinations
2. Preserves the standards of a typical XUnit framework
3. Open source - Free!!!
4. Can be hosted on a web server so that tests can be run across enterprise without different setups

Some facts about JsUnit

1. Unit tests in JsUnit are called Test Functions
2. Test Functions live in an HTML page called a Test Page
3. A Test Page is any HTML page that has a JavaScript “include” of jsUnitCore.js
4. jsUnitCore.js provides the assertion functions of JsUnit, e.g. assertEquals(comment, arg1, arg2)
5. JsUnit supports setUp() and tearDown()
6. A Test Suite Page declares a suite() function that returns a JsUnitTestSuite for grouping Test Pages
7. The JsUnit testRunner.html page runs Test Pages

Installation:

      For most of the open source tools I worked with, installation is a night mare. But JsUnit is pretty straight forward and is just 2 steps

1. Download installation zip from http://www.jsunit.net/
2. Unzip file in some local directory or on a webserver to host JsUnit

and we are done with installation. Really simple right :)
To verify the installation

1. go to http://xx.xx.xx.xx/jsunit/testRunner.html, where xx.xx.xx.xx is the IPAddress or Hostname of the server on which JsUnit is hosted
2. Enter xx.xx.xx.xx/jsunit/tests/TestPageTest.html
3. Hit run button
4. Progress should change to green

My First Test Case

I would like to unit test below function

function multiply(arg1, arg2) {
    return arg1*arg2;
}

Write a html page as below and put it in tests folder under JsUnit base folder

<html>
<head>
<script language="JavaScript" src="../app/jsUnitCore.js"></script>
<script language="JavaScript" src="math.js"></script>
<script language="JavaScript">

function testWithValidArguments() {
assertEquals("2 times 3 is 6", 6, multiply(2, 3));
assertEquals("Should work with negative numbers", -20, multiply(-4, 5));
}

function testWithInvalidArguments() {
assertNull("null argument", multiply(2, null));
assertNull("string argument", multiply(2, "a string"));
assertNull("undefined argument", multiply(2, JSUNIT_UNDEFINED_VALUE));
}

function testStrictReturnType() {
assertNotEquals("Should return a number, not a string", "6", multiply(2, 3));
}

</script>
</head>
<body>
This is a Test Page for multiplyAndAddFive(arg1, arg2).
</body>
</html>

Thats all you have to do and now you can run your tests using http://xx.xx.xx.xx/jsunit/testRunner.html.

-------------------------------------------------------------------------------------------------------------------  

 Testing can be used to show the presence of bugs,but never to show their absence!

   Edsger Dijkstra                                                             

Test Automation for ExtJS User Interface using Selenium

Test Automation of an application that has an ExtJS user interface can be tricky, because of randomly-generated default element IDs. The firm I am working for has ExtJS User Interface and Automation of UI Testing was about to drop because of randomly-generated element IDs. Unfortunately almost all the elements have the default random generated IDs and very few elements are overridden to get Custom set IDs. I managed to solve this issue.

Why is it difficult to Automate? 

To interact with DOM elements, Selenium Commands should specify the target elements using their xpaths.

click //input[@value='Google Search' and @type='button']

To determine the xpath we can use tools like xpath add-on for Firefox or record and get the xpath from Selenium IDE. But the biggest challenge here with ExtJS is this xpath changes every time page reloads as element IDs are randomly generated by default.

So the xpath generated using above mentioned ways will give some thing like

//div[@id='ext-gen77']/div[12]/table/tbody/tr/td[4]/div

This xpath will get changed every time you reload the page. So if you try to run the test it gives

[error] Element //div[@id='ext-gen77']/div[12]/table/tbody/tr/td[4]/div not found

How did I solve this?

To solve this Problem we need Firefox Browser with Firebug, Selenium IDE Add-On installed.

Record the Test Case using Selenium IDE

Open your Application UI which is developed in ExtJS. Right click on some element like Edit Button and say "Inspect Element"

Replace xpath which got recorded with xpath=//button[text()='edit'] because every time the screen gets reloaded, this xpath changes as Element ID is getting generated randomly. This xpath is determined by Tag name which is "button" and value of tag is "edit". 

Few times you may need to use class like xpath=//span[@class='x-menu-item']. This xpath is determined by Tag name which is "span" and value of class is "x-menu-itmt". 

First Generation - Need for Automation Testing

Before discussing any thing about Automation we should make sure that we are not having a wrong notion of

Automation is the replacement for Manual Testing   

Pain Points in Manual Testing

1. Remember and Re-run:
   Regression Testing will be increasing difficult due to hard to duplicate circumstances, seemingly unrelated features that may have been inadvertently broken due to the modification. 
   Also remembering every unusual test case from the previous release's testing and executing with high precision for the new release is a daunting task.
   
   
2. Spirit of Agile is lost:
   Developers waiting for QA for a day to know if there are any blockers in the given build. This happens if there is any issue in the last step of a work flow and as manual testing takes a day to complete Smoke.
   
   
3. Repetitive means boring:
   Repeating test case execution in different environments induces human errors like overlooking details, etc. and losing sight on obvious defects.
   
   
4. Out of time:
   Sequential execution of Test Cases on different environments is eating up the time.
   

Good Home work

Before automating, Test Cases should be documented, categorized and prioritized.

What should we automate?

1. To run a set of tests repeatedly, automation is need of the hour 
2. It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner
3. To execute test cases on different OS platforms and on different browsers. 
4. Parallel execution of automated tests can save a lot of time which can be used to test other critical features.

What should we test manually?

1. Test Cases executed very few times or test cases which are difficult to automate should be tested manually.
2. It allows the tester to perform more ad-hoc testing. 
3. If the tools have any limitations, those tests are manual.
4. If return on investment on automation is low as in some cases the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases is very less.

Criteria for automating

Automating a Test Scenario yields better ROI practically in following conditions (8 out of 54 conditions)

Importance	Cost to Automate	Technical Feasibility	Repeatability	Test Mode
High	Low	Yes	High	Automation
High	Low	Yes	Medium	Automation
High	Medium	Yes	High	Automation
High	Medium	Yes	Medium	Automation
Medium	Low	Yes	High	Automation
Medium	Low	Yes	Medium	Automation
Medium	Medium	Yes	High	Automation
Medium	Medium	Yes	Medium	Automation

Paros - Simplest way to secure your Web Application

Posted by: Hemanth Dande 
(Sr. SQA Engineer working at PlanetSoft, an Insurance Industry Company).
Contact Author: hemanth.dande@gmail.com


PAROS Configuration

Open Paros by clicking the shortcut key or by the below shown navigation

Navigate to Tools > Options and configure the settings as shown below

Configure the browser settings as shown below

Recording The session

Open the application, Navigate to each possible corners of the application in all possible
scenarios. Then it will be recorded as shown below.

Navigate to Analyse and click on Spider, below shown Spider pop up will display. Click on
Start

After completion of spider, navigate to Analyse and click on Scan or Scan All.

After completion, pop up will be displayed as shown below.

Click on OK in above pop up, Navigate to Reports and click on Last Scan Report. The it
displays the path of the report saved as shown in below screenshot

Click on OK, Report will display as shown below.

Software Test Automation - Generations

Many of us might have come across buzzwords like Keyword-driven, Data-driven, Capture-Replay and these days Model-driven Automation methodologies. Let us try to understand these buzzwords in a more systematic manner through the generations of Software Test Automation.

To Err Is Human

Software developers are Humans and they make mistakes. But is that a good excuse. Definitely not. This lead to inception of Quality Assurance(QA). 

But QA Engineers are also humans. They do make mistakes. Also humans are not meant for repetitive tasks as they can get bored, tired or they can skip things. This is what exactly happens during testing of an application. Also due to lack of time few test cases will be skipped. This showed the purpose of automating execution of test cases. 

Software Test Automation is not a Substitute for Manual Testing

Areas like Regression testing, Sanity and Smoke testing are apt for Automation as they give a huge Return On

Investment (ROI).  Trying to automate test cases related to new features or features in development is not a good idea and it does more bad than good.

First Generation in Software Test Automation(Agreement)

At this stage every one agreed that Automation is required and started automating test cases.Scripting languages like Shell Scripting, Windows Scripting, Perl, Ruby, Python are used to automate tasks which will make testing easy and increasing reliability by reducing human errors. Also used rudimentary tools to record and replay test case execution.

Maintaining loose scripts became a major problem here. Also in record and play the flexibility part is missing. Example change of some steps in Test Case due to change in application is not possible and it needs re-recording of whole test case.

Second Generation in Software Test Automation(Framework)

To fight with the pain points in first generation, Frameworks have been evolved which binds loose scripts lying here and there and also provides utilities like reporting, logging along with Test runner. Also they decouple activities like reading configuration parameters and passing information from one test case to other. Also Object Oriented Principles (OOP) is strictly followed while writing scripts eliminate duplication of code and to reap fruits of OOP concepts. Also tools provided a feature to allow edit of recorded test cases.

Maintaining test cases using scripts became difficult as the Person should be Good at QA with deep understanding of Product as well as he should be good at Coding. This is the Generation which started demanding huge coding skill set from QA. But in reality percentage of Engineers who are good at both QA and Coding is very very less as there is an attitude difference in QA(Breaking) and Coding(Making). 

Third Generation in Software Test Automation(Approach)
 

Extreme Engineering is in place by this time. People realized that Test Design is more important that automation Technology. Keyword-Driven approach got introduced which facilitates one to separate Coding from Test Case logic. This helps a person who is expert in Coding to Code and QA with good Domain knowledge to write test cases. Along with this one can use Data-Driven Approach where data is separated  (not hard coded) from Test cases so that a test case can be executed across different test data sets. Also Integrated Development Environment(IDE) are developed to write test cases. More about keyword-driven approach will be discussed in my next Post.

But Software Test Automation is not just meant for execution of Test Cases as this alone will not guarantee the quality of an application.

Fourth Generation in Software Test Automation(Big Picture)

Capturing test cases from requirements is a huge task and should be done with perfection. Otherwise some of the test cases will not get captured and can result in defective product. In the process of Evolution Model -Based Testing came into existence. According to Wikipedia Model-based testing is the application of Model based design for designing and executing the necessary artifacts to perform software testing

Load Tests using JMeter

Create Test Plan in JMeter

JMeter Documentation is available here

Discussion on Different Load Testing Tools

Brain Dumps are available here

Setup of EC2 Clients

1. Create a key-pair and copy it to your Linux Machine.
2. Rename it as rsa-gsg-keypair(optional).
3. Change Permissions using
   
   

   chmod 600 rsa-gsg-keypair

4. Launch an EC2 instance from Amazon Console
5. User guide for EC2 clients is available here
6. Get the Public DNS Name of the instance.
7. Now Log-in to the Instance using
   
   

   ssh -i id_rsa-gsg-keypair root@xxx.amazonaws.com

8. If required change Client's Time Zone to GMT using
   
   

   unlink /etc/localtime

   ln -sf /usr/share/zoneinfo/GMT /etc/localtime

9. Download JMeter using
   
   
   

   wget http://mirrors.axint.net/apache/jakarta/jmeter/binaries/jakarta-jmeter-2.4.tgz

10. Install JMeter
    
    
    

    tar –xzf jakarta-jmeter-2.4.tgz

11. Set path to JMeter
    
    
    

    export PATH=$PATH:/path_to_jmeter_bin_directory

12. Download and install Java Runtime Environment (JRE)
13. Get Load test Plan (jmx file of JMeter onto this client)
14. Increase Open files limit using
    
    

    ulimit –n 8192

15. Run Load test using
    
    

    jmeter.sh -n -t M2LoadTests.jmx -l sample.jtl -j sample.log

16. Copying JMeter results (jtl files) on half hourly intervals make crontab entry as
    
    

    */30 * * * * /bin/cp /root/sample.jtl /root/Client-1-`/bin/date +"%Y-%m-%d-%H-%M"`.jtl

JMeter as a Service

Information on JMeter as a Service is available here

Home Work (Benchmarking)

1. Host a simple "Hello world" HTML Page on the server on which you are going to perform Load tests
2. Try to perform load tests by hitting "Hello World" Page using 2 clients. (500 Threads from each client)
3. Increase number of threads by increasing no. of clients.
4. Keep on plotting results in a graph.
5. If it is a straight line then some thing is wrong with your setup like webserver setup (look at webserver conf file)
6. If it is some kind of Curve shooting up then wait for a U-Turn in the curve.
7. Get Numbers related to Latencies at this U-Turn and consider them as lowest Latencies with given setup.
8. Get Numbers related to Throughput at this U-Turn and consider them as highest Throughput one can achieve with given setup.
9. Also get average network latency for 100 pings using traceroute. This will be the minimum Network Latency.

Key parameters to be captured

1. Processor Utilization
   
   
   

   dstat with -cp switch

2. Memory utilization
   
   
   

   dstat with -m switch

3. Disk utilization
   
   
   

   dstat with -d switch

4. Network utilization
   
   
   

   dstat with -n switch

Points to Remember

1. Servers should not be put behind Load Balancer
2. Install dstat if required using
   
   
   

   emerge -av dstat if it is Gentoo

   
   
   
   

   yum if is RHEL

3. Monitor Server side System Resources using
   
   
   

   dstat -cndymlp -N total -D total -l --output SystemMon.csv