Posts tagged as:

Agile

Your Test Lifecycle Management Tool – What Should You be Looking Out For?

The software ecosystem and ALM

Software success is crucial to the success of your business.  And so the software ecosystem needs to be effectively and proactively managed. This requires adoption of ALM which is the business of designing, building, testing, delivering, managing and maintaining software.  Testing being a big part of this supply chain needs some special attention since that guides the path to software’s success.

Where is the TLM heading?

Testing has matured from a random click activity to a more organized science. It has a structure and a method to it– be it management of test cases, design, analysis or execution. Excels, Words and XMLs are suddenly becoming obsolete as the needs become more complex to manage and maintain.  Adoption of cloud systems and virtualized infrastructure are forcing business to move beyond their ‘walls’ towards a virtual server world which changes the equations further.

And so with such a rapidly moving market, organizations are unable to respond effectively with legacy tools. So for state-of-the-art software delivery, there is a need for a state-of –the-art TLM and ALM tools. The application lifecycle paradigms are diverging from traditional to agile and scrum and the heterogeneous nature of this ALM and TLM market has a several varied tools from vendors leads to a fluid market and needs to be  integrated seamlessly.

I got a chance to sit through an analyst briefing for our test management tool – QMetry and managed to gather some interesting data on  morphing test lifecycle management industry and what it would take to be market leaders in this space.

Making the case for a solid test management tool

First, as I mentioned above, there are a million TLM tools that need to integrate with each other seamlessly. These disparate tools could be performing defect management, automation, requirements management, change management, prototyping/WI reframing management etc. Your TLM tool needs to be an open platform system tool that integrates seamlessly with this fluid ecosystem.

It would be nice if the test management solution provides an intelligent testing platform with add-ons for a vertical solution. For example, HIPAA and FDA compliance regulations in the health care vertical, or some other specifics for travel or banking industry or the mobile vertical.

Specific to mobile vertical, a TLM solution needs to build out platform capabilities which would help to market mobile apps and websites  faster. It should test and monitor any mobile app or website – native, HTML5, or hybrid across multiple devices and platforms such as iOS, Android, Windows or test interactions from any part of the world.

A good TLM solution needs to offer solid reporting capabilities with out of box capabilities for metrics evaluation etc. There is an established need for query engine capabilities, archived audit trials, and custom field reporting as advanced functions to support advanced needs. A solid change management mechanism to house historical data and make way for new methodologies in agile and scrum is also required.

Hope this helps in your decision for choosing the right TLM tool for your business.

VN:F [1.9.10_1130]
Rating: 6.0/10 (3 votes cast)

  

{ 0 comments }

Agile Methodologies – Confessions of a QA Manager

What experiences should we work by? I think most, if not all, of us have worked in, around or through those SCRUM projects either

  1. Where a company or department has just adopted Agile
  2. Claim that previous releases, defects, infrastructure and current change is ok and needs to happen simultaneously;

They are very serious about keeping important processes moving with frequent fire drills. No matters what the cost of these overnight sessions are, all hands are on the deck via Skype or Messenger all across the globe as long as a ‘believed’ shippable package is ready for production in a 2-5 week sprint.

This is one example that I’ve lived through over and over again with a charter to introduce 1 new agile improvement technique per sprint before I joined InfoStretch. On top of that, this was deemed and touted as a ‘world-class’ organization with wise and brilliant leaders. There’s no question the latter was true but something seemed to get in the way of the ‘world class’ part.

What are the agile accelerators? Agile accelerators are those which:

(1) Meet the user needs created by an agile team that could synthesize functional and non-functional features effectively, under pressure, especially being humble enough to tackle two things that did not work well in a previous iteration.

(2) Be shippable quality and aligned to quorum of shared knowledge between end users, product management/biz analysts, development, test and operations.

(3) Be created per and only by acceptance criteria and as a side-effect of Automated Test Driven Development (ATDD).

(4) Flow through the life cycle smoothly even if changed, delayed or added by management or affected by resourcing through Fibonacci based point system and worthwhile 2-4 hour exhaustive planning sessions.

(5) Survive onslaught of heavy numbers of client facing defect fixes.

(6) Struggle between product management and testing or operations around handling backlogs and project roles.

(7) Which metrics are most helpful, ‘agile’ and beneficial?

I suggest these as warning signs in your project, if agile accelerators like the following:

(i)           are not welcomed,

(ii)          if introduced, not adopted

(iii)        are considered not agile enough criticized for being ‘too waterfall’

Somehow to consider a development life cycle practice in agile SCRUM and still have management as part of the core team and have inflexible rules of engagement, may be worse than a team focused on AGILE practice operating stagnantly.

Agile accelerators to turn a failed SCRUM-like project into a successful SCRUM or AGILE project:

1)   Master test strategy document stressing consistent user story card point velocity from sprint to sprint or iteration to iteration sometimes considered un-agile because we favor people and their skills over documentation from the agile manifesto.

2)   The MTS sets the ground rules, directly, concisely esp. around agile iterations, acceptance criteria, roles, responsibilities, life cycle (# of construction iterations, # release/system test iterations) and most importantly, the feature sets and criteria that make the package releasable at the end of the sprint cycle.

3)   Test environments planned, setup, data requested and managed

4)   Tests created and stored in test management tool for iteration test and system test

5)   Bidirectional traceability between story card – test case pass and acceptance criteria incorporated

6)   Tester collaborates with analyst & developer ‘The 3 Amigos!!’ to prepare acceptance criteria

7)   Functional tests are created during Iteration test and failures are fixed ‘in place’ according to acceptance criteria

8)   System tests are created for end to end scenarios per MTS

9)   Usability testing is performed where needed during iteration testing

10)  Triage process defined for customer facing issues/emergency with fixes in current release

11)    Issue found after iteration test are entered as defects and tracked

 

VN:F [1.9.10_1130]
Rating: 0.0/10 (0 votes cast)

  

{ 0 comments }

From Waterfall to Agile:7 mistakes you shouldn’t be making.

Past few years software industry has taken up the Agile buzzword in the same league as “2.0″,”social networking”,”cloud computing”.While some of this words are really taken up to attract investors or sound new-agey,Agile has not been abused as much due to it’s inherent back office use nature.However like all good things as the good word spreads it takes new meanings as well as a general incorrect use which has an opposite effect of what’s intended.

Some of the most common mistakes are made when an organization strives to move from the traditional waterfall to an Agile process.Here are some things to watch out for

  • Daily scrums turn into status meeting or long technical design discussions.
  • Daily scrums run longer than 15-20 minutes at the most.
  • Requirements and scheduling still happens at management level and not through team participation
  • The release cycles are not getting shorter.
  • Communication still flows top to bottom instead through team members
  • Teams are still seen in silos and boundaries are not disappearing.
  • Not enough tests that run automatically with every build.

This are just some of the things that you should look for when you are trying to build a true Agile team or process in your organization.Otherwise it would just be another buzz word to make the management feel good about it

Agile is not an easy process to adopt as well transformation of a culture used to the waterfall methodology is even harder.However if done the right way it can bring wonders to your organization and it’s highly satisfactory and rewarding for team members.

VN:F [1.9.10_1130]
Rating: 0.0/10 (0 votes cast)

  

{ 0 comments }

Importance of Dashboards in Continuous Integration

The inspiration for this post is KaChing’s wonderful DevOps dashboard . This dashboard empowers Kaching team to do about 50 deployments to production per day.It takes about 5 mins from a code commit to deployment.This brings to the forefront need of effective dashboards in each step in Agile process.While as an organization we might have different tools that provide a high level analytics,it’s important that all this data is collected and presented to do release level risk analysis and status analysis.Let’s check out some common metrics that we track while doing Agile development and see why it makes sense to bubble up this metrics to the Super Dashboard

Code development

Normally unit testing is written as code is being written.This is a compile time statistics and normally resides in your build server (e.g Hudson).This data should be bubbled up to Super dashboard.

Coverage statistics:We would want to measure whether all the code that was written is unit tested or not.Again a very good candidate to put up in the Super Dashboard

Integration Testing

You might be using different tools like FitNesse,RSpec,Cucumber but all this tools would have a statistic that you would normally track to promote the builds.The results from this testing should be consolidated and be part of the Super Dashboard

UI Testing

Again you might be using tools like Selenium,Watir,Silktest or other GUI tools.The pass/fail result ensures Test Completeness and should be made part of the Dashboard

Test Management tools

Several data points are collected as part of your Test Management tools like Quality Center,QMetry.This can be an important data point to understand if enough testing was done or not.Different release data points from this tool should be made part of the dashboard.

Requirements Management Tool

Metrics from tool like RallyDev,TargetProcess will ensure that a particular release is on track and will give the complete picture in terms of real progress(E.g good unit testing numbers don’t mean much if the requirements are not yet completed)

You might have other things that you would like to see in your Super Dashboard.The goal is to just look at it and turn on the switch on deployment.

VN:F [1.9.10_1130]
Rating: 8.0/10 (1 vote cast)

  

{ 0 comments }

Using business value as a criteria for requirement prioritization

As a followup to an excellent article by my colleague Marcus-Requirements-Building the foundation strong. ,In the Agile scheme of things,it becomes almost a necessity that requirements are well-defined and with enough information to prioritize and do iteration planning.One guideline that is often repeated to measure the requirement is

  • What is the business value of implementing this requirement?

Some of the examples are

  • Improves usability of the application
  • Will make the application faster
  • Adding this feature will make a workflow X easier to use.

On face value ,this makes perfect sense.I mean why do it if it’s not positively impacting a valuable feature in the application or not providing a valuable feedback,but if not used correctly this can often lead to dropping of important non-business facing requirements.Some of the examples of requirements which normally don’t pass the business value test is

  • Change the architecture
  • Technological changes that impacts the long term roadmap of application
  • Internal products such as metrics dashboard

To summarize,at any given time,it’s important to keep things in perspective and make sure that seemingly non-business value tasks are in fact important stories in life-cycle of the product.

VN:F [1.9.10_1130]
Rating: 0.0/10 (0 votes cast)

  

{ 0 comments }

Integration Testing:Test the right areas and discover bugs earlier.

Integration Testing:How to automate the core of your application?

What integration testing means can be different depending on your architecture,but for our discussion I would like to focus on role of integration testing in a service oriented architecture.For this discussion integration tests are primarily tests that do not require an actual application or browser to be launched.The tests should be able to run in a non-GUI environment.This is a testing layer that resides almost between unit testing and GUI testing.Depending on your architecture,this can be also the layer where your application API(public or testing) are defined and can be used to do significant business logic testing.

Why do integration testing?

Integration testing like all things in testing is not the ultimate solution,but part of an overall solution.Here are some key advantages

  • More stable than GUI tests
  • Useful in writing precise component level tests.
  • Drives better code design
  • Tester understands the internal workings of application allowing better writing of test cases
  • Easy to integrate in a continuous integration environment
  • Test the core of your application agnostic of UI.If your application has a large business logic area and a light UI layer,this is the right layer to do your automation.

What about the disadvantages?

  • Testers need to understand code and at times also write code.
  • Continuous integration framework though not necessary is nice to have to take full advantage of integration testing.Significant effort is required to get the whole system going.

Rather than discuss the technical details of how to do integration testing,I would like to share my experiences with two approaches that I took and my thoughts about it

Approach 1:API layers or interface layers

This was a Java application so there was no traditional API layer,but while programming there were some natural interfaces created by the application.But since they were written keeping the application and not testing needs in mind(which infact is not a bad thing,if you think about it).

The tests still need to be written using some framework.I used JUnit but in hindsight,I think TestNG is a better choice for integration testing.Since the API’s are not written for testing,Lot of time goes into understanding the limitations and actual functionality of the interfaces.This improved my understanding of the application but decreased my output in terms of actual testing to be written.Also when a functionality is not exposed,writing of the interface or exposing the functionality through existing interface falls on tester’s  shoulders and can be easy or hard depending on various factors.The tests were not plugged into a continuous integration architecture and hence keeping track of it ,fixing as soon as it failed turned out to not so easy task.

Approach 2:The other approach was using FitNesse.While FitNesse as a tool has it’s own advantages and disadvantages,from point of integration testing here are some of the advantages

  • Tests easy to write in Wiki format.
  • Drives better design of code when “fixture” code is getting written.
  • Tester gets better understanding of application writing component level tests.
  • Quick feedback and easy to see test results helps in finding bugs quickly.

While not exactly disadvantages,some of the pain points are

  • Developers/Testers need to write additional code to expose functionality for testing.
  • Working around FitNesse issues is not always straightforward.
  • Learning FitNesse takes time and effort.
  • Writing code to test actual code  is a change that entire organization has to learn to incorporate into planning.

Having said all of this,Integration testing becomes more important if you think about multiple browsers and different mobile devices which even though have separate UI will be hitting the same business logic.Any good integration testing strategy will give testing team more confidence and will be as important as any UI automation strategy moving forward.

Curious what is a good integration testing strategy for your organization.Infostretch can help you out.Contact us or follow us on twitter

VN:F [1.9.10_1130]
Rating: 0.0/10 (0 votes cast)

  

{ 0 comments }

Selenium and Hudson Integration:Keeping your Test Code separate from execution code

After my last post on integrating Selenium RC tests with Hudson,I had few folks asking me on how I organized my test execution code with selenium RC tests.

If you are like me you probably started with running Selenium RC tests on your local machine and didn’t give much thought about integrating with Hudson,but when it was time,there was lot of files moving around and unnecessary cruft of code.

To run this sample,you will need ANT and Java on your machine

So here is a sample TestNG runner code which is ready for Hudson to consume and at the same time having a clean separation between Selenium RC tests and execution.This structure should make things easier when multiple team members are writing test cases allowing easier test code checkin to your version control system.So look at the attached zip file.Please find instructions on how to use this in the README folder.I have also included a sample Selenium RC project which the current TestRunner configuration points to.

This code is not intended to replace the Eclipse+TestNG plugin and it’s use in dev environment is highly recommended.But this code does the next step of getting your Selenium RC tests ready to be used by Hudson.For now the test runner is overtly simplistic and can only run a single TestNG configuration file,but feel free to extend this and make your own modifications to suit your needs.

When you are ready for Hudson to run your Selenium RC + TestNG tests,have your Hudson job execute the STunner/bin/seleniumtestrunner.xml and sit back and watch your tests run.

If your Hudson is on a Unix machine,look at my earlier post on Integrating Selenium Tests with Hudson CI

If you want how to convert your hudson cluster to Selenium Grid ,this post should help you.

Let me know if it doesn’t work for you or other issues that you face.

Follow us on @infostretch on twitter to get more updates on Selenium as well how we can help your organization get Selenium ready.We offer multiple Selenium solutions that fits every project size and needs.

Download the Selenium TestNG Runner from here

Important:The bundle doesn’t contain the selenium-server.jar as the size of the bundle increases.So the very first thing you will need to do is to download selenium server and copy it under ‘server’ directory.

VN:F [1.9.10_1130]
Rating: 4.9/10 (9 votes cast)

  

{ 0 comments }

Introduction to FitNesse in 10 minutes

10 FitNesse questions answered in a simple language

If you are like me and have heard about FitNesse but never got your head wrapped around what exactly FitNesse does and who should use it.Here are some FAQ’s that I think will be useful for anyone starting on using or thinking about FitNesse.

1)What is FitNesse?

While here’s the official description of FitNesse,from testing perspective,it’s a automation tool that fits somewhere between unit testing and UI testing(manual or automation)

2)Why should I think about using FitNesse.?

If you are looking for a simple way to get quick feedback of the quality of your software,FitNesse can be of tremendous use.If your organization aims for quicker releases FitNesse can be an important tool along with unit testing and UI automation testing

3)Who should use FitNesse?

As the one minute description of  FitNesse suggests it’s for everyone.But Testers can use it to automate acceptance testing as well as test individual components.

4)What language do I write my tests in?

While initially this takes some time to get used to.The tests have to be written in Wiki language.Personally this is not the perfect way to write testcases,if you buy into the basic purpose of FitNesse you will understand the logic behind having Wiki as the language for test case writing

5)I looked at the documentation.What is this Slim and Fit business?

Here’s the official explanation through a diagram,but in simple terms they are two separate systems of writing FitNesse tests.If you are starting fresh,it’s recommended to go with Slim(However there is no complete escaping from Fit,the reasons for it will be discussed later)

6)What are Fixtures.Sounds intimidating.

Not to worry.Fixtures are nothing but simple Java classes that expose the applications/component’s functionality.

7)What is Table Styles.Is it necessary?

Yes it is necessary to understand and know what each of the Slim Table styles are.The testcases have to be written keeping this table styles in mind and it will also determine the Fixture code (class name,method name etc)

8)Where do I start?

Strongly recommend to read the official documentation.Because of it’s Wiki nature,it can get hard to have a linear narrative,but few hours spent and you will understand what’s important for you.Also read on Brett’s excellent tutorials on FitNesse.

9)Can I integrate FitNesse tests in my continuous integration cycle?

Ofcourse you can.Look at the FitNesse command line testrunner.There are also plugins available for Hudson.So check out your favorite CI and FitNesse might be available as a plugin

10)Give me the bottomline.What do I lose if I don’t use FitNesse?

Use FitNesse to fail fast.Discover and fix bugs earlier before it propagates to the UI.The learning curve is  a steep one,but the rewards do pay off handsomely once the basic structure is in place.Like any good tool you need to go through the pains of learning and optimizing for your organization’s needs.

There is much more to FitNesse than what’s mentioned above.To know more add your questions here or on twitter @infostretch.

VN:F [1.9.10_1130]
Rating: 5.5/10 (2 votes cast)

  

{ 1 comment }

GUI Test Automation:Why Selenium can be the tool of your choice

Let’s say your organization has now a stable set of manual testcases that is used for regression every time your product releases. As you are aware this set keeps on increasing and soon you are seeing that the regression test cycle is weighing down on your product release timelines. In these days of Agile practices, the obvious next place you look at is to automate testing  such that regression cycle is shortened. This is not the only benefit of automation, but we will leave that discussion to some other time. Also for purpose of this discussion, we will concentrate on GUI automation tools. So if you are responsible for GUI automation, the very first question that will come up is

“What is the best GUI test automation tool for my application”?

If you had asked this question in any QA forum or just done a plain search, atleast one of the answer to above question would be Selenium. To keep this discussion in perspective, I would like to discuss merits/demerits of Selenium being the tool of choice.

Merits of using Selenium

–Choice of using language of your choice.

This is an advantage which has an indirect benefit of plugging into your engineering architecture with ease. For example if your organization is a Java shop, you already have a knowledge and technical expertise available with range of tools (e.g IDE, build systems, reporting tools etc.) that your development branch uses. It would be just a matter of transferring this know-how to your QA team which they might already be familiar with it.

–Script once, Run on multiple browsers

While this is not exactly true in simplest sense of speaking, the advantage of  running your tests on multiple browsers with the same set of scripts is an enormous advantage when you have to test your application on different browser version

–Inherent applicability to AJAX

Selenium works in browser, with the core technology that is driving your test case working along side your application under test (AUT). This provides incredible control over testing of AJAX web applications .

–A community supported Selenium Software suite

Selenium users can and should take advantage of  Selenium IDE,A Firefox plugin which assists you in recording and running your tests.Selenium Grid allows you to take your tests and deploy it in a clustered environment optimizing use of your resources. For details please refer to this Selenium documentation

—Support multiple test frameworks

Contrary to common thinking, selenium is not a testing tool. It is more like a GUI driving library.  This is a blessing.  Selenium allows users to wrap the test scripts in the framework of their  choice. So if you like JUnit vs TestNG,you have a choice and also taking advantage of reporting capabilities of this framework that your development team might be already using.

—Integration with the ecosystem

If your requirement is integration with different suite of tools like Hudson, QMetry etc,there are already solutions available out there which will make the integration easier.With almost no investment in purchasing tools,you can create a test automation infrastructure right on par with a commerical suite of tools. There are also multiple solutions available for you if you want to run tests in parallel. Check out selenium Grid and saucelabs

Having said all of the above,what are the areas where Selenium falls short

—Supporting non-web applications

Selenium is designed and growing keeping in mind that it would be used to automate web applications. So if you have a windows application or a Java applet or applications that use Flash, it’s not easy to use Selenium. However other tools also do not do a better job of supporting this applications. Technically there can be other solutions to take care of this shortcoming from selenium itself.

—Maintenance readiness is not inbuilt

If you look at commercial tools, there would always be a concept of storing UI object definitions separately from test code, test data management and other nice utilities built into the tool. In Selenium, this is left to individual’s implementation. While both approaches have it’s pros and cons, a well-written Selenium framework can go a long way in giving you a head start in writing Selenium test cases.

One last point…..

Often Selenium is offered as a low cost  (opensource)  alternative to commercial tools like QuickTest Pro or SilkTest. While this being true in terms of obtaining the actual software, i think it’s imperative to understand that since writing and maintaining various functionalities that would have been offered as part of the commercial tool package now falls on individuals, this cost should be taken into consideration while making the final decision

Where does all this information leave you in terms of answering the original question? The answer relies on number of factors, ranging from management goals to technical requirements but Selenium should be considered an important candidate along with the commercial tools which not only requires a huge upfront investment but also significant investment in learning a new suite of tools, a team dedicated to specific skillset etc.

Infostretch can help you determine whether Selenium is the right tool for you or not.Contact us at 408-727-1100 or follow us on twitter @infostretch

VN:D [1.9.10_1130]
Rating: 8.6/10 (32 votes cast)

  

{ 9 comments }

Using Hudson as Selenium Grid for Continuous Integration

In my last post about Hudson integration with Selenium RC,I talked about how to use Hudson to drive your Selenium RC tests.This configuration would work fine in most scenarios.However this is not a optimized configuration if you want to take advantage of executing your tests in parallel.For e.g if you have two machines in your lab with FF 35,you might want to run RC tests on both machines at same time.

Selenium Grid offers this capabilities.Below I will discuss how to take advantage of this in a continuous integration process.

I strongly recommend that you try to setup Selenium Grid on your local machine first if you had no prior experience with Selenium Grid.There are some very good Selenium Grid demos available here

Step 1

Download and Install Selenium Grid plugin for Hudson.You should see a small icon on your Hudson home page.Clicking on which you would see the RC’s registered to this Selenium Grid

Step 2

Important before this step,make sure that you are able to ping the RC server from Hudson and viceversa.If they cannot see each other it will not work

Registering your RC to Selenium Grid.This took me quite a while to figure out,but the trick is in the way you register this RC.

ant -Denvironment=”/qalab_01/windowsxp_1/:*firefox”  -Dport=5555 -Dhost=10.63.87.192 -DhubURL=http://myhudson.com:4444 launch-remote-control

Let’s go through the parameters one by one,you should be familiar with the parameters if you did a local setup of Selenium Grid.

environment=Machine Name/Label Name:browerCommand (Look at my earlier post about Selenium Hudson Integration to understand how to setup machine names and label names)

port=Available port on the machine where you are launching RC

host=IP address of your RC machine

hubURL=location of your Hudson server (which also doubles up as Selenium Grid now)

Step 3

Repeat similar process for another RC and when you click on the Selenium icon on your Hudson page,you should see that there are two RC’s registered to this Selenium Grid.

Step 4

Configure your Hudson job to run on master(i.e Selenium Grid) and it will take care of assigning the jobs to correct RC

Some important resources that helped me during the setup are

Amit Easow’s post on Hudson integration

Unregistering the Selenium RC command by this Selenium Blog.The command is lifesaver and I am repeating a example here

http://localhost:4444/registration-manager/unregister?host=locahost&port=5556&environment=*iexplore

To give a recap,I used ANT,TestNG,Selenium Grid,Hudson to accomplish this.

One last thing,The Selenium server jar is part of the Selenium Grid plugin bundle.You might want to replace it with a version of Selenium server you like on the Hudson server.

Follow us on @infostretch

VN:F [1.9.10_1130]
Rating: 7.8/10 (13 votes cast)

  

{ 43 comments }

← Previous Entries