From the monthly archives:

November 2009

Quality Goals: Define and Measure Quality of your application

Often when we talk about quality of software,we define it in terms of number of testcases,bugs,requirement coverage but this numbers in itself are not enough to incrementaly measure the quality of the application as well as quality of testing.The quality goals are owned by QA but defined by entire team and should be as important artifact as a test plan.

What are Quality Goals?

It is collection of individual metrics that helps to quantify quality of application as well as quality of testing.

Why define and measure Quality Goals?

Agile process gives QA as well as management to change or prioritize tasks at end of each iteration.The quality goal metrics accomplishes two things

  • Establish protocol between product management and engineering what are the expectations from the product.
  • Empower QA to have product management/development prioritize between new feature development vs fixing old bugs

How to define Quality Goals

The answer to above question will largely rely on processes in your individual institution,but I will take the lowest common denominator which I believe should be present in any QA infrastructure

The first part is to define ways to measure if the testing is sufficient.If we are not doing the testing right,we lose the right to say that quality of application is sub-par.

Quality Goals for Testing

  • Written Testcase /Requirements ratio:This number should be greater than 1.Ofcourse this assumes that requirements are complete and sufficient.So for this discussion let’s assume that all the requirements that are known to entire team should qualify for this metrics.
  • Written Testcases/Executed testcases ratio:Again the ideal goal is this number should be 1,but remember quality goals are determined by prd management,developers,QA as a team,so you can estimate lower.
  • Bugs yet to be verified

Quality Goals for Application

  • Non-executed testcases:The number to achieve is 0.
  • High severity Open Bugs/Total Bugs
  • Medium severity Open Bugs/Total Bugs
  • Low severity Open Bugs/Total Bugs
  • Untargeted Bugs

The above metrics will vary depending on your own process and outputs that QA efforts produce.You can define it in terms of iteration,release milestones or combination of both.E.g it can look like following.The following figure shows

quality goals for measuring quality of testing

Written Testcase /Requirements ratio

Written testcase/Executed testcases

Iteration

1

0.8

Soft Code Freeze

1

0.9

Hard Code Freeze

1

1

Release

1

1

What to do with this numbers ?

The idea is to measure this against actual numbers and depending on that take corrective action to either achieve the goal or lower the goal(Ofcourse in practice,management will never like it if you say that we would be shipping with 50% of intended quality).The more often we measure this,we have the option to do something about quality of application.

Please add more metrics and ideas about how to quantify testing and application quality.

Next time a brief talk about integration testing.

PS:Just got a chance to view a presentation by Selenium author and WebDriver author.Glad to know that the Selenium 2.0 will merge the two.It is a huge step towards correcting some of the shortcomings of each tool.Presentation can be found here.Video is here

VN:F [1.9.10_1130]
Rating: 0.0/10 (0 votes cast)

  

{ 0 comments }

Agile Planning and role of QA

Backtracking a bit from my original post,I thought I should cover the planning stage in Agile process and what QA should do in those stages.Different companies have different names for it,but I will give a short explanation of each planning stage

1)Vision

Def:What is the product’s vision for next 2 years(or a long term period,depending on product).Should be defined in a short 2-3 line statement(management speak:elevator pitch)

Normally this is conceptualized by executive management/product management.Sometimes defined in terms of persona.(For e.g product X is normally used by people with age 20-25.Next 2 years make it target people with age 20-30).This can be part of a standard business plan or company strategy plan.Regardless QA gets little to do here/or probably not much to say.But if you are lucky enough to be part of this,as a QA person this can serve as the long term vision for the quality goals of the product and have the testing processes/activities updated according to it.For e.g if the vision is to grow from tens of users to thousands of users,as a QA Manager it’s time to start making noise about getting appropriate hardware for this long term growth.

2)Release planning

Once the vision is defined and list of requirements listed down,the team starts doing release planning.Product management starts writing the wishlist in form of user stories.Engineering(Dev/QA/IT) starts estimating how much time is required for each story.If questions arises,new stories are added.QA should make sure following is done as part of release planning

  • Start adding stories which are not defined for a product,but neverthless relevant to successful release(e.g automation/performance testing/migration etc).This way all necessary QA activities are correctly estimated for the release.
  • Define Quality Goals for the product for this release.This can be defined in terms of severity of bugs etc.I will try to elaborate on this in a different post.
  • Also tag stories which requires some kind of UI and screenshots will be helpful.Normally this is helpful to both Dev/QA,so either one can take a lead.QA can start writing better testcases if they have screenshots.Also it allows product management to visualize what they think and this helps in earlier detection of workflow issues,which UI component to choose etc.
  • If there were stories that didn’t make it in previous release,but is important for quality/maintainability of the product,push for those stories to be included in this release
  • If the product involves external hardware/software configuration(e.g OS or browsers),get the list narrowed down.Also be aware of yet unreleased OS/browser but available when product releases and ask product management of need to support it.Prospective customers also can define system requirements

At the end of this exercise,after couple of push/pull’s between product management and engineering management,we should have the following

  • Release date
  • Major milestones
  • Resource Allocation(QA lead/manager can now start doing resource planning)
  • Software/Hardware Matrix
  • Quality Goals
  • List of requirements to be finished for this release

Iteration Planning

Each release is divided typically into 2 or 3 weeks of iteration.Agile requires that and end of each iteration we should be able to release the product(but that’s hardly the case in practice),but regardless a good goal to keep in mind.Team gets together and starts pulling in stories from the backlog to be implemented in this iteration.QA needs to do following

  • Each story should have estimates for unit testing,bug fixing,optimization.
  • QA estimate should include writing testcases,executing testcases and bug verification
  • Make sure engineering doesn’t go overboard with story allocation and QA should try to maintain balance between implementing too much with low quality and implementing little with high quality.This is where the initial quality goals come into the picture,we could also have iteration quality goals and adjust iteration planning if we are continuously missing the quality goals.
  • As iteration progresses,and our initial estimate is wrong or we overshot ,bring this up in team meeting and have the story moved to next iteration or backlog.This is harder in practice as sometimes it’s hard for QA to explain why he/she feels that the story would not be completed and why it’s important to move it rather than keep it and think about it at end of iteration.
  • For each story to be marked completed,all bugs related to it should be fixed(or with reference to iteration quality goals).Help developer achieve this through continous bug reviews(once in a week or twice a week) with product management/engineering management to whittle through “fix it now” vs “ok to fix later” bugs.

Next post,I will go in details about Quality Goals.

VN:F [1.9.10_1130]
Rating: 9.0/10 (1 vote cast)

  

{ 0 comments }

Adopting Agile Methodology for SaaS

SaaS is still the new kid on the block and the friendship between SaaS and Agile brings forward few interesting aspects. The most important aspect in applying the Agile Methodology in any given organization is – its impact on the team structure. More precisely the distribution of the team is a critical success factor in enabling the implemented Agile process to either be a success, or be a foul strategy implementation.

Advantages:

Team Restructure: Team Restructuring can be advantageous if the distribution of the team allows all the senior resources to be divided into smaller teams and are paired with junior resources to help them out. This gives higher responsibilities to the senior members and the junior members get a chance to be mentored closely.

Identifying leaders: Agile methodology especially in the SaaS environment will provide numerous opportunities for team members to demonstrate their strengths. This should be used by the senior management to groom the key members according their strengths and future aspirations.

Long Term Investment: Implementation of Agile Methodology by a company which is developing/testing a SaaS based application would need modifications over the span of time. Agile allows different teams to much more involved in the process leading to better awareness about product design and inter team dependencies

Challenges observed:
Meeting the Deadlines: Agile methodology claims that the time taken for Developing, deploying and Testing is cut down to almost half than what is available in the Traditional System. The developers and testers need to be cognizant about performance and security aspects and the added boundaries of timelines may add to the workload.

Delayed ROI: For the Top Executives, the top brass would want to see far reaching results in terms of impact and ROI for the new methodology to be implemented. Less ROI in the initial stages may deny the methodology of the executive support required to withstand initial roadblocks. SaaS model already has longer breakeven periods for providers and any further delays would be watched very closely.

VN:F [1.9.10_1130]
Rating: 10.0/10 (1 vote cast)

  

{ 2 comments }

Testing in Agile process

Past few years,many companies have started adopting Agile process for their software development.I will try to share some experiences and related technologies that QA can use to make the transition from the waterfall model to Agile model.For this particular post let’s dive right into the first stage of waterfall model and a parallel process in Agile.

Requirements

Waterfall model requires that requirements are listed before any design/implementation begins(There are variations of it like prototyping which involves some coding).But the basic concept is to have requirements cleaned up before full-scale development begins.This usually requires a dedicated person who needs to work with the user/owner to get the requirements and write them down in a format understood by Dev/QA(Usually MS Word or other graphic design tools like Visio etc).

The Agile process actually turns this concept upside down and throws in the concept of being “Agile” in accomodating new requirements during the release.The idea is to plan for the big picture but keep refining details as we go and as part of a team rather than being owned by a single person.

From my personal experience for a QA person

Advantages

  • Requirements are often discussed/refined as a team(standups/scrums) and covers the business/technical aspects of the requirement.This leads to better understanding of the product by each team member and avoids late in the game objections
  • Agile process often requires having an entry and exit criteria for stories(a compression of things to do in a particular release/iteration).This makes sure that the requirements are well defined and have a measurable way to tell whether the requirements were actually completed or not
  • Having a concept of backlog definitely helps,where new ideas don’t get sidetracked but are kept for future releases(so that it can be sidetracked in future :) ).While it can be argued that something similar can be achieved in waterfall model,normally a Agile management software(like Rally,TargetProcess etc) ensures proper management and transparency not available in Waterfall model

Disadvantages

From my perspective,the disadvantages or weaknesses does not stem from the process,but gaps in correctly implementing the process.However some of them are easy traps and can be perceived as Agile’s weakness in this area(SW Engineering Rule 1:When a product fails,blame the process not the people :) )

  • If big picture requirements are not clear,the details within will be muddled as well.Normally for new products the SW architecture takes a path based on the intial requirements.If you start changing the requirements(as allowed in Agile) during the release,it’s very hard for Engineering to adapt to those changes as lot of effort has already gone into the intial requirements development and testing.
  • Sizing requirements and estimating how long it will take is a big people dependent task.Normally QA gets the short shrift ,since it’s logically the last task in marking a story as done,any delay in the prior task(development) is going to hurt QA timelines.Lot of time QA doesn’t even get to execute even one testcase in the whole iteration and QA is struggling to finish the task.Ofcourse this can be improved by better estimation and scheduling,but it’s harder in practice.There are ways to get QA getting hands dirty earlier(like writing integration tests,which I will cover later)

Last,What’s the role of QA when requirements are getting written

  • Stay involved when the big picture requirements are written.It’s your chance to make sure testing estimates don’t get overlooked
  • Ask questions(and lots of it) when the story gets picked up for implementation(Assumption is the mother of all screwups-Anonymous) .It will save QA lots of time later
  • When new stories are added during the iteration,make sure that QA time is counted and also get commitment from development to have initial estimate include bug fixing,optimization etc.
VN:F [1.9.10_1130]
Rating: 9.0/10 (1 vote cast)

  

{ 0 comments }