From the monthly archives:

September 2010

Leveraging open source automation tools to reduce costs and optimize test process

With the shrinking development cycles, pressure of reducing the time to go-to-market and repetitive processes, automation is no longer optional. Also, there is no relaxation on the quality of the final products; companies just can’t afford to launch products that are not ready for the prime time. While test management tools like QMetry, Testopia, Quality center etc. provide a great way to manage and improve the overall test process, various test automation tools help further optimize it. Therefore, it is not surprising that the demand for test automation tools has increased drastically and more so for less expensive ones, given the current economic situation.
This has given rise to a number of open source test automation tools (See the list below). These open source tools offer an inexpensive way to automate the testing process. However, we still need to evaluate them against the commercially available tools that seem to offer greater predictability and support. Tools like QTP, Rational Robot have served the test automation needs of QA teams for a long time, but there have been many challenges and constraints that users have been facing with these tools. Some issues with proprietary tools that need to be addressed are:

- These tools are highly expensive, often times outside the budget of smaller QA teams
- As the application undergoes revision over time, maintenance costs of test scripts rises sharply
- Restrictions on licensing
- Lack of support for scripting language
- Dependency on add-ons

As far as the open source testing tool market is concerned, with so many players jumping onto the bandwagon, it is a herculean task for new entrants to differentiate and find a place for themselves The number of open source automation tools makes it hard for users to identify the best for their needs. At Infostretch, we evaluated almost all the tools out there on different parameters and discovered various strengths and weaknesses associated with them. (I’ll be publishing those findings in my next blog, stay tuned). We have been helping our clients in selecting the best tools and in developing various frameworks to address their specific needs. We spend most of our time in understanding the nature of application, business objectives, and the features that are required of the automation tools. This information is critical for selecting the right tool and developing robust frameworks.

In the later blogs on automation, I will be talking about our research findings, frameworks, testing on the cloud and open source test automation tools in general. In the meanwhile, if you have any questions or suggestions regarding any of these topics, please feel free to write to me at Durgesh.kaushik@infostretch.com.

List of Open source testing tools for functional testing (Source: http://www.opensourcetesting.org/)

Abbot Java GUI Test Framework
Anteater
Apodora
Arbiter
Autonet
AutoTestFlash
Avignon
Blerby Test Runner
Canoo WebTest
Capedit
Concordion
Crosscheck
csvdiff
CubicTest
Cucumber
DBFeeder
DbFit
DejaGnu
Dogtail
Doit: Simple Web Application Testing
Eclipse TPTP
EMOS Framework
Enterprise Web Test
Expect
expect-lite
Frankenstein
GITAK
GNU/Linux Desktop Testing Project
Harness
httest
IdMUnit
Imprimatur
ItIN – Infopath testing in .Net
ITP
ItsNat, Natural AJAX
ivalidator
Jacobie
Jameleon
jDiffChaser
Jemmy
JFunc: JUnit Functional Testing Extension
JSystem
JUnitum
jWebUnit
Latka
Linux Test Project
LogiTest
LReport
MActor
Marathon
MaxQ
miTester for SIP
Mockito
MozUnit
OLVER – Open Linux VERification
Om
Om
org.tigris.mbt
PAMIE
Pounder
pywinauto
QAliber
QAT (Quality Assurance Tests)
QMTest
Rasta
Robot Framework
Sahi
Samie
Sauce Labs: cloud-based Selenium service
Scalable Test Platform
Selenium
SharpRobo
soapui
Software Automation Framework Support
Software Testing Automation Framework (STAF)
Solex
SWAT (Simple Web Automation Toolkit)
SWTBot
SymbioTeam Lite
Systin
T-Plan Robot
tclwebtest
TestGen4Web
TestPlan
TextTest
Tomato
Toster – The Object-oriented Sofware Testing Environment
Watij
WatiN
Watir
Web Form Flooder
WebDriver
WebInject
WebTst
WET
Win32::IEAutomation
XML Test Suite

Other Resources:

http://www.slideshare.net/sebastian_bergmann/testing-with-phpunit-and-selenium

http://www.slideshare.net/hsplmkting/fitnesse-watir-and-ruby-based-test-automation-system

VN:F [1.9.10_1130]
Rating: 8.0/10 (10 votes cast)

  

{ 0 comments }

Meux – Automation Tool for Mobile Devices

I was highly excited to read the introduction of the Meux tool from Jamo Solutions. For a while now, I have been working on mobile QA, both manual and automation. Automation was always carried out using simulation tools such as Fiddler and Bayden UA Pick. But for certain testing the actual device became indispensible and that’s when manual QA had to kick in. I always hoped for a tool that would run my automation scripts directly on the phone.I was highly excited to read the introduction of the Meux tool by Jamo Solutions. It promised to do exactly what I was desperately looking for and that made it my new best friend.

Meux tool by Jamo Solutions available at http://www.jamosolutions.com/documents/downloadproduct.php is an add-in to QTP that records and plays back automation scripts on a mobile application just as it would on a regular desktop application. A pre requisite of having the Meux agent installed on the device and the QTP Add In installed on QTP machine is required. Both of these are available as a part of the installation package. The Meux Device Manager acts as the bridge between the add in and the agent. The installation and set up would require some extra technical expertise with port numbers being used by the agent clashing with the the port numbers already in use on the machine.

After the initial hiccups of setting up, I was highly thrilled to try out the application and see how useful it can be in my mobile automation tasks. Currently it supports only Windows Mobile, Blackberry RIM OS and Android OS with the limitations on the versions of the same. Recording on mobile application is almost as simple as on a desktop application. However the details and the excellence of the same can be debated upon. Meux tool does not support any customization of the Mobile OS which is very common on all the Windows Mobile Devices by HTC. As a workaround many function calls have been provided. But I am pretty concerned with the re usability of scripts across same OS on different devices or different version of the OS. With my concentration mainly on browser testing, I wasn’t highly impressed with the same.

The tool does not support many of the native functions on the mobile browser which are found on the regular browser. The browser and page objects have very limited number of native inbuilt functioanilities and much is left to the discretion of the coder. For instance, the GetChildObjects method found on a regular browser is absent on the mobile browsers inbuilt functionalities. This pretty much defeats the purpose of descriptive programming. To verify the presence of certain web object, you would have to get an array collection of all the objects on the page and then parse through each of them to find what you are looking for. This is not the most effective way of working on a highly dynamic page with high number of objects. Also, the object spy does not work on the mobile browser.

The analysis of the tool can be summarized through the following SWOT analysis of the same.

Strengths :
• Works as a bridge in between QTP Automation Tool and Mobile Device
• Records and Plays back automation scripts on mobile devices

Weaknesses:
• Does not support many important native functionalities of the web browser
• Does not support any customizations on the OS
• Object Spy does not work

Opportunities:
• Can re run same script across multiple devices with the same OS
• Can replace manual QA

Threats:
• Supports limited versions of the OS
• No agent for iPhones as yet
• Can behave highly unexpected at times

Having tried this, for the time being I would still continue using our current options viz. DeviceAnywhere and Perfecto Mobile from Nexperience. Perfecto Mobile provides automation scripting across various devices and platforms with a high level of re usability. But having said that, I am going to keep a keen eye on the development progress of Meux with heavy interest considering its excellent potential.

Another thing that I would like point out here is the excellent support from the Meux team especially by Jacques Wouters. In spite of being on different sides of the Atlantic and having a disastrous time difference, the support team made sure they were available when we needed it.

VN:D [1.9.10_1130]
Rating: 8.7/10 (15 votes cast)

  

{ 1 comment }

Time Estimation to Test an Application !!!

Estimation is one of the vital factors for success of any project. Precise estimation and desired flow of execution of project will create good impression on client which eventually helps in generating repeat business. Sales department usually forces operation or project management for well defined and quick turnaround for an estimation. Many a times overestimates or delay in estimation costs loss of project and clients trust.

Managers play with few variables like requirements, time and resources (people, money) to decide estimation. Efficient estimators always consider unexpected changes with these variables while defining estimation. Extent of understanding of scope, domain/industry experience, strength and capabilities of teams involve in project are elements which usually made high impact on estimation.

Good estimation must include stuff which are not to be covered in project execution. Client will admire it and it might facilitate him to take any business/technology related decision. They always appreciate the company which will provide them accurate estimation and proper breakdown of the tasks/modules that need to be completed. It holds their trust that service provider properly understood the requirement and will develop and test the project in desired manner. Today’s most of the managers would like to go for Work Breakdown Structure (WBS). Estimate each and every activity, combine all, add some buffer and finalize the estimation.

Converting data into information is very important. When service provider received all requirements from client then they need to refine it in details as much as possible. It would be the best to give couple of days (depends on project size) to developers and testers to review and comment so all team will come on same platform of understanding. On proper understanding define the baseline that team should build throughout the releases. Define the indicators or responses that need to be estimated depending on factors. The baselines could be a simple guess for the first iteration that will be updated in the next iterations expecting some convergence and better accuracy.

Testing an application can also be measured based on:

1. Time to write a test (just the time to write the test procedure if manual and/or writing the test procedure and test script if automated). Time to write a test will initially be estimated using simple guess (where you apply some kind of factors). Just note somewhere your initial estimation and compare with the actual time it took. Do this for all the tests you’re going to write. It will give you refined metrics on the factor to apply to your initial estimations.

This should be split in different types of tests as your estimations will be more or less accurate depending on the category. In addition, if you have actual figures for each type of tests, you can easily apply similar figures for the next similar tests you still have to write. The next figures should be based on the past ones… experience counts!!

2. Time to run the test – It will initially be estimated using simple guess as well but if you’re using a good test management application, you’ll get the execution time automatically tracked by the system and some metrics computed for you (based on the last N executions of the same test). So the test management application should be able to give you in advance how much time it will take to run a complete test campaign – with very accurate numbers.

One can also identify the type of testing required as not all the projects required all kind of testing. Once categorize different types of testing like compatibility, performance, system integration etc. then identify high level scenarios and estimate those based on past experience. Add some buffer time to it and present the estimation.

In all approaches experience plays a vital role. Estimation might be different for same application if it will be carried out by different managers. Their experiences with similar project, understanding of domain and industry may vary which could end up in different estimations. Risk taking ability of managers also makes a difference in judgment. Research and learning about new concepts, confidence in team could bring optimistic estimation. It has always being the challenge to project management as how close their estimates are to actual.

References:

• http://bright-green.com/docs/howto_estimate.html
• Linkedin Software QA group discussions

VN:F [1.9.10_1130]
Rating: 6.0/10 (3 votes cast)

  

{ 0 comments }

A primer to Mobile Operating Systems

Mobile Operating systems and Application Platforms

 Mobile devices are evolving into increasingly sophisticated and general purpose computers and this has led to the development of a variety of platforms and operating systems in the mobile space. Today the operating system lying inside your mobile device is equally important as the device manufacturer (OEM). The various features, GUI, processing speed and most importantly the applications available for your device will depend on the underlying OS.

The jargons like Mobile operating system and Mobile application platforms give rise to doubts and I have tried in this article to address any such doubts. 

Mobile device manufacturers bring devices based on a variety of platforms and OS’s to cater to various user segments giving rise to a number of Mobile OS and Application platforms. A Mobile OS manages the hardware and software resources of a mobile device similar to a computer OS. Some OS platforms cover the entire range of the software stack while others may only include the lower levels (typically the kernel and middleware layers) and rely on additional software platforms to provide a user interface framework.

 Application development platforms, such as Brew, Symbian UIQ, Android, LIMO, ALP, Qtopia or WIPI provide programming resources for native and Java applications. Today’s phones are expected to run a growing range of software such as internet browsers, navigation packages, games and music/video players. Application platforms are supposed to provide a ground for them.  Application platforms are built over lower-level kernel operating systems (OS) such as Rex, Linux or compact real-time OS’s such as Nucleus. Software running on the device accesses platform resources through a set of application programming interfaces or APIs.

 To maximise the available market, a Mobile developers would typically develop for multiple platforms but this can be a difficult and time-consuming task as multiple platforms use different API calls for common OS operations and accessing OS resources such as accessing memory and files.

Mobile OS stack

  There is a variety of both, Mobile OS and the Mobile application platforms in mobile space with a few names enjoying most of the market share. Here is a  of major mobile operating systems and Application platforms. Examples of Mobile operating system with complete software stack are Windows Mobile and Android.

 Mobile (Kernel) Operating Systems

 Some Major Mobile Operating Systems (OS) used in smart phones are: 

Symbian OS from the Symbian Foundation

    Symbian has the largest share in most markets worldwide, but lags behind other companies in the relatively small but highly visible North American market. It has been used by many major handset manufacturers, including BenQ, Fujitsu, LG, Mitsubishi, Motorola, Nokia, Samsung, Sharp, and Sony Ericsson. Prior to 2009 Symbian supported multiple user interfaces, i.e. UIQ from UIQ Technologies, S60 from Nokia, and MOAP from NTT DOCOMO. As part of the formation of the Symbian platform in 2009 these three UIs were merged into a single platform which is now fully open source. It has received some adverse press attention due to virus threats (namely Trojan horses).

Symbian has a microkernel architecture, which means that the minimum necessary is within the kernel to maximize robustness, availability and responsiveness. It consists of  kernel and middleware components of the software stack. The upper layers are supplied by application platforms like S60, UIQ, and MOAP.It contains a scheduler, memory management and device drivers, but other services like networking, telephony and file system support are placed in the OS Services Layer or the Base Services Layer. The inclusion of device drivers means the kernel is not a true microkernel.

RIM Blackberry OS

    Blackberry OS is the proprietary software platform, created by Research In Motion, for their Black Berry phones. OS is focused on easy operation and was originally designed for business. The Blackberry operating system provides multitasking and supports specialized input devices that have been adopted by RIM for use in its handhelds, particularly the track wheel, trackball, and most recently, the track pad and touch screen. 

The Blackberry platform is perhaps best known for its native support for corporate email, through MIDP 1.0 and, more recently, a subset of MIDP 2.0, which and allows complete wireless activation and synchronization with Microsoft Exchange, Lotus Domino, or Novell GroupWise email, calendar, tasks, notes, and contacts, when used in conjunction with Blackberry Enterprise Server. The operating system also supports WAP 1.2. Recently it has seen a surge in third-party applications and has been improved to offer full multimedia support.

 iPhone OS from Apple Inc.

    The iPhone uses an operating system called iPhone OS, which is derived from Mac OS X. iPhone OS has four abstraction layers: the Core OS layer, the Core Services layer, the Media layer, and the Cocoa Touch layer. The operating system uses roughly 500 megabytes of the device’s storage. iPhone app store is the online market for downloading iPhone applications. Third party applications were not officially supported until the release of iPhone OS 2.0 on July 11th 2008.

 Windows Mobile from Microsoft

    Windows mobile is based on the Windows CE 5.2 kernel, and features a suite of basic applications developed using the Microsoft Windows API. It is designed to be somewhat similar to desktop versions of Windows, feature-wise and aesthetically. The current version is called “Windows Mobile 6.5″.  Third-party software development is available for this with several options for developers to use when deploying a mobile application. This includes writing native code with Visual C++, writing managed code that works with the .NET Compact Framework, or Server-side code that can be deployed using Internet Explorer Mobile or a mobile client on the user’s device.

    The two improved variants of this operating system, Windows Mobile 6 Professional (for touch screen devices) and Windows Mobile 6 Standard, were unveiled in February 2007. It has been criticized for having a user interface which is not optimized for touch input by fingers; instead, it is more usable with a stylus. However, unlike iPhone OS, it does support both touch screen and physical keyboard configurations. On February 15th, 2010 Microsoft unveiled its next-generation mobile OS, Windows Phone 7. The new mobile OS includes a completely new over-hauled UI called “Metro”. It includes full integration of Microsoft services such as Zune, Xbox Live and Bing.

 Linux operating system

    Linux is strongest in China where it is used by Motorola, and in Japan, used by DoCoMo. Rather than being a platform in its own right, Linux is used as a basis for a number of different platforms developed by several vendors, including Android, LiMo, Maemo, Openmoko and Qt Extended, which are mostly incompatible. PalmSource (now Access) is moving towards an interface running on Linux. Another platform based on Linux is being developed by Motorola, NEC, NTT DoCoMo, Panasonic, Samsung, and Vodafone.

 Palm webOS from Palm Inc. and Palm OS/Garnet OS from Access Co.

Palm webOS is Palm’s next generation operating system. PalmSource traditionally used its own platform developed by Palm Inc.  Underneath the graphical user interface sits a software stack that has much in common with mainstream Linux distributions for desktop computers. webOS uses, among other components, the Linux kernel, Upstart, GStreamer as multimedia framework, libpurple for instant messaging, Simple DirectMedia Layer (SDL) for game programming, PulseAudio for audio management, and WebKit.Applications for webOS are written in JavaScript, HTML, AJAX, and some other common ‘web’ technologies, which is handled by WebKit, as well as C and C++. It supports streaming video in RTSP, H.263, and H.264 formats.Palm publishes a webOS software development kit called Mojo which can be accessed from the launcher screen

 Bada from Samsung

The Bada operating system has a kernel configurable architecture, which allows the use of either the Linux kernel, or another Real-time operating system (RTOS) kernel. The Linux kernel is the preferred choice for smart phones, while RTOS is used for cheaper feature-phones, due to its smaller memory footprint.

It has a web browser based on the open-source WebKit, and also features Adobe Flash. Both WebKit and Flash can be embedded inside native Bada applications. Bada offers interactive mapping with Point of interest (POI) features, which can also be embedded inside native applications. Bada supports various sensors, such as motion sensing, vibration control, face detection, accelerometer, magnetometer, tilt, and GPS, which can be incorporated into applications. Native applications are developed in C++ with the Bada SDK, and the Eclipse based IDE. Bada can also run Java ME applications (MIDP profile).

The bada mobile phone operating system is still in development, and Samsung expects handsets to be available in the second half of 2010.The first device to run Bada is called ‘Wave’ and was unveiled to the public at Mobile World Congress 2010, Wave is a fully touch screen phone running the new mobile operating system.

MeeGo from Nokia and Intel

At the 2010 Mobile World Congress in Barcelona, Nokia and Intel both unveiled ‘MeeGo’ a brand new mobile operating system which would combine the best of Moblin and the best of Maemo to create a truly open-sourced experience for users across all devices.

Android from Google Inc.

    Android was developed by Google Inc. Android is an Open Source, Linux-derived platform backed by Google, along with major hardware and software developers (such as Intel, HTC, ARM, and eBay, to name a few), that form the Open Handset Alliance. This OS, though very new, already has a cult following among programmers eager to develop apps for its flexible, Open Source, back end. Android promises to give developers access to every aspect of the phone’s operation. This lends many to foresee the promise of further growth for the Android platform.

So this was a brief primer to Major Mobile operating systems available in market today. The next part in this series will cover the Mobile application development platforms in the market.

Anuj Gupta

VN:F [1.9.10_1130]
Rating: 9.3/10 (6 votes cast)

  

{ 0 comments }