From the monthly archives:

May 2012

Part 1 – MSFT Getting Their Act Together in the App, Tablet and Smartphone Market – Is There a Strategy?

The app market according to Gartner is expected to reach $58 billion by 2014 and Microsoft wants a slice of this BIG pie. And with the announcement of the Windows touch tablet, the wheels continue to get oiled but it is still a steep hike to meet the Androids and the iOSs. Microsoft was sitting on the side-lines for quite some time when developers were pouring their efforts into iOS and Android to establish a healthy app ecosystem.

IDC forecasted less-than-stellar performance for their Windows 8 OS in 2012. Largely irrelevant for traditional PCs, and effectively no upgrade activity from Windows 7 to Windows 8 in that form factor.

So what’s their strategy to play in this market with the Windows 8 tablet and their Windows Store and their Windows Phone? How has their game evolved to compete with the titans?

Luring developers to build ‘metro’ style applications

Getting into the game slowly but surely, they reached out to developers to build their next best application for them. Especially since developers are reluctant to invest time and money in an app for an unsure marketplace, Microsoft found other tactics to entice them.  Incentives like giving away free phones and the promise of prime spots in the app store and Windows phone advertising were all part of that package. They also tried running some contests to get more app submissions to their Windows store.

They also presented some generous revenue sharing models for developers by taking smaller cuts than Google and Apple on successful apps. Although it would take 30% on all apps, it will reduce its commission on strong selling apps. The revenue share base is 70 percent, but when an app achieves $25k in revenue — aggregated across all sales in every market — that app moves to 80 percent revenue share for the lifetime of that app.

They promised a more transparent certification process with more guidance and feedback to developers on how they could monetize better. Their premise – it would help developers spend more time to write more innovative applications and less time to navigate through all the approval processes.

Has this helped them to out-market the Droids and the iOSs? The answer is no. Why not?

Relationship with Carriers

Despite spending millions in marketing, Windows phone still does not seem to be a popular choice. Microsoft believes that it’s got nothing to do with the technology; it is not being able to gain the support of their device manufacturers who haven’t push this hard enough as Android. Microsoft’s focus on end-user first did not really work in a world where Google tied up with carriers and device manufacturers closely.  Let’s dive a bit into this ecosystem.

Carriers own the customer, sales and the physical pipe. They also own the marketing money. Device manufacturers own the hardware, industrial design and not the customer, OS providers own the customer experience but since they don’t build the hardware, they are not directly connected to the customer.  Users own the disposable income and are highly influenced by advertising. Developers deliver most of the end user benefit and target platforms that show the most promise and definitely care about monetization

Apple successfully cut the device manufacturer our and forced the carriers to provide pipe and nothing more than that. Google was successful because it worked with carriers and device manufacturers to provide tons of choice to the user by making their platform open source and letting developers innovate in the end users best interest.

Windows however put a lot of emphasis on end user experience and did not care so much about the device or the carrier. But, it forgot that end users are very influenced by carriers who own the marketing money and spend billions of dollars to gain mindshare of their customers. So despite being a better product, Windows phone could not sell like hot cakes.

Will it’s partnership with Nokia help to get them ahead in this race? What should they have done better with their app strategy which Android and Apple did not do? Will Windows 8 help them put on a better show? Let’s find out in part 2 of this blog.

 

 

VN:F [1.9.10_1130]
Rating: 9.0/10 (1 vote cast)

  

{ 0 comments }

Talking Load and Performance Testing with Rob Thevenet

We chatted with Rob Thevenet our Sr. QA Manager about his experience and challenges with performance and load testing for enterprise software. Rob is a seasoned QA guru and has built automation and continuous frameworks to support performance automation and performance testing for HP, Virgin Mobile and Sun Run.  At InfoStretch he leads the automation and performance competencies, rolling out new projects all the time.

Interviewer – Rob, thanks for taking the time to talk to us today. We wanted to do a technical deep dive into some of the issues that you come across repeatedly during performance and load testing preparation and planning.

Rob – No problem, I will be happy to answer any questions you might have. Thanks for having me.

Q1. What are Enterprise, Load and Performance (L&P) test objectives?

Rob – Some or all of the below can be included as test objectives and each require their own type of approach, metrics and tool sets

  • Understand application or subsystem/component behavior under different operating conditions
  • Disaster Recovery (DR) preparation and/or Business continuity (BC) assurance
  • Model (mirror or extrapolate)
  • Implement (tune)
  • Benchmark (pre-test, pre-prod, etc.) specific subsystems or modules

 

Q2. What issues / constraints often need be overcome to setup, execute, measure, extend the test?

Rob – Technical gap or debt, environment contention and/or dependency

 

Q3. Which Non-Functional Requirements (*NFRs) and related conditions are in play for the test effort?

Rob – The following potential conditions apply to most if not all NFRs (most listed below) and serve as possible parameters for applying to common usage cases by particular user types including system and/or operational activities including database re-indexing, backups, replication, etc.:

(a)    Minimal usage (standup, idle, acquiesce then activate minimally)

(b)   Normal operation usage levels

(c)    Peak operation usage levels

(d)   Stress/Breakage usage levels

*List of NFRs – Availability, Business Continuity, Capacity, Configuration, Data Integrity (Data Currency, Data Integrity, Data Restoration, Data Retention

Disaster Recovery), Error Handling, Globalization, Internationalization, Localization, Logging, Longevity, Manageability, Messaging

Portability, Reliability, Scalability, Security, Testability, Usability, Volume

 

Q4. What are some Key Metrics to collect and what makes their analysis interesting,

Rob – Throughput, Latency, Transaction roundtrip time and Response times (for system, subsystems*, components); per SLA or Industry accepted thresholds**

*subsystems = Operating System, Application, Hardware, Software, Driver, Firmware, Memory, Disk I/O, Cache, Network, Processor, etc.

**SLA or industry accepted thresholds;

 

Q5. Which tools/methods can ensure a successful test effort?

Rob – Virtualization and Database snapshots help to provide s/w, h/w, f/w and state interchangeability, as well as decreased setup time enabling shorter Root Cause Analysis (RCA) cycles for issues found in test or production

-Full system infrastructure monitoring using Splunk, HP Mercury Sitescope, Topaz, IBM , Oracle Empirix, and/or Silk Central

-Unique Instance Report (UIR)

-Sorted Message type files (contain # of instances per source, message data, log entry of its containing log or event file)

 

Q6. What acceptance criteria are essential to a successful test with valid results?

Rob – Access to Application / System Under Test (AUT/SUT), REST, SOAP, etc. APIs to drive, check status, etc.

-Framework design and implementation supporting L&P test infrastructure client approval sign off

-Functional Stability

-NFR acceptance criteria defined (Agile, at least 1 NFR acceptance criteria per card)

-Implement an ongoing L&P test effort to reoccur monthly to quarterly in frequency supporting regular release cycle certification

 

Q7. What are protocols encountered in L&P testing and special conditions or methods to consider for handling?

Rob – Web(HTTP/HTTPS/HTML), LDAP, COM/DCOM, OBIEE, IEEE 802.11 (mobile standards) <OSI model>

 

Q8. What types of L&P issues manifest? How can failure analysis pinpoint issues?

Rob –  Database deadlocks, failed/stopped transactions, refused connections, HTTP level 200, 300, 400 or 500 messages, load generation agent script/code compilation errors.

 

VN:F [1.9.10_1130]
Rating: 0.0/10 (0 votes cast)

  

{ 0 comments }

Nasdaq’s Million Dollar Bug, or Was it a Billion?

Facebook’s record breaking trading volume and a delayed 30 minute start in the trading of their IPO stock. An incident that will be talked about for years as the largest debacle in the history of US trading possibly after the ‘flash crash’ in 2010 where a trillion dollars were wiped out. I am pretty sure that Facebook’s trading fiasco might have made rounds in your coffee house and water cooler discussions, and I am sure that it will be talked about for years to come with a lot of heartburn.

The impact that this had on the trading community was obviously pretty severe causing mayhem, decreased level of confidence in equity markets, and significant irreversible damage. Now this is what we call the ‘million dollar bug’ in the testing world. A bug that goes undiscovered, leading to a dysfunctional trading system, instilling a FUD (fear, uncertainty and disorder) in the minds of investors, and negatively impacting the credibility of large trading houses like NASDAQ. Not a pretty situation.

Granted that these bugs are difficult to discover as part of the standard operating procedure in the QA cycle and granted that the damage cannot be rolled back, once the bug is in action. But, can the bug be avoided? Not by spending a million dollars after the bug hits you, but possibly before? You can listen to our webcast on the million dollar bug syndrome, but I’ll summarize what we want to say. There are three routes to take with these kind of special bugs. Tactical – address the disruption in a timely manner, strategic – find the root cause and address the situation or preventive – make sure that the problem does not recur.

Most times, you might be unable to prevent these from occurring, primarily because it is driven by Murphy’s Law and a lot of other dependencies which are usually out of your control. BUT, you can minimize the impact of it by better preparing your QA, technology and business teams.

Interestingly, Rutesh, our CEO and founder who presents this session cites an example of an online brokerage firm which was hit by an unknown bug and halted all trading activity with losses of more than 60k trades in less than 30 minutes. And, the approach to deal with this is was what we call the war room approach to find the root cause and create a solid back-up system to make sure that all lost trades are executed at the customer’s discretion. Of course, it’s very important to arm your QA teams with the right ammunition to create solid QA processes, which we call out-of-the-box squad approach which are used to identify the deadly bugs and create enough readiness to fight them, as soon as they’re spotted. NASDAQ’s system failure was unavoidable probably, but some of the not-so-pleasant outcomes could have been taken care of.

VN:F [1.9.10_1130]
Rating: 4.3/10 (3 votes cast)

  

{ 0 comments }

Emulators – How Good?

IDC predicts that the volume of smartphones is estimated to reach over 500 million in shipments by 2014. These smartphones will run millions of applications embedded in them. A lot of these applications will communicate with complex ERP and CRM systems which will require rigorous security testing and verification. And, all these applications will be used on many OS’s which then will need a good look at their functionality and integration capabilities.  Unlike the PC based environment, the mobile world comprises of a zillion devices with diverse hardware and software configurations presenting a unique challenge in QA and designing testing strategies.

Testing is and will be quintessential for mobile applications and testing spend will exponentially grow along with the volume of smartphones – almost to about 20billion by 2015 per IDC. The first question that then comes to mind – do we need to test an application on all these devices or can we use emulators/simulators to simulate the real device experience. Diversity in this environment with multiple device types, browsers, hardware configurations and network related challenges might make actual device testing a little difficult, but some key attributes of applications have to be tested on real devices only. Emulators can only be used to test some device independent features.

Which route should you go? Well, it depends – on the type of application, network and firewall configurations, on the QA strategy, current and immediate needs, cost and resource availability. But let’s take a look at the comparison matrix to analyze the emulator vs. the real device experience to arrive at a more definitive conclusion

Emulators

Real devices
The application can be installed through apk file backup which is a time consuming process and it requires a real device The application can be directly installed in the device.
Real time applications (i.e. GPS based, server based, motion, multitouch or tilt control based) cannot be tested on emulators All types of application can be very accurately tested on a real device.
Emulator is slower than a real device. So, the app performance cannot be measured accurately and slower speed of emulators effects productivity A real device is always more accurate than an emulator
Communication with the emulated device may be blocked by a firewall program running on computer because the emulator can act like a normal application on workstation. There are no firewall restrictions on a real devices
There is no support for USB connections, SD card insert/eject, battery charge level, device-attached headphones etc. Real devices supports all of this without any restriction

To firmly establish this premise, we conducted an experiment by testing top 75 apps in the market on emulators as well as real devices. Here’s what we found, emulators are suitable for initial stages or only to test some device independent features. But to ensure the release of a zero defect application, real devices are important and a very critical part of the testing cycle.

 

VN:F [1.9.10_1130]
Rating: 8.7/10 (3 votes cast)

  

{ 0 comments }

Shifting Equations in the App Store World – Who Benefits?

Have you ever guessed, what makes the ‘Androids’ and the ‘iPhones’ sell like hot cakes? I am sure you’d all agree that it’s the “Apps” that makes the meat and let these phones thrive. It would be interesting to probe into the workings of this very dynamic market and understand the strategic, some smart, and some self-defeating moves that players made to survive, succeed and perish.

Shifts in the ‘balance of power’

Apple guarded its flag with the Mac App store, and Google followed suite with an open source model to disrupt this with Android. And, it rightfully did, until Jeff Bezos changed the equation and introduced the Amazon marketplace with the ‘free app of the day’ drive, slashing down prices significantly on some its high volume, exclusive, top performing apps

Did Google feel that its sales were cannibalized? Well no, to the contrary Eric Schmidt sticks to his open-source spirit when he says that the goal of stores is to make money for the developers and not for Google to make revenue.

Android became an OS of choice for many manufacturers to power their devices quickly and easily enter the market. Riding on the Android bandwagon, like many others AMZ powered its Kindle Fire and fringe benefits; its own appstore got super busy with its rising Kindle Fire sales (only next to Apple) because it didn’t come pre-loaded with Google play.

Now, AMZ makes its next move and lures developers to sell in their marketplace rather than Googles and it succeeds. Developers make a lot more there than in the Google Marketplace.  Next, it slashes down the prices of apps significantly to compete with Google, and making the end users happy.

Developers sway away from Google Play in search of greener pastures.

No doubt the developer community has benefited greatly by having a choice of rival appstores. This exponentially increased their exposure to more end users and also gave them a greater bite of the revenue. For example Amazon Appstores’ per user revenue is higher than that of Apple App Store and Google Play! Also Google Play doesn’t allow an application to be switched between free to paid and back (though you can sell two different versions of the same app)

Google checkout restricts developers from distributing apps in more than 9 countries. Developers bypass Google Play to sell in other more lucrative stores. No surprise there.
Google Play is a HUGE marketplace with a zillion apps living there. Clearly then, finding an app of choice is quite a challenge. This has spurted another ecosystem of app search stores like Androlib and Appbrain, which use intelligent algorithms to navigate the Google store.

The Flip side, there always is one.

AMZ appstore is new and is built as a reactive move to Google Play’s weaknesses. AMZ’s success is largely tied to the Kindle Fire sales. Soon, AMZ might be in Google’s not-so-good-looking shoes especially because of the lowering adoption rate of Kindle Fire. With a steep spike of 16% market share, the growth has tapered down to 4%, more so with an increase in the number of different types of tablets with better form factors and features, Fire gets a tough competition to survive.

And as the Android fragmentation continues, the appstore wars worsen further. Developers would not want to post a new flavor of their app for each Android device and confirm to the legal regulations. They end up spending more time marketing and selling their app than creating apps which defeats the entire purpose of Android’s open source model.

But in this entire conundrum, what we have to remember is that the end users are driving this market. And, it will continue to grow and explode as long as they are downloading these apps for their various needs. Developers will monetize on this no doubt and why not. After all, it is not them who set the rules, what have they got to lose?

VN:F [1.9.10_1130]
Rating: 5.0/10 (1 vote cast)

  

{ 0 comments }

Performance Testing for Mobile Apps – Is Your App Ready for the Worst?

Mobile apps are here to stay but the challenges to test these remain the same. We have already gone through the list of challenges when testing a mobile app – OS and device fragmentation, form factor variations, bandwidth issues, network speeds, recording user actions and load requirements which are varied in mobile. Testing web apps is equally challenging, since there are other set of factors like browser compatibility, form factors, geo location features etc. specific to the web that need to be considered.

No matter what the type of app is, native, hybrid or web based, it has to go through a very rigorous performance testing cycle to ensure that it can perform, under all conditions.

There are also some major performance application issues on the server and client side which need to be addressed. Key issue – mobile application testing for performance testing is NOT the same as traditional performance testing. Mobile is a whole new game and the testing requirements for all attributes are much steeper due to the unpredictability.

On the server side we need to understand variations in response times, streaming resource intensive packets, delays in delivery of messages, application crashes etc. On the client side – we need to address the usual discrepancy of application behavior on various platforms and handsets, memory and CPU consumption, loading speed and battery issues.

Let’s first clear some of the definitions that we often come across in the performance testing world. It’s important to understand the key differences of these terms. It’s mostly used interchangeably and considered to be same, but actually it’s not as their goals are different.

Testing Type Key Goals
Performance Testing · Establish a smooth, controlled, stable environment with a clear defined set of expectations (e.g. expected load, acceptable response time & throughput & record repeatable measurements· Eliminate required bottlenecks, while increasing load & establish a baseline for various load levels
Load Testing · Increase load constantly via automated tools to operate at predefined level & hence it’s part of the process of Performance testing [Incremental or Sequential]· Aim to break the software, but keep the software working at the highest load possible on peak hours

· Rectify bugs identified to ensure performance testing baseline is met
Stress Testing · Test the software under unpredictable scenarios like increasing load three times, sudden failure of DB connectivity etc. and try to crash the software or see how software reacts under unexpected failures/scenarios.

 

There are some tools in the market that can help you with the performance/load, and stress testing conundrum in the mobile and web based apps space.

  1. Recent announcement by MicroFocus about SilkPerformer 9.0 (2012)  supports Mobile Web & Native apps testing on iOS, Android & BB platforms
  2. HP LoadRunner supports mobile web & native apps testing on iOS, Android, BB & WM platforms
  3. NeoLoad supports mobile web & native mobile apps
  4. JMeter supports mobile web app testing on simulators
  5. IBM Rational Performance Tester (RPT) supports mobile web app testing as well

But no matter what your mobile app is like, you need a strong testing team behind you to first plan the detailed diagnostics and determine the capacity. Then, bring the know-how to run the tests by selecting the right load generation and monitoring tools to make sure that your app can handle the worst, at all times.

 

VN:D [1.9.10_1130]
Rating: 6.7/10 (6 votes cast)

  

{ 0 comments }

HTML5 – The New Paradigm to Redefine Mobility Strategy

The mobile applications market is moving at the speed of light. While smartphones created huge ripples in form of many players and platforms to alter the ecosystem, tablets are now entering the space to alter the mobility game even more. No matter which you space you play in – enterprise or consumer, mobility with native, hybrid, and/or web based apps has to be a part of your survival strategy.

Gartner predicts that in the next two years, 60% of the enterprise mobile applications and 40% of consumer mobile applications will be web applications. HTML5 is an emerging standard which has the potential to build rich web applications using CSS3 and Javascript and is predicted to take over a large piece of this pie. Some big enterprises are already ahead on the adoption curve and some are already rethinking their mobility strategy to integrate new technologies. This new paradigm brings significant advantages by lowering the time and cost for production, providing ease to work across multiple platforms, and making it very simple to port applications to native device code.

But there’s never a free lunch as they say. Designing web apps in HTML5 does require an in-depth UI/UX  look at some of the key usability attributes like browser compatibilities, screen orientations, forms factors, frameworks etc.  There are some ‘browser and animation tradeoffs’ that need to be made and some unique challenges taken care of when adopting HTML5. But we at InfoStretch are convinced that if you have a good UX team in place, most of these issues can be worked with.

Join Ray Matsil, our UI/UX designer tomorrow as he uncovers some of these challenges and his experiences in working around those to create sleek and stylish web apps in HTML5.  This will be your chance to get all your questions around designing HTML5 based apps answered and also understand some of the advantages this platform provides over native solutions.

Register today, and if for some reason you’re unable to attend, we will be recording this session and it will be available as an on-demand webinar.

Visit - http://www.infostretch.com/Resources/webinars.php

VN:F [1.9.10_1130]
Rating: 7.0/10 (2 votes cast)

  

{ 0 comments }