Applying The Principles of QA Automation to OTT Apps

What is Unit Testing?

Unit testing is a methodology which requires diligence and a relentless approach to testing software. In this article, we’re going to discuss our approach to QA testing across a vast array of TV applications.

Unit testing is a component of Test-Driven Development (TDD). Unit tests are critical as it ensures that smaller units of software are free of defects before they are reintroduced back into the main branch. The creation of an exhaustive list of test cases is vital to investigate the threshold of any software unit. Unfortunately, the hours required to perform such testing has increased dramatically over the past few years as the sheer number of TV devices is staggering.

Float Left Interactive builds TV apps for a vast array of set-top boxes, SmartTVs, and mobile platforms. These devices include but are not limited to Roku, Samsung Smart TV, Apple TV and Android TV. Each one of these platforms has a series of models and chipsets that we are required to support when deploying a new app. On average, we create roughly 300–400 test cases per device model.

So Many Tests; So Little Time

3 platforms * 3 models * 300 test cases = 2700 test cases per app that would need to be tested and verified before sending along for certification and publishing! Also, device manufacturers often have several firmware versions in the market, and our apps much perform correctly on each one of them. Additional firmware doubles and sometimes triples the number of hours required to test properly.

Automation of QA

By now you can imagine that the hours required to support the testing is difficult to manage. For this reason, Float Left has developed a comprehensive solution for reducing the human-factors of QA testing.

Aside from the obvious benefits of testing automation, is that faults in software will sometimes remain unseen during an initial test. Automation allows our us to perform the same test multiple times to ensure the unit is functioning as expected. It is incredibly tedious with a human performing this task.

Unit Testing for TV applications is much more challenging as compared to testing for a website or mobile application as the engineer can inspect each element and create a script accordingly. Automation scripts for TV applications offer little or no feedback as it is digital and dynamic for each channel. At Float Left, we have made it possible to create an automated unit test for almost all test cases required for TV applications.

Advertising, channel behavior, channel graphics, UI, performance, search, video, and audio functions are all converted into test cases regardless the device. We take care of each of the above aspects in detail before any channel is live to our client’s viewers. It also illustrates how our QA team investigates and decides to pass or fail a particular test case. We describe each test suite and their components in detail below:

1. Advertising: One of the primary sources of funding for any channel is advertising. Advertisements can be bifurcated depending upon the location. We have the potential to test each ad for any location. Ads are inserted either client-side or server-side during playback. Running automated tests for proper ad insertion has proven to be a challenge. Regardless, we always verify the test cases related to advertisements.
2. Channel Behavior: It is imperative for all TV apps to run properly on any device model, including slow network or slower devices. There could be numerous apps getting published, having an update, demanding a necessary hardware to run, software update or combination of items listed. Any modification of the device should not affect the functioning of the app. We test rigorously on SD, HD, FHD, 4K. In order words, our QA team is capable of testing each app any mode, any device or any firmware. With automation, we test all of these modes simultaneously. The amount of time saved coupled with the accuracy of the results are highly effective.

Authentication is also another area we spend a lot of time trying to get accurate results. Defects or faults during these critical processes could be catastrophic for our clients. It could also create legal issues surrounding entitlement.

1. Channel UI and Graphics: Graphics are one of the most critical components of any channel, device or website. Graphics, UI and modern look of the app or device attracts an audience. Testing becomes challenging as there are a lot of test cases surrounding user-interface. User focus, animation, duplication, fonts style, size, alignment and various other attributes have to be taken care before the channel is live to an audience.

Automation testing is performed by navigating to each element and saving the screenshot for each test. The user interface or graphics differ from one device to the next or other resolution types. Linking or device activation, verifying the focus, episode picker, poster layout, and show detail screens are some of the examples related to the user interface that is tested manually and by automation. Our QA team pays close attention to these critical areas since it is directly reflective to what a viewer could see within the application.

1. Video and Audio: It is imperative for any app that is delivering audio and video to a viewer that they two be synchronized irrespective of the device or speed of the internet. Using automation, we can monitor the bit rates and how the video or app reacts to it. Audio and video synchronization issues are a common failure point within TV apps. Tests need to pass irrespective of the internet service provider or the speed of the connection. We can identify the location of potential issues through our automation by the use of screenshots, which are taken when problems arise – in this example when audio and video become out of sync. Before delivery of an app to either the client or platform certification, we ensure every aspect of the app is verified and working properly.
2. System Compatibility and Localization: All the test cases designed for a TV application are expected to pass through various devices. At Float Left, we support a vast array of diverse platforms and devices. Each device may have different sizes and resolutions. For example, we might test a Roku device on a 10” TV to 100” TV. We automate or manual test each of the above devices and check system capability. Each of the device goes into regression testing to check its compatibility.

Localization testing is necessary as there might be some text or poster which might have missed to translate into required language. Also, depending on the location of the ISP, some contents gets filtered. By automation, we can have screenshots of each screen and can study later while the regression is testing can run in parallel. The allows us to regression test and verify localization simultaneously.

1. Deep links and search functionality: Deep linking allows manufacturers to advertise apps in various ways throughout the various app stores. In some cases, specific media files can be loaded immediately upon app launch. Using our automation scripts, we can also verify that deep-linking is working properly. Our clients provide us with specific pieces of metadata for the items that they want to showcase. We believe we are the first company that has automated this process in such a way as to ensure proper validation of all deep linking requirements with all manufacturers.

In many cases, TV apps have a need for searching across vast libraries of content. With this in mind, it’s vital for us to test if a particular search string will retrieve the proper set of results. We can test search using our automation scripts. We send the search function pre-defined characters and verify the results. Various factors such as selecting only available content, or navigating it to the correct screen if a viewer selects an item from the results should NOT crash the app.

In Summary

At Float Left, we are committed to creating innovative ways to help brands launch TV services in the most efficient way possible. We are always striving to discover new ways to reduce cost and increase the quality of our products and services.