Due to the fiercely competitive nature of the mobile app landscape, the need for teams to adopt practices such as CI/CD (Continuous Integration and Continuous Delivery) and automated testing in order to maintain productivity and efficiency is no longer a debate. The question now is whether you should host the mobile testing process yourself or use one of the many cloud-based mobile testing tools that have blossomed.
While many startups are embracing the convenience of cloud-based mobile testing, a general overview of on-premise vs. cloud-based mobile testing will help you choose an approach that’s right for your company’s needs.
In this article, we’ve broken down the pros and cons of on-premise vs. cloud-based mobile testing, so you can be confident about your next steps in choosing an approach for your team and project.
What is on-premise mobile testing?
On-premise (or self-hosted) mobile testing happens when an organization installs and accesses all mobile testing software from its own server or infrastructure. As a result, the organization is completely responsible for setting up the servers/infrastructure, as well as maintaining and scaling it, among other tasks, to ensure that everything remains operational.
A typical example of an on-premise mobile testing setup would be installing an automation server like Jenkins on a machine like a Mac mini. Then, when new code is pushed or a test deployment is requested, Jenkins creates a build and runs all the associated automated tests. These tests can either run on actual physical devices connected to the Mac mini or a virtual device emulator.
Pros of on-premise mobile testing
The major appeal of on-premise mobile testing is that it gives you complete control over the infrastructure.
- Full control & customizations: with an on-premise mobile testing setup, you have the ability to do whatever you want and however you want. Cloud-hosted mobile testing setups usually have preconfigured pipelines that can help you save time. However, they can be quite constraining when you start to walk off the beaten path. You’ll be forced to settle for the most appropriate package offered by the vendor, even if it doesn’t quite suit your needs. So, if your processes are specialized, an on-prem mobile testing setup may be a better choice.
- Better latency: with traditional infrastructure, an on-premise setup allows you to move data faster since it’s within your network. If your codebase was big or you produced large containers full of binaries, moving these bits across the open internet was slow. However, with today’s technology, cloud-based performance can be just as fast and reliable.
- Security: with on-premises, security is the responsibility of your team since you own the infrastructure and must be careful about every aspect related to security of the infrastructure, services, or tools. But with a cloud-based (hosted) solution, you need to do your due diligence when choosing a third-party vendor. Choose a vendor that offers 2FA, SSO, and a firewall and take security seriously. For some specific sectors with rigid compliance and regulatory requirements, running the tests on-premise may be a necessity to limit access to data.
Cons of on-premise mobile testing
Going the on-premise way means that you’ll have to set up, maintain, and secure everything yourself. So, before you go forward, consider whether your organization has an existing network and server infrastructure. If not, is there capital to purchase new infrastructure? Does the organization also have human resources personnel with the technical knowledge or resources to handle such tasks?
- Huge setup costs: setting up on-premise mobile testing is much more expensive in terms of both time and money. You’ll have to pay not only for hardware but also for electricity and other operational costs necessary to keep the infrastructure running. There are also human hours to consider. Instead of working on enhancing your business solution, your employees will be spending time customizing the setup to operate with your present configuration.
- High maintenance requirements: with an on-premise mobile testing setup, you’ll have to keep all hardware and software up to date. As your software’s requirements change, you’ll also need to update the configuration and purchase new infrastructure. That’s a lot of time that could have been spent further refining your actual product.
- Need for dedicated support: in the case where your developers cannot handle the added responsibility of managing the setup (either because they do not have the necessary knowledge or have a lot on their plates), you’ll need to hire or contract specialized system administrators or DevOps engineers. This further adds to the total cost of ownership.
- Your security is in your hands: with an on-premise setup, you are responsible for the whole system’s security. And, if your team does not have adequate knowledge and expertise, they may make mistakes that’ll end up as critical security issues.
What is cloud-based mobile testing?
In a cloud-based (or cloud-hosted) mobile testing setup, you’ll make use of mobile testing tools developed, maintained, and hosted by third-party organizations that give customers access to the tools over the internet for a fee.
A typical cloud-based mobile testing setup will comprise of mobile CI/CD tools like Bitrise in collaboration with a managed cloud device farm like Sauce Labs, BrowserStack, LambdaTest, Kobiton, AWS Device Farm or Firebase Test Lab and use a test automation framework like Appium, Espresso, XCUITest or Detox.
Pros of cloud-based mobile testing
Cloud-based setups are much easier to establish because the bulk of the tasks are handled by the third-party provider. All that is required from you is minimal configuration to make the tools suit your workflow needs.
- Value-add features: Cloud-based testing solutions provide several advanced built-in features and pre-configured pipelines essential for effective mobile testing that would normally take months to build in-house.
- Transparent cost of ownership: With a cloud-based mobile testing setup, the cost is as straightforward as a predefined subscription fee. For example, LambdaTest offers subscription plans starting at $15 per month.
- Automatic scalability & parallelization: When necessary, cloud-based platforms can scale up to split up work and run long processes in parallel in order to save time.
- No/low maintenance: Cloud-based providers handle everything from security, maintenance, outages, upgrades, and improvements in the solution. This allows your team to focus on optimizing its testing process and also improving your core product.
Cons of cloud-based mobile testing
- Provider lock-in: How much effort and cost do you need to expect in order to move should the need arise? Migrating from one mobile testing platform to another may be difficult due to differences across vendor platforms.
- Possibility of a data breach: With cloud-based services, you are reliant on the service provider to keep your pipelines and workflows secure. While cloud service providers adhere to the highest security standards and industry certifications, storing sensitive data and files with third-party providers does introduce some risk. Therefore, it’s important to pay strict attention to user access management and limit access to the least privilege when using cloud-based platforms.
Final verdict: which makes sense for you?
Deciding which option will be best for you is all about finding out what works for the peculiarities and reality of your project/team. If you are a small- to medium-sized team, definitely use cloud services. You’ll be able to save time and money while increasing agility and scalability. However, your business may require the use of an on-premise solution due to industry regulation, so ensure that you check compliance and regulatory requirements.
Still, the future of the internet is the cloud. The cloud is still widely regarded as the most convenient and cost-effective method of managing extensive infrastructure. For example, N26, the Germany-based international digital bank, uses the Bitrise cloud-based CI/CD despite the sensitiveness of its data.
So, even if your niche requires data to stay on your own server, you can adopt a hybrid approach (host your mobile testing solutions yourself on private cloud servers) to enjoy the best of both worlds. Bitrise offers both enterprise-private solutions as well as fully managed cloud-based CI/CD.
Use a cloud-based mobile testing tool with cloud CI/CD
At Bitrise we believe that test automation is a vital part of mobile CI/CD. Without it, we can’t release our mobile apps frequently with confidence. Because of that are providing more than 330 integration steps to help developers and test engineers to accelerate their development and testing life cycle including managed cloud device farms like Sauce Labs, BrowserStack, LambdaTest, Kobiton, AWS Device Farm, and Firebase Test Lab to use them with different test automation frameworks like Appium, Espresso, XCUITest or Detox.
If your team decides to use any of the above cloud devices, just in easy steps you can insert the integration step into your CI/CD pipeline and start the build.
Additionally, if your team decided to use codeless testing tools, our platform also supports different tools from different partners.
Sounds interesting for you and your team. Book a personalized demo today to discover how Bitrise can help you test more quickly and reliably and save your team thousands of hours each month.
Thank you and happy building!
Further reading
- It's time to move your CI/CD to the cloud
- 10 of the most popular mobile automation testing tools
- The 5 main benefits you can expect from migrating your app to cloud CI
- Lessons from engineers for migrating your mobile CI from Jenkins to Bitrise
- 19 automated mobile testing resources to take you from beginner to expert
- Android UI testing with AWS Device Farm and Bitrise
- The top four mobile app testing mistakes (and how to avoid them)
- React Native E2E UI testing with Detox and Bitrise
- Testing on the CI
%202.png)
