The gray area between Installation Testing and Application Testing
Installer testing and application testing are two completely different subjects, even though they can seem very similar.
In this article, we'll discuss what we would refer to as "the gray area of testing" – a subject that is not always covered in training materials.
To get a better understanding of what I mean by "the gray area", you can review our Installer Testing Guide free ebook. There, we talk about the steps and best practices that you should follow to ensure your installer package works properly - after we’ve completed creating the installer package: MSI, EXE, or MSIX.
Windows Installer offers the possibility to debug MSI installer packages during the build process by using ICE Validators, which makes it much easier to catch any errors within the package prior to the testing phase.
Additionally, Advanced Installer integrates a set of best practice ICEs that check your MSI package for common errors that might appear during the creation and fix them automatically.
But first, let’s see what is the difference between the application testing and installation testing.
Application testing vs Installation testing - What is the difference?
What is application testing (software testing)?
Application testing refers to the functionality test of your application. In this scenario, all the possible settings, buttons and extra features that your application is offering should be tested to confirm 100% working condition.What is installation testing?
Installation testing refers to the package testing. The delivery technology does not matter (EXE, MSI, MSIX, APPX, APP-V), what matters is that your application package is installed successfully and performs all the desired system configurations. Of course, all possible scenarios like upgrade, maintenance and uninstall should be considered.
Why should application testing always go after installation testing?
You might ask me, Alex, if we already have all of these ICEs, best practices and an Installer testing guide, why are we still talking about application testing?
Well, here is where the fun begins.
A successful installation of a package does NOT guarantee the correct functionality of the application itself. Here are two examples:
- The application itself contains an error in the code (either in a DLL, EXE, etc) and it can't be fixed through packaging.
- The application is wrongly configured or the package lacks data that is required for the application to function as expected.
When it comes to the first scenario – developers only need to fix the code and push it through a patch or a new product version upgrade. However, an IT professional who runs into this scenario can only report it to the product vendors.
Now, let’s focus on the second scenario since many companies use automation in all areas, including application packaging.
However, many companies rely on the return codes that the installer hands out and consider the application good to go.
But, even if an installer has a successful return code (like 0), we shouldn't assume that everything will be working as expected since it could be wrongly configured or the package could be lacking data.
Why? Here are some use cases.
Why you should never skip application testing - real life scenarios
Per-user settings
Let’s consider this scenario: you have VLC Media Player as a repackaged application that you want to push into the infrastructure. You also have as a request some changes that you need to apply to VLC Media Player. For instance, let’s say you disabled updates and some codec settings.
As we previously discussed in the Packaging Series: Convert VLC Media Player to MSIX article, VLC Media Player settings are per-user, meaning some kind of self-healing action must be implemented in the package.
But, those types of actions are usually difficult to trace if they work correctly, meaning that the VLC might get installed but the settings won’t be populated for the users as expected.
This is what we call an “application test”, since somebody must actively open up VLC Media Player and check if all the required settings are there after the installation.
Additionally, each option and functionality that the application offers must be tested. In our VLC example, we should test if the videos are being displayed properly, if subtitles are working, if we can open a network stream or a capture device, etc.
Registry keys
A much better example in terms of functionality testing are the SAP GUI and other SAP GUI add-ons.
In some cases, only if a single registry key is not configured properly in the package, the application or the add-on will not work. And believe me when I say, there is no simple way to determine if you entered the right key during the package's development.
Missing dependency
Another scenario is where an application is installed, returns error code 0 but when launched it gives an error saying that a containing module cannot be opened.
That happens because the application is missing a prerequisite.
In this case, the installer is not properly configured to detect if the prerequisite is present on the machine during the setup wizard, so it continues with the installation anyway.
Conclusion
We often see that the “application testing” step is completely skipped, or users are not sure of what they're doing or what they need to test. These kinds of tests should be performed by a small group of users that are familiar with the use of the application.
Since the role of an IT pro is mainly to check for consistency and for all the data to be correctly placed inside the package, if he encounters an application error that cannot be fixed through packaging, the only solution is to get in touch with the product providers.
That's why, in an enterprise environment, it's better to have an acceptance test on a pilot group of users before performing a mass rollout.
Now that you have a better idea of the difference between application testing and package testing and their importance, you can start reducing the number of issues within your application.
Subscribe to Our Newsletter
Sign up for free and be the first to receive the latest news, videos, exclusive How-Tos, and guides from Advanced Installer.