Software testing is one of the most important activities in the life cycle of any software, since with it we can detect easily if the software works or if it doesn’t. Software Testing is a process in which you perform evaluations of the software and the system to check if it follows the requirements, works as expected and is bug-free.
Types of Software Testing
To understand the way software testing works you need to know that there are many different types of testing and each one with its own subtypes. In this blog we will mainly talk about the software testing approaches and the software testing levels.
Software Testing Approaches
White Box Testing.- This type of testing is done by via looking at the structure of the code, you test the internal part of it, designing test cases for it.
Black Box Testing.– This type of testing is done mainly by looking at the inputs and outputs of the system or the program. You design test cases which only look at the input and the output, and see if the expected output matches the real output of the program.
Grey Box Testing.- In this type of testing you perform a mixture of Black box and a mixture of White Box testing, you choose tests that align with what you want out of the program.
Software Testing Levels
Unit Testing.- This level of testing is done via testing each module of the program as separate units, ensuring that each function and every unit of the application works correctly as expected by its own.
Integration Testing.- In this type of testing you check if the connections or transfer of data within the units works as expected.
System Testing.- In this type, you check the system as a whole, ensuring that the whole system works as is expected when all the units work together as one.
Acceptance Testing.- In this type of testing you mainly check if the system follows the requirements of the client. Checking if it has met the criteria for delivery.
You could also divide software testing into functional and non functional, where in one you test the functional requirements of the program and in the other you test the non functional ones.
Roles of Software Testing
QA Manager.- The one who manages the whole team related to testing, ensuring they are all working as a team and following the guidelines of the company and of the client.
QA Analyst.- The one whose main job is analyzing the results of the tests, looking for where the bugs are, what is happening, and the one who also designs the tests.
Tester.- The person whose in charge of testing the system using the test cases.
Other jobs.- Sometimes in software testing you get other roles like the one in charge of the automation of the tests or the one who tests the performance of the system.
Test Case Design
Since Test Cases are the base of testing, you need to ensure that they are good and are well designed. To do that there are different techniques which you can use to design them. These techniques are defined into 3 categories:
Designing test cases using the requirements specification and black box testing.- Examples: Use Case Testing, Decision Table Testing.
Designing test cases from the structure of the system or its components. Examples: Branch coverage, path coverage.
Designing test cases based on the tester’s experience or intuition. Examples: Error guessing, Exploratory Testing.
One of the main ways to test is using a testing environment, which is an environment designed solely for testing before launching to the live server. It allows to deploy application and ensure it is bug free before pushing to the product.
Software testing process Types and levels of testing Activities and roles in testing Testing environments Test Case Design techniques (open and closed views) Process for control and management of defects in artifacts
Software testing has been described previously, but only by name, and maybe the stages at which it should be applied; however, this post will be dedicated to it (as if it wasn’t implied by the title). Software Testing can be defined as:
“Process to check if software meets requirements and its devoid of defects by executing software components and/or using automated tools”
Software testing, is great to make a product cost-effective, secure, of quality, and to assure that meets the customers expectations.
Software Testing Process
The stages of process testing can be diffetent for each project, but it can be broke down into 5 basic ones:
Planning: the scope of the test is determined, impact analysis is performed, all resources are listed, and dates for testing are planned (of course they are estimates).
Creation of Scenarios: expected outputs are prepared. And test scenarios are created.
Preparation of Test Environment and Data Creation: all preparations are made in order to start the test. Basically, setting all up.
Run the Scenarios: the created test scenarios are run.
Reporting the Results: Pretty self-explanatory, check the results and compare them to the expected output, to design the next test cycle.
I should be noted that this is a process that should be applied on all stages of the Software Development Life Cycle (SDLC).
Types of Testing
There are three main categories:
Functional Testing: testing the components that are essential for the functional requirements. This includes Unit testing, Integration testing, User acceptance testing, among others.
Non-Functional Testing: testing the components that are essential for the non-functional requirements. This includes Performance testing, Endurance testing, Load testing and so on.
Maintenance Testing: testing the product in order to keep it functional or improve it. This includes Regreesion testing and Maintenance testing.
Lists are always welcomed, so of course the different levels of testing are going to be listed. But before it should be noted that this is not a strict set of levels that should always be followed, this is more of a list of most relevant levels that provide a comple testing process:
Unit Testing (Level 1): Testing by small blocks of code. For this to work these blocks should be able to be tested separately, and in a considerable small amount of time.
Integration Testing (Level 2): Test groups of blocks of code to check if their functionality as a whole meets the requirements.
System Testing (Level 3): Test the system as a whole.
Acceptance Testing (Level 4): Test user scenarios, to confirm that the user accepts the product.
Activities and Roles in Testing
There is a designated testing team, or at least it should be, that is divided in different roles, the two main ones are, the test lead and the testers. But here, 3 roles will be given:
Software Tester Role: designs test case scenarios, conducts them, analyses the results and reports the observations. Interacts with clients to better understand the product, and do modifications, and also creates the documentation. Ensures that all tests are carried out accordingly to standards and procedures. There are many different roles for a software tester, thus, why so many responsabilities were listed.
Software Test Manager Role: represent the team, interact with customers, recruit software test staff, supervise all testing activities, schedule testing activities, select test tools, improve test processes with the help of metrics, and check the quality of the requirements.
Software Test Automator Role: design procedures and test cases for automated software testing, design reusable automated scripts, and ensure that all automated processes are done by the defined standards.
A test environment enables the creation of identical environments for testing, in order to control the most variables in the testing stage. A controlled test enviroment can provide useful feedback to the behavior of the application. There are some key elements that should be completed to create the test environments:
Implementation of test beds (data created and inserted to the environment)
Setup of the database
Configure the environment
Select hardware and software to run the application
Configure the network
Document all actions
There is a difference between test environment and staging environment, since stanging environment is the exact replica of the production level of the application, so is used to check if it is ready for deployment. Basically, a test environment is used to test specific parts of the producto while staging environment is for the product as a whole.
To manage the test environment there is also a checklist
A repository with all the updated versions of the environments
Create new environments as required
Monitoring the environment
Updating/Deleting outdated environments
Check for errors in the environments
Coordinate to solve all the errors in the environments
Test Case Design Techniques
There are three main types of design techniques:
Specification-Based Technique: a black box technique. These tests are done to know how the system behaves. This is more about creating test cases based on the output.
Structure-Based Technique: a white box technique. Tests that are done to have a level of coverage of control flows. Done by code analysis.
Experience-Based Technique: this is entirely based on the tester’s expertise. The tester designs tests based on previous experiences, guesses, and different known techniques.
From these 3 types, many techniques can be chosen, such as, error guessing, boundary value analysis, equivalence class partitioning, decision table based testing, state transition, and many others.
Defects, or bugs, are very relevant to software testing, since one of the main objectives of software testing is to reduce them. The defect management process goes as:
Discovery: the team tries to discover as many bugs as possible.
Categorization: all the bugs are classified, in order to decide the priorities of the team. A critical error should be fixed immediately. A high level error is the next priority, since it affects the main features. A medium level error causes damage that is not urgent, but no insignificant. Finally, a low level bug is very minor.
Defect resolution: a process to fix the bugs, divided in four stages:
Assignment: assing the bug to a developer to solve it
Schedule fixing: schedule by when it should be fixed
Fixing: fix the bug, while tracking the progress
Report resolution: report what the solution was.
Verification: verify that the bug is resolved.
Closure: Change bug status to close.
During all process, the bug should reported to the management team for feedback. Now, to measure the quality test execution, there are to parameters: Defect Rejection Ratio (Number of defects rejected over total of defects raised) and Defect Leakage Ratio (Number of defects missed over total defects of software). The value of these measures is better the smaller it is.
Verification and validation has existed for a while now, this is good for us because people have created many tools to make our jobs way easier. Testing nowadays is much easier and less exhausting than before and I will tell you of some of the most popular tools used for verification and validation. These tools each have their purpose and beware, they are not the only options out there, I will simply talk about the ones I’m most familiar with.
First, I would like to talk about git. Git a tool used for version control. If you are unfamiliar with this term, version control is a system used to record changes to a file or set of files over time. Git is probably the most popular tool used in V&V because of its usefulness when doing any kind of project. If you ever need to go back to some specific version of your files, you can. Let’s say you accidentally mess up something in your code by accident, with git you can simply go back to the last version where your code worked and not lose all your progress. It also allows people to work simultaneously on one project, letting you put everything together once the separate tasks are done. And if any conflicts are detected, it helps you fix them to ensure that the project works properly. There are other tools that can be used for version control but I would be lying if I said I’ve used them. Some of the most popular ones I found where Mercurial, CVS, and SVN, among others. From my experience, this is an essential tool and taking the time to learn how to use it properly is extremely worth the time, as it will make your life easier and your projects, in a way, quicker to manage.
From what I understand, it is harder to talk about only one testing tool since there are many out there used for very different purposes. We have test management tools, test execution tools, test data preparation tools, and the list goes on. In your projects, you will be required to use several different tools depending on what you want to accomplish. In most cases you are responsible for creating the test cases, these tools are meant to help you have a controlled place with all your tests on a particular part of the project and they can also help you automatize your tests. This is especially useful, it saves you time and once the test are done you know your code will properly do its job, that is if the tests were written correctly of course. Some testing tools can also be used while you are writing your code, rather than when you’re done. A very popular tool and one that I’ve used before is Postman. Postman helps you test your API to make sure that your endpoints are working properly. As I stated before, you must identify the parts of your project that need to be tested before you can decide which tools are required. Fortunately for us, there are tools for any kind of process imaginable when developing software, so it should only be a matter of investigating which appeals the most to your needs.
The last kind of tools I wanted to talk about were tools for process administration of V&V. The only problem is that I’ve never really had much experience with any tools of this sort, at least not that I’m aware of. What I know is that these tools, unlike the other tools that focus on a specific part of your projects functionality, are meant to help you in the process of validating and verifying your project.
Whatever your project may be and its purpose, you will always benefit from using tools to help you verify and validate its functionality. No matter which technologies you use, you should always look for a way in which you can prove that what you did is working as intended, otherwise there might be more problems in the future. Releasing code that you know works, even if it takes more time, is always better than releasing something that you didn’t fully test and having to fix it later.
The software testing process, although not exactly the same for every methodology, can be generalized to contain the main and most important points that help us have an effective testing experience.
First we need to define the test strategy and test plan. Some of the things you can use to describe your project’s testing scope are:
Systems that need to be tested
Focused features and functions of the project
And many more
With no strategy and plan, every project will find it difficult to be productive. This will help you define your entry and exit criteria for testing, which are important as a control for your team. If requirements lack specificity, the will not enter testing. If tested code doesn’t meet the specific quality standards, the code will never move onto the next phase.
Second, you need the test design. In here you will have a collection of test cases necessary to validate the system. Test design relies on test experience, testers’ knowledge of the system being tested and prevailing testing practices. It is focused on uncovering and fixing major bugs, rather than any other superficial aspects or flaws.
The third phase is test execution. There are many ways in which you can execute tests, for this step you will have to adapt your testing to the adequate amount of testing until you are sure that your system is fully functional and mostly bug-free. Understanding your test environment requirements is key to be able to decide your testing strategy.
Finally, we have the test closure. Here is where your exit criteria comes into play because it signals the completion of the test cycle and readiness for your product’s release. To ensure test closure, all your requirements must be met, both technical and business. You need to have passed a minimum percentage pass rate, best practice is targeting 90% of your tests passed. Every critical defect must be fixed. These steps are just a general view of how your testing process should look like.
There are four levels of software testing that you must take into account. These are:
Unit testing: Program is submitted to assessments that focus on specific components of the software to see if they are fully functional. The purpose of this round of testing is to determine whether the software works as designed. A unit could refer to a function, an individual program or a procedure.
Integration testing: This allows individuals the opportunity to combine every unit in a program and test them together. It is designed to find interface defects between every module or function.
System testing: The system testing level is the first in which the application is tested as a whole. Its goal is to evaluate whether the system has complied with all of the requirements and to see if it meets the quality standards.
Acceptance testing: This is the final level of testing and is used to determine whether a system is ready for its release or not. Requirements may change during the development cycle so you need to test in order to see if the process meets the business’ needs. After this level is passed, the software can be delivered to production.
There are several roles and activities to be done in testing, each of which have certain responsibilities that need to be done. The main roles are:
Test manager: The test manager is usually hired when there are many testing groups. Some of the major roles of a test manager are: preparing test strategy, preparing the test budget, defining the levels and test cycles, developing the strategy for test documentation, metrics and reporting, and many more.
Test leader: Every testing group must be led by a test leader. The test leader performs the roles of test manager when they are absent and have their own responsibilities including: preparing the test plan at each test level, defining the objectives, assigning roles and providing schedules to testers, etc.
Testers: Testers group can have different levels of testers and roles like performance testers, automation testers, etc. Some of their responsibilities are: gathering the test requirements, set up and verify the test environment, automate the tests, and many more.
If you would like to see a more extensive list of the activities and responsibilities that each has, check the external links below.
Something we need to address too are testing environments. Testing environments are a setup of software and hardware in which the testing teams execute test cases. They support test execution with hardware, software, and a network configured. Test environments are configured as needed by the application under test. It is important to have them properly setup in order to ensure that the software is being tested successfully. There are many things to be taken into account when setting up your test environment and in order to avoid this blog becoming too long, I will leave a link below explaining what you must do in order to assure that your process environment is adequate.
Another important point you need to understand is software testing techniques. Their purpose is to help you design better test cases. They help reduce the number of test cases to be executed while increasing test coverage and helping you identify test conditions. Some of the most important testing techniques are:
Boundary Value Analysis (BVA)
Equivalence class partitioning
Decision table based testing
As you can see, software testing is an extensive topic that requires research in several different branches in order for you to successfully do your testing. It is important for you to at least have a general idea of what their purposes are and to follow them if you want your test process to be much more effective and fluent.
In this blog post I’d like to write about an experience I had at work.
As I mentioned in another post, we recently underwent an agile transformation at my workplace. We went from something resembling a waterfall model to an agile one. I don’t know if it’s industry standard, but this meant that the QA team and the dev team became one. This caused some problems, chief among them that the dev team did not have experience with the testing tools of the QA team. One of this tools is called ReadyAPI.
I was working on fixing a defect that was sized as a 3, and one of the requirements of the definision of done was to test that the defect was actually solved. I won’t get into the details of my task, but the problem was that because of the specifics of my task, it was impossible to perform automated testing, and I did not have the slightest idea on how I could test it appropriately.
I got in touch with a coworker that was part of the QA team and asked him for help. Long story short he was able to write a test specific to my case, and we had a chat about it. He told me that he was annoyed that the product owner had sized the defect as a 3. I mentioned to him that the coding involved on my part was akin to a 3 point story, but then he mentioned that if the coding was a 3, there was no more room to include the testing effort. He also walked me through the test he wrote and told me that it took quite a bit of effort. I certainly would not have been able to write this in any reasonable amout of time, and would have changed the defect into a 13 if I had to do it by myself.
Thanks to this little interaction I saw that the testing part of the stories we take is not taken into account as much as the development. I think this would not be an issue if reliability in our product wasn’t as important as new features, but I would argue that because of the nature of the product, it is more important than the new features. I don’t see why the leadership team would leave so little time to testing, but demand intricate graphs and test coverage before a release.
Recently at my work we have been going through an agile transformation, and one of the goals was to lose the development team and QA team, and just have a team that did all of the development and QA. Thanks to this I have learned more about testing methodologies and tools. I have been taking a look at all of the tests that our software goes through and its incredible. This agile transformation made me realize that the old way of designing software has a lot of issues. I was talking with the person who used to be the head of QA and he was talking about how much redundancy and useless test cases we had because dev and QA were separated. Us in the dev team had written unit tests that the QA team had no knowledge of, so they wrote manual and automated test cases that valildated the same thing.
We were also talking about what tools we used to do the testing. We currently use a suite called ReadyAPI which is extremely expensive, and alot of the team dislikes it, both on the QA and the dev sides. There have been talks about taking the time to develop our own testing software using Python or JS but mangement does not want this, since they see it as not necesary. I hope we can one day get this project greenlit since I think working on a testing suite would be a very valuable experience.
Both concepts are crucial to mantain good software quality that complies with the standards we talked about before. But we have to understand what each concept means, and what are their differences before we dive deep into them. Let’s see their formal definitions according to the IEEE-STD-610:
Verification : “A test of a system to prove that it meets all its specified requirements at a particular stage of its development”.
Validation: “An activity that ensures that an end product stakeholder’s true needs and expectations are met.
With the above definitions, we can see that Verification is more about making sure we are building the software correctly with good practices, standards, and following a good methodology that allows it to meet the goals set at the start of the development cycle. Meanwhile Validation is more about making sure that the product being built is one that benefits the client, a product that ensures that the client’s needs are being met, because we may have a good product that works, but it may not solve the problems of our client.
When do we implement it into our SDLC?
The verification and validation process is one that should be present in each stage of the SDLC, we can’t allow errors, bugs, miscommunication to keep piling on until the end, because we may even find we have too many which may make the software completely useless since it would become much more expensive to fix. That’s why Verification and Validation must be done at each stage, to find errors on the planning phase, on the design phase, on the coding phase, on each process that the team does, to ensure that the errors never become too costly or difficult to fix, and also ensuring that the software being built is adequate for our client, that it is what our client needs.
Standards for V&V
The main standard used in the industry is the IEEE 1012-2016 which incorporates both verification and validation as a rigorous process carried since the beginning until the end.
The standard defines and talks about all the tests that should be done to comply with the standard, talking about tests like: System Integration, Acceptance Testing, Software Integration Testing, Component Testing and more.
The standard also talks about continuous review, how every test, every test result and every process done on V&V must be reviewed by the whole team working on the software to ensure of its quality and make recommendations for the next phase so the errors, or improvements that could be made are done. It should have a constant appeareance on each stage of the SDLC.
Just as for the SDLC, you have to start with planning, make a good planning of the Verification and Validation process, one that covers fully the objectives, needs of the client and requirements of the software.
The IEEE-1012 standard suggests making a document stating the objectives that should be met by the Verification and Validation, what products are being covered, who is involved and on which role, the activites, processes, tasks and tests that will be done to verificate and validate and how will the report of results will be done. The main part of the document should be about the planning of tests and their design and the analysis that will be performed based on those tests.
As we see, the most important part are the tests for the V&V, they need to be built according to the goals of the software, but according to the IEEE standard, they should have the following types of test almost by default:
Software Component Tests
Software Integration Tests
Software Qualification Tests
Software Acceptance Tests
All the above tests are required to test and analyze almost everything about the software, and with those we can be almost completely sure that we will achieve our goals. Each test is in the same level of importance, and some are even preventive to detect certain risks or menaces that although not present on the software on that moment, they may arise later.
After the design, testing and analysis is finished, a good report must be made, which will be reviewed by the whole software development team. The report should present documentation about how the tests were performed, the results observed from the tests and conclusions and recommendations that can be made from their results. This process is very important, tests aren’t useful if you don’t learn anything from them! The reviewing should reveal and offer good insights for the whole team to implement them and improve the software on the next stages of it. They should cause change towards a better direction.
Remember Plan -> Design -> Test -> Review -> Success!
Types of Tests
Unit tests only test a single part of an implementation. Basically a unit, theres no dependencies or integrations involved.
Integration tests concern about the functionality of the code with another third party, this could be a database, a server, another module, etc.
Functional test makes sure the system works as the user expects it to behave.
Frequency of Tests
The smaller and more isolated the tests the more frequent they become because of their simplicity, so functional tests are written just a few times.
Test will help build a software which is less prone to defects latter in the future and more well structured.
What I learnt using jest
I used an already built simple project with jest with the configuration being done, what my objective was is learning the basics structure of the jests test and how it is applied in a project.
First I made the simplest test ever to just see how a test is written, it was a simple arrow function which returns Hello as a string. All I need to do with the function is export to later imported in a test file.
Later imported the function in a test file which expects the output to be Hello, it shows that it ran correctly, the good thing about jest is the feedback given by the framework and every time a change is made and save the test run automatically.
The test has a describe to grab the function being imported, the it first parameter gives a description of what the test should do, and the second is a functions with the expect which is the test case that will run. We expect hello() to have “Hello” as an output.
To give another example which is more likely to happen when coding in a project I used a simple technique call TDD (Test Driven Development) where tests are written first and the code is done after to make it comply with the tests being made. This makes it easier to make the code reliable for the project, not just making it work for some cases, but all cases possible. The function will have to remove all names that begin with a letter s.
I wrote three kind of tests, one that should see if all name with S are removed, one that keeps all names without a letter S as a first letter, and one that takes attention to the lower and upper case letters. With that done I can make the function. Each expect has a matcher toContain or a negated toContain.
As an example I omitted the removal of upperCase letters. Just as an example of how jest can tell me what went wrong and help me figure out what the code needs to have.
In the third case I see that “scott” passed and “Scott” did not pass, which I can tell that I’m not accounting for upper case letters. I can also see the expected array given by the method and the result. By adding a toLowerCase function to the name array the problem is fixed. Another useful case for TDD is if I made my function to have a for loop instead of a filter I can later try to optimize the code and see how well its made by the feedback of the tests cases.
By making tests code can be thought out more, make functions smaller to making them easily testable with unit tests, have code that works in all cases so that each part of the project wil not fail in the future with unforeseen problems and can save a lot of time. When I was working in my social service making a webpage the automatic test runs could have saved me a lot of time becuase I made the tests case manually by inputting values that came to my head one by one and later forgot which one I did. With a tool like jest I can have my tests organize and easily run them whenever I need.
I will be learning more about Jest in the future and will write about my progress.
Everything that was required for task one I have already done previously. I am an avid Linux user, been using it as my main OS since 2017. I started as many do, using ubuntu but quickly lost interest since I saw it as a clunkier and worse version of windows. I could not see the true value of Linux in a distro like Ubuntu. I started to look for different distros and found some posts about Arch Linux elitism. I was intrigued so I started to research this distro. What I found was a nigh impossible distro to install, use and learn, I was game. Since I did not want to lose all of my files, I first tried installing arch in a VM. It was a total failure, and after a few days of trying I gave up. I started to look up guides besides the official install guide in the Arch Wiki (which was a mistake) and found a distro called Antergos. Antergos was a distro based on Arch with an actual GUI installer. I was frustrated enough at this point that I gave up trying to install Arch and decided to go with Antergos. I used Antergos for quite a while, and with it, learned Arch. The difference between Arch and Ubuntu was like night and day. Everything was under my direct control. I really felt like the administrator of the machine. I was even able to change the screen brightness from the terminal. I was in love. Then the NVidia drivers came and changed everything. I broke my installation about 3 times before I finally got my display drivers working. Doing all this configuration and installation and messing it up was very frustrating, but I kept throwing myself at it. The process of going from default drivers for the intel iGPU to a bumblebee installation with me deciding what program runs with intel and what program runs with Nvidia took me about a week at first. I could do it in probably 30 minutes today. This learning and breaking and getting better and faster is what I wanted from Linux, and thanks to Arch I got it.
Since I was already confident in my abilities to install and use arch, I decided that for this task I would install Gentoo. What a big mistake. If I thought the Arch installation process was clunky and convoluted, I never imagined a distro would include kernel compilation in the installation process. I quickly gave up on Gentoo since I did not want to slam my head against the keyboard for a week again, and I decided to go a little different, more professional route, RHEL. I had heard about RHEL, and how it was the main distro for the enterprise, and it was not free. Well, it is now free for development use and I decided to try it, so RHEL will be the distro I use for this class.
For the DB, I went with a standard MariaDB installation, since I don’t really care about the DB.
For the shell, I have been using bash for my entire time in Linux, so I decided I would try zsh.