Tools for Verification and Validation

Now that all the concepts are set, it is time to discuss the tools that can be used for the different stages that were mentioned in the previous entries.

Tools for version control

Version control is useful to keep track of the changes made to a code, is great for working in teams. It keeps the history of all the modifications, and can undo a change if necessary. Now, to discuss some of these tools.

1. Git

Probably the most widely used in current times. Is a free, cross-platform and open source distributed control tool. It is great for both small and big projects. It is fast, has a full history tree, that is also available offline, has a pretty good interface. It is robust and easily maintainable, and has a variety of commands, and even comes with its own command line. There are many cons to use git, although not everything is perfect, since the history log can be hard to understand, does not support keyword expansion and timestamp preservation, and is more limited for Windows users.

2. Concurrent Versions System (CVS)

It is another widely used revision control system, and has been the tool of choice for many developers since the 80s. Although it is not as feature rich as other tools (this doesn’t mean this tool is not robust), it is fairly easy to learn how to use it. It makes it pretty simple to keep all to date, can allow multiple developers to work on a project parallely, has a and even has a pretty big community that gives it support, since it is open source. However, it comes with its tolls; poor support for distributed source control, no integrity check for source code, it has expensive branch operations, and has no automatic check-outs and commits.

3. Subversion (SVN)

Free and open source, and a popular alternative to CVS, trying to fix all the problems that were found in it and at the same time still being compatible with it. It prevents corruption the the database using atomic operations, where all or none of the changes made to the source code are applied, but no partial changes will affect the original code. Also, SVN fixed the cost of branch operations. SVN has better support for Windows, and can use good GUI tools, and even has support for empty directories. It also is easy to administer and to integrate with IDE and Agile tools. But, it does not store the modification time of files, does not support signed versions, is not good for filename normalization, and is still slower than git.

4. Mercurial

It is a free, open-source distributed tool, written in python to directly compete with git. It has a high performance, robustness and scalability. It has advanced branching and merging capabilities, it is decentralized, easy to learn, lightweight, and simple, and handles both plain text and binary files robustly. It also has an integrated web interface. But, all the add-ons must be written in python, partial check-outs are not allowed, and can become problematic with additional extensions.

5. Bazaar

It is a free, open-source, and distributed tool (what a surprise), that scales and adapts according to the needs of the user. It is similar to git, and mercurial, and can be centrilized or decentralized. Provides free hosting services, it is easy to use, has a high efficiency, speed, and supports tracking of directories. Although, it doesn’t support partial check-outs, and does not preserve timestamps.

6. Monotone

It is a decentralized peer-to-peer tool that focuses on integrity over performance. It is intended for distributed operations. It employs cryptographic primitives to track file revisions and authentications. It can import CVS projects, and uses netsync, a very efficient and robust custom protocol. It requires very low maintenance, has good documentation, it is easy to learn, works great with branching and merging, and has a stable GUI. Although it suffers in performance for some operations, and cannot commit or check-out from behind a proxy (due to a non-HTTP protocol).

Tools for testing

Testing tools offer many options to be used: tools for automation, for functional testing, for non-functional testing, and even for agile testing. I will not discuss every single one of them, but I will put a link with a list of 50 of them (or you can check out the references for even more examples): https://www.guru99.com/testing-tools.html#7

1. Selenium

The number one testing framework, to perform various web application tests on many platforsm and operating systems. It can help to create tests for different programming languages: Java, PHP, C#, Python, Groovy, Ruby, and Perl. Offers record and playback features without the need of learning Selenium IDE. It is the base for most of the software testing tools. It is powerful,and supports multiple frameworks like Data-Driven, KeyWord Driven, Cucumber, JUNIT, NUNIT, and Test NG. And the best part is that it is free to use.

2. Eggplant

Alson known as Eggplant’s Digital Automation Intelligence, is primarily aimed at application testing and GUI testing. It offers an AI-driven test automation for functional, usability and performance testing. It can provide user and business focused metrics for quality and impact on the end-user. Eggplant uses and image based approach, so, by using a single script, test on multiple platforms can be performed (Windows, Mac, Linux, Solaris, among others). As a plus, it has a very intuitive interface, although it is not free. Also, it seems to work better on MacOS than Windows.

3. Katalon Studio

These test automation tool covers API, Web, Desktop and mobile testing. It has a nicely sorted set of features for recording actions, creating test case, generating and executing test scripts, reporting results, and many others. It runs on Windows, MacOS and Linux. Also can be integrated with other tools such as Jira, qTest, Kobiton, Git, Slack, etc. It is not free, although offers a free version for individual testers. It is very easy to learn, even for non-QA testers. For now, it is very limited on the programming languages it accepts.

4. SauceLabs

Is a selenium cloud based tool that offers automated testing over cross-browsers and multiple platforms. It has support for mobile and desktop apps. It has a reputation of reducing testing time to a huge extent (it is widely used by big companies). However, the cloud service is a bit slow, and outages in the software causes the odd test to fail. It is priced, but there are free licences for open source projects.

5. OWASP

The Open Web Application Security Project (OWASP) is a tool for security testing, to help projects to be reliable and trusted. It helps to verify security, validate all types of inputs, implement authentication controls and all the appropriate access controls. It is an open source framework, and it offers many guides and documentation to assit the execution of tests.

6. LAMBDATEST

A scalable cloud-based browser testing platform to perform manual and automated tests on a widely number of desktop and mobile browsers. It supports parallel testing.

Tools for process administration of V&V

Here is the awkward moment, in which a lot of time was spent on a topic that is not very specific (first I found an article from 2006, which means that is not going to have an updated list). These tools are used not only for administrating processes, since most of them have more features than just keeping track of all the tasks that were, are, or will be performed. But they are very useful, and widely used, also, since the trend right now is agile development, most of these tools have a focus on agile:

Jira

Jira is a great software management tool that is widely used in many companies. It is very robust and offers many features that help the management of tasks and processes, and it can be integrated with other tools.

Notion

A very innovative tool that serves as a all-in-one workspace. It supports many features, for productivity, design, and of course, task management. An alternative to Jira, and great for managing the status of processes. It also supports markdown.

References

Version Control Tools. (2020). Top 5 Free and Open-Source Version Control Tools in 2020. GeeksforGeeks. Retrieved from https://www.geeksforgeeks.org/top-5-free-and-open-source-version-control-tools-in-2020/

Version Control Tools. (2020). 15 BEST Version Control Software (Source Code Management Tools). Software Testing Help. Retrieved from https://www.softwaretestinghelp.com/version-control-software/

Version Control Tools. (2015). Top 5 open source version control tools for system admins. FileCloud. Retrieved from https://www.getfilecloud.com/blog/2015/02/top-5-open-source-version-control-tools-for-system-admins/#.X7xSG2hKiUk

Testing Tools. (2020). Top 50 Software Testing Tools in 2020. Guru99. Retrieved from https://www.guru99.com/testing-tools.html#7

Satasiya, P. (2019). Top 10 Automated Software Testing Tools. DZone. Retrieved from https://dzone.com/articles/top-10-automated-software-testing-tools

Testing Tools. (2020). Top 20 Best Automation Testing Tools In 2020 (Comprehensive List). Software Testing Help. Retrieved from https://www.softwaretestinghelp.com/top-20-automation-testing-tools/

Boog, J. (2020). 10 Latest Software Testing Tools QAs Are Using In 2020. QA Lead. Retrieved from https://theqalead.com/tools/software-testing-tools/

Process Administration Tools. (n.d.). A story of tools and the future of work. Notion. Retrieved from https://www.notion.so/about

Process Administration Tools. (n.d.). Jira Software. Atlassian. Retrieved from https://www.atlassian.com/software/jira?&aceid=&adposition=&adgroup=109687540304&campaign=10332064761&creative=443576046641&device=c&keyword=jira&matchtype=e&network=g&placement=&ds_kids=p55122863591&ds_e=GOOGLE&ds_eid=700000001550060&ds_e1=GOOGLE&gclid=CjwKCAiA2O39BRBjEiwApB2Iku6tiVe8LVTlnp_BSWfHFts3jV44ueKHzvrTlWiNrZi4Fi5RQeV_OxoCw9EQAvD_BwE&gclsrc=aw.ds

Verification and Validation of Software

It is time to talk about two very important concepst within software quality, and those are Verification and Validation (V&V). But as always, these terms have to be defined first through the software engineering approach. So let’s start with Verification:

“Confirmation by the analysis of objective evidence that proves that a product, service or system fulfills specified requirements.”

So basically, to ensure that a product meets its requirements according to the team for the customer. Now, with Validation:

“Confirmation by the analysis of objective evidence that proves that a product, service or system fulfills specified requirements for an specific use or application.”

In other words, to ensure that a product accomplishes its intended use in its intended environment. There can be a confusion between those two terms, but worded like that, its specific purpose can be differenciated easily.

V&V in the life cycle of software development

Even though V&V is constatly referred to the testing phase of the life cycle, it is essential across all phases:

PhaseVerificationValidation
RequirementsIt should be verified that each requirement is doable, in scope, testable, as well as having a purpose and being unique.Requirements should be validated with the client to understand them better and eliminate ambiguity.
Architecture and DesignThe architect and designer should verify that the product meets the non-functional requirements and the scalability.Everything regarding design and architecture should be validated with the client.
ImplementationUnit and integration tests are done to the codem as well as peer review and static code analysis.Requirements should be checked with the client, and since this is the phase where major changes are done, these changes should be validated too.
TestingVerify that everything is working as expected, all test cases have been passed, and designs and non-functional requirements are met.Ensure that the customer accepts the product. That all requirements are met, it is ready for use, and all validation and error messages are clear.
DeploymentAuditing of installation. Make sure everything works as expected with each release.Make sure that installation and configuration is working, also that integration had no problem with internal and external systems.
MaintainanceVerify that everything is fine within integration.Ensure that everything runs smoothly, and understand errors in order to create new requirements.
International standards for V&V of software

There are many standards for V&V, that includes ISO/TS 17033, ISO 14021, IEC, many from IEEE, and countless others. And each one has very detailed documentation that describes each method with detail. There are some that would be great to discuss a little bit more:

  • ISO/IEC 17029: Provides requirements for validation and verification of ethical processes with a focus on environmental issues. Even the software industry should take these aspects into account. It describes how verifiers can avoid conflict of interest, ensure the competence of the verifiers. It verifies ethical claims, sustainability reports, and validates due dilligence approaches and environmental claims.
  • IEEE 1012-2016: Processes are used to determine if the development products satisfies its intended use and user needs, in every part of the software development cycle. Applied to software, systems, and hardware being developed, maintained, or reused.
Planning V&V

A system validation and verification plan should be stated since the begining of the software development. In order to have a good v&v it is important to plan it taking into account who are going to be the main stakeholders, since everything should be validated with them. As stated in the previous point, every part of the life cycle should take v&v into account, and that requires planning of the needs, goals and objectives.

There are important tools that help the v&v planning:

  • Project Management Plan (PMP): Defines the overall project activities, milestones and events.
  • Systems Engineering Management Plan (SEMP): establishes the overall plan for the technical development of the system.
  • Master Integration Verification and Validation Plan (MIVVP): implements the v&v philosphy of the project.
  • System Requirements Document or Specification (SRD/SRS): contains the system’s requirements.
  • Requirements Verification Matrix (RVM): a matrix that lists the requirement with relevant data such as its verification method, responsible organization, level, phase and requirement rationale, as well as a short description for the success criteria.
  • System Integration, Verification and Validation Plan (SIVVP): implements the project’s MIVVP process for a given component.
Administration of a V&V plan

Although planning is very important, it is not the end of it, the plan should be implemented. And to help administering the v&v plan there are some helpful tools:

  • Verification Requirement Definition Sheet (VRDS): documents the details of the specific verification activities identified in the RVM.
  • Task Definition Sheets (TDS): lists tasks to be performed to accomplish the needed verification activities defined in the VRDS.
  • Task Performance Sheet) (TPS): contains the actual procedure used to perform the verification or validation activities defined in the TDS.
References

Sami, M. (2018). Validation and Verification in SDLC. Mohamed Sami. Retrieved from https://melsatar.blog/2018/08/06/validation-and-verification-in-sdlc/

Verification and Validation. (n.d.). Verification vs Validation. Software Testing. Retrieved from https://softwaretestingfundamentals.com/verification-vs-validation/

Standard for V&V. (n.d.). P1012 – Standard for System, Software, and Hardware Verification and Validation. IEEE. Retrieved from https://standards.ieee.org/project/1012.html#:~:text=Verification%20and%20validation%20(V%26V)%20processes%20are%20used%20to%20determine%20whether,specified%20for%20different%20integrity%20levels.

Illner, C. (2019). VERIFICATION AND VALIDATION: WHY THE NEW STANDARD ISO/IEC 17029 MATTERS. DQS-CFS. Retrieved from https://dqs-cfs.com/2019/10/verification-and-validation-why-the-new-standard-iso-iec-17029-matters/

Wheatcraft, L. (2012). Thinking Ahead to Verification and Validation. Requirements Experts. Retrieved form https://reqexperts.com/wp-content/uploads/2015/07/thinking-ahead-to-verification-and-validation.pdf

V&v plan. (n.d.). System Validation and Verification Plans. UC Berkley. Retrieved from https://connected-corridors.berkeley.edu/developing-system/system-validation-and-verification-plans

Images from https://undraw.co/

V&V

In order to understand verification and validation of software we must first define it. According to the IEEE, V&V is the process of determining whether:

  • Requirements for a system are complete and correct.
  • Products of each development phase fulfill the requirements and conditions imposed by the previous phase.
  • Final systems comply with specified requirements.

In other words, V&V is the process of eliminating software defects from every part of the software life cycle, on a stage by stage basis before testing.

As with pretty much everything regarding software quality, there are a lot of perspectives and perceptions as to what V&V is. This is why having a definite international standard is hard, there are a lot of existing definitions and each vary in some way. As stated before, the IEEE has defined a technique for V&V but there are other alternatives like the ISTQB or the HTB one. As for approaches, V&V can be used in different ways: testing, life cycle frameworks, and life cycle methodologies. Testing is pretty self explanatory, life cycle frameworks are the use of techniques to help eliminate software defects in each stage, as stated before. The difference between this and life cycle methodologies is that life cycle methodologies are streamlined to include nothing but the most effective and minimum software defect elimination techniques that are recommended by the life cycle frameworks. This is just a quick overview of these approaches, I hope it is enough to understand the basic difference between them. If not, I encourage you to look further into it as the differences in them are more than what they actually share.

Planning is the first step for a successful use of this technique. There are many things to consider, I found this link with a document describing the outline for V&V planning and the recommended content you should look for in it. As you can see, it is a very simple outline but does a good job explaining how the planning is divided and what you should be doing for each step. Administering

External links:

V&V

Planning

How to get that little blue check mark on your site or profile… I mean, validating and verifying software (V&V)

As always, let’s get to what we’re gonna be talking about in this post first:

  1. V&V in the life cycle of software development
  2. International standards for V&V of Software
  3. Planning V&V
  4. Administration of a V&V Plan

Before everything, we shall ask our main questions.

What is verification and validation?

Just by hearing those two words, we can get a pretty basic definition, which would be verification: checking that something completes all the requirements it needs and validation: the process that we follow to verificate a product, thing or whatever you could think.

When talking about the life cycle in software, validation and verification is a step used during almost the whole process of development, so we’ll see in the following table what happens in each one, very briefly:

PhasesValidationVerification
RequirementsValidate requirements along with the client to ensure proper understanding of what’s going to be done.Verifying that all requierements are unique and doable.
Architecture and DesignAll types of design that might end up being used have to be validated with the costumer, as to meet their expectations.We verify that all designs are meeting the non-functional requirements (like performance or usability)
ImplementationFor each requirement done, we have to validate with the client if that’s what they wanted.When doing unit and integration tests, or while having peer review.
TestingCheck that all functionality meets the client expectations and that all messages are clear.All test cases need to pass this part, as to ensure that everything is working as intended.
DeploymentAll software after installation should be working as intented.Configuring environment and running smoke tests.

As to international standards, ISO has always been there to mantain a certain level needed for validating different things, including software. As to what I could say about this, is simply that by having a certain level required for stuff to be valid and verified by certain organisms, we can mantain a same procedure for developing new stuff, such as software, I’ll also leave this article here so that people can indulge in really technical stuff if they feel like it, because I dont feel like it.

https://dqs-cfs.com/2019/10/verification-and-validation-why-the-new-standard-iso-iec-17029-matters/

Planning and administrating V&V, the real challenge

For this, we have a term that already exists and depicts pretty well what was to be done. Verification and Validations Plans. This pretty much works by going hand with hand on your normal software life cycle, but implementing everything that’s needed to make a V&V correctly:

  • Explaining the test and objective of the project.
  • Define what items you’re gonna be testing, using the required documentation.
  • What features need to be tested? Along with all the data you’ll need.
  • What DOESN’T need to be tested? This should be a small list, and you can include items that need too much time and effort to be properly tested at earlier phases.
  • How is your testing team gonna approach the requirements? By defining a pass/fail criteria, you can prevent later confusion on how to validate and verificate diverse stuff.
  • An approach for handling any mistakes and errors that could stop your testing, otherwise you’ll lose time on making all test cases needed, with their design specification and procedures.
  • Environmental needs (I think this one pretty much speaks for itself? Please don’t run some software on a all mighty computer that your client does NOT have).
  • Approval from the client for all requirements.
  • Operation and maintenance in case a step was missed or something went wrong.
  • An open space of possible upgrades or replacement.

So, this was it for Validation and Verification, if you have anything to share, like experiences, knowledge on the topic, or just a plain comment on how you dislike my blog, please share in the comments so that other people can also enjoy a different perspective!

The Two V’s (Verification and Validation)

What does Verification and Validation mean?

Both concepts are crucial to mantain good software quality that complies with the standards we talked about before. But we have to understand what each concept means, and what are their differences before we dive deep into them. Let’s see their formal definitions according to the IEEE-STD-610:

Verification : “A test of a system to prove that it meets all its specified requirements at a particular stage of its development”.

Validation: “An activity that ensures that an end product stakeholder’s true needs and expectations are met.

With the above definitions, we can see that Verification is more about making sure we are building the software correctly with good practices, standards, and following a good methodology that allows it to meet the goals set at the start of the development cycle. Meanwhile Validation is more about making sure that the product being built is one that benefits the client, a product that ensures that the client’s needs are being met, because we may have a good product that works, but it may not solve the problems of our client.

When do we implement it into our SDLC?

The verification and validation process is one that should be present in each stage of the SDLC, we can’t allow errors, bugs, miscommunication to keep piling on until the end, because we may even find we have too many which may make the software completely useless since it would become much more expensive to fix. That’s why Verification and Validation must be done at each stage, to find errors on the planning phase, on the design phase, on the coding phase, on each process that the team does, to ensure that the errors never become too costly or difficult to fix, and also ensuring that the software being built is adequate for our client, that it is what our client needs.

The more late an error is detected, the more expensive it is to be fixed

Standards for V&V

The main standard used in the industry is the IEEE 1012-2016 which incorporates both verification and validation as a rigorous process carried since the beginning until the end.

The standard defines and talks about all the tests that should be done to comply with the standard, talking about tests like: System Integration, Acceptance Testing, Software Integration Testing, Component Testing and more.

The standard also talks about continuous review, how every test, every test result and every process done on V&V must be reviewed by the whole team working on the software to ensure of its quality and make recommendations for the next phase so the errors, or improvements that could be made are done. It should have a constant appeareance on each stage of the SDLC.

If you want to dive deeper into what the standard talk about you can visit the following website: https://katie.mtech.edu/classes/esof411/Schedule/IEEE_Standard_System_Software_HardwareV&V_160pages.pdf

How do I actually implement it into my SDLC?

Just as for the SDLC, you have to start with planning, make a good planning of the Verification and Validation process, one that covers fully the objectives, needs of the client and requirements of the software.

Document

The IEEE-1012 standard suggests making a document stating the objectives that should be met by the Verification and Validation, what products are being covered, who is involved and on which role, the activites, processes, tasks and tests that will be done to verificate and validate and how will the report of results will be done. The main part of the document should be about the planning of tests and their design and the analysis that will be performed based on those tests.

Test Design

As we see, the most important part are the tests for the V&V, they need to be built according to the goals of the software, but according to the IEEE standard, they should have the following types of test almost by default:

  • Design Evaluation
  • Interface Analysis
  • Traceability Analysis
  • Criticality Analysis
  • Software Component Tests
  • Software Integration Tests
  • Software Qualification Tests
  • Software Acceptance Tests
  • Hazard Analysis
  • Risk Analysis
  • Security Analysis

All the above tests are required to test and analyze almost everything about the software, and with those we can be almost completely sure that we will achieve our goals. Each test is in the same level of importance, and some are even preventive to detect certain risks or menaces that although not present on the software on that moment, they may arise later.

TEST!!!

Report Results

After the design, testing and analysis is finished, a good report must be made, which will be reviewed by the whole software development team. The report should present documentation about how the tests were performed, the results observed from the tests and conclusions and recommendations that can be made from their results. This process is very important, tests aren’t useful if you don’t learn anything from them! The reviewing should reveal and offer good insights for the whole team to implement them and improve the software on the next stages of it. They should cause change towards a better direction.

Remember Plan -> Design -> Test -> Review -> Success!

Sources: