It is a free, cross-platform and open-source distributed version control set of tools, which provides strong support for non-linear development and is able to handle from small to very large projects. Some important features git offers are local branching, convenient staging areas, and multiple workflows. It also offers the entire history tree.
Concurrent Versions System (CVS)
It is a version control system in which you can record the history of source files and documents. It handles conflicts between two programmers by only allowing for the latest version of the code to be worked on and updated.
Mercurial
It is a free, open-source and distributed source control management tool that is able to handle projects of any size, while offering an easy, intuitive interface. It is a major competitor of git, supported in various operating systems like Mac, Windows and Linux. It is known for its high performance, robustness, and scalability, and it is capable of branching and merging.
Tools for software testing
XRAY
It is a Manual and Automated Test Management App for QA. It lives inside and seamlessly integrates with Jira. Some of its features include traceability between requirements, tests, defects and executions, reusable preconditions, organization of tests in folders, tracking of progress, test environments, built-in REST API, CI integrations with Bamboo or Jenkins, and built-in reports.
Testpad
It is a manual test tool that prioritizes pragmatism over process. It uses checklist-inspired test plans that can be adapted to a wide range of testing styles. Some key features it has include guest testers that don’t need an account, simple to be used, keyboard-driven editor, drag’n’drop organization of test plans, adding of new tests, and Jira integration.
Practitest
It is an end-to-end test management tool. Some of its features are third-party integrations with bug trackers and automation tools, reusing of tests, unique hierarchical filter trees, anti-bug duplicates, and visualization of data with advanced dashboards and reports.
Tools for V&V
GDPro
It is a tool that generates UML diagrams from code or code from UML diagrams. It uses reverse engineering and supports C++ and Java.
WinA&D
It is a tool that uses verification reports to check consistency between class diagrams and project dictionary. It supports data modeling.
AxiomSys
For small to medium projects. It is a tool that can trace any type of information to the processes, modules and data items in the model. It validates trace files.
Testing is the process of evaluating a system or its individual components with the intent to find whether it satisfies the specified requirements or not. It consists of executing a system in order to identify any gaps or errors. It can be defined as analyzing a software item to detect the differences between existing and required conditions.
Software Testing Process
Phase 1: test strategy and test plan
It is about describing the scope for testing for a project. The things to be defined in this phase are:
The system to be tested
Main features and functions of the project
Non-functional requirements
Test approach
Key processes to follow
Tools
Documentation
Test environment requirements
Risks and dependencies
Test schedule
Approval workflows
Entry and exit criteria
Phase 2: test design
After having a test strategy and plan, the next step is to create a test suite, which is a collection of test cases that are necessary to validate the system being built, against its original requirements.
Phase 3: test execution
Tests can be executed in many different ways. As a single System Integration Test, as User Acceptance Test phases, as part of Agile sprints, with exploratory tests, or using a test-driven development methodology. An adequate amount of software testing has to be done to ensure the system is relatively bug-free.
Phase 4: test closure
The exit criteria needs to be considered for marking the completion of the test cycle and the project’s readiness for release. Some general exit criteria may be:
Requirements coverage testing by 100%
Minimum percentage of test cases to be passed
All critical defects fixed
A Test Summery and Defects Analysis can be done to provide statistics about the testing process.
Types of Testing
Manual Testing
The tester takes over the role of an end-user and tests the software to identify any unexpected behavior or bug. Some manual testing phases include unit and integration testing, system testing, and user acceptance testing. It also includes exploratory testing.
Automation Testing
The tester writes scripts and uses another software to test the product. Automation Testing is used to re-run the test scenarios that were performed manually, quickly, and repeatedly. It is also used to test the application from load, performance, and stress point of view.
TutorialsPoint
Levels of Testing
There are two main levels of software testing: Functional and Non-functional Testing.
Functional Testing
The application is tested providing input, then the results are examined to confirm that they conform to the functionality it was intended for. Testing an application for functionality involves five steps:
Determination of the functionality meant to be performed
Creation of test data
Determination of the expected output based on the test data
Writing of test scenarios and execution of test cases
Comparison of expected and actual results
Unit Testing
It is performed by developers on the individual pieces of source code assigned areas. The goal is to isolate each part of the program to show that individual parts are correct in terms of their functionality.
Integration Testing
It is the testing of combined parts of an application. It can be done in two ways: Bottom-up or Top-down.
System Testing
Like its name indicates, it tests the system as a whole. Once all the components are integrated, the application is tested thoroughly to verify it meets the functional specifications. It is tested in an environment that is very close to production.
Regression Testing
It is done to ensure that a change, such as a bug fix, hasn’t resulted in another fault being uncovered in the application.
Acceptance Testing
The QA team has a set of pre-written scenarios and test cases that will be used to test the application to make sure it meets the intended specifications and client’s requirements.
Alpha Testing
Unit testing, integration testing and system testing when combined together is known as alpha testing.
Beta Testing
A sample of the intended audience tests the application. It is also known as pre-release testing.
Non-functional Testing
It involves testing a software from the requirements which are nonfunctional in nature but important nevertheless.
Performance Testing
It is used to identify any bottlenecks or performance issues. It is a testing type in terms of the speed, capacity, stability and scalability aspects of the product.
Load Testing
It is about testing the behavior of a software when applying maximum load in terms of software accessing and manipulating large input data. It identifies the maximum capacity of software and its behavior at peak time.
Stress Testing
It is about testing the behavior of a software under abnormal conditions, like when taking away some resources, in order to identify breaking points.
Other kinds of testing levels are Usability Testing, Security Testing and Portability Testing.
Software Testing Roles
Test Lead/Manager
Responsible for:
Defining the testing activities
Test planning
Check the team has the necessary resources to perform the testing activities
Verify testing is going hand in hand with the development at all phases
Prepare the status report of testing activities
Interactions with customers
Updating the project manager about the status of testing activities
Test Engineer / QA Testers
Responsible for:
Understanding what needs to be tested
Decide how it is going to be tested
Inform the test lead about what resources will be required for the testing activities
Develop test cases and prioritize testing activities
Execute test cases and report defects, defining severity and priority for each defect
Carry out regression testing
Software Test Environment Setup
Setup of test server with support of needed applications
Network setup
Test PC setup for running the software product
Provide bug reporting tools
Create test data for testing environment
Test Case Design Techniques
Boundary Value Analysis (BVA): based on testing at the boundaries between partitions.
Decision Table Based Testing: for every function, a table is created to list down all types of combinations of inputs and its respective outputs.
State Transition: the testing team provides both positive and negative input test values to evaluate the system behavior.
Error guessing: the test analysts use their experience to guess potential problematic parts of the application and write test cases to expose those errors.
It is a complete process that results in carefully examining a software product, usually undertaken by project personnels, managers, users, customers, or user representatives who inspect the software in order to determine the positive and negative aspects of a program. It assists software engineers in validating the quality, functionality, and other vital features and components of the software, by involving the testing of the software product and ensuring that it meets the requirements stated by the client.
It’s a systematic examination done by one or more individuals who work together to find and resolve errors and defects in the software during the early stages of the Software Development Life Cycle. Software review is used to verify various documents like requirements, system designs, codes, test plans, and test cases. With the assistance of software review, the team can verify whether the software is developed as per the requested requirements or not, and make the necessary changes before its release in the market.
Types of Software Reviews
Peer Review
It is the process of assessing the technical content and quality of the product, usually conducted by the author of the work product along with some other developers. The quality is also checked by other members of the team. It has the following types:
Code review: the computer source code is examined in a systematic way.
Pair programming: it is a code review where two developers code together at the same platform.
Walkthrough: members of the development team are guided through the product by the author and the participants ask questions and make comments about defects.
Technical review: a team of qualified individuals examines the software product for its client’s use and identifies technical defects from specifications and standards.
Inspection: the reviewers follow a well-defined process to find defects.
Software Management Review
Management reviews identify consistency with and deviations from plans, or adequacies and inadequacies of management procedures. It evaluates the work status. In this section, decisions regarding downstream activities are taken.
Software Audit Review
It is a type of external review in which one or more critics, who are not a part of the development team, organize an independent inspection of the software product and its processes to assess their compliance with stated specifications and standards. This is done by managerial level people.
Review Stages Workflow
Tutorialspoint
Software Design Review
A systematic, comprehensive, and well-documented inspection of design to check whether it meets the specified requirements. Generally, the review process is carried out in three steps, which correspond to the steps involved in the software design process.
First, a preliminary design review is conducted with the customers and users to ensure that the conceptual design (which gives an idea to the user of what the system will look like) satisfies their requirements. Next, a critical design review is conducted with analysts and other developers to check the technical design (which is used by the developers to specify how the system will work) in order to critically evaluate technical merits of the design. Finally, a program design review is conducted with the programmers in order to get feedback before the design is implemented.
Verification concerns about the correctness of process. Its objective is to determine if the software at a given phase of the development life cycle satisfies the requirements established during previous phases. It attempts to identify faults and errors that could eventually rise to the failure of the product. The question to answer with verification is, are we implementing the software right?
Validation
Validation concerns about the correctness of product. It is the process of evaluating the software at the end of its development to ensure that it is free from failures and it complies with the requirements. Its objective is to make sure that the product actually meets the client’s needs and that the specifications are correct, by verifying that the software works properly for correct inputs and gives error messages for wrong ones. The question to answer with validation is, are we implementing the right software?
Verification ensures that the system complies with an organization’s standards and processes, relying on review and non-executable methods. On the other hand, validation physically ensures that the system operates according to plan by executing its functions through a series of tests that can be observed and evaluated.
V&V in the Software Development Life Cycle
Phases
Validation
Verification
Requirements
Work directly with the customer to better understand the requirements, using analysis techniques.
Verify that each requirement is unique, has a purpose and origin, and is doable, in scope and testable.
Arquitecture and Design
Consult available prototypes and process designs with the customer to ensure their expectations.
Verify the software designs are meeting the non-functional requirements.
Implementation
Validate changing, unclear or incomplete requirements that arise as the software progresses.
Start doing unit and integration testing for the code, as well as peer reviews and static code analysis.
Testing
Build the user acceptance test cases and validate them with the customers.
Test the requirements and ensure that all test cases have been passed, all functionalities are working fine and there is enough code coverage.
Deployment
Making sure software is working after installation and configuration.
Auditing of the environment, running of smoke tests, and release of the test report.
Support
Validate there are no logical errors or something broken, as well as changes and new requirements.
Verify there is no downtime, data issues, and that the logs are clear, as well as that there is nothing broken in integration.
Standard
IEEE 1012-2016 – IEEE Standard for System, Software, and Hardware Verification and Validation
The scope of V&V processes encompasses systems, software, and hardware, and it includes their interfaces. This standard applies to any of them being developed, maintained, or reused. The term software also includes firmware and microcode, and each of the terms system, software, and hardware include documentation. V&V processes include the analysis, evaluation, review, inspection, assessment, and testing of products.
The Capability Maturity Model Integration (CMMI) is a process improvement approach that identifies the weaknesses and strenghts of the processes in place. It is a collection set of best practices that help improve quality and efficiency of the software delivered. It was first developed at Carnegie Mellon University, initially as CMM, a model of matured practices. CMMI was built combining the best components of individual disciplines of CMM. The objectives of CMMI are to produce quality products and services, create value for stockholders, enhance customer satisfaction, increase market share and gain an industry-wide recognition for excellence.
A mature process is one that is well-defined, repeatable, measured, analyzed, improved, and effective. A maturity level is a well-defined evolutionary plateau towards achieving a mature software process. In a Staged Representation approach there are five maturity levels designated by the numbers 1 through 5. This approach uses predefined sets of process areas to define an improvement path for an organization. This improvement paths are described by the maturity levels. It provides a proven sequence of improvements and a pre-defined roadmap for organizational refinement. One can not divert from the sequence of steps.
The five maturity levels are:
1. Initial
Process unpredictable, poorly controlled, and reactive.
2. Managed
Process characterized for projects and is often reactive.
3. Defined
Process characterized for the organization and is proactive.
4. Quantitatively Managed
Process measured and controlled.
5. Optimizing
Focus on continuous process improvement.
In a Continuous Representation approach, there are six capability levels designated by the numbers 0 through 5. In this approach, an organization selects a specific process area and makes improvements based on it. It provides flexibility to choose the processes for improvement and the amount of improvement required. The capability level is a well-defined evolutionary plateau used to characterize improvement relative to an individual process area, consisting of related specific and generic practices for it. The capability levels are cumulative, and they are the following:
0 – Incomplete
1 – Performed
2 – Managed
3 – Defined
4 – Quantitatively Managed
5 – Optimizing
Process Areas
A Process Area is a cluster of related practices in an area that, when implemented collectively, satisfy a set of goals considered important for making significant improvement in such area. They can be grouped into four categories: process management, project management, engineering and support. There are two categories for goals and practices: generic and specific.
TSP/PSP
Personal Software Process
A personal process, designed for individual use, for developing software that includes defined steps, forms and standards. It provides a framework for characterizing and managing your personal work, and to help improve personal performance. It allows you to estimate and plan your work, meet your commitments, understand current performance and be better equipped to improve your capability.
It provides a proven basis for developing and using an industrial-strength personal process, governed by the principle that the key to quality is the individual developer’s skill, commitment, and personal process discipline. As a software professional, one is required to measure, track and analyze their work, learn from their previous performance and incorporate the lessons learned into their personal practices. It is designed to gather and analyze accurate and complete data on your work and use the results to improve personal performance.
PSP Flow
2006 by Carnegie Mellon University, Introduction to PSP and TSP
Team Software Process
Framework that seeks to create a balance between processes, products, and team work. The goal is to integrate independent, high performance teams to plan and register their work, establish goals, and be owners of their plans and processes, as well as accelerate the continous improvement of processes, and provide a guide to managers on how to monitor and motivate its team and help them acquire maximum productivity.
TSP actions
Establish precise metrics that permit the measurement between an initial and final state of a team’s project.
Define the roles and responsibilities according to the characteristics and objectives of the work to be done.
Provides suggestions to solve common problems.
Provides a framework and structure of processes to work on projects.
Phases that start on each release.
Main Roles in TSP
Project leader
Development manager
Planning manager
Quality manager
Support manager
The developers can also take one of the roles above.
ISO-15504
It is the international standard named as Software Process Improvement Capability Determination. It is a tool with the objective of proposing and developing a model for the evaluation of software processes. It establishes the requirements for an evaluation of processes and the models of evaluation, spanning the areas of process improvement, capability, and maturity.
Structure
There are 10 parts conforming this norm:
Vocabulary and concepts
Realization of an evaluation
Conducting an evaluation
Guide about the use for the improvement of the process and the determination of its capability
An example of an evaluation model of processes in the life cycle of software
An example of an evaluation of the life cycle of the system
Evaluation of organizational maturity
An example model of evaluation of processes for the management of IT services
Profiles of objective process
Security extension
It extends the Capability levels of the CMM. To position a process in a capability level, it uses the following rating scale, based on the percentage of completed attributes:
N, not implemented (0-15%)
P, partially implemented (15-50%)
L, widely implemented (50-85%)
F, completely implemented (> 85%)
MOPROSOFT
It is a model that provides a set of best practices, integrated processes, workflows, roles and products to serve as a framework for companies in the software industry. It is done considering three levels of organization: High Direction, Management, and Operation, containing in each the processes that conform the norm in the following way:
High Direction: contains the process of Business Management
Management: contains the processes of Process Management, Projects Management, and Resources Management. This last one contains other three sub-processes: Human Resources and Work Environment, Services, and Infrastructure and Organizational Knowledge.
Operation: contains the process of Administration of Specific Projects and Development and Maintenance of Software.
IDEAL method
It is an improvement model for the CMMI product suite, serving as a roadmap for initiating, planning, and implementing process improvement actions in this standard. It is named after the five phases required to perform a change initiative : Initiating, Diagnosing, Establishing, Acting, Learning.
Initiating
Lay the groundwork for a successful improvement. Current way of doing business needs to change, and the appropriate resources to do so are allocated.
Diagnosing
Where you are vs where you want to be. Analysis to determine what are the current practices and highlight potential areas of improvement opportunities.
Establishing
Plan how objectives will be reached. Recommendations of the analysis phase are prioritized, and plans are developed to conduct these activities.
Acting
Do the work that was established in the plan. Implement the designed activities.
Learning
Learn from the experience and improve. Collect lessons that can be applied to future rounds of the IDEAL life cycle.
The structure of the cycle is the following:
Plays-In-Business (2014). See references.
References
Carnegie Mellon Software Engineering Institute (2006). Personal Software Process for Engineers: Part I. Introduction to PSP and TSP.
Defining Software Quality can be a tricky task, since it is an abstract concept that can have many different definitions, depending on the source. A variaty of standards, measures and characteristics exist to determine whether a software project is considered to be of good quality or not quite. However, one common factor present in several of the definitions is how well does the software conform to its functional and non-funtional requirements, which are established based on the needs and desires of the customer. This is called fitness for purpose, and it refers to the fulfillment of a product’s requirements. This is reflected at the same time through its functional quality and its structural quality. Other important aspects of quality software is that it is delivered on time, withi the budget, and it is maintainable.
Functional Quality
Based on the functional requirements, it determines how well the software satisfies a given design or specifications document.
Structural Quality
It determines how well the non-functional requirements are implemented in order to support the functional requirements.
The reason why quality is a lot more difficult to completely ensure for software products compared to physical manufactured products is because of its inherent complexity and visibility. The product complexity refers to the large amount of operational modes it permits, since software packages allow for millions of operational possibilities and often it is hard to correctly assure them all. The product visibility refers to how defects in software are most times not detectable by sight, like it is for industrial products during the manufacturing process, hence being invisible.
I am currently taking a semester long class (August – December 2020) on the subject of Software Quality and Testing, and in this blog I am documenting what I learn and from which sources.
The topics I’ll cover will be divided into six different entries, with several subtopics in each one.
They are the following:
Software Quality
Models and Standards for Software Process Improvement
Verification and Validation of Software (V&V)
Software Review
Software Testing
Tools for V&V
I’ll be discussing each one of those in detail in posts to come.
