SE 433 Software Testing Quality Assurance Lecture
In this lecture series, Dennis Mumaugh covers various topics related to software testing and quality assurance. The content includes discussions on black box testing, JUnit, test case design, assignment guidelines, and insightful thoughts on bug prevention in software development. The assignments focus on developing parameterized JUnit tests and designing test suites using black-box techniques. The lecture series emphasizes the importance of designing tests as a bug prevention strategy in the software development lifecycle. Stay updated with lectures, assignments, and valuable insights in software testing and quality assurance.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
SE 433/333 Software Testing & Quality Assurance Dennis Mumaugh, Instructor dmumaugh@depaul.edu Office: CDM, Room 428 Office Hours: Tuesday, 4:00 5:30 April 18, 2017 SE 433: Lecture 4 1 of 101
Administrivia Comments and feedback Announcements Solution to Assignment 1 and 2 have been posted to D2L Hints Look at the Java documentation. For example: http://docs.oracle.com/javase/8/docs/api/java/lang/Double.html Look at the examples (JUnit2.zip) mentioned on the reading list and provided in D2L. In solving a problem, try getting the example working first. April 18, 2017 SE 433: Lecture 4 2 of 101
SE 433 Class 4 Topic: Black Box Testing, JUnit Part 2 Reading: Pezze and Young: Chapters 9-10 JUnit documentation: http://junit.org An example of parameterized test: JUnit2.zip in D2L See also reading list April 18, 2017 SE 433: Lecture 4 3 of 101
Assignments 4 and 5 Assignment 4: Parameterized Test The objective of this assignment is to develop a parameterized JUnit test and run JUnit tests using Eclipse IDE. [See JUnit2.zip] Due Date: April 25, 2017, 11:59pm Assignment 5: Black Box Testing Part 1: Test Case Design The objective of this assignment is to design test suites using black- box techniques to adequately test the programs specified below. You may use any combination of black-box testing techniques to design the test cases. Due date: May 2, 2017, 11:59pm April 18, 2017 SE 433: Lecture 4 4 of 101
Thought for the Day More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest. Boris Beizer April 18, 2017 SE 433: Lecture 4 5 of 101
Assignment 1 Lessons Learned The assignment requirements may have some unwritten items: Being able to test the program Programs need to defend themselves from Illegal input Boundary conditions April 18, 2017 SE 433: Lecture 4 6 of 101
Assignment 1 Part I, Grading rubric Part I: 10 Points Compiles and runs and handles standard tests Handles invalid cases -1 Not a triangle (a + b < c) Degenerate triangle (a + b = c), aka line Handles illegal input -1 Negative numbers Floating point numbers Extra large integers and MAXINT [See above triangle test]. Text If the code handles most of the special condition give it the point. April 18, 2017 SE 433: Lecture 4 7 of 101
Assignment 1 Part I, Grading rubric For triangles: 10 + 20 = 30 hence it is a line 12 + 8 < 30 hence it is not a triangle Negative length triangles are bad. Also triangles with long numbers such as 2147483647. This number is 2^31 - 1 ) i.e. MAX_VALUE A constant holding the maximum value an int can have, 231-1 . The problem is in testing for illegal triangles one must check a + b <= c and we get arithmetic overflows. [See example on next slide.] I got bit the first time myself. The problem in using 21474836471 is simply that it is too large April 18, 2017 SE 433: Lecture 4 8 of 101
Assignment 1 Lessons Learned Consider: public class Triangle { public static void main(String[] args) { int a = 2147483647; int b = 2147483647; int c = 2147483647; System.out.println("Not a triangle"); } else { System.out.println("Triangle");} } } Result: Not a triangle if (a + b <= c) // Check a c b { April 18, 2017 SE 433: Lecture 4 9 of 101
Case Study Knight Capital High Frequency Trading (HFT) April 18, 2017 SE 433: Lecture 4 10 of 101
Case Study Knight Capital: High Frequency Trading (HFT) The Knight Capital Groupis an American global financial services firm. Its high-frequency trading algorithms Knight was the largest trader in U.S. equities with a market share of 17.3% on NYSE and 16.9% on NASDAQ. April 18, 2017 SE 433: Lecture 4 11 of 101
Case Study Knight Capital Aug 1, 2012. In the first 45 minutes, Knight Capital's computers executed a series of unusually large automatic orders. spit out duplicate buy and sell orders, jamming the market with high volumes of trades that caused the wild swings in stock prices. By the end of day: $460 million loss Trading Program Ran Amok, With No Off Switch In two days, the company's market value plunged by 75% April 18, 2017 SE 433: Lecture 4 12 of 101
Case Study Knight Capital: What Happened? "Zombie Software" Blamed for Knight Capital Trading Snafu A new algorithmic trading program had just been installed, and began operation on Aug 1. A dormant legacy program was somehow "inadvertently reactivated" Once activated, the dormant system started multiplying stock trades by one thousand Sent 4 million orders when attempting to fill just 212 customer orders Knight s staff looked through eight sets of software before determining what happened. April 18, 2017 SE 433: Lecture 4 13 of 101
Case Study Knight Capital: The Investigation and Findings SEC launched an investigation in Nov 2012. Findings: Code changes in 2005 introduced defects. Although the defective function was not meant to be used, it was kept in. New code deployed in late July 2012. The defective function was triggered under new rules. Unable to recognize when orders have been filled. Ignored system generated warning emails. Inadequate controls and procedures for code deployment and testing. Charges filed in Oct 2013 Knights Capital settled charges for $12 million April 18, 2017 SE 433: Lecture 4 14 of 101
Regression Test April 18, 2017 SE 433: Lecture 4 15 of 101
Software Evolution Change happens throughout the software development life cycle. Before and after delivery Change can happen to every aspect of software Changes can affect unchanged areas break code, introduce new bugs uncover previous unknown bugs reintroduce old bugs April 18, 2017 SE 433: Lecture 4 16 of 101
Regression Test Testing of a previously tested program following modification to ensure that new defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It should be performed whenever the software or its environment is changed. It applies to testing at all levels. April 18, 2017 SE 433: Lecture 4 17 of 101
Regression Test Keep a test suite Use the test suite after every change Compare output with previous tests Understand all changes If new tests are needed, add to the test suite. April 18, 2017 SE 433: Lecture 4 18 of 101
Test Driven Development (TDD) April 18, 2017 SE 433: Lecture 4 19 of 101
Test Early Testing should start as early as possible design test cases Test early has several advantages independence from design & code discover inconsistencies and incompleteness of the specifications serve as a compendium of the specifications April 18, 2017 SE 433: Lecture 4 20 of 101
Test Driven Development Test driven development (TDD) is one of the corner stones of agile software development Agile, iterative, incremental development Small iterations, a few units Verification and validation carried out for each iteration. Design & implement test cases before implementing the functionality Run automated regression test of whole system continuously April 18, 2017 SE 433: Lecture 4 21 of 101
Process of Test Driven Development Tests should be written first (before any code) Execute all test cases => all fail Implement some functions Execute all test cases => some pass Repeat implement and re-execute all test cases Until all test cases => pass Refactoring, to improve design & implementation re-execute all test cases => all pass Every time changes are made re-execute all test cases => all pass April 18, 2017 SE 433: Lecture 4 22 of 101
Black Box Testing April 18, 2017 SE 433: Lecture 4 23 of 101
Black Box View The system is viewed as a black box Provide some input Observe the output April 18, 2017 SE 433: Lecture 4 24 of 101
Functional Testing: A.k.a.: Black Box Testing Derive test cases from the functional specifications functional refers to the source of information not to what is tested Also known as: specification-based testing (from specifications) black-box testing (no view of the code) Functional specification = description of intended program behavior either formal or informal April 18, 2017 SE 433: Lecture 4 25 of 101
Systematic vs. Random Testing Random (uniform) testing Pick possible inputs randomly and uniformly Avoids designer bias But treats all inputs as equally valuable Systematic (non-uniform) testing Select inputs that are especially valuable Choose representatives Black box testing is systematic testing April 18, 2017 SE 433: Lecture 4 26 of 101
Why Not Random Testing? Non-uniform distribution of defects Example: Program: solve a quadratic equation: a x2 + b x + c = 0 Defect: incomplete implementation logic does not properly handle special cases: b2 - 4ac = 0 and a = 0 Failing values are sparse in the input space needles in a very big haystack. Random sampling is unlikely to choose a=0.0 and b=0.0 April 18, 2017 SE 433: Lecture 4 27 of 101
Systematic Partition of Input Space Failures are sparse in the space of possible inputs ... ... but dense in some parts of the space Failure No failure The space of possible input values If we systematically test some cases from each part, we will include the dense parts Functional testing is one way of drawing lines to isolate regions with likely failures April 18, 2017 SE 433: Lecture 4 28 of 101
The Partition Principle Exploit knowledge in problem domain to choose samples for testing Focus on special or trouble-prone regions of the input space Failures are sparse in the whole input space ... ... but we may find regions in which they are dense (Quasi*-) Partition testing Separates the input space into classes whose union is the entire space *Quasi because the classes may overlap April 18, 2017 SE 433: Lecture 4 29 of 101
The Partition Principle Desirable case for partitioning Input values that lead to failures are dense (easy to find) in some classes of input space Sampling each class in the quasi-partition by selecting at least one input value that leads to a failure, revealing the defect Seldom guaranteed, depend on experience- based heuristics April 18, 2017 SE 433: Lecture 4 30 of 101
Black Box Testing Exploiting the functional specification Uses the specification to partition the input space e.g., specification of roots program suggests division between cases with zero, one, and two real roots Test each partition, and boundaries between partitions No guarantees, but experience suggests failures often lie at the boundaries (as in the roots program) April 18, 2017 SE 433: Lecture 4 31 of 101
Why Black Box Testing? Early. can start before code is written Effective. find some classes of defects, e.g., missing logic Widely applicable any description of program behavior as spec at any level of granularity, from module to system testing. Economical less expensive than structural (white box) testing The base-line technique for designing test cases April 18, 2017 SE 433: Lecture 4 32 of 101
Early Black Box Testing Program code is not necessary Only a description of intended behavior is needed Even incomplete and informal specifications can be used Although precise, complete specifications lead to better test suites Early test design has side benefits Often reveals ambiguities and inconsistency in spec Useful for assessing testability And improving test schedule and budget by improving spec Useful explanation of specification or in the extreme case (as in XP), test cases are the spec April 18, 2017 SE 433: Lecture 4 33 of 101
Functional versus Structural: Classes of faults Different testing strategies (functional, structural, fault- based, model-based) are most effective for different classes of faults Functional testing is best for missing logic faults A common problem: Some program logic was simply forgotten Structural (code-based) testing will never focus on code that isn t there! April 18, 2017 SE 433: Lecture 4 34 of 101
Functional vs. Structural Test Functional test is applicable in testing at all granularity levels: Unit test (from module interface spec) Integration test (from API or subsystem spec) System test (from system requirements spec) Regression test (from system requirements + bug history) Structural test is applicable in testing relatively small parts of a system: Unit test April 18, 2017 SE 433: Lecture 4 35 of 101
Steps: From specification to test cases 1. Decompose the specification If the specification is large, break it into independently testable features to be considered in testing 2. Select representatives Representative values of each input, or Representative behaviors of a model Often simple input/output transformations don t describe a system. We use models in program specification, in program design, and in test design 3. Form test specifications Typically: combinations of input values, or model behaviors 4. Produce and execute actual tests April 18, 2017 SE 433: Lecture 4 36 of 101
From specification to test cases Functional Specifications Independently Testable Feature Representative Values Model Test Case Test Cases Specifications April 18, 2017 SE 433: Lecture 4 37 of 101
An Example: Postal Code Lookup Input: ZIP code (5-digit US Postal code) Output: List of cities What are some representative values to test? April 18, 2017 SE 433: Lecture 4 38 of 101
Example: Representative Values Simple example with one input, one output Correct zip code With 0, 1, or many cities Malformed zip code Empty; 1-4 characters; 6 characters; very long Non-digit characters Non-character data Note prevalence of boundary values (0 cities, 6 characters) and error cases April 18, 2017 SE 433: Lecture 4 39 of 101
Summary Functional testing, i.e., generation of test cases from specifications is a valuable and flexible approach to software testing Applicable from very early system specs right through module specifications (quasi-)Partition testing suggests dividing the input space into (quasi-)equivalent classes Systematic testing is intentionally non-uniform to address special cases, error conditions, and other small places Dividing a big haystack into small, hopefully uniform piles where the needles might be concentrated April 18, 2017 SE 433: Lecture 4 40 of 101
Basic Techniques of Black Box Testing April 18, 2017 SE 433: Lecture 4 41 of 101
Single Defect Assumption Failures are rarely the result of the simultaneous effects of two (or more) defects. April 18, 2017 SE 433: Lecture 4 42 of 101
Functional Testing Concepts The four key concepts in functional testing are: Precisely identify the domain of each input and each output variable Select values from the data domain of each variable having important properties Consider combinations of special values from different input domains to design test cases Consider input values such that the program under test produces special values from the domains of the output variables April 18, 2017 SE 433: Lecture 4 43 of 101
Developing Test Cases Consider: Test cases for input box accepting numbers between 1 and 1000 If you are testing for an input box accepting numbers from 1 to 1000 then there is no use in writing thousand test cases for all 1000 valid input numbers plus other test cases for invalid data. Using equivalence partitioning method, above test cases can be divided into three sets of input data called as classes. Each test case is a representative of respective class. We can divide our test cases into three equivalence classes of some valid and invalid inputs. April 18, 2017 SE 433: Lecture 4 44 of 101
Developing Test Cases 1. One input data class with all valid inputs. Pick a single value from range 1 to 1000 as a valid test case. If you select other values between 1 and 1000 then result is going to be same. So one test case for valid input data should be sufficient. 2. Input data class with all values below lower limit. I.e. any value below 1, as a invalid input data test case. 3. Input data with any value greater than 1000 to represent third invalid input class. So using equivalence partitioning you have categorized all possible test cases into three classes. Test cases with other values from any class should give you the same result. April 18, 2017 SE 433: Lecture 4 45 of 101
Equivalence Classes Equivalence classes are the sets of values in a (quasi-) partition of the input, or output domain Values in an equivalence class cause the program to behave in a similar way: failure or success Motivation: gain a sense of complete testing and avoid redundancy First determine the boundaries then determine the equivalencies April 18, 2017 SE 433: Lecture 4 46 of 101
Determining Equivalence Classes Look for ranges of numbers or values Look for memberships in groups Some may be based on time Include invalid inputs Look for internal boundaries Don t worry if they overlap with each other better to be redundant than to miss something However, test cases will easily overlap with boundary value test cases April 18, 2017 SE 433: Lecture 4 47 of 101
Selecting Data Points Determining equivalence classes for each input variable or field Single input variable Normal test Select one data point from each valid equivalence class Robustness test Include invalid equivalence class April 18, 2017 SE 433: Lecture 4 48 of 101
Selecting Data Points Multiple input variables Weak normal test: Select one data point from each valid equivalence class Strong normal test: Select one data point from each combination of (the cross product of) the valid equivalence classes Weak/strong robustness test: Include invalid equivalence classes How many test cases do we need? April 18, 2017 SE 433: Lecture 4 49 of 101
Example of Selecting Data Points Suppose a program has 2 input variables, x and y Suppose x can lie in 3 valid equivalence classes: a x < b b x < c c x d Suppose y can lie in 2 valid equivalence classes: e y < f f y g April 18, 2017 SE 433: Lecture 4 50 of 101
