Testing and maintenance
| English | Chinese | Pinyin |
|---|---|---|
| run-time error | 运行时错误 | yùn xíng shí cuò wù |
| syntax error | 语法错误 | yǔ fǎ cuò wù |
| logic error | 逻辑错误 | luó jí cuò wù |
| dry run | 手工跟踪 | shǒu gōng gēn zōng |
| walkthrough | 走查 | zǒu chá |
| white-box testing | 白盒测试 | bái hé cè shì |
| black-box testing | 黑盒测试 | hēi hé cè shì |
| integration testing | 集成测试 | jí chéng cè shì |
| stub | 桩 | zhuāng |
| alpha testing | α测试 | α cè shì |
| beta testing | β测试 | β cè shì |
| acceptance testing | 验收测试 | yàn shōu cè shì |
| test strategy | 测试策略 | cè shì cè lüè |
| test plan | 测试计划 | cè shì jì huà |
| normal data | 正常数据 | zhèng cháng shù jù |
| abnormal data | 异常数据 | yì cháng shù jù |
| extreme data | 极端数据 | jí duān shù jù |
| boundary data | 边界数据 | biān jiè shù jù |
| corrective maintenance | 纠正性维护 | jiū zhèng xìng wéi hù |
| adaptive maintenance | 适应性维护 | shì yìng xìng wéi hù |
| perfective maintenance | 完善性维护 | wán shàn xìng wéi hù |
| regression testing | 回归测试 | huí guī cè shì |
Thirty-seven seconds
- On 4 June 1996 the first Ariane 5 rocket lifted off from French Guiana. Thirty-seven seconds later it veered off course and destroyed itself. The payload was four satellites worth $370 million.
- The cause was one line of code, reused from Ariane 4, that converted a 64-bit number into a 16-bit integer. Ariane 5 flew faster, the number was bigger, and the conversion overflowed: a run-time error 运行时错误 in a routine that was not even needed after lift-off.
- The code had never been tested with Ariane 5's flight data. Nobody had chosen test data at the extremes of the new rocket's range.
- This lesson is the three kinds of error, the methods of testing, how to choose test data, and how a program is kept alive after release.
Three kinds of error
- A syntax error 语法错误 breaks the grammar of the language: a missing bracket, a misspelled keyword. It is found at translation, so the program will not run until it is fixed.
- A run-time error happens while the program runs: division by zero, a file that does not exist, an array index out of range. The program crashes or raises an exception; the fix is a check before the risky operation.
- A logic error 逻辑错误 lets the program run and produce wrong results:
+for-, an off-by-one loop, conditions in the wrong order. Nothing flags it; only testing and tracing reveal it.
Found at translation, found at run time, found only in the output
Match each kind of error to how it shows up.
Syntax = caught at translation; logic = wrong output; run-time = crash while running.
Worked example: find and correct the errors
Total ← 0
FOR i ← 1 TO Count
Total ← Total + Marks[i]
NEXT i
Average ← Total / Count
IF Average > 50 THEN
OUTPUT "Pass"
- Syntax: the
IFhas noENDIF. The translator rejects it. Run-time: ifCountis 0 the division fails; guard it withIF Count > 0 THEN. Logic: if a pass is 50 or more,> 50fails a student on exactly 50; it should be>= 50. - Name the type, say why it is that type, and give the correction. Three parts, three marks.
A program divides a total by the number of entries and crashes when the input file is empty. This is a:
Division by zero happens while the program runs and stops it. A guard such as IF Count > 0 fixes it.
Testing methods: reading the code
- A dry run 手工跟踪 traces the code on paper, writing each variable's value in a trace table after each line.
- A walkthrough 走查 is a team review: the programmer explains the code line by line while colleagues look for faults.
- White-box testing 白盒测试 designs tests from the code's internal structure, so that every statement, branch and loop is exercised. Black-box testing 黑盒测试 designs tests from the specification only: feed inputs, compare the outputs with what was expected, without looking at the code.
Outside looking in, or inside looking at every path
Designing test cases from the specification only (inputs and expected outputs), ignoring the code inside, is called ______-box testing.
Black-box tests from the spec; white-box uses the code's internal structure to cover statements and branches.
Testing methods: assembling and releasing
- Integration testing 集成测试 combines modules that were tested separately and tests the interfaces between them. A stub 桩 stands in for a module that is not yet written, returning fixed values so the rest can be tested top-down.
- Alpha testing α测试 is done in-house by the developers before release. Beta testing β测试 gives a limited group of real users the program in their own environment.
- Acceptance testing 验收测试 is done by the customer, against the requirements, to decide whether the product is fit for purpose.
Software process lab
Classify development examples by the stage or tool they belong to.
Worked example: which method for which situation
- The reporting module is finished but the database module it calls does not exist yet: test it with a stub that returns fixed data.
- Two modules pass their own tests but fail when the output of one feeds the other: integration testing of the interface.
- The software is complete; the company wants faults found in real conditions before general release: beta testing by a limited group of users.
- The customer decides whether to pay: acceptance testing against the agreed requirements. Name the method and what it is for.
Match each situation to the testing method it needs.
Stub for a missing part, beta for real users, acceptance for the customer, integration for the joins.
Test strategy and test plan
- A test strategy 测试策略 is the high-level approach: which kinds of testing will be done, by whom, when, and what must pass before the next stage.
- A test plan 测试计划 is the detailed list of tests. For each: the purpose, the input data, the expected output, and a column for the actual output when the test is run.
- A test without an expected result is not a test. It only shows what the program did, not whether that was right.
A test plan lists, for each test, the input data and the expected output.
Purpose, input, expected output and a column for the actual output. The strategy is the high-level approach; the plan is the detailed list.
Choosing test data
- Normal data 正常数据: typical valid values inside the range, which should be accepted and processed correctly.
- Abnormal data 异常数据: values that should be rejected, out of range or the wrong type.
- Extreme data 极端数据: the largest and smallest values still accepted, at the edges of the range. Boundary data 边界数据: the pairs that straddle each edge, the accepted extreme and the rejected value just outside it, where off-by-one errors hide.
Inside, outside, and right on the line
For a field that accepts marks 0–100, which are the boundary test values?
Boundary data sits at the edges of the valid range (and just outside) — where off-by-one errors hide.
Worked example: test data for a mark from 0 to 100
| Kind | Data | Expected result |
|---|---|---|
| normal | 50, 75 |
accepted and processed |
| abnormal | -10, 200, "abc" |
rejected: out of range or wrong type |
| extreme | 0, 100 |
accepted: the smallest and largest valid values |
| boundary | -1 and 0, 100 and 101 |
-1 rejected, 0 accepted; 100 accepted, 101 rejected |
- Every row needs the expected result; a table of inputs alone scores half. The extremes are accepted:
-1is boundary, not extreme.
A field accepts marks from 0 to 100. Which values are extreme test data? Select all that apply.
Extreme values are the smallest and largest still accepted. -1 is rejected, so it is boundary or abnormal; 50 is normal.
Maintenance
- Most of a program's lifetime cost is spent after release. Corrective maintenance 纠正性维护 fixes faults found in use.
- Adaptive maintenance 适应性维护 keeps the program working in a changing environment: a new operating system, a new API, a change in the law.
- Perfective maintenance 完善性维护 improves a program that already works: faster performance, a new feature users asked for. A program may need all three throughout its life.
Fix it, keep it working, make it better
Corrective maintenance fixes faults, perfective maintenance improves features, and adaptive maintenance keeps the software working in a changed environment.
Three maintenance types: corrective (fix bugs), perfective (enhance), adaptive (new OS/hardware/rules).
A payroll program is changed because the tax law changed. Which kind of maintenance is this?
The program was not faulty and is not being improved; its environment changed. That is adaptive maintenance.
Amending an existing program
- Read the existing code until you understand the algorithm and the data flow. Find where the change belongs: which subroutine, which lines.
- Make the change as small as possible; do not rewrite working code. Update every related part: each caller of a changed parameter list, each routine that uses a changed data structure.
- Test the new behaviour and the old: regression testing 回归测试 checks that nothing that used to work has broken. Then document the change.
After changing a program, regression testing checks that:
Regression testing re-runs old tests to confirm existing behaviour still works after a change.
Put the steps of amending an existing program in order.
Understand, locate, change small, propagate, test everything, record. Skipping the regression test is how a fix breaks something else.
Marks that slip away
- Extreme is accepted; abnormal is rejected.
0and100are extreme;-1and101are boundary values on the rejected side. - A logic error does not crash the program. If it crashed, it was a run-time error.
- Alpha is in-house; beta is real users outside. A stub replaces a missing module; it is not a test method for finished code.
- A test plan row without an expected output earns nothing. Regression testing follows every change.
You've got it
- syntax errors stop translation · run-time errors crash a running program · logic errors run and give wrong output, found only by testing
- methods: dry run, walkthrough, white-box (from the code), black-box (from the specification), integration (interfaces, with stubs for missing modules), alpha (in-house), beta (real users), acceptance (the customer)
- test data: normal accepted, abnormal rejected, extreme the accepted edges, boundary either side of each edge; every test has an expected result
- maintenance: corrective fixes, adaptive keeps up with the environment, perfective improves; amend small, update callers, regression test