Testing and maintenance
Introduced| English | Meaning |
|---|---|
| run-time error/rʌn taɪm ˈerə/ | A problem that occurs while a program executes and prevents an operation from completing normally. |
| syntax error/ˈsɪntæks ˈerə/ | A mistake that breaks the grammatical rules of a programming language. |
| logic error/ˈlɒdʒɪk ˈerə/ | A mistake in an algorithm or program that produces an incorrect result despite valid syntax. |
| dry run/draɪ rʌn/ | Tracing an algorithm by hand using example inputs and recording changes in its values. |
| walkthrough/ˈwɔːkθruː/ | A team review: the programmer explains the code line by line while colleagues look for faults. |
| white-box testing/waɪt bɒks ˈtestɪŋ/ | white-box testing |
| black-box testing/blæk bɒks ˈtestɪŋ/ | black-box testing |
| integration testing/ˌɪntɪˈɡreɪʃn ˈtestɪŋ/ | integration testing |
| stub/stʌb/ | stub |
| alpha testing/ˈælfə ˈtestɪŋ/ | Testing of an early software version within the developing organisation before wider external testing or release. |
| beta testing/ˈbiːtə ˈtestɪŋ/ | Testing of a pre-release software version by selected external users in realistic conditions. |
| acceptance testing/əkˈseptəns ˈtestɪŋ/ | acceptance testing |
| test strategy/test ˈstrætədʒi/ | The high-level approach: which kinds of testing will be done, by whom, when, and what must pass before the next stage. |
| test plan/test plæn/ | The detailed list of tests. |
| normal data/ˈnɔːml ˈdeɪtə/ | Typical valid input values used to check that a program works as expected. |
| abnormal data/əbˈnɔːml ˈdeɪtə/ | Invalid or unsuitable input values used to check a program's validation and error handling. |
| extreme data/ekˈstriːm ˈdeɪtə/ | Valid input values at the boundaries of the permitted range, used to test boundary behaviour. |
| boundary data/ˈbaʊndəri ˈdeɪtə/ | boundary data |
| corrective maintenance/kəˈrektɪv ˈmeɪntənəns/ | corrective maintenance |
| adaptive maintenance/əˈdæptɪv ˈmeɪntənəns/ | adaptive maintenance |
| perfective maintenance/pəˈfektɪv ˈmeɪntənəns/ | perfective maintenance |
| regression testing/rɪˈɡreʃn ˈtestɪŋ/ | Repeating checks after a change to detect whether previously working behaviour has been broken. |
Thirty-seven seconds
- On 4 June 1996 the first Ariane 5 rocket lifted off from French Guiana. Thirty-seven seconds later it veered off course and destroyed itself. The payload was four satellites worth $370 million.
- The cause was one line of code, reused from Ariane 4, that converted a 64-bit number into a 16-bit integer. Ariane 5 flew faster, the number was bigger, and the conversion overflowed: a run-time error 运行时错误 in a routine that was not even needed after lift-off.
- The code had never been tested with Ariane 5's flight data. Nobody had chosen test data at the extremes of the new rocket's range.
- This lesson is the three kinds of error, the methods of testing, how to choose test data, and how a program is kept alive after release.
Three kinds of error
- A syntax error 语法错误 breaks the grammar of the language: a missing bracket, a misspelled keyword. It is found at translation, so the program will not run until it is fixed.
- A run-time error happens while the program runs: division by zero, a file that does not exist, an array index out of range. The program crashes or raises an exception; the fix is a check before the risky operation.
- A logic error 逻辑错误 lets the program run and produce wrong results:
+for-, an off-by-one loop, conditions in the wrong order. Nothing flags it; only testing and tracing reveal it.
Found at translation, found at run time, found only in the output
Match each kind of error to how it shows up.
Difficulty 2/5 · 140 coins + a small bonus on first completion today
Syntax = caught at translation; logic = wrong output; run-time = crash while running.
Worked example: find and correct the errors
Total ← 0
FOR i ← 1 TO Count
Total ← Total + Marks[i]
NEXT i
Average ← Total / Count
IF Average > 50 THEN
OUTPUT "Pass"
- Syntax: the
IFhas noENDIF. The translator rejects it. Run-time: ifCountis 0 the division fails; guard it withIF Count > 0 THEN. Logic: if a pass is 50 or more,> 50fails a student on exactly 50; it should be>= 50. - Name the type, say why it is that type, and give the correction. Three parts, three marks.
A program divides a total by the number of entries and crashes when the input file is empty. This is a:
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Division by zero happens while the program runs and stops it. A guard such as IF Count > 0 fixes it.
Testing methods: reading the code
- A dry run 手工跟踪 traces the code on paper, writing each variable's value in a trace table after each line.
- A walkthrough 走查 is a team review: the programmer explains the code line by line while colleagues look for faults.
- White-box testing 白盒测试 designs tests from the code's internal structure, so that every statement, branch and loop is exercised. Black-box testing 黑盒测试 designs tests from the specification only: feed inputs, compare the outputs with what was expected, without looking at the code.
Outside looking in, or inside looking at every path
Designing test cases from the specification only (inputs and expected outputs), ignoring the code inside, is called ______-box testing.
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Black-box tests from the spec; white-box uses the code's internal structure to cover statements and branches.
Testing methods: assembling and releasing
- Integration testing 集成测试 combines modules that were tested separately and tests the interfaces between them. A stub 桩 stands in for a module that is not yet written, returning fixed values so the rest can be tested top-down.
- Alpha testing α测试 is done in-house by the developers before release. Beta testing β测试 gives a limited group of real users the program in their own environment.
- Acceptance testing 验收测试 is done by the customer, against the requirements, to decide whether the product is fit for purpose.
Software process lab
Classify development examples by the stage or tool they belong to.
Worked example: which method for which situation
- The reporting module is finished but the database module it calls does not exist yet: test it with a stub that returns fixed data.
- Two modules pass their own tests but fail when the output of one feeds the other: integration testing of the interface.
- The software is complete; the company wants faults found in real conditions before general release: beta testing by a limited group of users.
- The customer decides whether to pay: acceptance testing against the agreed requirements. Name the method and what it is for.
Match each situation to the testing method it needs.
Difficulty 2/5 · 140 coins + a small bonus on first completion today
Stub for a missing part, beta for real users, acceptance for the customer, integration for the joins.
Test strategy and test plan
- A test strategy 测试策略 is the high-level approach: which kinds of testing will be done, by whom, when, and what must pass before the next stage.
- A test plan 测试计划 is the detailed list of tests. For each: the purpose, the input data, the expected output, and a column for the actual output when the test is run.
- A test without an expected result is not a test. It only shows what the program did, not whether that was right.
A test plan lists, for each test, the input data and the expected output.
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Purpose, input, expected output and a column for the actual output. The strategy is the high-level approach; the plan is the detailed list.
Choosing test data
- Normal data 正常数据: typical valid values inside the range, which should be accepted and processed correctly.
- Abnormal data 异常数据: values that should be rejected, out of range or the wrong type.
- Extreme data 极端数据: the largest and smallest values still accepted, at the edges of the range. Boundary data 边界数据: the pairs that straddle each edge, the accepted extreme and the rejected value just outside it, where off-by-one errors hide.
Inside, outside, and right on the line
For a field that accepts marks 0–100, which are the boundary test values?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Boundary data sits at the edges of the valid range (and just outside) — where off-by-one errors hide.
Worked example: test data for a mark from 0 to 100
| Kind | Data | Expected result |
|---|---|---|
| normal | 50, 75 |
accepted and processed |
| abnormal | -10, 200, "abc" |
rejected: out of range or wrong type |
| extreme | 0, 100 |
accepted: the smallest and largest valid values |
| boundary | -1 and 0, 100 and 101 |
-1 rejected, 0 accepted; 100 accepted, 101 rejected |
- Every row needs the expected result; a table of inputs alone scores half. The extremes are accepted:
-1is boundary, not extreme.
A field accepts marks from 0 to 100. Which values are extreme test data? Select all that apply.
Difficulty 2/5 · 140 coins + a small bonus on first completion today
Extreme values are the smallest and largest still accepted. -1 is rejected, so it is boundary or abnormal; 50 is normal.
Maintenance
- Most of a program's lifetime cost is spent after release. Corrective maintenance 纠正性维护 fixes faults found in use.
- Adaptive maintenance 适应性维护 keeps the program working in a changing environment: a new operating system, a new API, a change in the law.
- Perfective maintenance 完善性维护 improves a program that already works: faster performance, a new feature users asked for. A program may need all three throughout its life.
Fix it, keep it working, make it better
Corrective maintenance fixes faults, perfective maintenance improves features, and adaptive maintenance keeps the software working in a changed environment.
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Three maintenance types: corrective (fix bugs), perfective (enhance), adaptive (new OS/hardware/rules).
A payroll program is changed because the tax law changed. Which kind of maintenance is this?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
The program was not faulty and is not being improved; its environment changed. That is adaptive maintenance.
Amending an existing program
- Read the existing code until you understand the algorithm and the data flow. Find where the change belongs: which subroutine, which lines.
- Make the change as small as possible; do not rewrite working code. Update every related part: each caller of a changed parameter list, each routine that uses a changed data structure.
- Test the new behaviour and the old: regression testing 回归测试 checks that nothing that used to work has broken. Then document the change.
After changing a program, regression testing checks that:
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Regression testing re-runs old tests to confirm existing behaviour still works after a change.
Put the steps of amending an existing program in order.
Difficulty 3/5 · 200 coins + a small bonus on first completion today
Understand, locate, change small, propagate, test everything, record. Skipping the regression test is how a fix breaks something else.
Marks that slip away
- Extreme is accepted; abnormal is rejected.
0and100are extreme;-1and101are boundary values on the rejected side. - A logic error does not crash the program. If it crashed, it was a run-time error.
- Alpha is in-house; beta is real users outside. A stub replaces a missing module; it is not a test method for finished code.
- A test plan row without an expected output earns nothing. Regression testing follows every change.
You've got it
- syntax errors stop translation · run-time errors crash a running program · logic errors run and give wrong output, found only by testing
- methods: dry run, walkthrough, white-box (from the code), black-box (from the specification), integration (interfaces, with stubs for missing modules), alpha (in-house), beta (real users), acceptance (the customer)
- test data: normal accepted, abnormal rejected, extreme the accepted edges, boundary either side of each edge; every test has an expected result
- maintenance: corrective fixes, adaptive keeps up with the environment, perfective improves; amend small, update callers, regression test