Add failure summary to ConsoleOutputRecorder

[tienquocbui]

Overview

This PR implements a failure summary section for ConsoleOutputRecorder to help users quickly locate and review test failures in large test suites.

Fixes #1355
Fixes rdar://168698184

---

Architectural Changes Implemented

1. State Management

• Uses existing HumanReadableOutputRecorder state with single Locked<Context>
• Fits naturally with existing Graph traversal logic
• No additional state in ConsoleOutputRecorder (stateless design preserved)

2. Data Structure

• Added lightweight TestData.IssueInfo struct with: sourceLocation, description, isKnown, severity
• Issues stored as array in TestData.issues (each issue has its own source location)
• Test metadata (displayName, testCaseArguments) stored once per test in TestData
• Existing issueCount dictionary preserved for efficient parallel tracking

3. New TestRunSummary Type

• Dedicated type for failure summary generation
• Breaks down logic into focused methods: header(), formatFailedTest(), fullyQualifiedName(), formatIssue()
• Collects failures from Graph and formats them independently
• fileprivate initializer uses HumanReadableOutputRecorder.Context types

4. API Improvements

• generateFailureSummary() returns Optional<String> (nil when no failures)
• Uses expandedDescription() for normal output, expandedDebugDescription() when verbose
• ConsoleOutputRecorder calls this at runEnded event with blank line spacing

5. Output Features

• Fully qualified suite/test paths (addresses Add failure summary section to console output for easier debugging #1355)
• Custom display names shown in quotes (e.g., "Custom Display Name")
• Parameterized test arguments displayed once per test (e.g., (value → 7))
• Each issue listed with indentation, description, and source location
• Visual separation with blank lines before and after summary

Example Output

Regular Test Failure:

✗  TestingTests/FailureSummaryDemoTests/MathTests/"Division fails"
  - (result: Swift.Int → 3) == (4: Swift.Int → 4)
    at TestingTests/FailureSummaryDemoTests.swift:26

Parameterized Test Failure (shows which argument failed):

✗  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
  (value → 7)
  - (value % 2: Swift.Int → 1) == (0: Swift.Int → 0)
    at TestingTests/FailureSummaryDemoTests.swift:72

Test with Custom Display Name:

✗  TestingTests/FailureSummaryDemoTests/ParameterizedTests/"This is a custom display name"
  - (value: Swift.Int → 42) == (100: Swift.Int → 100)
    at TestingTests/FailureSummaryDemoTests.swift:83

Complete Test Output

Click to expand full test run output

(base) buitienquoc@Kelvins-MacBook-Pro swift-testing % swift test --filter "FailureSummaryDemoTests" --disable-xctest
[1/1] Planning build
Building for debugging...
[257/257] Linking swift-testingPackageTests
Build complete! (39.83s)
􀟈  Test run started.
􀄵  Testing Library Version: 6.3-dev (d8b140d780dc2da - modified)
􀟈  Suite "Failure Summary Demo" started.
􀟈  Suite "Math Operations" started.
􀟈  Suite "String Operations" started.
􀟈  Test "Addition works correctly" started.
􀟈  Test "String comparison passes" started.
􀟈  Test "Multiple string failures" started.
􀟈  Test "String concatenation fails" started.
􀟈  Suite "Array Operations" started.
􀟈  Test "Division fails" started.
􀟈  Test "Array equality" started.
􀟈  Suite "Parameterized Tests" started.
􁁛  Test "Addition works correctly" passed after 0.001 seconds.
􁁛  Test "String comparison passes" passed after 0.001 seconds.
􁁛  Test "Array equality" passed after 0.001 seconds.
􀟈  Test "Array contains element" started.
􀟈  Test "This is a custom display name" started.
􀟈  Test "Check even numbers" started.
􀟈  Test "String length validation" started.
􀟈  Test case passing 1 argument value → 4 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 6 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 7 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 10 to "Check even numbers" started.
􀟈  Test case passing 1 argument text → "hello" to "String length validation" started.
􀟈  Test case passing 1 argument text → "world" to "String length validation" started.
􀟈  Test case passing 1 argument text → "a" to "String length validation" started.
􀟈  Test case passing 1 argument value → 2 to "Check even numbers" started.
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:46:7: Expectation failed: "a" == "b"
􀢄  Test "String concatenation fails" recorded an issue at FailureSummaryDemoTests.swift:36:7: Expectation failed: (greeting + name → "HelloWorld") == "Hello World"
􀢄  Test "This is a custom display name" recorded an issue at FailureSummaryDemoTests.swift:83:7: Expectation failed: (value → 42) == 100
􀢄  Test "String length validation" recorded an issue with 1 argument text → "a" at FailureSummaryDemoTests.swift:77:7: Expectation failed: (text.count → 1) >= 5
􀢄  Test "Array contains element" recorded an issue at FailureSummaryDemoTests.swift:57:7: Expectation failed: (numbers → [1, 2, 3, 4, 5]).contains(10)
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:47:7: Expectation failed: ("hello".count → 5) == 10
􀢄  Test "Division fails" recorded an issue at FailureSummaryDemoTests.swift:26:7: Expectation failed: (result → 3) == 4
􀢄  Test "Check even numbers" recorded an issue with 1 argument value → 7 at FailureSummaryDemoTests.swift:72:7: Expectation failed: (value % 2 → 1) == 0
􀢄  Test "This is a custom display name" failed after 0.003 seconds with 1 issue.
􀢄  Test "Array contains element" failed after 0.004 seconds with 1 issue.
􀟈  Test case passing 1 argument text → "swift" to "String length validation" started.
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:48:7: Expectation failed: ("swift".uppercased() → "SWIFT") == "swift"
􀢄  Test "String concatenation fails" failed after 0.005 seconds with 1 issue.
􀢄  Suite "Array Operations" failed after 0.005 seconds with 1 issue.
􀢄  Test "Division fails" failed after 0.005 seconds with 1 issue.
􀢄  Test "String length validation" with 4 test cases failed after 0.004 seconds with 1 issue.
􀢄  Test "Check even numbers" with 5 test cases failed after 0.004 seconds with 1 issue.
􀢄  Suite "Math Operations" failed after 0.005 seconds with 1 issue.
􀢄  Test "Multiple string failures" failed after 0.005 seconds with 3 issues.
􀢄  Suite "Parameterized Tests" failed after 0.005 seconds with 3 issues.
􀢄  Suite "String Operations" failed after 0.005 seconds with 4 issues.
􀢄  Suite "Failure Summary Demo" failed after 0.007 seconds with 9 issues.
􀄵  /// Demo tests showcasing the refactored failure summary feature.

Test run had 7 failed tests with 9 issues:
􀢄  TestingTests/FailureSummaryDemoTests/MathTests/"Division fails"
  - (result: Swift.Int → 3) == (4: Swift.Int → 4)
    at TestingTests/FailureSummaryDemoTests.swift:26
􀢄  TestingTests/FailureSummaryDemoTests/ArrayTests/"Array contains element"
  - (numbers: Swift.Array<Swift.Int> → [1, 2, 3, 4, 5]).contains(10: Swift.Int → 10)
    at TestingTests/FailureSummaryDemoTests.swift:57
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
  (value → 7)
  - (value % 2: Swift.Int → 1) == (0: Swift.Int → 0)
    at TestingTests/FailureSummaryDemoTests.swift:72
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testStringLength(text:)/"String length validation"
  (text → "a")
  - (text.count: Swift.Int → 1) >= (5: Swift.Int → 5)
    at TestingTests/FailureSummaryDemoTests.swift:77
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/"This is a custom display name"
  - (value: Swift.Int → 42) == (100: Swift.Int → 100)
    at TestingTests/FailureSummaryDemoTests.swift:83
􀢄  TestingTests/FailureSummaryDemoTests/StringTests/"Multiple string failures"
  - ("a": Swift.String → "a") == ("b": Swift.String → "b")
    at TestingTests/FailureSummaryDemoTests.swift:46
  - ("hello".count: Swift.Int → 5) == (10: Swift.Int → 10)
    at TestingTests/FailureSummaryDemoTests.swift:47
  - ("swift".uppercased(): Swift.String → "SWIFT") == ("swift": Swift.String → "swift")
    at TestingTests/FailureSummaryDemoTests.swift:48
􀢄  TestingTests/FailureSummaryDemoTests/StringTests/"String concatenation fails"
  - (greeting + name: Swift.String → "HelloWorld") == ("Hello World": Swift.String → "Hello World")
    at TestingTests/FailureSummaryDemoTests.swift:36

􀢄  Test run with 10 tests in 5 suites failed after 0.007 seconds with 9 issues.
(base) buitienquoc@Kelvins-MacBook-Pro swift-testing % 

[grynspan]

Would it make sense to factor the new code into a dedicated TestRunSummary type? (I'm not saying you must, just asking.)

[tienquocbui]

@grynspan thanks for the suggestion! I discussed that with Stuart and I've refactored the failure summary logic into a dedicated TestRunSummary type. Would you mind taking another look when you get a chance? Thanks!

[stmontgomery]

We met about this offline, and I made a handful of commits of my own which I've just pushed to the PR branch after discussion. Beyond that, there is one problem in how the fully-qualified name is formed for parameterized test functions, and cases within them, in the failure summary section, and @tienquocbui it sounds like you will address that (thanks!)

[tienquocbui]

@stmontgomery I've addressed the parameterized test display issue. Here's what changed:

Before:

✗ FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
(value → 7)
   - (value % 2 → 1) == 0
        at FailureSummaryDemoTests.swift:70:13

After:

✗ FailureSummaryDemoTests/ParameterizedTests/"Check even numbers"
(value → 7)
   - (value % 2 → 1) == 0
        at FailureSummaryDemoTests.swift:70:13

[stmontgomery]

This could potentially be a suite, for example if the issue was recorded by a TestScoping trait applied to a suite, so ideally we would conditionalize the word "Test" to use "Suite" instead in that scenario