Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle tests marked as inconclusive #2405

Merged
merged 19 commits into from
Apr 10, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
8e1413e
Adds handling of It's marked as inconclusive
csandfeld Nov 19, 2023
5ec791c
Removes **DEPRECATED** from Inconclusive parameter help text
csandfeld Nov 19, 2023
7eb17cd
Adds deprecation notice to pester report when Set-ItResult -Pending i…
csandfeld Nov 19, 2023
f5626bf
Fix typo and add changes to fix failing P tests
csandfeld Nov 20, 2023
53acb19
Moves deprecation notice to end of Pester report
csandfeld Jan 19, 2024
79d8327
Fix InconclusiveCount typo
csandfeld Jan 19, 2024
6710d33
Adds missing increment of OwnInconclusiveCount
csandfeld Jan 19, 2024
b9237e8
Increment InconclusiveCount in Pester4Result object
csandfeld Jan 19, 2024
7c219c2
Update example in Set-ItResult comment based help
csandfeld Feb 3, 2024
9ba12d5
Test for Inconclusive tests and count in ResultObject
csandfeld Feb 3, 2024
a21a550
Adds inconclusive and skipped tests to nunit report 'ts' files
csandfeld Feb 3, 2024
ce3da79
Suppress output and use consistent configuration style
csandfeld Apr 5, 2024
b77ce8a
Adds skipped and inconclusive testcases to schema validation test
csandfeld Apr 5, 2024
82e93f3
Refactors nunit reports inconclusive count and skipped count tests
csandfeld Apr 5, 2024
6cf60c6
Removes unnecessary assertions
csandfeld Apr 5, 2024
066ca5d
Set Inconclusive result in NUnit reports
csandfeld Apr 7, 2024
9d80eb5
Add cheaper check for deprecation, and a test, remove INCONCLUSIVE fr…
nohwnd Apr 10, 2024
c156209
Format because
nohwnd Apr 10, 2024
d5a5e1a
Fix pending detection on failing run
nohwnd Apr 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions src/Pester.RSpec.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,9 @@ function Add-RSpecTestObjectProperties {
$TestObject.Result = if ($TestObject.Skipped) {
"Skipped"
}
elseif ($TestObject.Inconclusive) {
"Inconclusive"
}
elseif ($TestObject.Passed) {
"Passed"
}
Expand Down Expand Up @@ -160,6 +163,9 @@ function PostProcess-RspecTestRun ($TestRun) {
"Skipped" {
$null = $TestRun.Skipped.Add($t)
}
"Inconclusive" {
$null = $TestRun.Inconclusive.Add($t)
}
default { throw "Result $($t.Result) is not supported." }
}

Expand Down Expand Up @@ -236,6 +242,7 @@ function PostProcess-RspecTestRun ($TestRun) {
$TestRun.PassedCount = $TestRun.Passed.Count
$TestRun.FailedCount = $TestRun.Failed.Count
$TestRun.SkippedCount = $TestRun.Skipped.Count
$TestRun.InconclusiveCount = $TestRun.Inconclusive.Count
$TestRun.NotRunCount = $TestRun.NotRun.Count

$TestRun.TotalCount = $TestRun.Tests.Count
Expand Down
11 changes: 10 additions & 1 deletion src/Pester.Runtime.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -687,7 +687,12 @@ function Invoke-TestItem {
Write-PesterDebugMessage -Scope Skip "($path) Test is skipped."
nohwnd marked this conversation as resolved.
Show resolved Hide resolved
}
$Test.Passed = $true
$Test.Skipped = $true
if ('PesterTestInconclusive' -eq $Result.ErrorRecord.FullyQualifiedErrorId) {
$Test.Inconclusive = $true
}
else {
$Test.Skipped = $true
}
}
else {
$Test.Passed = $result.Success
Expand Down Expand Up @@ -2194,6 +2199,7 @@ function PostProcess-ExecutedBlock {
$b.OwnFailedCount = 0
$b.OwnPassedCount = 0
$b.OwnSkippedCount = 0
$b.OwnInconclusiveCount = 0
$b.OwnNotRunCount = 0

$testDuration = [TimeSpan]::Zero
Expand Down Expand Up @@ -2245,6 +2251,7 @@ function PostProcess-ExecutedBlock {
$b.FailedCount = $b.OwnFailedCount
$b.PassedCount = $b.OwnPassedCount
$b.SkippedCount = $b.OwnSkippedCount
$b.InconclusiveCount = $b.OwnInconclusiveCount
$b.NotRunCount = $b.OwnNotRunCount
}
else {
Expand All @@ -2266,6 +2273,7 @@ function PostProcess-ExecutedBlock {
$b.PassedCount += $child.PassedCount
$b.FailedCount += $child.FailedCount
$b.SkippedCount += $child.SkippedCount
$b.InconclusiveCount += $child.OwnInconclusiveCount
fflaten marked this conversation as resolved.
Show resolved Hide resolved
$b.NotRunCount += $child.NotRunCount
}

Expand All @@ -2274,6 +2282,7 @@ function PostProcess-ExecutedBlock {
$b.PassedCount += $b.OwnPassedCount
$b.FailedCount += $b.OwnFailedCount
$b.SkippedCount += $b.OwnSkippedCount
$b.InconclusiveCount += $b.OwnInconclusiveCount
$b.NotRunCount += $b.OwnNotRunCount

$b.Passed = -not ($thisBlockFailed -or $anyTestFailed -or $anyChildBlockFailed)
Expand Down
2 changes: 2 additions & 0 deletions src/csharp/Pester/Container.cs
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ public static Container CreateFromBlock (Block block) {
FailedCount = block.FailedCount,
PassedCount = block.PassedCount,
SkippedCount = block.SkippedCount,
InconclusiveCount = block.InconclusiveCount,
NotRunCount = block.NotRunCount,
TotalCount = block.TotalCount,
ErrorRecord = block.ErrorRecord ?? new List<object>(),
Expand Down Expand Up @@ -56,6 +57,7 @@ public static Container CreateFromFile(FileInfo file)
public int FailedCount { get; set; }
public int PassedCount { get; set; }
public int SkippedCount { get; set; }
public int InconclusiveCount { get; set; }
public int NotRunCount { get; set; }
public int TotalCount { get; set; }
public List<object> ErrorRecord { get; set; } = new List<object>();
Expand Down
2 changes: 2 additions & 0 deletions src/csharp/Pester/Run.cs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ public static Run Create()

public int PassedCount { get; set; }
public int SkippedCount { get; set; }
public int InconclusiveCount { get; set; }
public int NotRunCount { get; set; }
public int TotalCount { get; set; }

Expand Down Expand Up @@ -47,6 +48,7 @@ public static Run Create()

public List<Test> Passed { get; set; } = new List<Test>();
public List<Test> Skipped { get; set; } = new List<Test>();
public List<Test> Inconclusive { get; set; } = new List<Test>();
public List<Test> NotRun { get; set; } = new List<Test>();
public List<Test> Tests { get; set; } = new List<Test>();

Expand Down
1 change: 1 addition & 0 deletions src/csharp/Pester/Test.cs
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ public Test()
public DateTime? ExecutedAt { get; set; }
public bool Passed { get; set; }
public bool Skipped { get; set; }
public bool Inconclusive { get; set; }

public TimeSpan UserDuration { get; set; }
public TimeSpan FrameworkDuration { get; set; }
Expand Down
1 change: 1 addition & 0 deletions src/csharp/Pester/ToStringConverter.cs
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ static string ResultToString(string result)
"Passed" => "[+]",
"Failed" => "[-]",
"Skipped" => "[!]",
"Inconclusive" => "[?]",
"NotRun" => "[ ]",
_ => "[ERR]",
};
Expand Down
2 changes: 1 addition & 1 deletion src/en-US/about_Pester.help.txt
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ LONG DESCRIPTION
The output ends with a summary of the test results.

Tests completed in 3.47s
Passed: 20 Failed: 1 Skipped: 0 Pending: 0 Inconclusive: 0
Tests Passed: 20, Failed: 1, Skipped: 0, Inconclusive: 0, NotRun: 0

However, because Pester uses Write-Host, it does not write to the output
stream (stdout), so there are no output objects to save in a variable or
Expand Down
2 changes: 1 addition & 1 deletion src/functions/It.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
In addition to using your own logic to test expectations and throw exceptions,
you may also use Pester's Should command to perform assertions in plain language.

You can intentionally mark It block result as inconclusive by using Set-TestInconclusive
You can intentionally mark It block result as inconclusive by using Set-ItResult -Inconclusive
command as the first tested statement in the It block.

.PARAMETER Name
Expand Down
22 changes: 14 additions & 8 deletions src/functions/Output.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ $script:ReportStrings = DATA {

TestsPassed = 'Tests Passed: {0}, '
TestsFailed = 'Failed: {0}, '
TestsSkipped = 'Skipped: {0} '
TestsSkipped = 'Skipped: {0}, '
TestsPending = 'Pending: {0}, '
TestsInconclusive = 'Inconclusive: {0}, '
TestsNotRun = 'NotRun: {0}'
Expand Down Expand Up @@ -61,6 +61,7 @@ $script:ReportTheme = DATA {
Discovery = 'Magenta'
Container = 'Magenta'
BlockFail = 'Red'
Warning = 'Yellow'
}
}

Expand Down Expand Up @@ -309,6 +310,10 @@ function Write-PesterReport {
)
# if(-not ($PesterState.Show | Has-Flag Summary)) { return }

if ($RunResult.Tests.ErrorRecord.FullyQualifiedErrorID -contains 'PesterTestPending') {
Write-PesterHostMessage '**DEPRECATED**: The Set-ItResult -Pending parameter is deprecated. It will be removed in a future version of Pester.' -ForegroundColor $ReportTheme.Warning
}

fflaten marked this conversation as resolved.
Show resolved Hide resolved
Write-PesterHostMessage ($ReportStrings.Timing -f (Get-HumanTime ($RunResult.Duration))) -Foreground $ReportTheme.Foreground

$Success, $Failure = if ($RunResult.FailedCount -gt 0) {
Expand Down Expand Up @@ -345,12 +350,12 @@ function Write-PesterReport {
# else {
# $ReportTheme.Information
# }
# $Inconclusive = if ($RunResult.InconclusiveCount -gt 0) {
# $ReportTheme.Inconclusive
# }
# else {
# $ReportTheme.Information
# }
$Inconclusive = if ($RunResult.InconclusiveCount -gt 0) {
$ReportTheme.Inconclusive
}
else {
$ReportTheme.Information
}

# Try {
# $PesterStatePassedScenariosCount = $PesterState.PassedScenarios.Count
Expand All @@ -374,6 +379,7 @@ function Write-PesterReport {
Write-PesterHostMessage ($ReportStrings.TestsPassed -f $RunResult.PassedCount) -Foreground $Success -NoNewLine
Write-PesterHostMessage ($ReportStrings.TestsFailed -f $RunResult.FailedCount) -Foreground $Failure -NoNewLine
Write-PesterHostMessage ($ReportStrings.TestsSkipped -f $RunResult.SkippedCount) -Foreground $Skipped -NoNewLine
Write-PesterHostMessage ($ReportStrings.TestsInconclusive -f $RunResult.InconclusiveCount) -Foreground $Inconclusive -NoNewLine
Write-PesterHostMessage ($ReportStrings.TestsTotal -f $RunResult.TotalCount) -Foreground $Total -NoNewLine
Write-PesterHostMessage ($ReportStrings.TestsNotRun -f $RunResult.NotRunCount) -Foreground $NotRun

Expand Down Expand Up @@ -843,7 +849,7 @@ function Get-WriteScreenPlugin ($Verbosity) {
if ($PesterPreference.Output.Verbosity.Value -in 'Detailed', 'Diagnostic') {
$because = if ($_test.FailureMessage) { ", because $($_test.FailureMessage)" } else { $null }
Write-PesterHostMessage -ForegroundColor $ReportTheme.Inconclusive "$margin[?] $out" -NoNewLine
Write-PesterHostMessage -ForegroundColor $ReportTheme.Inconclusive ", is inconclusive$because" -NoNewLine
Write-PesterHostMessage -ForegroundColor $ReportTheme.Inconclusive "$because" -NoNewLine
Write-PesterHostMessage -ForegroundColor $ReportTheme.InconclusiveTime " $humanTime"
}

Expand Down
16 changes: 11 additions & 5 deletions src/functions/Set-ItResult.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
backwards compatible

.PARAMETER Inconclusive
**DEPRECATED** Sets the test result to inconclusive. Cannot be used at the same time as -Pending or -Skipped
Sets the test result to inconclusive. Cannot be used at the same time as -Pending or -Skipped

.PARAMETER Pending
**DEPRECATED** Sets the test result to pending. Cannot be used at the same time as -Inconclusive or -Skipped
Expand Down Expand Up @@ -58,13 +58,9 @@

$result = $PSCmdlet.ParameterSetName

[String]$Message = "is skipped"
if ($Result -ne 'Skipped') {
[String]$Because = if ($Because) { $Result.ToUpper(), $Because -join ': ' } else { $Result.ToUpper() }
}
nohwnd marked this conversation as resolved.
Show resolved Hide resolved
if ($Because) {
[String]$Message += ", because $Because"
}

switch ($null) {
$File {
Expand All @@ -81,15 +77,25 @@
switch ($result) {
'Inconclusive' {
[String]$errorId = 'PesterTestInconclusive'
[String]$message = "is inconclusive"
break
}
'Pending' {
[String]$errorId = 'PesterTestPending'
[String]$message = "is pending"
break
}
'Skipped' {
[String]$errorId = 'PesterTestSkipped'
[String]$message = "is skipped"
break
}
}

if ($Because) {
[String]$message += ", because $Because"
}

throw [Pester.Factory]::CreateErrorRecord(
$errorId, #string errorId
$Message, #string message
Expand Down
2 changes: 1 addition & 1 deletion src/functions/TestResults.NUnit25.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ function Write-NUnitTestResultAttributes {
$XmlWriter.WriteAttributeString('errors', '0')
$XmlWriter.WriteAttributeString('failures', $Result.FailedCount)
$XmlWriter.WriteAttributeString('not-run', $Result.NotRunCount)
$XmlWriter.WriteAttributeString('inconclusive', '0') # $Result.PendingCount + $Result.InconclusiveCount) #TODO: reflect inconclusive count once it is added
$XmlWriter.WriteAttributeString('inconclusive', $Result.InconclusiveCount)
$XmlWriter.WriteAttributeString('ignored', '0')
$XmlWriter.WriteAttributeString('skipped', $Result.SkippedCount)
$XmlWriter.WriteAttributeString('invalid', '0')
Expand Down
Loading