Performance

Picket performance work is measured with BenchmarkDotNet for engine-level changes and with scanner-oracle scripts for end-to-end behavior.

Optimization is a late-project activity. Do not do broad hot-path rewrites until the relevant feature set is complete, unless a measured regression blocks correctness, feasibility, or CI reliability.

Run the engine benchmarks with:

dotnet run -c Release --project benchmarks/Picket.Benchmarks -- --filter "*SecretScanBenchmarks*"

Run the report writer benchmarks with:

dotnet run -c Release --project benchmarks/Picket.Benchmarks -- --filter "*ReportWriterBenchmarks*"

Run the strict compatibility regex-component benchmarks with:

dotnet run -c Release --project benchmarks/Picket.Benchmarks -- --filter "*GitleaksRegexPipelineBenchmarks*"

Current benchmark scenarios cover:

• native default rules over the embedded Gitleaks config,
• strict Gitleaks-compatible rules over the same file,
• the focused native Google API key rule over the embedded Gitleaks config as a local hosted-alert parity proxy,
• mapped native rules over the sanitized GitHub secret-scanning oracle fixture in tests/fixtures/github-secret-scanning,
• native default rules over credential-analyzer tests,
• steady-state and fresh-rule-set scans for native default and strict Gitleaks-compatible rules,
• complete native default, strict Gitleaks-compatible, and mapped GitHub-alert regex compilation, including deferred rule, path, and allowlist regexes,
• direct Scout candidate-regex searches and the complete retained-finding compatibility pipeline over the same representative input,
• compatibility JSON report writer throughput,
• native JSON, JSON Lines, SARIF, HTML, and TOON report writer throughput across deterministic 1, 100, and 1000 finding report sizes.

Run benchmarks before and after hot-path changes and keep the output in ignored BenchmarkDotNet.Artifacts/ or artifacts/ directories. Do not commit machine-specific benchmark output unless it has been normalized into a reviewed fixture.

End-to-End Scanner Harness

scripts/Measure-ScannerPerformance.cs measures Native AOT scanner processes from a checked-in JSON scenario. The default strict-compatibility scenario is benchmarks/scenarios/gitleaks-compatible-tracked.json. It creates an immutable copy of the selected Git-tracked files, runs Picket and the pinned Gitleaks binary from the same working directory, and removes the generated corpus and reports after measurement.

The native incremental scenario is benchmarks/scenarios/native-cache-tracked.json. It compares the same Picket binary and native scan with cache disabled and with an initially empty secret-hash-only cache. Its one recorded cold run populates the cache before the warmup and warm rounds. Report parity is mandatory, and bounded CPU diagnostics record scan inputs, findings, cache hits, misses, and writes for every run. Canonical parity excludes raw line, match, and secret fields because secret-hash-only cache hits intentionally omit them, plus matchSha256 because the missing match context cannot be reconstructed. Secret hashes, stable fingerprints, full report hashes, and all other finding properties are still recorded and compared.

Publish the current scanner, identify the two direct executable paths, build the file-based app once, and run the scenario:

dotnet publish src/Picket.Cli/Picket.Cli.csproj --configuration Release --no-restore -p:PublishProfile=release-speed -r win-x64 -o artifacts/performance/tools/picket
$env:PICKET_BIN = (Resolve-Path artifacts/performance/tools/picket/picket.exe).Path
$env:PICKET_GITLEAKS_BIN = (Resolve-Path artifacts/tools/gitleaks.exe).Path
dotnet build ./scripts/Measure-ScannerPerformance.cs --nologo --verbosity quiet
dotnet run --file ./scripts/Measure-ScannerPerformance.cs --no-build -- -ScenarioPath ./benchmarks/scenarios/gitleaks-compatible-tracked.json -FailOnParityDifference

Run the cache scenario with the same PICKET_BIN value:

dotnet run --file ./scripts/Measure-ScannerPerformance.cs --no-build -- -ScenarioPath ./benchmarks/scenarios/native-cache-tracked.json -FailOnParityDifference

Run the capability-separated native filesystem scenario after setting direct paths for Picket, TruffleHog, and Kingfisher:

$env:PICKET_TRUFFLEHOG_BIN = (Resolve-Path ../trufflehog/trufflehog.exe).Path
$env:PICKET_KINGFISHER_BIN = (Resolve-Path ../kingfisher/target/release/kingfisher.exe).Path
dotnet run --file ./scripts/Measure-ScannerPerformance.cs --no-build -- -ScenarioPath ./benchmarks/scenarios/native-filesystem-competitors.json

This scenario does not use -FailOnParityDifference. It holds the corpus, filesystem source mode, live-verification state, persistent-cache state, and JSON Lines output category constant, and disables competitor update checks, but each scanner retains its own built-in rules, offline filtering, decoders, archive handling, ignores, and report schema. Finding counts and timings describe those complete tool-native capabilities; they do not establish an equivalent winner.

Run the native report fan-out scenario with the same PICKET_BIN value:

dotnet run --file ./scripts/Measure-ScannerPerformance.cs --no-build -- -ScenarioPath ./benchmarks/scenarios/native-report-writing-tracked.json -FailOnParityDifference

The control writes JSON Lines. The fan-out variant writes the same primary JSON Lines finding set plus SARIF, HTML, and TOON. The harness requires the primary finding sets to match, verifies every additional file exists, and records their aggregate byte count and content manifest hash.

On Unix-like systems, set PICKET_BIN and PICKET_GITLEAKS_BIN to the corresponding executable paths before running the same dotnet build and dotnet run commands. A scenario falls back to executable names on PATH when its environment variable is not set.

The result schema is picket.performance-result.v1. Each capture records:

• scenario, corpus manifest, Picket commit, tool repository commit, executable byte count, SHA-256, and version output,
• OS, architecture, processor, effective processor count, GC-visible memory, filesystem, runner type, .NET runtime, and SDK,
• wall time, child-process CPU time, peak child-process working set, exit code, output byte counts and hashes, report byte count and hash, and finding count,
• a canonical finding-set hash for parity groups, so report ordering does not create a false difference,
• optional bounded diagnostic artifact metadata and non-secret scan-input, finding, cache-hit, cache-miss, and cache-write counters.

Picket scenarios set RequireRepositoryCommitInVersion. A measurement fails before its first timed process when the Native AOT binary version does not identify the selected repository commit. Corpus staging reads exact blobs from the recorded Git HEAD; dirty tracked working-tree bytes cannot enter a result that claims to represent that commit.

The default schedule records one pre-warmup run, discards one warmup round, then records five warmed rounds. Tool order rotates between rounds to reduce fixed order and thermal bias. Every scanner invocation is a fresh process. “Warm” therefore describes warmed host and filesystem state, not retained in-process scanner caches. A genuinely cold OS-cache measurement requires a fresh host or an explicit host-level cache reset and must be recorded in the scenario conditions.

The harness invokes executables directly through ProcessStartInfo.ArgumentList. Do not benchmark dotnet run, a shell wrapper, or a build command as if it were scanner time. It stores only hashes and byte counts for scanner stdout, stderr, and canonical findings. Generated reports can contain secrets and are deleted by default. -KeepWork is an explicit debugging option and must be used only for a trusted artifact location.

Tools that emit reports on standard output use the scenario’s ReportSource value. The harness streams stdout directly into the ephemeral report file and does not retain decoded report text in the result.

Only tools in the same ParityGroup are required to produce the same canonical finding set. Native comparisons with TruffleHog, Kingfisher, or another scanner must omit a parity group unless rules, decoding, verification, source traversal, history behavior, ignores, and report filtering have actually been aligned. Capability-separated measurements remain valid, but they are not presented as an equivalent winner/loser comparison.

Reviewed Release Baseline: 2026-08-02

The current reviewed baseline uses the Picket v0.2.8 release-speed Windows x64 Native AOT executable from commit 52c27a3d46b4e1fbea7dac1ca67ce98db76b9a72, Scout.Text.Regex 0.6.1, and Gitleaks 8.30.1 from commit 4c232b5014f7618360bd992b4c489cb055881c6b. The Picket executable is 14,984,704 bytes with SHA-256 42d6543fbf6d404cef4866d144af2d5605e6ade43f7c597b7233d4309c554755.

Every scenario staged the same 630 tracked files from src/ and tests/, totaling 5,791,233 bytes. The corpus manifest SHA-256 was dfceda5dbc76f574e7c2fe05cf196370dd3f822fd5a261371db83c85766e59da. The host ran Windows 10.0.26200 on NTFS with an AMD64 Family 26 Model 68 processor, 32 effective processors, 47.13 GiB of GC-visible memory, .NET 10.0.10, and SDK 10.0.302. Host-level filesystem caches were not reset, so pre-warmup values are not cold OS-cache measurements.

The strict-compatible scenario produced 47 findings from both scanners with canonical finding-set SHA-256 dc587f66d6cbe7b607b149d660184651d9c449c3d1e68b446b906ae15cabd403. The parity gate passed.

Tool	Pre-warmup elapsed	Warm elapsed median	Warm elapsed p95	Warm CPU median	Warm peak working set median
Picket	976.02 ms	978.48 ms	1,007.87 ms	2,750.00 ms	115.61 MiB
Gitleaks	687.27 ms	755.88 ms	787.35 ms	2,671.88 ms	106.73 MiB

Picket’s warm elapsed median was 29.45% higher than Gitleaks in this run. This result supersedes the earlier public speed claim and remains scoped to this corpus, scanner mode, host, and tool versions.

The native cache scenario preserved the same 74 canonical findings with and without its secret-hash-only cache. Its canonical finding-set SHA-256 was 4464003d4be600488e1ff7e83a2bb1b9986183a71e01b4c9ad83faa7a749bdb7. Every warm cache-enabled run reported 614 hits, zero misses, and zero writes.

Cache mode	Pre-warmup elapsed	Warm elapsed median	Warm elapsed p95	Warm CPU median	Warm peak working set median
Disabled	1,294.18 ms	1,226.05 ms	1,262.27 ms	3,671.88 ms	144.66 MiB
Secret-hash-only	1,346.50 ms	448.31 ms	453.35 ms	406.25 ms	75.00 MiB

The populated cache reduced the warm elapsed median by 63.43%, or 2.73 times, while preserving the required canonical finding set.

Reviewed Capability Baseline: 2026-08-02

The capability-separated scenario used the same Picket binary, TruffleHog commit f2cd191b97098913a07522227d2b5e40e57252f4 (v3.95.8-1-gf2cd191b9), and Kingfisher commit 78904df5ea7354a7dc3700e3c41a124524d23083 (1.105.0).

Tool	Findings	Pre-warmup elapsed	Warm elapsed median	Warm elapsed p95	Warm CPU median	Warm peak working set median
Picket native	74	1,189.39 ms	1,186.10 ms	1,197.28 ms	3,687.50 ms	146.98 MiB
TruffleHog filesystem	32	1,421.95 ms	1,407.71 ms	1,427.26 ms	421.88 ms	119.04 MiB
Kingfisher filesystem	23	3,400.38 ms	226.27 ms	230.94 ms	671.88 ms	184.37 MiB

The finding sets are not equivalent. Each scanner retained its own built-in rules, offline filtering, decoding, archive handling, and ignore behavior. The scenario therefore has no parity group and does not establish a universal winner.

Reviewed Report-Writing Baseline: 2026-08-02

Both report-writing variants produced 74 primary JSON Lines findings with canonical finding-set SHA-256 117d0e72ee8c005c92dfd83cecfa6d7bc4becb29db69647e345ee6be289f29ef. The fan-out variant produced three additional deterministic reports totaling 862,860 bytes. Their content manifest SHA-256 was 86b224dddd1dc59a50b37bee3847be07c9b5f2f5e988ff403a5576ac552e7475 in every retained round.

Report set	Pre-warmup elapsed	Warm elapsed median	Warm elapsed p95	Warm CPU median	Warm peak working set median
JSON Lines	1,232.57 ms	1,177.34 ms	1,198.85 ms	4,234.38 ms	144.26 MiB
JSON Lines, SARIF, HTML, and TOON	1,212.81 ms	1,195.61 ms	1,206.32 ms	4,218.75 ms	147.14 MiB

The additional writers increased the warm elapsed median by 18.27 ms, or 1.55%. This is an end-to-end result; use ReportWriterBenchmarks for isolated writer throughput and allocations.

The detailed July 12 and July 31 measurements used older Picket commits, Scout versions, and smaller corpora. They remain available in repository history but are not repeated here because their differing inputs make them unsuitable as a direct before-and-after comparison with this release baseline.

Release automation writes release-artifacts.json after assembling all final payloads. The deterministic manifest records each archive, installer, NuGet bundle, Marketplace package, and package-manager bundle with its exact byte count and SHA-256. Mutable checksum sidecars are excluded. The manifest is included in checksums.txt, attested, and published with the release so package size regressions can be compared without downloading and unpacking every asset.

Steady-state scan scenarios compile deferred regexes during global setup. The fresh-rule-set scenarios create a new compiled rule set for every operation and therefore include candidate regex compilation on first use. Compilation scenarios force every deferred regex so they measure actual Scout compilation, not only Picket rule-wrapper and fingerprint construction.

Each scan pass searches configured rule keywords once with a shared Scout Aho-Corasick automaton and maps overlapping matches back to candidate rules. Rules without keywords remain unconditional. Before parallel filesystem work starts, Picket compiles every deferred rule, path, and allowlist regex and builds the shared keyword prefilter so the first worker does not serialize the other workers behind lazy initialization. Strict compatibility directory timing excludes config and regex compilation, then includes keyword-prefilter construction, traversal, and matching, following the upstream directory command boundary. Process wall-clock measurements still include all startup and config work.

Filesystem, baseline file, and strict Git-history fragment evaluation is bounded by work-item count, effective CPU availability, and current memory headroom. Environment.ProcessorCount honors processor affinity and CPU limits. Picket uses GC.GetGCMemoryInfo() to measure the bytes remaining below the GC high-memory-load threshold. Each worker requires 64 MiB of that headroom, and the selected degree is the lowest of the work-item, processor, and memory limits. This absolute budget avoids serializing scans merely because a large-memory host has crossed a relative pressure percentage while retaining a conservative reserve for each worker. Small workloads use one worker, and unavailable GC metrics fall back to the processor and work-item limits. Filesystem and baseline results are merged in source order, so their report bytes do not depend on scheduling. Strict Git-history scans match Gitleaks’ concurrent completion model: added-line fragments enter a queue capped at one pending fragment per worker, findings are emitted as workers complete, final reports are merged in source order for stable compatibility bytes, and the entire patch is never retained. Checkpointed scans commit serially because a checkpoint low-water mark must always identify a complete source-manifest prefix.

For incremental-scan changes, run with --cache-dir and opt-in diagnostics. The cpu.json, mem.json, and trace.jsonl artifacts include scanInputs, findings, cacheHits, cacheMisses, and cacheWrites counters. Parallel filesystem scans also record effectiveProcessorCount, scanWorkers, scanMemoryLoadBytes, scanHighMemoryLoadThresholdBytes, and scanMemoryHeadroomBytes. These fields are the preferred evidence for cache hit-rate and worker-selection changes.

Large Local Files

Local files larger than 100,000 bytes are read through pooled, bounded fragments. Strict Gitleaks-compatible commands reproduce the pinned Go buffered reader: primary reads request up to 100,000 bytes, read-ahead consumes at most 25,000 bytes through a 4 KiB buffer while seeking a blank-line boundary, and buffered remainders become short primary reads. Fragments do not overlap.

Binary classification uses only the first 100,000 bytes, before safe-boundary read-ahead, matching the compatibility source reader. Binary files therefore stop after one bounded probe even when native caching is enabled.

Native filesystem and baseline scans also inspect a combined window containing the current fragment and the final 64 KiB of the preceding fragment. The overlap expands backward to a line boundary when one is available, and duplicate findings from the standalone and combined windows are removed. Source positions remain absolute, and blobSha256 identifies the complete file.

This path does not allocate an array proportional to the file length, so local files beyond the managed single-array limit can be scanned or rejected as binary without a full-buffer failure. A positive --max-target-megabytes continues to skip files above the requested cap.

For repository-level comparison:

• use scripts/Capture-CompatibilityOracle.cs for Gitleaks parity,
• use scripts/Capture-GitHubSecretScanningOracle.cs for sanitized hosted GitHub secret-scanning alert metadata,
• use scripts/Compare-GitHubSecretScanningOracle.cs to compare native JSONL scan output against mapped hosted alert classes and locations,
• use picket git . --profile picket --report-format jsonl when captured hosted locations include commit SHAs,
• compare native scans against the GitHub alert classes and locations only after confirming whether differences are compatibility allowlists, history-only alerts, or true rule gaps.

Fair Competitor Comparisons

End-to-end speed comparisons are useful only after the behavior being compared is feature complete. Compare Picket with Gitleaks, TruffleHog, Kingfisher, and other local references only when the scenario can be described precisely enough to review.

Each comparison must record:

• tool name, version, commit SHA, and command line,
• Picket commit SHA, build profile, and runtime identifier,
• OS, CPU, memory, filesystem, runner type, and .NET SDK,
• source payload, repository commit range, path filters, and ignored paths,
• rule/config set and whether compatibility or native behavior is selected,
• output format, report destination, redaction, and baseline/cache settings,
• whether verification, source enumeration, archive traversal, decoding, and history scanning are enabled,
• cold-cache and warm-cache timing separately.

Do not compare a scan-only Picket run against a competitor run that also performs network verification, source enumeration, archive expansion, or full git history unless those costs are explicitly the scenario being measured. When capabilities cannot be made equivalent, report them as separate measurements instead of a single winner.

Scout Escalation

Scout is a dependency through NuGet packages. If profiling attributes a material bottleneck to a Scout package, capture a minimal reproducer, benchmark or trace, exact package version, command line, input shape, and expected impact. Open a concise Scout issue with that evidence.

The full GitleaksRegexPipelineBenchmarks default job for Picket v0.2.8 and Scout.Text.Regex 0.6.1 measured the current regex paths on .NET 10.0.10:

Method	Mean	Managed allocation
Keyword candidate selection	163.7 us	0 B
Scout search with captures	63.17 ms	14,320 B
Scout captures over prepared candidates	62.61 ms	14,320 B
Scout find without captures	62.23 ms	6,720 B
Scout find then capture	62.49 ms	14,320 B
Complete Picket compatibility pipeline	56.61 ms	50,792 B

All four Scout execution paths returned the same raw match count during setup. Older package-specific measurements remain in repository history but are not a current baseline because the regex implementation and benchmark input changed.

If the Scout bottleneck is critical enough to block Picket’s feature-complete path, pause Picket implementation work and fix Scout first. Non-critical Scout performance issues stay tracked, but Picket optimization waits until the late-project hardening phase.