8000
Skip to content

fix(audit): surface Codex firewall blocks from agent-stdio.log and populate action_minutes#23889

Merged
pelikhan merged 4 commits intomainfrom
copilot/cli-tools-test-audit-fix
Apr 1, 2026
Merged

fix(audit): surface Codex firewall blocks from agent-stdio.log and populate action_minutes#23889
pelikhan merged 4 commits intomainfrom
copilot/cli-tools-test-audit-fix

Conversation

Copy link
Copy Markdown
Contributor
Copilot AI commented Apr 1, 2026

Codex run audits produced empty metrics (turns=0, token_usage absent) and no firewall_analysis even when the agent exited due to a firewall-blocked domain — the only evidence being a --allow-domains warning buried in a 10+ MB agent-stdio.log.

Changes

Firewall root cause from agent log

  • firewall_log.go: Adds extractFirewallFromAgentLog() which scans agent-stdio.log for the Codex CLI's firewall warning pattern and synthesizes a *FirewallAnalysis with the blocked domain(s):
    [WARN] chatgpt.com is not in the allowed domains. To allow access, add --allow-domains chatgpt.com to your command.
    
  • audit.go: Calls extractFirewallFromAgentLog after proxy log analysis and merges results (or sets as primary when no proxy logs exist)
  • firewall_log.go (AddMetrics): Fixed to also merge BlockedDomains/AllowedDomains lists — previously only merged request counts

Firewall findings now name specific domains

  • generateFindings: Firewall finding description now reads "Agent attempted to access blocked domain: chatgpt.com" instead of the generic count message
  • generateRecommendations: Threshold lowered from >10 to >0; example now includes the exact network.allowed YAML snippet with the blocked domain(s)

action_minutes always populated

  • audit_report.go (buildAuditData): MetricsData.ActionMinutes is now derived from run.Duration, so it appears in audit output regardless of whether token/turn metrics were extractable from the log

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw -buildtags /home/REDACTED/.do--show-toplevel git rev-�� --show-toplevel git /usr/bin/git 38/001/test-simpgit -tests e_modules/.bin/s--show-toplevel git (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw git /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git b/workflows git /usr/bin/git git (http block)
  • https://api.github.com/orgs/test-owner/actions/secrets
    • Triggering command: /usr/bin/gh gh api /orgs/test-owner/actions/secrets --jq .secrets[].name --local cfg 64/pkg/tool/linux_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api /orgs/test-owner/actions/secrets --jq .secrets[].name --show-toplevel sh /usr/bin/git "prettier" --wrigit x_amd64/vet /usr/sbin/sh git rev-�� --show-toplevel sh /usr/bin/git k/gh-aw/gh-aw/.ggit x_amd64/vet ache/go/1.25.0/x--show-toplevel git (http block)
    • Triggering command: /usr/bin/gh gh api /orgs/test-owner/actions/secrets --jq .secrets[].name --show-toplevel git /usr/bin/git --show-toplevel git /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git --show-toplevel git /usr/bin/infocmp git (http block)
  • https://api.github.com/repos/actions/ai-inference/git/ref/tags/v1
    • Triggering command: /usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq .object.sha /home/REDACTED/work/gh-aw/gh-aw/.github/workflows/ace-editor.md ache/go/1.25.0/xconfig /usr/bin/git ub/workflows (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq .object.sha --show-toplevel git /usr/bin/git --show-toplevel 64/pkg/tool/linurev-parse /usr/bin/git git conf�� --get remote.origin.url /usr/bin/git --show-toplevel git /usr/bin/git git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v3
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq .object.sha --noprofile l /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/compile --ignore-path ..git nternal/testdepsrev-parse 64/pkg/tool/linu--show-toplevel /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/compile -o /tmp/go-build2957268215/b422/_pkg_.a -trimpath /usr/bin/git -p github.com/githurev-parse -lang=go1.25 git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq .object.sha licyMinIntegrityOnlyCompiledOutput3910426824/001 git /usr/bin/git l bash /usr/bin/git git rev-�� 076257326/001 l /usr/bin/git --show-toplevel bash 64/bin/node git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v5
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha -unreachable=false /tmp/go-build787724694/b267/vet.cfg ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha -v x_amd64/compile /usr/bin/git y-frontmatter.mdgit /tmp/go-build787rev-parse ules/.bin/node git rev-�� --show-toplevel /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/vet /usr/bin/git -bool -buildtags 0/x64/bin/npm git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha --show-toplevel /opt/hostedtoolcowner=github /usr/bin/git rite '../../../*git -buildtags 0/x64/bin/node git rev-�� --show-toplevel bash 86_64/node vaScript17924294git -tests /home/REDACTED/nod--show-toplevel git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha --show-toplevel 64/pkg/tool/linu-buildtags /usr/bin/git HEAD (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha b.actor }}, Repo: ${{ github.repository }} -buildtags /usr/bin/git -errorsas -ifaceassert -nilfunc git conf�� --get remote.origin.url /usr/bin/git json' --ignore-pgit 724694/b124/vet.rev-parse ache/go/1.25.0/x--show-toplevel git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha --show-toplevel /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linuremote.origin.url /usr/bin/git 1403-14362/test-git arison_test.go es/.bin/node git rev-�� --show-toplevel ortcfg /usr/bin/docker agentic-observabgit d/gh-aw/capitalirev-parse 0/x64/bin/node docker (http block)
  • https://api.github.com/repos/actions/github-script/git/ref/tags/v8
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha --local cfg (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha FETCH_HEAD cfg 64/pkg/tool/linux_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha h ../../../.prettierignore (http block)
  • https://api.github.com/repos/actions/setup-go/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq .object.sha /tmp/shared-actions-test1353585078 config /usr/bin/git remote.origin.urgit (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq .object.sha --show-toplevel git /usr/bin/git --show-toplevel /bin/sh /usr/bin/git git rev-�� --show-toplevel git /usr/bin/infocmp --show-toplevel ache/go/1.25.0/xrev-parse /usr/bin/git infocmp (http block)
  • https://api.github.com/repos/actions/setup-node/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq .object.sha --show-toplevel x_amd64/asm /usr/bin/git k/gh-aw/gh-aw/pkgit k/gh-aw/gh-aw/pkrev-parse ache/go/1.25.0/x--show-toplevel git -C /tmp/gh-aw-test-runs/20260401-131403-14362/test-3875295959 rev-parse /usr/bin/git @{u} 724694/b098/vet.rev-parse cfg git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq .object.sha --show-toplevel git /usr/bin/infocmp --show-toplevel node /usr/bin/git infocmp -1 xterm-color git /usr/bin/git --show-toplevel git inPathSetup_Goro--show-toplevel git (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v0.1.2
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.1.2 --jq .object.sha /tmp/shared-actions-test1353585078 config /usr/bin/git remote.origin.urgit (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.1.2 --jq .object.sha --show-toplevel 3sE-jS52FoHl /usr/bin/git 01 node /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git --show-toplevel git /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.0.0 --jq .object.sha FwWM8pAbU **/*.cjs 7268215/b400/vet.cfg l --ignore-path ../../../.pretti--show-toplevel ortcfg -c g/testutil/tempdir_test.go x_amd64/vet 0/x64/bin/node rror --local x_amd64/vet 0/x64/bin/node (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.0.0 --jq .object.sha --show-toplevel git /usr/bin/git --show-toplevel 64/pkg/tool/linurev-parse (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.2.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.2.3 --jq .object.sha 1403-14362/test-3875295959 **/*.cjs 7268215/b401/vet.cfg **/*.json --ignore-path ../../../.pretti--show-toplevel sh -c "prettier" --write '../../../**/*.json' '!../../../pkg/workflow/go1.25.0 x_amd64/vet /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/compile rror --local x_amd64/vet /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linu1 (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.2.3 --jq .object.sha --show-toplevel git /usr/bin/infocmp --show-toplevel 64/pkg/tool/linurev-parse /usr/bin/git infocmp -1 xterm-color git /usr/bin/git --show-toplevel /opt/hostedtoolcrev-parse /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/1/artifacts
    • Triggering command: /usr/bin/gh gh run download 1 --dir test-logs/run-1 /tmp/go-build787724694/b039/vet.-ifaceassert de_modules/.bin/sh (http block)
    • Triggering command: /usr/bin/gh gh run download 1 --dir test-logs/run-1 git /usr/bin/git --show-toplevel 64/pkg/tool/linurev-parse /usr/bin/git git rev-�� runs/20260401-131807-19472/test-1640349388/.github/workflows git /usr/bin/git l bash /usr/bin/git git (http block)
    • Triggering command: /usr/bin/gh gh run download 1 --dir test-logs/run-1 git /usr/lib/git-core/git --show-toplevel git /usr/bin/git /usr/lib/git-corstatus main�� run --auto /usr/bin/git --detach git /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/12345/artifacts
    • Triggering command: /usr/bin/gh gh run download 12345 --dir test-logs/run-12345 /tmp/go-build787724694/b028/vet.-w x_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh run download 12345 --dir test-logs/run-12345 git /usr/bin/git l_request_branchgit 64/pkg/tool/linurev-parse /usr/bin/git git rev-�� - 8000 -show-toplevel git /usr/bin/git --show-toplevel bash /usr/bin/git git (http block)
    • Triggering command: /usr/bin/gh gh run download 12345 --dir test-logs/run-12345 git /usr/bin/git --show-toplevel (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/12346/artifacts
    • Triggering command: /usr/bin/gh gh run download 12346 --dir test-logs/run-12346 -trimpath ules/.bin/sh -p github.com/stretconfig -lang=go1.17 ache/go/1.25.0/xTest User -uns�� */*.ts' '**/*.json' --ignore-patgo1.25.0 /tmp/go-build787724694/b160/vet.-c=4 x_amd64/link -c=4 9Ebxh1pAtSxdT/Cz/tmp/js-hash-test-3773855507/test-hash.js -importcfg x_amd64/link (http block)
    • Triggering command: /usr/bin/gh gh run download 12346 --dir test-logs/run-12346 git /usr/bin/git --show-toplevel x_amd64/compile /usr/bin/git git rev-�� --show-toplevel resolved$ /usr/bin/git --show-toplevel go /usr/bin/git git (http block)
    • Triggering command: /usr/bin/gh gh run download 12346 --dir test-logs/run-12346 gh /usr/bin/git /repos/github/ghgit --jq /usr/bin/git git add . git /usr/bin/git /usr/bin/git git /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/2/artifacts
    • Triggering command: /usr/bin/gh gh run download 2 --dir test-logs/run-2 /tmp/go-build787724694/b010/vet.-test.run=^Test bin/sh (http block)
    • Triggering command: /usr/bin/gh gh run download 2 --dir test-logs/run-2 git /usr/bin/git --show-toplevel 64/pkg/tool/linurev-parse /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git --show-toplevel bash /usr/bin/git git (http block)
    • Triggering command: /usr/bin/gh gh run download 2 --dir test-logs/run-2 e/git /usr/bin/bash --show-toplevel git /usr/bin/git bash /tmp�� /usr/bin/git git /usr/bin/git --show-toplevel git /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/3/artifacts
    • Triggering command: /usr/bin/gh gh run download 3 --dir test-logs/run-3 /tmp/go-build787724694/b040/vet.cfg tions/node_modules/.bin/sh (http block)
    • Triggering command: /usr/bin/gh gh run download 3 --dir test-logs/run-3 git /usr/bin/git --show-toplevel aw.test /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git --show-toplevel bash /usr/bin/git git (http block)
    • Triggering command: /usr/bin/gh gh run download 3 --dir test-logs/run-3 git /usr/bin/git --show-toplevel git /usr/bin/git git rev-�� HEAD git /usr/bin/git user.name Test User /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/4/artifacts
    • Triggering command: /usr/bin/gh gh run download 4 --dir test-logs/run-4 /tmp/go-build787724694/b041/vet.cfg ules/.bin/sh (http block)
    • Triggering command: /usr/bin/gh gh run download 4 --dir test-logs/run-4 git /usr/bin/git --show-toplevel 64/pkg/tool/linuinit /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git --show-toplevel bash /usr/bin/git git (http block)
    • Triggering command: /usr/bin/gh gh run download 4 --dir test-logs/run-4 git /usr/bin/git --show-toplevel git /usr/bin/git git bran�� --show-current git /usr/bin/git user.email test@example.comrev-parse /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/5/artifacts
    • Triggering command: /usr/bin/gh gh run download 5 --dir test-logs/run-5 -trimpath h -p testing/internalrev-parse -lang=go1.25 /opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linuscripts/**/*.js -ato�� */*.ts' '**/*.json' --ignore-path ../../../.pret.prettierignore -buildtags ache/go/1.25.0/x64/pkg/tool/linux_amd64/link -errorsas -ifaceassert (http block)
    • Triggering command: /usr/bin/gh gh run download 5 --dir test-logs/run-5 git /usr/bin/git --show-toplevel 64/pkg/tool/linurev-parse /usr/bin/git git rev-�� /v1.2.3 git /usr/bin/git -aw-actions/git/git bash /usr/bin/git git (http block)
    • Triggering command: /usr/bin/gh gh run download 5 --dir test-logs/run-5 git /usr/bin/git --show-toplevel git /usr/bin/git git log --oneline -10 /usr/bin/git add origin /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/workflows
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path d -n 10 cfg x_amd64/compile (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 100 (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 6 (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v0.47.4
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v0.47.4 --jq .object.sha --show-toplevel /opt/hostedtoolcache/go/1.25.0/x-tests /usr/bin/git ormat:cjs --silegit /tmp/go-build787rev-parse bin/sh git rev-�� --show-toplevel /opt/hostedtoolcache/go/1.25.0/x--json /usr/bin/git rity633572020/00git /tmp/go-build787show-ref /opt/hostedtoolc--verify git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v0.47.4 --jq .object.sha --show-toplevel infocmp /usr/bin/git xterm-color /tmp/go-build295rev-parse /usr/bin/git git chec�� .github/workflows/test.md git /usr/bin/git --show-toplevel git /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq .object.sha ty-test.md /tmp/go-build787724694/b091/vet.-c=4 n-dir/sh INVALID,NEW -j DROP l2/Ut612OBXueNp0eLGih9P/MnNd7ePdzEVMfEX_ucvu -uns�� .js' --ignore-path .prettierignore --log-level=e!../../../pkg/workflow/js/**/*.json /tmp/go-build787724694/b239/vet.cfg tions/setup/js/node_modules/.bin/node (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq .object.sha --show-toplevel /bin/sh /usr/bin/git Nbxn/BFpAecVysl8git x_amd64/vet (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.2.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq .object.sha re --log-level=error cfg 64/pkg/tool/linux_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq .object.sha --show-toplevel git /usr/bin/git runs/20260401-13git -tests /usr/bin/git git rev-�� ub/workflows git e (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq .object.sha tmp/TestGetNpmBinPathSetup_GorootOrdering2242553594/001/go/1.25.0/x64"; export PATH="$(find "/tmgit git /usr/bin/git --show-toplevel git r,url,status,conclusion,workflow--show-toplevel git rev-�� --show-toplevel git /usr/bin/git /tmp/TestGuardPogit rev-parse /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v2.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq .object.sha --noprofile cfg 64/pkg/tool/linux_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq .object.sha --noprofile cfg x_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq .object.sha --show-toplevel git /usr/bin/git --show-toplevel l /usr/bin/infocmp--show-toplevel git rev-�� --show-toplevel infocmp (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v3.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq .object.sha k/gh-aw/gh-aw/pkg/cli/actions_test.go cfg x_amd64/asm k/gh-aw/gh-aw/pk/opt/hostedtoolcache/go/1.25.0/x64/pkg/tool/linux_amd64/vet k/gh-aw/gh-aw/pk-atomic k/gh-aw/gh-aw/pk-bool x_amd64/asm k/gh�� k/gh-aw/gh-aw/pk-errorsas k/gh-aw/gh-aw/pk-ifaceassert ache/go/1.25.0/x-nilfunc (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq .object.sha --show-toplevel git /usr/bin/git --get remote.origin.urrev-parse /usr/bin/git gFvSngM/bUaI0OuDt-3eoMbXNQPo rev-�� rity1969874094/001 git /usr/bin/git user.email l /opt/hostedtoolc--show-toplevel git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq .object.sha port PATH="$(find /opt/hostedtoolcache -maxdepth 4 -type d -name bin 2>/dev/null | tr '\n' ':')$git git /usr/bin/git user.email test@example.comjs/fuzz_sanitize_output_harness.cjs /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git --show-toplevel git /usr/bin/git git (http block)
  • https://api.github.com/repos/nonexistent/action/git/ref/tags/v999.999.999
    • Triggering command: /usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq .object.sha ty-test.md /tmp/go-build787724694/b088/vet.cfg cal/bin/bash (http block)
    • Triggering command: /usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq .object.sha --show-toplevel node /usr/bin/git licyBlockedUsersgit --write .cfg git rev-�� --show-toplevel sh (http block)
  • https://api.github.com/repos/nonexistent/repo/actions/runs/12345
    • Triggering command: /usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion (http block)
    • Triggering command: /usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion --show-toplevel x_amd64/compile /usr/bin/git git rev-�� runs/20260401-131807-19472/test-3666546285/.github/workflows git /usr/bin/git --show-toplevel bash /usr/bin/git git (http block)
    • Triggering command: /usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion /repos/github/ghgit --jq /usr/bin/git grep -c ^From [0-9a-f]\{40\} /tmp/gh-aw/aw-feature-branch.patch 0/x64/bin/node --show-toplevel gFvSngM/bUaI0OuDrev-parse /usr/bin/git git (http block)
  • https://api.github.com/repos/owner/repo/actions/workflows
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo x_amd64/compile (http block)
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/pkg/tool/linu--ignore-path (http block)
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo /usr/bin/git runs/20260401-13git x_amd64/compile 7268215/b388/vet--show-toplevel git rev-�� --show-toplevel node /usr/bin/git UpdateDiscussiongit --write /opt/hostedtoolc--show-toplevel git (http block)
  • https://api.github.com/repos/owner/repo/contents/file.md
    • Triggering command: /tmp/go-build2957268215/b396/cli.test /tmp/go-build2957268215/b396/cli.test -test.testlogfile=/tmp/go-build2957268215/b396/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true (http block)
    • Triggering command: /tmp/go-build2604940350/b001/cli.test /tmp/go-build2604940350/b001/cli.test -test.paniconexit0 -test.timeout=10m0s -test.count=1 0/x6�� --show-toplevel ache/go/1.25.0/x64/pkg/tool/linux_amd64/compile /usr/bin/git 7268215/b431/_pkgit --write 7268215/b431=> git rev-�� --show-toplevel sh /usr/bin/git Zx0M/uEMuWn9Pe8igit x_amd64/vet /home/REDACTED/wor--show-toplevel git (http block)
    • Triggering command: /tmp/go-build2632375156/b001/cli.test /tmp/go-build2632375156/b001/cli.test -test.paniconexit0 -test.timeout=10m0s -test.count=1 -test.run=. --show-toplevel git ache/go/1.25.0/x64/bin/node licyTrustedUsersnode git /usr/bin/git bin 2>/dev/null | tr '\n' ':')$PATH"; [ -n "$GOROOT" ] && expo ache�� --show-toplevel nly /usr/bin/git --show-toplevel git /tmp/go-build2604940350/b389/wor--show-toplevel git (http block)
  • https://api.github.com/repos/test-owner/test-repo/actions/secrets
    • Triggering command: /usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name --local cfg (http block)
    • Triggering command: /usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name --show-toplevel sh /usr/bin/git runs/20260401-13git x_amd64/vet /usr/local/sbin/--show-toplevel git rev-�� --show-toplevel sh /usr/bin/git k/gh-aw/gh-aw x_amd64/vet ache/node/24.14.--show-toplevel git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name --show-toplevel git /usr/bin/git --show-toplevel git /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git --show-toplevel git /usr/bin/git git (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI and others added 2 commits April 1, 2026 13:26
…pulate action_minutes

- Add extractFirewallFromAgentLog() to scan agent-stdio.log for Codex CLI
  firewall-blocked domain warnings (--allow-domains pattern)
- Merge agent-log firewall findings into FirewallAnalysis in AuditWorkflowRun
- Fix AddMetrics to also merge BlockedDomains/AllowedDomains lists
- Populate MetricsData.ActionMinutes from run.Duration in buildAuditData so
  it is always shown even when token/turn metrics are zero
- Improve firewall key_findings to include specific blocked domain names
- Lower firewall recommendation threshold from >10 to >0 with domain-specific example
- Add tests for all new functionality

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/8a45d621-34c3-4c93-bcc8-f0e7a1faeb5e

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…romAgentLog

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/8a45d621-34c3-4c93-bcc8-f0e7a1faeb5e

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix audit report by adding missing metrics and firewall analysis fix(audit): surface Codex firewall blocks from agent-stdio.log and populate action_minutes Apr 1, 2026
Copilot AI requested a review from pelikhan April 1, 2026 13:32
@pelikhan pelikhan marked this pull request as ready for review April 1, 2026 14:19
Copilot AI review requested due to automatic review settings April 1, 2026 14:19
Copy link
Copy Markdown
Contributor
Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves audit reporting for Codex runs by surfacing firewall blocks that only appear in agent-stdio.log, ensuring firewall metrics merge domain lists, and always populating action_minutes even when log-derived metrics are missing.

Changes:

  • Parse Codex firewall “add --allow-domains …” warnings from agent-stdio.log and merge into firewall analysis.
  • Enhance firewall findings/recommendations to include the specific blocked domain(s) and trigger recommendations on any block.
  • Populate action_minutes in audit output based on run duration, plus add/extend unit tests.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
pkg/cli/firewall_log.go Adds agent log firewall extraction and improves firewall metric merging.
pkg/cli/firewall_log_test.go Adds unit tests for agent log extraction and domain merging/deduping.
pkg/cli/audit.go Merges agent-log-derived firewall analysis into the main audit flow.
pkg/cli/audit_report.go Populates MetricsData.ActionMinutes from run duration.
pkg/cli/audit_report_test.go Adds tests for action_minutes and firewall domain messaging in findings/recommendations.
pkg/cli/audit_report_analysis.go Updates firewall finding/recommendation text to include blocked domains and trigger on any block.
Comments suppressed due to low confidence (3)

pkg/cli/firewall_log.go:458

  • extractFirewallFromAgentLog assumes agent-stdio.log is located at logsPath/agent-stdio.log, but other parts of the CLI resolve logs by searching subdirectories (e.g. nested artifact paths). This will miss firewall warnings when the artifact layout places the log in a subfolder; consider reusing the existing log-file resolution logic (or walking logsPath) to locate agent-stdio.log reliably.
func extractFirewallFromAgentLog(logsPath string, verbose bool) *FirewallAnalysis {
	agentStdioPath := filepath.Clean(filepath.Join(logsPath, "agent-stdio.log"))
	content, err := os.ReadFile(agentStdioPath) // #nosec G304 -- path is cleaned via filepath.Clean and logsPath is a trusted run output directory

pkg/cli/firewall_log.go:467

  • This reads the entire agent-stdio.log into memory via os.ReadFile and then splits it, which is unnecessarily heavy for multi‑MB logs. Consider scanning the file line-by-line (similar to parseFirewallLog), which avoids loading the full log and reduces peak memory usage.
	agentStdioPath := filepath.Clean(filepath.Join(logsPath, "agent-stdio.log"))
	content, err := os.ReadFile(agentStdioPath) // #nosec G304 -- path is cleaned via filepath.Clean and logsPath is a trusted run output directory
	if err != nil {
		// File not present is normal (agent didn't run, or run used a different log path)
		firewallLogLog.Printf("No agent-stdio.log found at %s: %v", agentStdioPath, err)
		return nil
	}

	blockedDomainsSet := make(map[string]bool)
	for line := range strings.SplitSeq(string(content), "\n") {
		if matches := agentLogAllowDomainsPattern.FindStringSubmatch(line); len(matches) > 1 {

pkg/cli/firewall_log.go:497

  • The synthesized analysis sets TotalRequests/BlockedRequests to the number of unique blocked domains (and sets each domain’s Blocked count to 1). These fields are used elsewhere as request counters (e.g. performance metrics NetworkRequests), so this will undercount when the same domain is blocked multiple times. Consider incrementing request counters per matched warning line and tracking per-domain blocked counts separately from the deduplicated BlockedDomains list.
	analysis := &FirewallAnalysis{
		TotalRequests:    len(blockedDomains),
		AllowedRequests:  0,
		BlockedRequests:  len(blockedDomains),
		RequestsByDomain: make(map[string]DomainRequestStats),
	}
	analysis.SetBlockedDomains(blockedDomains)
	for _, d := range blockedDomains {
		analysis.RequestsByDomain[d] = DomainRequestStats{Blocked: 1}
	}

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

// adding a blocked domain to the allow-list.
// Example: "add --allow-domains chatgpt.com to your command"
// Example: "add --allow-domains chatgpt.com,other.com to your command"
agentLogAllowDomainsPattern = regexp.MustCompile(`--allow-domains\s+([\w.\-]+(?:,[\w.\-]+)*)`)
Copy link
Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agentLogAllowDomainsPattern only matches bare hostnames ([\w.\-]+) and will not capture valid --allow-domains entries that include protocol prefixes (e.g. https://... / http://...), wildcard patterns (*), or ports (:443). Consider widening the capture to accept the full allow-domains token(s) up to whitespace (then split by commas) so agent log extraction works with the same domain formats supported elsewhere.

This issue also appears in the following locations of the same file:

  • line 456
  • line 457
  • line 488
Suggested change
agentLogAllowDomainsPattern = regexp.MustCompile(`--allow-domains\s+([\w.\-]+(?:,[\w.\-]+)*)`)
// Note: capture the full --allow-domains token up to whitespace (may include protocol, wildcards, ports).
agentLogAllowDomainsPattern = regexp.MustCompile(`--allow-domains\s+([^\s]+)`)

Copilot uses AI. Check for mistakes.
Comment on lines +277 to +281
// Populate ActionMinutes from run duration so it is always visible even
// when token/turn metrics are zero (e.g. Codex runs that exit early).
if run.Duration > 0 {
metricsData.ActionMinutes = run.Duration.Minutes()
}
Copy link
Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ActionMinutes is described and used elsewhere as billable Actions minutes (rounded up), but here it’s set to run.Duration.Minutes() which can be fractional and inconsistent with math.Ceil(run.Duration.Minutes()) used in other code paths. Consider setting metricsData.ActionMinutes from run.ActionMinutes when available, or computing it as ceil(duration minutes) to match existing semantics.

Copilot uses AI. Check for mistakes.
blockedDomains := processedRun.FirewallAnalysis.GetBlockedDomains()
var example string
if len(blockedDomains) > 0 {
example = fmt.Sprintf("Add the blocked domain(s) to your workflow frontmatter: `network:\\n allowed:\\n - %s`", strings.Join(blockedDomains, "\n - "))
Copy link
Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The recommendation example mixes literal \\n sequences with actual newlines from strings.Join(..., "\n - "), and wraps the result in single backticks (inline code), which typically can’t contain newlines and will render poorly. Consider formatting the YAML snippet with real newlines (no escaped \\n) and using a representation that doesn’t rely on inline code spans for multiline content.

Suggested change
example = fmt.Sprintf("Add the blocked domain(s) to your workflow frontmatter: `network:\\n allowed:\\n - %s`", strings.Join(blockedDomains, "\n - "))
example = fmt.Sprintf(
"Add the blocked domain(s) to your workflow frontmatter:\n\n```yaml\nnetwork:\n allowed:\n - %s\n```",
strings.Join(blockedDomains, "\n - "),
)

Copilot uses AI. Check for mistakes.
@pelikhan
Copy link
Copy Markdown
Collaborator
pelikhan commented Apr 1, 2026

@copilot Apply comments

… ActionMinutes, fix YAML example

- firewall_log.go: Use `[^\s]+` capture for --allow-domains to support protocols, wildcards, ports
- audit_report.go: Use math.Ceil(Duration.Minutes()) for ActionMinutes fallback; prefer run.ActionMinutes when pre-set
- audit_report_analysis.go: Format firewall recommendation example as proper YAML code block
- Update tests to reflect ceil semantics and new YAML example format

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/1b3bf4ce-eb19-4cee-8acc-121ddbf8d5a5

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author
Copilot AI commented Apr 1, 2026

@copilot Apply comments

Applied all three review comments in commit abe3bbe:

  1. agentLogAllowDomainsPattern: Changed to [^\s]+ to capture the full --allow-domains token including protocols, wildcards, and ports.
  2. ActionMinutes: Now uses run.ActionMinutes when pre-set (by the orchestrator), falling back to math.Ceil(run.Duration.Minutes()) to match the rounding semantics used elsewhere.
  3. Recommendation example: Reformatted to a proper multi-line YAML code block (```yaml\nnetwork:\n allowed:\n - <domain>\n```).

Tests updated to cover the ceil rounding and pre-set ActionMinutes precedence.

@pelikhan pelikhan merged commit 596a23f into main Apr 1, 2026
@pelikhan pelikhan deleted the copilot/cli-tools-test-audit-fix branch April 1, 2026 14:40
Copilot stopped work on behalf of pelikhan due to an error April 1, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[cli-tools-test] audit: Codex run audit missing token/turn metrics and firewall failure root cause

3 participants

0