block-log-audit
Skill1k starsChanged 14 days agoApache-2.0View on GitHub
Audit the server (carbon) and test-side logs of one integration-v2 test block after a run: account for every ERROR/WARN, attribute it to the scenario that caused it, and root-cause it from code, logs and product bytecode. Finds assertions that pass for the wrong reason, coverage gaps, resource leaks, cleanup/hygiene defects, readiness races and product bugs that a green run hides. Mandatory gate for new or changed integration tests (test-coverage Phase 6); also use to audit a CI run block by block.
Trending
Steady talk.about the whole repository
Who's talking about wso2/product-apim
- Stack Overflow“WSO2APIM v4.2: WebSocket connection timed out by GLOBAL_TIMEOUT config”Stack Overflow · Jul 6, 2023 · 2 votes
- Stack Overflow“how to apply bug fixes on open source wso2 API Manager 4.0.0 and wso2 Micro Integrator 1.2.0”Stack Overflow · Jan 20, 2023 · 2 votes
- wso2/carbon-apimgt“Fix service catalog and publisher permissions to required fields”GitHub · Apr 17, 2021 · 2 reactions
- wso2/carbon-apimgt“Update onboarding pages”GitHub · Apr 9, 2021 · 2 reactions
- wso2/carbon-apimgt“Adding scope role mapping to admin portal”GitHub · Jan 23, 2020 · 2 reactions
The biggest 5 of 76 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 76
Talked about on GitHub and Stack Overflow (76 times).
On GitHub · 50
- Apr 17, 2021Fix service catalog and publisher permissions to required fields — wso2/carbon-apimgt2 reactions
- Apr 9, 2021Update onboarding pages — wso2/carbon-apimgt2 reactions · 1 comment
- Jan 23, 2020Adding scope role mapping to admin portal — wso2/carbon-apimgt2 reactions
- Aug 29, 2019Improve configuration page user experience — wso2/carbon-apimgt2 reactions · 4 comments
- Oct 17, 2024http_access_log.enabled = false does not really disable the http access — wso2/api-manager1 reaction
45 more on GitHub
- Jun 15, 2021Clear details when configuring the DCR via APIM with WSO2 IS as KM — wso2/docs-apim1 reaction · 2 comments
- Aug 6, 2020[3.1.0] Wrong `user_store` type configurations for JDBC User Store — wso2/docs-apim1 reaction
- Feb 27, 2020Fix swagger ui payload not getting sent error in POST request — wso2/carbon-apimgt1 reaction
- Feb 12, 2020Java (Maven): Actually fix the use of insecure protocol to download/upload artifacts — github/securitylab1 reaction · 10 comments
- Oct 8, 2019JWT Token Revocation [Upgrade Needed] — wso2/product-microgateway1 reaction
- Oct 2, 2019Add empty value validation to rest API required fields — wso2/carbon-apimgt1 reaction · 1 comment
- Sep 12, 2019Fixing login loop when no permission for publisher — wso2/carbon-apimgt1 reaction
- Nov 9, 2026Fail over to fallback models on timeouts and connection errors — wso2/carbon-apimgt0 reactions · 20 comments
- Oct 13, 2026Move/Remove unused sample codes from product-apim repository — wso2/api-manager0 reactions
- Oct 12, 2026Show total complexity of the selected query in the GraphQL try-out Query Complexity Analyzer — wso2/apim-apps0 reactions · 2 comments
- Sep 21, 2026Document per-policy compliance affecting severities — wso2/docs-apim0 reactions · 11 comments
- Sep 14, 2026[4.6.0] Document severity-based governance compliance — wso2/docs-apim0 reactions · 1 comment
- Aug 21, 2026[4.11.x] Return a 404 response for invoking APIs with incorrect tenant domains — wso2/carbon-kernel0 reactions · 3 comments
- Aug 19, 2026Remove warning for path parameters in WebSub topic operations — wso2/apim-apps0 reactions · 2 comments
- Aug 18, 2026Stop Info Severity Rule Violations from Failing Governance Compliance — wso2/carbon-apimgt0 reactions · 1 comment
- Aug 5, 2026[4.11.x] Port tomcat template configs for web portal securing — wso2/carbon-kernel0 reactions · 1 comment
- Jul 1, 2026Fix UI issues in admin and devportal for custom keymanager configs — wso2/apim-apps0 reactions · 4 comments
- May 12, 2026Add test case for backend throughput throttling validation — KenathKapilarathna/product-apim0 reactions
- Sep 3, 2025Issue — ranuka-laksika/docs-apim0 reactions · 1 comment
- Sep 2, 2025Issue — ranuka-laksika/docs-apim0 reactions
- Aug 27, 2025Need to replace Slack with Discord in README.md — wso2/docs-apim0 reactions
- Jan 23, 2024Tenant Support for Integration Tests Added for Gateway Policy Feature — wso2/api-manager0 reactions
- Dec 20, 2023Unable to build docker image of Api Manager 4.1 — wso2/api-manager0 reactions · 3 comments
- Mar 9, 2023[mTLS] Wait approximately 10 minutes to allow the cert to be loaded into memory in mTLS authentication — wso2/docs-apim0 reactions
- Feb 17, 2023[Improvement]: Support proxy support in data publishing with Choreo analytics — wso2/api-manager0 reactions
- Sep 20, 2022Missing link in documentation version 3.1 — wso2/docs-apim0 reactions
- Jul 5, 2022Implement Integration test for Websocket transport to validate closed backend connections — wso2/api-manager0 reactions
- Apr 15, 2022Improve UI/UX related to Call Interceptor Service Policy by supporting an `Enum Array` input type — wso2/api-manager0 reactions
- Nov 19, 2021Defined allowed cipher for WSS endpoints in APIM 3.2.0/APIM 4.0.0 — wso2/docs-apim0 reactions · 1 comment
- Jun 23, 2021Create template for docs-ei repo — wso2/docs-apim0 reactions · 1 comment
- Apr 9, 2021Petstore swagger URL wrong — wso2/docs-apim0 reactions
- Apr 8, 2021Need to update the docs: Role Permissions -> Scope Assignment — wso2/docs-apim0 reactions
- Apr 5, 2021Add information to change bindaddress when ssl reload happens in windows — wso2/docs-apim0 reactions
- Apr 2, 2021Fix API Property based filtering — wso2/carbon-registry0 reactions · 1 comment
- Mar 29, 2021Need update docs related to customizing Store/GW urls for tenants — wso2/docs-apim0 reactions · 1 comment
- Mar 15, 2021Include --optimize when starting Key Manager profile — wso2/docs-apim0 reactions
- Feb 22, 2021Update config-catalog with Registry Handler configurations — wso2/docs-apim0 reactions
- Feb 4, 2021Add more information to Enabling Self Signup page for tenants — wso2/docs-apim0 reactions
- Dec 14, 2020Fix invalid url error message — wso2/carbon-kernel0 reactions
- Nov 18, 2020Details on provisioning handler implementation to keep all system roles without being deleted during the provisioning process — wso2/docs-apim0 reactions · 1 comment
- Apr 6, 2020Add role based validation on JWT grant configuration guide — wso2/docs-apim0 reactions
- Jan 9, 2020Set publisher AuthManager ServerURL to localhost — wso2/kubernetes-apim0 reactions
- Nov 26, 2019Fixing issue #6673 — wso2/carbon-apimgt0 reactions · 1 comment
- Nov 20, 2019Fixes documents issues — wso2/carbon-apimgt0 reactions · 1 comment
- Nov 18, 2019Fix issue #6690 — wso2/carbon-apimgt0 reactions · 1 comment
On Stack Overflow · 26
- Jul 6, 2023WSO2APIM v4.2: WebSocket connection timed out by GLOBAL_TIMEOUT config2 votes
- Jan 20, 2023how to apply bug fixes on open source wso2 API Manager 4.0.0 and wso2 Micro Integrator 1.2.02 votes · 2 answers
- Oct 16, 2018How to build/package wso2 api manager Identity Server as Key Manager2 votes
- May 25, 2023Enable AI-based Recommendations for the WSO2 Developer Portal1 vote
- May 25, 2023WSO2 APIM New Install Won't Call APIs - APILoggerManager Errors1 vote
21 more on Stack Overflow
- Nov 8, 2022WSO2 Api Manager 4.1.0 logging continuously Message dispatched to the main sequence. Invalid URL., RESOURCE = / HEALTH CHECK URL = /1 vote
- Oct 6, 2021Problems with authentication login in Pizza Shack WSO2 API Manager1 vote
- May 10, 2021Wso2 APIManager 3.2.0 How can download Analytics for it?1 vote
- Aug 25, 2020How to enable Jaeger opentracing in WSO2-APIM 3.1.0?1 vote
- Aug 12, 2020Wso2 carbon/apimgt open source code download1 vote
- Jun 30, 2020WSO2 API update REST API throwing exception1 vote
- Jun 28, 2020WSO2 Api Manager - Sandbox/Production keys1 vote
- Mar 18, 2020When clicking on the Forgot password link of a service provider oauth2 login, the user is redirected to the Carbon login page1 vote · 1 answer
- Dec 18, 2019WSO2 APIM 3.0.0: Unable to generate access token from devportal1 vote
- Nov 13, 2019SOAP API pass through in WSO2 API manager v3.0.01 vote
- Mar 13, 2019WSO2 API Manager: Creating SOAP or websocket API changes existing APIs types1 vote · 1 answer
- Jul 16, 2018wso2am 2.2.0 publisher in HA setup1 vote
- Jun 19, 2018WSO2 APIM Application Registration urn:approve action not found1 vote · 1 answer
- Nov 21, 2016monetisation labels won`t work as espected for AM 2.01 vote
- Jun 13, 2023WSO2: What are the differences between some of WSO2's repositories?0 votes · 1 answer
- Jan 28, 2023Extending the group ID extractor in WSO2 APIM for Custom User Claim0 votes
- Oct 8, 2020Issue in SOAP API publishing in wso2 3.0.00 votes
- Mar 26, 2020WSO2 APIM 3.0 Configuration proxy problem0 votes
- Feb 27, 2020java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ObjectMapper cannot be found by org.wso2.carbon.apimgt.keymgt_6.4.500 votes · 1 answer
- Feb 14, 2018WSO2 Api Manager - Generating REST APIs from SOAP0 votes
- Feb 12, 2018Deploying WSO2 API Manager .war on tomcat0 votes · 1 answer
In this repository3 other files it gives its agents.
- parity-checkerSkill
- suite-layout-consultantSkill
- test-coverageSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
13 rules. About 0 are common, 7 are distinctive, and 6 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 31
Evidence or it did not happen: Every claim cites the exact log line (with timestamp) and file:line, or product bytecode (javap -c -p -l).
- Distinctive: line 34
Count everything, explain everything: Every distinct normalised ERROR at any time and WARN after startup ends up in exactly one section.
- Distinctive: line 39
Fix the cause, never the symptom: Never relax an assertion, never reduce coverage, no static sleeps (poll a real signal), no retry that …
- Very specific: line 41
Audit the complete block, as the suite runs it: Every run the audit relies on (the run audited and every verification run) executes the …
- Distinctive: line 48
When the block's own behaviour depends on its neighbours (4-lane CPU contention, the shared node backend), add a concurrent run with the …
- Very specific: line 70
Run scripts/digest_block_log.py <label>.carbon.log <label>.test.log to normalise (UUIDs, _<hex8>_<n> unique suffixes, numbers, timestamps) …
- Very specific: line 89
Put each message (or group with one root cause) in exactly one class — references/defect-patterns.md has the recurring patterns of each:
- Very specific: line 118
Use references/report-template.md exactly: per-block header with counts, sections 0/A–H (and I–K when fixing), tables with block-prefixed …
- Distinctive: line 125
no item in A, B, C or D, no E item without a readiness/robustness fix, and no open question (H) about those scenarios.
- Distinctive: line 126
every ERROR/WARN they produce is in G with a verified reason, or is a catalogued product issue (F) the change does not make worse.
- Distinctive: line 128
every new negative assertion is exact (status AND error code/message field where the product provides one) and has been shown able to fail.
- Distinctive: line 142
Do not run the full suite from this skill.
- Very specific: line 143
Never stage or commit; leave changes for the developer.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Integration-V2 Block Log Audit
A green block is not proof that its tests test what they claim. A scenario can pass because the product rejected the request for an unrelated reason, because a helper accepts any error, or because a poll eventually succeeded after a race; cleanup can leak or re-delete; a fixture can reach a real external host. These defects are visible in the block's logs long before they cause a flaky failure. This skill accounts for every ERROR and WARN a block produced and turns each into one of: expected and correctly asserted, benign with a stated reason, or a finding with a root cause and a fix.
Audit one block per invocation (one <test name="IntegrationV2-..."> in the suite file that ran:
testng-v2.xml for all-in-one, testng-v2_distributed.xml for distributed). Rules you must obey
when proposing or making test changes: all-in-one-apim/modules/integration-v2/tests-integration/cucumber-tests/CLAUDE.md.
Operating principles
- Dig until the root cause is proven. Time and tokens are unlimited; a quick pass is not an audit. A failure or finding is "open" (section H) only after every available source has been read for it: the failing resource's identity traced through the test log and every server/container log of the block, the step and helper code, the product source at the shipped version and the bytecode on any stack trace. "Unresolved" with unread evidence left is an incomplete audit, not a result.
- Evidence or it did not happen. Every claim cites the exact log line (with timestamp) and file:line, or product
bytecode (
javap -c -p -l). A plausible story is not a root cause. If evidence is missing, the item is an open question (section H) with the evidence that would settle it, never a finding. - Count everything, explain everything. Every distinct normalised ERROR at any time and WARN after startup ends up in exactly one section. Startup WARN noise is catalogued separately. Silence is not a verdict: "no ERROR in the block" is checked, not assumed.
- Asserted is not the same as expected. A server ERROR is "expected" only if a step deliberately provokes it AND the scenario asserts the outcome that proves it, and the asserted outcome could not have come from another cause.
- Fix the cause, never the symptom. Never relax an assertion, never reduce coverage, no static sleeps (poll a real signal), no retry that re-sends a deterministically rejected or non-idempotent request, no recovery-as-fix.
- Audit the complete block, as the suite runs it. Every run the audit relies on (the run audited and every
verification run) executes the block's whole
<test>element verbatim: all of its runners, itsthread-count, its runner order and its parameters/overlays. Never a subset, never only the new or changed runner, never a temporary single-runner block (that is for timing insuite-layout-consultant, not for this audit). Many defects exist only in the full block: runners on parallel lanes sharing one server (leaks one runner leaves that another trips over, name collisions, tenant-global config changed under a sibling), cleanup that deletes a sibling's shared fixture, ordering dependencies ("keep X last"), restarts or global flips that a sibling observes, and load-dependent races. When the block's own behaviour depends on its neighbours (4-lane CPU contention, the shared node backend), add a concurrent run with the heaviest neighbouring blocks. - A fix is not done until a run proves it. Re-run the complete block; show the log line that disappeared or appeared and
the block result. For a race, run it more than once. For an assertion strengthened against a wrong-reason pass,
prove it can fail (mutation, see
references/evidence-and-pitfalls.md).
Inputs
- The run's full Maven/CI log. Split it per block with
scripts/split_block_logs.py <raw log> <suite xml> <out dir>(<label>.carbon.log: APIM container lines;<label>.test.log: the block's TestNG-thread lines;<label>.is.log: the block's external IS, attributed best-effort — verify by content). A distributed block (testng-v2_distributed.xml) has no<label>.carbon.log: each of its containers gets<label>.<container>.log(apim-cpcontrol plane,apim-gwgateway,apim-tmtraffic manager,mysql); a block's own extra containers (e.g.gw-ctrl/gw-rt) are split the same way. Use the raw log; the script strips ANSI with an exact pattern. - The code at exactly the revision that ran (a worktree at that commit plus any uncommitted content that ran).
- The product binaries of the image that ran (OSGi bundles, REST webapps, default
conf/), for bytecode evidence. - Known causes:
references/known-product-issues.mdandreferences/benign-catalogue.md— cite their IDs instead of re-deriving, but re-verify that the log matches the catalogued signature before citing.
Procedure
1. Inventory
Run scripts/digest_block_log.py <label>.carbon.log <label>.test.log to normalise (UUIDs, _<hex8>_<n> unique
suffixes, numbers, timestamps) and count every ERROR at any time and WARN after server startup, with first/last
timestamps. For a distributed block run it once per
container log (the test log with any one of them); a request crosses containers (deploy on apim-cp, route and
handshake on apim-gw, throttling on apim-tm), so follow each failing resource through all of them. A block that
boots an external IS also has <label>.is.log: digest it too (same WSO2 Carbon started in startup boundary) and
count its post-startup ERROR/WARN with the others; a federated or key-manager failure often shows only there. Startup
WARN noise (catalogued in references/benign-catalogue.md) is listed once and set aside; an ERROR is a finding even
before startup completes. Read the block's definition in the suite file that ran (parameters, overlays, runners, any
XML comment explaining why it is separate) and each runner's
@CucumberOptions(features = ...).
2. Attribute
Tie every distinct message to the scenario, Examples row/actor and step that caused it. Other blocks run
concurrently and their Cucumber output is interleaved: attribute by resource identity (${UNIQUE:...} suffixes, API
contexts, app names, tenant), not by time alone. Test-thread and container timestamps are both available; state
which clock you aligned. See references/evidence-and-pitfalls.md §Attribution.
3. Classify
Put each message (or group with one root cause) in exactly one class — references/defect-patterns.md has the
recurring patterns of each:
- A. Assertion passing for the wrong reason (highest priority): the asserted status/body was produced by a different cause than the scenario claims.
- B. Coverage gap: the scenario does not assert what its title/description claims, or a documented behaviour is left untested (status-only negative, substring where an exact code is needed, scope limits).
- C. Resource leak: something created is never removed, or cleanup cannot remove it.
- D. Test hygiene: cleanup re-deletes, duplicate registration, literal unresolved placeholders, comment/config drift.
- E. Test-side robustness: polls that start before a deploy/route/key is live, missing readiness gates, retries of deterministic or non-idempotent failures, fixtures contacting real external hosts, order dependence.
- F. Product bug: the product misbehaves (NPE, swallowed exception, cache race, wrong status). Prove it from bytecode plus the log sequence; state whether a sound test-side mitigation exists without relaxing anything.
- G. Expected / benign: one table with message, count and the reason it is harmless or asserted.
- H. Open questions: what could not be proven and what evidence would prove it.
4. Root-cause
For every non-benign item: read the step definition and helper, the feature lines, the cleanup path
(utils/ResourceCleanup.java), the readiness/heal gates, the block's overlays, and the product bytecode on any stack
trace. State the causal chain end to end (step → client → gateway/REST → product class → log line).
5. Fix and verify (when the invocation includes fixing — always under test-coverage)
Implement each test-side fix you are confident about, minimally and in the style of the surrounding code; comments
state what the code IS, never history. Product bugs: propose, never patch the product. Re-run the block; record Tests
run/Failures and the log evidence per fix. If a feature file changed, regenerate the coverage tree
(python3 all-in-one-apim/modules/integration-v2/docs/devs/render_coverage_tree.py, require invalid: 0 and
not-executed: 0).
6. Report
Use references/report-template.md exactly: per-block header with counts, sections 0/A–H (and I–K when fixing),
tables with block-prefixed IDs (<label>-A1), file:line and timestamped log evidence in every row, "None found" plus
what was checked for empty sections.
Gate criteria (when used as the test-coverage Phase 6 gate)
The block passes the gate only when, for the scenarios added or changed by the work under review:
- no item in A, B, C or D, no E item without a readiness/robustness fix, and no open question (H) about those scenarios;
- every ERROR/WARN they produce is in G with a verified reason, or is a catalogued product issue (F) the change does not make worse;
- every new negative assertion is exact (status AND error code/message field where the product provides one) and has been shown able to fail;
- the cleanup sweep logs no WARN indicating a failed or unverified delete, deregistration, or unsubscribe, including
may leak,may have leaked,Cleanup failed to delete,Failed to delete ... on the external IS,cannot delete,deleteUser ... returned HTTP,deregister ... returned HTTP,Cleanup failed to deregister,STILL EXISTS,NOT deleted, an IS sweep skipped despite registered resources, a WebSub unsubscribe or removal that failed, or persisted WebSub rows that remain, differ, or could not be verified after unsubscribe, and the server logs no not-found ERROR from cleanup re-deletes; - the complete block (every runner, verbatim
thread-count, order and parameters) ran green after the fixes, with the run cited in the report; a run of a subset of the block does not count. Pre-existing findings in untouched scenarios are reported but do not block; list them separately.
Scope boundaries
Do not run the full suite from this skill; one block (or a focused suite of the blocks under review) at a time.
Never stage or commit; leave changes for the developer. Delete every temporary focused suite (zz-*.xml) and scratch file
the invocation created before finishing, and confirm with git status --porcelain. On a shared Docker host, never prune, restart the engine, or
stop containers/processes you did not start. Follow the repository's lock or concurrency convention for Docker and
Maven runs.
References
references/report-template.md— the report format and table schemareferences/defect-patterns.md— recurring A–F patterns, how to detect each, and the proper fixreferences/benign-catalogue.md— startup noise and messages that are harmless, with the conditions that make them soreferences/known-product-issues.md— established product bugs with signatures (re-verify before citing)references/evidence-and-pitfalls.md— attribution, bytecode reading, running a block, and traps that silently produce wrong conclusionsscripts/split_block_logs.py— split a run log into per-block carbon/test/IS logsscripts/digest_block_log.py— normalised ERROR/WARN inventory with counts and first/last timestamps
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

