agentleFS
Sign inStart free
MicrosoftKnown publishermicrosoft/vscode-python-environments/.github/skills/run-smoke-tests/SKILL.md

run-smoke-tests

Skill146 starsChanged 9 months agoMITView on GitHub

Run smoke tests to verify extension functionality in a real VS Code environment. Use this when checking if basic features work after changes.

We split this skill into the rules it gives an agent

8 rules. 0 are common, 5 are distinctive, and 3 are very specific to this project.

  • 0

    Common

    Most of the 5 edited versions we read keep these.

  • 5

    Distinctive

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run smoke tests to verify the extension loads and basic functionality works in a real VS Code environment.

    line 6
  2. Distinctive:

    After making changes to extension activation code

    line 10
  3. Distinctive:

    After modifying commands or API exports

    line 11
  4. Distinctive:

    Before submitting a PR to verify nothing is broken

    line 12
  5. Distinctive:

    When the user asks to "run smoke tests" or "verify the extension works"

    line 13
  6. Very specific:

    Create a new file in src/test/smoke/ with the naming convention *.smoke.test.ts:

    line 74
  7. Very specific:

    Use waitForCondition() instead of sleep() for async assertions

    line 106
  8. Very specific:

    Extension builds: Run npm run compile before tests

    line 121

“Common” is measured: kept by at least 6 in 10 of the 5 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Run smoke tests to verify the extension loads and basic functionality works in a real VS Code environment.

When to Use This Skill

  • After making changes to extension activation code
  • After modifying commands or API exports
  • Before submitting a PR to verify nothing is broken
  • When the user asks to "run smoke tests" or "verify the extension works"

Quick Reference

ActionCommand
Run all smoke testsnpm run compile && npm run compile-tests && npm run smoke-test
Run specific testnpm run smoke-test -- --grep "Extension activates"
Debug in VS CodeDebug panel → "Smoke Tests" → F5

How Smoke Tests Work

Unlike unit tests (which mock VS Code), smoke tests run inside a real VS Code instance:

  1. npm run smoke-test uses @vscode/test-cli
  2. The CLI downloads a standalone VS Code binary (cached in .vscode-test/)
  3. It launches that VS Code with your extension installed
  4. Mocha runs test files inside that VS Code process
  5. Results are reported back to your terminal

This is why smoke tests are slower (~10-60s) but catch real integration issues.

Workflow

Step 1: Compile and Run

npm run compile && npm run compile-tests && npm run smoke-test

Step 2: Interpret Results

Pass: 4 passing (2s) → Extension works, proceed.

Fail: See error message and check Debugging section.

Running Individual Tests

To run a specific test instead of the whole suite:

# By test name (grep pattern)
npm run smoke-test -- --grep "Extension activates"

# Or temporarily add .only in code:
test.only('Extension activates without errors', ...)

Debugging Failures

ErrorCauseFix
Extension not installedBuild failed or ID mismatchRun npm run compile, check extension ID
Extension did not become activeError in activate()Debug with F5, check Debug Console
Command not registeredMissing from package.jsonAdd to contributes.commands
Timeout exceededSlow startup or infinite loopIncrease timeout or check for blocking code

For detailed debugging, use VS Code: Debug panel → "Smoke Tests" → F5

Adding New Smoke Tests

Create a new file in src/test/smoke/ with the naming convention *.smoke.test.ts:

import * as assert from 'assert';
import * as vscode from 'vscode';
import { waitForCondition } from '../testUtils';
import { ENVS_EXTENSION_ID } from '../constants';

suite('Smoke: [Feature Name]', function () {
    this.timeout(60_000);

    test('[Test description]', async function () {
        // Arrange
        const extension = vscode.extensions.getExtension(ENVS_EXTENSION_ID);
        assert.ok(extension, 'Extension not found');

        // Ensure extension is active
        if (!extension.isActive) {
            await extension.activate();
        }

        // Act
        const result = await someOperation();

        // Assert
        assert.strictEqual(result, expected, 'Description of what went wrong');
    });
});

Key patterns:

  • Use waitForCondition() instead of sleep() for async assertions
  • Set generous timeouts (this.timeout(60_000))
  • Include clear error messages in assertions

Test Files

FilePurpose
src/test/smoke/activation.smoke.test.tsExtension activation tests
src/test/smoke/index.tsTest runner entry point
src/test/testUtils.tsUtilities (waitForCondition, etc.)

Prerequisites

  • CI needs webpack build: The extension must be built with npm run compile (webpack) before tests run. The test runner uses dist/extension.js which is only created by webpack, not by npm run compile-tests (tsc)
  • Extension builds: Run npm run compile before tests

Notes

  • First run downloads VS Code (~100MB, cached in .vscode-test/)
  • Tests auto-retry once on failure

Download now (ZIP)

  • Other versions5 projects use a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 5

    1. Kripu77/software-factory★ 7same name
    2. cogine-ai/cogine-dev-skillset★ 4same name
    3. dhanji4U/skills★ 2same name
    4. ElyzeSolutions/skills★ 1same name
    5. ArchitektApx/cursor-skills-for-claude★ 0same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • Used alongsideProjects with it also use loop-on-ci, check-compiler-errors and 4 more.

    Other files people use alongside this one.

    Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
    • cogine-aiSkill

      loop-on-ci

      Monitor PR checks and fix failures until green. Uses gh pr checks as the source of truth for PR-attached checks.

      Also in 5 projects that use this

    • cogine-aiSkill

      check-compiler-errors

      Run compile and type-check commands and report failures

      Also in 3 projects that use this

    • jaydubya818Skill

      make-pr-easy-to-review

      Use when a correct pull request needs clearer history, description, or reviewer guidance.

      Also in 3 projects that use this

    • jaydubya818Skill

      get-pr-comments

      Use when pull-request feedback needs to be fetched and summarized for action.

      Also in 3 projects that use this

    • +1 more from cogine-ai
    • +1 more from jaydubya818
  • In this repository8 other files it gives its agents.

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free