cloudai-workload-contribution

This skill is a set of guidelines when working on workloads implementation

Implementation guidelines

• User configs are trusted data. Don't over-validate test definitions
• The workload implementation is intended to be pass-through, which means that CloudAI:
  • defines benchmark shape (processes, installables, etc.)
  • translates TOML test config into workload interface submission so that user can use the workload fully
  • doesn't (re-)define underlying workload parameters unless required for workload submission and clean code

• CloudAI cannot support every possible cluster-specific hardware/software setup in terms of benchmark startup/finalization. Prefer generic solutions instead of specific technologies support in the workloads implementation. The generic solutions are:

  • container mounts
  • installables
  • pre/post-srun scripts (custom per workload; some workloads already support it)
  • pre/post-tests for heavy lifting hooks that need an srun
• Fetching workloads sources to understand how to use them. Use results/vendor-src folder for it. When working under a worktree, re-use this folder from the main repo checkout. Be aware of the checkout version. Have a single checkout for one code source (switch checkouts)
• Make the most of parent CloudAI command generation classes so that the workload supports all the builtin features, like single-sbatch, sbatch directives, pre/post-test hooks, DSE/CloudAIGym. When it's too complicated to support one
  • do not (80-20 rule)

Testing guidelines

• Don't produce too many unit-tests covering a small feature. More tests != better
• Prefer maintaining end-to-end tests (tests/test_acceptance.py). Cover distinct behavior and compatibility risks; avoid redundant cases rather than limiting the number of tests
• Existing workload TOML configs from conf/ must stay supported. One may extend them with new features; add scenario test cases for the new behavior and affected compatibility paths
• If a workload may result in diverse execution shapes (number of processes and their orchestration), then prepare that many test cases (not scenarios)