Agent skill portability testing should ask what the environment can actually provide. Copying a SKILL.md file does not install a command, create an input or make an output directory writable. Our local documentation-checking lab makes those boundaries visible.
We exercised seven controlled scenarios with one unchanged Python checker. Two passed, three stopped on missing prerequisites, and two failed their functional checks. All seven matched the expected classification. These results describe a deterministic local harness, not seven agent clients or an AI success rate.
Why agent skill portability testing needs a contract
GitHub’s 12 November 2026 guide to agent skills separates reusable instructions from the host’s tools, context and permissions. Its Data Forge example keeps project configuration in a manifest and uses seven workflow phases with human checkpoints.
The guide also describes Proof Agent’s static diff review. Its demonstration returns evidence-based findings but does not execute tests. That distinction matters: an instruction package can describe a check without proving that a particular host ran it. Our experiment therefore retains the command result and the actual local output separately.
One small task, explicit boundaries
The fictional task checks whether a project’s documentation contains a required heading. A versioned JSON input names that heading. The contract declares the local Python command, input path and output path. The checker may read the declared files and write its result; it has no network or credential task.
The result records the input’s SHA-256, the schema version, the required heading and whether the heading was found. Our runner then checks those fields against the fixture. This makes a plausible-looking output file insufficient on its own. It also gives a reviewer a concrete artifact to inspect.

How the seven scenarios were constructed
First, we ran the same checker in two complete fictional project folders. Both contained the version-one input, a documentation file with the required heading and a writable output directory. Each run returned PASS and produced a result that matched the fixture.
Next, one copy declared a command that did not exist. Another lacked its input file. A third denied writing to its disposable output directory. Each stopped as BLOCKED. The denied-write case recorded the actual permission error; the workflow did not bypass that restriction to produce a success result.
Finally, one input declared an unsupported schema version, and one document lacked the required heading. Those cases returned FAIL. The first stopped before writing a result. The second retained a valid result with found set to false. A useful failure report can therefore be a correctly produced artifact.

A file is evidence only after checking its meaning
The two passing runs and the missing-heading run wrote result files. All three matched our artifact checks. However, only two reported that the required heading existed. We therefore distinguish verified evidence from a successful task outcome.
For example, a result can truthfully say found is false. Calling that file “verified” means the record agrees with its input and document. Calling the task PASS would contradict it. This distinction helps prevent a workflow from treating the mere presence of a report as completion.

What we did not test across agent clients
We included a sample SKILL.md to explain the intended task and stopping rules. The harness calls the Python checker directly. It does not demonstrate that an agent client recognised the skill, loaded its references, interpreted the instructions correctly or selected the right tool.
Consequently, execution in different agent clients remains untested. The compatibility register has separate fields for the client, version, skill recognition, reference loading, available runtime, permissions and output evidence. Fill them only after an observed run. Copying a file into another client’s directory is not that run.
We also generated no client-specific packages, so this lab makes no package-drift claim. If a future project produces several client formats from one source, a separate comparison should check whether those generated files still match. That is an additional test with its own evidence.
A practical environment-contract checklist
Before running a reusable workflow, name the required files and their schema. Next, identify the real command and version the host will provide. Then declare the output destination and allowed actions. Finally, define what counts as PASS, FAIL, BLOCKED and UNTESTED, including the evidence needed for each.
Tech Consulting Tailored to Your Coding Journey
Get expert guidance in coding with a personalized consultation. Receive unique, actionable insights delivered in 1-3 business days.
Get StartedKeep secrets out of that manifest. If a workflow needs an external account, record the kind of access required rather than copying a token into a shared package. Actual authorization must come from the relevant user and service. A local configuration file cannot create it.
Use the same evidence discipline in other projects: our PostgreSQL restore experiment checks application answers after command success, while the incident-interval project keeps measurement scope separate from arithmetic.
Download the contract and observed results
- All seven executed environment-contract results (CSV)
- Environment-contract field dictionary (CSV)
- Blank agent-client compatibility register (CSV)
- Executed and untested scope register (CSV)
- Local checker, fixtures, sample skill and retained logs (ZIP)
Methodology and limits
We executed the harness on 15 November 2026 using Python 3.9.6 on macOS arm64. Each scenario used a fresh temporary fictional project. The missing-command path was deliberately absent. The denied-write fixture changed only its own temporary directory and restored that directory’s permissions for cleanup after recording the blocked result.
The seven expected classifications matched, and three emitted artifacts passed the stated checks. These are narrow engineering observations. They do not establish cross-client compatibility, model independence, security against malicious inputs or reliability for a larger agent workflow. Extend the contract and tests when the task’s actual boundaries change.
