My first open source contribution to the Elixir ecosystem! https://x.com/i/article/2109686585922887680
Covenant: Ensure your Elixir code keeps its promises
If nobody reads every line of code generated by our agents, what do we still review to know that what they're building is what we want?
My answer is the Contract. In Impatient Programming, the methodology based on how we work by at ZAR, every component carries a written Contract: what it promises, why each odd-looking value is what it is, and numbered obligations, each one proved by an automated test. The Contract is meant to be comprehensive enough that you could safely delete a component and rebuild it from the Contract alone, even if that's far from an everyday occurrence.
I'm currently rewriting one of our major internal systems in Elixir and did not find an open-source testing library that met my exact needs, so I built Covenant, a Contract layer for Elixir. Version 0.1.0 is on Hex as covenant, MIT licensed.
What it does
Covenant reads a Contract written in plain Markdown and YAML under contracts/, and fails the build when the Contract stops being true.
An obligation looks like this:
And the test that proves it names it:
Covenant.Case checks those tags when the module compiles. An untagged test, an unknown obligation id, or an id from another component is a compile error at the test's line. You can't add a contract test that proves nothing, and you can't cite an obligation that doesn't exist.
Then run mix covenant.check. Here it is on the example project in the repo:
That fails on a malformed obligation, a failing contract test, and an active obligation with no passing test. A skipped or excluded test doesn't count as passing. It prints contract coverage: the share of promises that are proved, not the share of lines that ran. contracts/coverage.json has no timestamp, so you can commit it and diff it in review.
Decisions that check their own premises
Real world contracts tend to accumulate values that look arbitrary: a cap of 100, a timeout of 30 seconds, a retry count of 3. Covenant gives each one a decision record with the reason, the rejected alternatives, and who approved it. A record can also state conditions that mix covenant.conditions checks against the running project:
When someone changes the default to 150 without touching the record, the record turns suspect and the build says so. The decision and the code can't drift apart quietly.
Mutation testing, credited to promises
Call mix covenant.mutate billing.invoices to break the component's code one small change at a time (flip a comparison, drop a guard, return nil) and run the component's contract tests against each broken version. Every kill is credited to the obligations on the tests that caught it.
That gives you a kill matrix per obligation instead of one score per file. You see which promises are actually guarded, which tests catch nothing on their own, and which surviving mutants are holes in the Contract rather than gaps in the tests. Every survivor has to be killed by a new test or sorted in a file with a reason a reviewer can check (equivalent, dead code, a decision deliberately left open). There is no score threshold to game.
Mutants are compiled in memory and hot-swapped into one booted worker per job, so it's fast: no recompiling the project per mutant. Each worker runs from a private copy of your project and checks after every mutant that the VM is still healthy.
Regeneration packs
The real test of a Contract is whether you can rebuild the component from it. mix covenant.regen billing.invoices writes a pack with everything an agent needs to rebuild one component and none of the old implementation: the obligations, the decision records, the system intent, the contract tests, and stubs for the public modules. Hand it to a fresh agent, rebuild, and run the contract tests. Every difference after the rebuild is a sentence that your Contract was missing.
It holds itself to the same standard
Covenant has its own Contract: 96 obligations across ten components, every test tagged, mix covenant.check in CI on Elixir 1.18 and 1.19. Its mutation engine is mutated with Covenant itself: 1,541 mutants, and every one of the 64 survivors is sorted in the repo with a reason you can check.
Siblings
Covenant is one of a family of Impatient Programming tools: flinch is the mutation gate for Go, qualm flags changes an experienced reviewer would push back on, and exhale catches duplication in Ruby. Covenant also reads the smaller Contract layout those tools share, so a polyglot team can keep one Contract.
Try it
Start with one component you care about. Write three obligations, tag the tests that prove them, and run mix covenant.check. Then run mix covenant.mutate on it and read the survivors. Each one is a promise you thought you'd made and hadn't.
- GitHub: https://github.com/tools4imps/covenant
- Hex: https://hex.pm/packages/covenant
- Docs: https://hexdocs.pm/covenant
- Impatient Programming: https://impatientprogramming.org
It's 0.1. The known limits are listed in the README. Issues and pull requests are welcome, and I'd especially like to hear from anyone who tries it on an umbrella app or a big Phoenix codebase.