Blyx

ModelGate

Release gate and CLI · 0.1.2

Compare two binary classifiers on the same observations and stop a release when mistake costs or Brier score regress.

Install ModelGate in your editor

Extension RahulChaube.blyx-modelgate · 0.1.0

The extension includes a runnable Blyx example, function completions and the framework source. For the modular project, extract the starter, open its folder and run blyxc native-run main.blyx with the native toolchain installed.

Create an empty .blyx file, type blyx-modelgate-demo and accept the completion. Choose Blyx: Run File to run the example.

For an offline installation, choose Extensions: Install from VSIX for Blyx Language first, then for ModelGate.

Download the companion Blyx Language VSIX

Run the command-line tool

Extract the archive for your machine and run the included CSV example. Use .\modelgate.exe in PowerShell. The input header is label,baseline,candidate.

./modelgate fixtures/regression.csv --fn-cost 5

Returns exit code 1. Both models get three rows right, but the candidate misses the more costly positive.

brew install rahulchaube1/blyx/modelgate

On Windows x64, add the Blyx Scoop bucket once, then install the tool from PowerShell.

scoop bucket add blyx https://github.com/Rahulchaube1/scoop-blyx
scoop install blyx/modelgate
Scoop manifest

Install with Conda

Create an environment containing ModelGate 0.1.2, its Blyx source and a runnable example. Run the second command with Blyx 0.2.1, Rust and a host linker installed.

conda create -y --prefix ./blyx-env --override-channels -c https://rahulchaube1.github.io/blyx-packages blyx-modelgate=0.1.2=0
blyxc native-run ./blyx-env/share/blyx/modelgate/main.blyx

To use this package in your application, copy share/blyx/modelgate/vendor/modelgate from the environment into your project's vendor directory.

Install the full collection with Homebrew or Scoop

Import into a Blyx project

Install Blyx 0.2.1, Rust 1.85 or newer and a host linker. From the directory containing your entry file, clone this tagged release.

git clone --depth 1 --branch v0.1.2 --recurse-submodules https://github.com/Rahulchaube1/blyx-modelgate.git vendor/modelgate

Save this as main.blyx beside vendor and run blyxc native-run main.blyx.

use vendor::modelgate::gate;
fn main() {
    let baseline_cost = 1.0 / 4.0;
    let candidate_cost = 5.0 / 4.0;
    println!(baseline_cost);
    println!(candidate_cost);
    println!(gate_passes(0.105, 0.105, baseline_cost, candidate_cost, 0.0));
}

Expected output

0.25
1.25
false

API and input requirements

Use paired labels and probabilities in [0,1]. A missed positive can carry a different cost from a false alarm. The sample has two models at 75% accuracy. With a miss costing five times as much, cost per prediction rises from 0.25 to 1.25.

APIUse
gate_passes(baseline_brier, candidate_brier, baseline_cost, candidate_cost, tolerance)Apply the gate to validated per-row metrics.
--threshold, --fn-cost, --fp-cost, --brier-toleranceConfigure costs and the maximum increase in mean Brier score.

Develop and contribute

Open the tagged source to read the implementation and examples. Include the smallest reproducible Blyx program, compiler version and expected result when reporting an issue.