How we eval every release.
Lalakai measures every Hunter release through a comprehensive evaluation suite designed to quantify reasoning, coding, tool use, and security capabilities. Our evaluation framework is reproducible, transparent, and designed to give you confidence in the numbers.
In this post we walk through the exact metrics we track, the datasets we use, and how you can run the same evaluations on your own machine.
Before release
No Lalakai model ships until it meets our standards, and we set those standards before the release. Security and safety are part of them for every model we make, not only the ones trained for cybersecurity.
We test internally on every training run, not only the final one. If a change to the data or the training method makes a model worse, we see it on that run. A model that doesn't meet our standards doesn't ship.
Security testing.
Our internal security team reviews the security tests before we use them.
Safety on every model
Safety on every model Our security models get the most attention, but the same practices apply to all of our models.
Intended and out-of-scope uses. Every model card says what the model is for and what it isn't for. For Hunter 1.7B, out of scope includes offensive use, testing systems you aren't authorized to test, and acting on its output without human review.
Stated limitations. The Hunter 1.7B card says it was trained on 1,024-token sequences and that its behavior on longer inputs hasn't been tested. The Hunter 4B card says it can hallucinate and can produce unsafe content when prompted adversarially, and that you should add your own filtering in production.
A person reviews the output. Both cards say security output should be checked by a qualified person before anyone acts on it.
Curated training data
We don't train on scraped web data. We build curated datasets with explicit reasoning traces, security-focused examples, and verified code, and every example goes through several rounds of review.
Hunter 4B was trained on about 180,000 curated, deduplicated examples, and Hunter 1.7B on 2,500 curated cybersecurity examples.
Verifiable downloads
Hunter 4B is released as GGUF files, which store only the model's weights, with SHA-256 checksums so you can confirm a download matches what we published.
Local by default
The models run on your machine, so your prompts, logs, and code don't go to us.
Benchmarks
Hunter 4B and Hunter 1.7B don't have published benchmark results. Our evaluation of them has been internal. The next iterations of these models, and every new model after them, will come with benchmarks.