gus

Run Needle, Cactus Compute's on-device tool-calling model, inside your Go program. Text goes in, tool calls come out. The engine is linked into your binary and the weights download once on first use, so there's no Python, server or extra library to install.

• Tool calls: declare your tools as Go structs, the model picks the right ones and fills the arguments from what the user said.
• Extraction: turn messy text into a typed struct, like an invoice or a booking.
• Embeddings: a vector per sentence, for local search and routing.
• Speech: Whistle transcribes 16 kHz audio, or turns a spoken request straight into tool calls.

The models are small (35 MB for Needle, 17 MB for Whistle) and run on the CPU. A typical request takes 50–200 ms on a laptop.

Install

go get code.rbel.co/rubiojr/gus

You need Go 1.27 or newer, cgo and a C compiler (gcc or clang), and one of:

• linux/amd64 or linux/arm64 (glibc)
• darwin/arm64
• android/arm64, built with the NDK's clang as CC (the fyne and gomobile tools do that); the engine links the NDK's static libc++

Everything else is vendored. Other platforms, and CGO_ENABLED=0, fail to compile with an error saying so. Alpine and Intel Macs are out because upstream doesn't ship a static engine for them.

Quick start

package main

import (
	"context"
	"fmt"
	"log"

	"code.rbel.co/rubiojr/gus"
)

type weatherArgs struct {
	City string `json:"city" desc:"the city to report on"`
}

func main() {
	weather := gus.NewTool("get_weather", "Get the current weather for a city.",
		func(ctx context.Context, args weatherArgs) (any, error) {
			return map[string]any{"city": args.City, "temp_c": 27, "sky": "clear"}, nil
		})

	agent, err := gus.New(gus.WithTools(weather))
	if err != nil {
		log.Fatal(err)
	}

	resp, err := agent.Run(context.Background(), "how's the weather in Madrid and in Tokyo?")
	if err != nil {
		log.Fatal(err)
	}
	for _, r := range resp.Results {
		fmt.Println(r.Call.Name, string(r.Call.Arguments), r.Value)
	}
}

get_weather {"city":"Madrid"} map[city:Madrid sky:clear temp_c:27]
get_weather {"city":"Tokyo"} map[city:Tokyo sky:clear temp_c:27]

The first run downloads needle3.cact (35 MB, checksum verified) into your cache directory. After that it starts in a fraction of a second.

Tools

NewTool reads the arguments from a struct: json tags name them, pointer and omitempty fields are optional, and a few tags add constraints:

type setThermostat struct {
	Temperature int    `json:"temperature" desc:"target temperature in Celsius" min:"5" max:"30"`
	Mode        string `json:"mode,omitempty" enum:"heat,cool,auto"`
}

Supported tags: desc, enum, min, max, minlen, maxlen, minitems, maxitems, pattern and format.

Got a tools.json already? It unmarshals straight into []gus.Tool. Set Handler on each tool if you want Run to execute it.

Needle works best with a handful of focused tools. Cactus has a good guide on designing tools for Needle.

Tools can also be WebAssembly plugins that run in a sandbox. See Plugins.

Run or Complete

Run is the whole loop: it executes your handlers, feeds the results back to the model and returns every result. It won't execute a call whose arguments the request doesn't support (the model made them up). Those come back with gus.ErrUngrounded and the handler never runs.

Complete gives you one turn and leaves the calls to you. Use it when you want to decide what runs:

resp, err := agent.Complete(ctx, "dim the living room to 30")
switch {
case resp.Refused():
	// Off-topic, or no tool covers it: you get no calls rather than a guess.
case resp.Confidence != nil && *resp.Confidence >= 0.7 && len(resp.Ungrounded) == 0:
	// Sure enough: execute resp.Calls.
default:
	// Unsure: show resp.Calls (or resp.Suppressed) to the user and ask.
}

Confidence is calibrated, so the threshold is yours to pick. It's nil with fine-tuned weights that carry no confidence head.

resp.Timing has the engine's account of the turn: prompt and generation speed in tokens per second, peak memory, and for spoken requests the time to the first transcribed token.

Facts and conversations

Pass environment facts with gus.WithFacts("locale: en-US; device: phone"). Needle reads them as facts, not instructions, so don't write a system prompt there. The local date is added for you, so "tomorrow at 7" resolves.

Every call starts fresh by default. gus.WithConversation() keeps the history, so follow-ups work:

chat, _ := gus.New(gus.WithTools(setLights), gus.WithConversation())
chat.Complete(ctx, "dim the living room to 30")
chat.Complete(ctx, "make it 60 instead") // still the living room

Extraction and embeddings

type Invoice struct {
	Vendor string  `json:"vendor"`
	Total  float64 `json:"total"`
}

inv, err := gus.Extract[Invoice](ctx, "Invoice from Acme Corp, $1,200.00")
// &{Acme Corp 1200}

vec, err := gus.Embed(ctx, "turn on the kitchen lights") // 3072 floats

Extract returns gus.ErrNothingExtracted when there's nothing to extract, and a *gus.ExtractionError when the values aren't in the text.

Speech

Whistle takes 16 kHz mono audio, up to 30 seconds per call, in English, German, French, Spanish, Italian, Dutch and Polish. Its weights (17 MB) download on first use. gus/wav decodes WAV files.

f, _ := os.Open("clip.wav")
pcm, rate, _ := wav.Decode(f) // rate must be gus.SampleRate

t, _ := gus.Transcribe(ctx, pcm)
fmt.Println(t.Text) // Turn off the kitchen lights.

// Or audio in, tool calls out, in one go:
resp, _ := agent.CompleteAudio(ctx, pcm, gus.WithLanguage("en"))
fmt.Println(resp.Transcript.Text, resp.Calls)

gus.WithLanguage, gus.WithKeywords and gus.WithWordTimestamps tune all three calls. With none, Whistle detects the language.

For live audio, create a gus.NewStream(), Push about a second of audio at a time and Close it when you're done.

Models and offline use

The pinned weights download on first use into ~/.cache/gus on Linux or ~/Library/Caches/gus on macOS. That's the only thing gus writes to disk. Move it with gus.SetCacheDir(dir) or GUS_CACHE; the former wins. Android has no user cache directory, so an app must call SetCacheDir with a directory of its own, such as its files directory, before the first call.

To get the weights ahead of time, in a Dockerfile for example, call gus.Fetch(ctx, gus.Text, gus.Speech) or run gus fetch -speech. gus.CachePath(gus.Text) tells where they land. A gus release that pins new weights downloads them once and removes the old revision next to them.

To show a download, set a hook before the first call that may need one:

gus.SetProgress(func(kind gus.Kind, received, total int64) {
	fmt.Printf("\r%v model %d%%", kind, received*100/total)
})

It reports once as a download starts, as bytes arrive and once when they are all in, from the downloading goroutine. gus fetch prints the same on stderr.

gus.BuildInfo() says what a binary links: the gus version, the engine release and the pinned weights with their repos, commits and checksums. gus version prints it.

• GUS_OFFLINE=1 never downloads. Missing weights give gus.ErrNoModel.
• HF_ENDPOINT points at a Hugging Face mirror.
• gus.LoadFile(ctx, path) uses your own .cact instead, like a fine-tune. The file is memory-mapped, so loading costs no memory beyond the engine's own copy.
• gus.Load(ctx, data) does the same for bytes you already have, like weights embedded with //go:embed.

One engine per process

The engine holds one text model, one speech model and one conversation per process. Any goroutine can call gus, but calls run one at a time. A few things follow from that:

• Run keeps the engine for the whole loop. Other calls wait, and their context bounds the wait. A call that has started can't be interrupted.

• A tool handler can't call Complete, Run or Extract while its Run is going. You get gus.ErrBusy instead of a deadlock. Don't push those calls to another goroutine and wait on it, though: that one does deadlock. Embed and the speech functions work fine.

• When another agent takes over the engine, a Conversation agent gets gus.ErrPreempted on its next call, then starts afresh.

• One live transcription stream at a time. A new stream takes over an open one, which gets gus.ErrPreempted the same way, so a forgotten stream can't block speech.

• Embedding and transcription don't disturb a conversation.

• On Linux each model starts a pool of worker threads, one per CPU the process may use minus one, pinned to their cores. The engine caps the pool (11 on a 16-thread laptop). They sleep when idle and stay until the process exits. The pool follows the CPU affinity, so a container pinned to a few cores (--cpuset-cpus, taskset) gets a matching pool instead of a full one fighting over them.

• There is no Close. The engine can't unload a model, so the weights stay in memory and the goroutine that drives the engine stays until the process exits. If your tests check for leaked goroutines with goleak, ignore it:

  goleak.VerifyTestMain(m, goleak.IgnoreTopFunction("code.rbel.co/rubiojr/gus.engineLoop"))
  

• Agent is a struct, not an interface. To test code that uses it without the engine and the weights, declare the interface your code needs and satisfy it with a fake:

  type completer interface {
  	Complete(ctx context.Context, text string) (*gus.Response, error)
  }
  

  *gus.Agent satisfies it, and so does a fake that returns the Response you want. See Example_stub in the package docs.

CLI

go install code.rbel.co/rubiojr/gus/cmd/gus@latest

gus complete -tools tools.json "turn off the kitchen lights"
gus run -plugin weather.wasm "how's the weather in Madrid?"
gus inspect weather.wasm
gus tools install weather.wasm      # then gus run "..." finds it
gus tools install ./weather         # or build it from source first
gus tools install example.com/weather@latest
gus listen -tools tools.json clip.wav
gus listen -tools tools.json -mic      # speak, then Ctrl+C
gus transcribe -timestamps clip.wav
gus transcribe -mic                    # live, until Ctrl+C
gus devices
gus fetch -speech
gus version

complete and listen print the calls the model makes. run also executes them and prints the results: the tools come from -plugin files and the installed plugins, since a tools.json tool has nothing to run. inspect prints what a plugin declares before you run it.

-mic records from the default microphone, or from the one -device names (gus devices lists them). listen -mic records until Ctrl+C or 30 seconds, then answers. transcribe -mic prints the words on stderr as it hears them and the whole transcript as JSON on stdout when you stop it. Microphones are recorded at their own rate, since many take no other, and resampled to the 16 kHz Whistle needs. WAV files can be at any rate too.

The CLI records through PortAudio, so building it needs the PortAudio and JACK development files. The library doesn't: go get code.rbel.co/rubiojr/gus brings no audio libraries in.

Platform	Packages
Fedora	portaudio-devel pipewire-jack-audio-connection-kit-devel
Debian, Ubuntu	portaudio19-dev libjack-jackd2-dev
macOS (Homebrew)	portaudio

Plugins

A tool can be a WebAssembly plugin: a Go program built for wasip1 against gus/wasm/sdk, run by gus/wasm in a sandbox with no filesystem, no environment and no sockets. The one way out is HTTPS to the hosts the plugin declares, and the host makes those requests on its behalf. A weather tool reaches the weather API and nothing else, whoever wrote it.

//go:build wasip1

package main

import (
	"context"
	"encoding/json"
	"net/http"
	"net/url"

	"code.rbel.co/rubiojr/gus/wasm/sdk"
)

type args struct {
	City string `json:"city" desc:"the city to report on"`
}

func init() {
	sdk.Hosts("wttr.in")
	sdk.Tool("get_weather", "Current weather for a city.",
		func(ctx context.Context, a args) (any, error) {
			resp, err := http.Get("https://wttr.in/" + url.PathEscape(a.City) + "?format=j1")
			if err != nil {
				return nil, err
			}
			defer resp.Body.Close()
			var out struct {
				Current []map[string]any `json:"current_condition"`
			}
			return out.Current, json.NewDecoder(resp.Body).Decode(&out)
		})
}

func main() {}

GOOS=wasip1 GOARCH=wasm go build -buildmode=c-shared -o weather.wasm .
gus inspect weather.wasm
gus run -plugin weather.wasm "how's the weather in Madrid?"

Or in your program:

p, err := wasm.LoadFile(ctx, "weather.wasm", wasm.WithConfig(map[string]string{"API_KEY": key}))
agent, err := gus.New(gus.WithTools(p.Tools()...))

• The arguments struct follows the rules of NewTool, tags included: the plugin and gus share the schema code. The host never calls main, so declare everything from init.
• net/http works as is inside the plugin: the SDK routes every request through the host, so a library that takes an http.Client, or uses the default one, works unchanged. Requests from several goroutines overlap.
• sdk.Keys declares the configuration the plugin reads with sdk.Config, such as an API key, which stays with the host: wasm.WithConfig supplies it, and the CLI reads it from the environment. A plugin reads the keys it declares and no other.
• Each call runs in a fresh instance, which costs a few milliseconds, so calls share nothing and a call that dies takes nothing down. The compiled code is cached under the gus cache directory: a 14 MB plugin loads in about 2.5 s the first time and 0.1 s after that.
• The limits per call are 30 s, 128 MB of memory and 8 MB per response, HTTPS only, with every redirect checked against the declared hosts. A plugin that ignores its deadline is stopped. wasm.WithTimeout, wasm.WithMemoryLimit, wasm.WithMaxResponse and wasm.WithAllowedHosts, which refuses a plugin that declares more than you allow, change them.
• The plugin's stderr and log output go to the host's stderr, or where wasm.WithStderr says. The clock and random numbers work.
• Plugins run on wazero, in pure Go. go get of gus brings it in only when you import gus/wasm. A Go plugin is 4 MB, or 15 MB with net/http; the SDK does not need TinyGo, though it would shrink them.

examples/tools/headlines and examples/tools/aemet are two plugins on real libraries: news headlines from RSS feeds, and Spanish weather forecasts behind an API key.

Installing plugins

Install a plugin once and every gus run, complete or listen picks it up, unless you pass -installed=false. A tool given with -plugin or -tools wins over an installed one of the same name:

gus tools install aemet.wasm
gus tools config aemet AEMET_API_KEY=...   # stored for it, private to you
gus tools list
gus tools disable aemet                    # installed, but out of the toolset
gus tools enable aemet
gus tools remove aemet

Needle works best with a handful of tools, so keep the enabled set small and park the rest with disable.

install also builds plugins from Go source, with the go command, for you:

gus tools install .                                  # the plugin in this directory
gus tools install example.com/tools/weather@latest   # fetched like go install would
gus tools install example.com/tools/weather@v1.2.0

It checks first that the package is a main package using gus/wasm/sdk, then builds it for wasip1 and installs it under its package name. A remote plugin installs at the exact version resolved, which gus tools list shows; run install again to upgrade. Like go install, a remote install refuses a module with replace directives in its go.mod, since a remote build would ignore them: clone it and install the directory. Programs can do the same with build.Dir and build.Remote from gus/wasm/build, then r.Install.

The same registry is a package, so your program can share the installed plugins with the CLI, or keep its own:

r := registry.Open()                          // the CLI's: ~/.config/gus/tools, or $GUS_TOOLS
r = registry.Open(registry.WithDir(appDir))   // or your program's own directory
ts, err := r.Load(ctx, registry.WithConfigFunc(os.LookupEnv))
defer ts.Close(ctx)
agent, err := gus.New(gus.WithTools(ts.Tools()...))

Load takes the enabled plugins. registry.WithPlugins("aemet") picks for one load without changing anything, while r.Enable and r.Disable persist, so every program sharing the directory agrees. r.Install, r.Remove, r.SetConfig, r.List and r.Get do what the commands do. registry.WithWasmOptions passes wasm options to every plugin.

The files live behind a small registry.Store interface: a directory by default or with WithDir, any backend of yours with WithStore, and any fs.FS read-only with WithFS, so a program can ship its tools embedded:

//go:embed tools/*.wasm
var tools embed.FS

sub, _ := fs.Sub(tools, "tools")
ts, err := registry.Open(registry.WithFS(sub)).Load(ctx)

A plugin's settings and configuration live in a JSON file next to it, which may hold API keys, so the directory store keeps the files private to the user.

Status

I've only run it on linux/amd64. linux/arm64 builds and links, but I haven't run it on real hardware yet. darwin/arm64 should work, but it's untested.

License

gus is MIT. The vendored Needle engine in lib/ is Cactus Compute's, Apache-2.0 (see lib/LICENSE). On Linux it's linked with parts of LLVM's libc++ (Apache-2.0 with LLVM exception). The Needle and Whistle weights are downloaded from Hugging Face under their own terms.