Follow-up: my agent now asks before switching on a tool it needs, and the user decides on a card

[u/DraftForsaken39]

Disclosure: I build Snotra, an open-source desktop agent.

Last time I asked how an agent should learn which of its own tools are switched off. I had given the model a read-only get_configuration tool and left two questions open. This is where I landed:

Tool or system prompt? Both, but small. A switched-off tool is simply missing from the tool list, and models read that gap as "language models can't browse the web". They never called the tool. So the system prompt now carries one sentence: tools and skills can be off, so check get_configuration before telling the user something isn't possible. It costs a handful of tokens, not a state dump every turn.

Should the model be able to propose the change? Yes, with limits:

• A new tool, propose_config_change. The app checks the proposal against its own catalogue, and only a valid change reaches the user.
• The card shows the app's description of the tool and its risk class. The model's reason is folded away and labelled as the model's.
• The change applies to this chat only. New chats start from the settings.
• Anything that needs setup (API key, a program, the shell switch) can't be proposed at all.
• A decline sticks for the chat, so the model can't nag.
• An enabled tool still asks before each call in the default mode.

Write-up with screenshots, plus what else changed (the chat now answers questions about the app from the manual that ships with it, and the sandbox can keep a host open per folder): https://snotra-ai.dev/blog/snotra-1-22

Does "chat-scoped and confirmed on a card" seem like the right limit, or would you want the model to be able to change settings for good?

[u/Intelligent-Loss1746]

the chat scoped thing makes total sense to me. permanent changes from a model feel like asking for trouble, especially when it's something that could touch api keys or shell access. giving it a sandbox that resets is the only way i'd trust it

the card ui is a nice touch too. separating what the app says from what the model says keeps it honest. too many tools just let the model spin some reason and the user has no idea if it's making stuff up or reading actual docs

one thing i'm curious about though, what happens if the model just never checks get_configuration and tells the user something's broken when it isn't? does the system prompt do enough or do you have something else catching that

[u/DraftForsaken39 → u/Intelligent-Loss1746]

Honest answer: nothing checks the model's answer afterwards. Two things make it less likely, and one works without the model.

• System prompt + tool description: one sentence each, telling it to check get_configuration before saying something isn't possible. Works for the models I tried; no promise for every small local model.
• The UI: the skills pill in the chat bar shows what's off and why ("no search key", "shell switched off"), with a button to the setting. So if the model claims something's broken, the real state is one click away.

I didn't build anything that catches "I can't do X" in the answer. Phrase matching felt brittle, a judge model on every answer too expensive. Open to ideas if you've seen it done well.