I've been experimenting with the local model (LM) that Apple made available in the macOS Tahoe. It is just less than 4Gb and if you have that OS version, it will sit there mostly unused.
So I've created a CLI app that used that LM to accept natural language requests and execute the appropriate SF CLI command.
Examples:
SFAI list my orgs
SFAI open org SFResourcefulBear
SFAI show last 5 contacts
That experiment was done in 3 phases:
1️⃣ use LM as the primary driver of logic with one large prompt. The model had to understand the request, choose the SF CLI command, build the arguments, run the command, and summarize the result. It worked sometimes, but very unreliably.
2️⃣ hybrid planner, reducing the LM role. The program now produced a shortlist of candidate SF commands for the model to pick the best fit. Still fragile though.
3️⃣ LM for value extraction only. In the latest version the command decision is 100% deterministic and the model was asked only to provide the values from the user input such as org alias, record ids, etc.
Since that model is small, that is reasonable. It can be used just as a value extractor.
That kind of mimics what has happened with Agents. Now we have Agent Script that adds the determinism of programmatic instructions.
I think as people start to become aware of issues of relying too much on LLMs and have more realistic expectations - or the token costs go up dramatically, the prevailing pattern will be "do no let the model do everything".
How much can we delegate to the model? What is the right sized useful role the model can play without making the system fragile?
