I know a guy who was working on something like this, they just had the call to the model loop until the response met whatever criteria the code needed (e.g. one single number, a specifically formatted table, viable code, etc) or exit after a number of failed attempts. That seemed to work pretty well, it might mess up from time to time but it’s unlikely to (with the right prompt) do so repeatedly when asked again.
I’m currently a guy working on something like this ! It’s even simpler as you can have structured output on the chatgpt API. Basically you give it a JSON schema and it’s guaranteed to respond with JSON that validates against that schema. Spent a couple weeks hacking at it and i’m positively impressed, I have had clean JSON 100% of the time, and the data extraction is pretty reliable too.
The tooling is actually reaching a sweet spot right now where it makes sense to integrate LLMs in production code (if the use case makes sense and you haven’t just shoe-horned it in for the hype).
That’s fucking badass thanks for the pointer this might prove useful. In the structured output department i’m hearing great things about dotTxt’s outlines which lets you constrain output according to a regex, but i haven’t tested it yet.
Honestly the use case i’m working on is pretty mind blowing. User records an unstructured voice note like “i am out of item 12, also prices of items 13 & 15 is down to 4 dollars 99, also shipping for all items above 1kg is now 3 dollars 99” and the LLM will search the database for items >1kg (using tool calling) then generate a JSON representing the changes to be made. We use that JSON to make a simple UI where the user can review the changes - then voilà it’s sent to the backend which persists the change in database. In the ideal case the user never even pulls up the virtual keyboard on their phone, it’s just “talk, check, click, done”.
Human in the loop systems with LLMs really nicely deal with a lot of their problems. Very cool! Do you have specific change “types” that the system is able to propose? I guess restricting the response to the right types is covered by your JSON schema?
I know a guy who was working on something like this, they just had the call to the model loop until the response met whatever criteria the code needed (e.g. one single number, a specifically formatted table, viable code, etc) or exit after a number of failed attempts. That seemed to work pretty well, it might mess up from time to time but it’s unlikely to (with the right prompt) do so repeatedly when asked again.
I’m currently a guy working on something like this ! It’s even simpler as you can have structured output on the chatgpt API. Basically you give it a JSON schema and it’s guaranteed to respond with JSON that validates against that schema. Spent a couple weeks hacking at it and i’m positively impressed, I have had clean JSON 100% of the time, and the data extraction is pretty reliable too.
The tooling is actually reaching a sweet spot right now where it makes sense to integrate LLMs in production code (if the use case makes sense and you haven’t just shoe-horned it in for the hype).
This works well too, and with many different models: https://github.com/guardrails-ai/guardrails
That’s fucking badass thanks for the pointer this might prove useful. In the structured output department i’m hearing great things about dotTxt’s outlines which lets you constrain output according to a regex, but i haven’t tested it yet.
Fair play to Open AI - I still think LLMs are overhyped, but they’re moving things along constantly in impressive ways.
Honestly the use case i’m working on is pretty mind blowing. User records an unstructured voice note like “i am out of item 12, also prices of items 13 & 15 is down to 4 dollars 99, also shipping for all items above 1kg is now 3 dollars 99” and the LLM will search the database for items >1kg (using tool calling) then generate a JSON representing the changes to be made. We use that JSON to make a simple UI where the user can review the changes - then voilà it’s sent to the backend which persists the change in database. In the ideal case the user never even pulls up the virtual keyboard on their phone, it’s just “talk, check, click, done”.
Human in the loop systems with LLMs really nicely deal with a lot of their problems. Very cool! Do you have specific change “types” that the system is able to propose? I guess restricting the response to the right types is covered by your JSON schema?