@fasterandworse@dgerard I mean, it is absurd. But it is how it works: an LLM is a black box from a programming perspective, and you cannot directly control what it will output.
So you resort to pre-weighting certain keywords in the hope that it will nudge the system far enough in your desired direction.
There is no separation between code (what the provider wants it to do) and data (user inputs to operate on) in this application 🥴
That’s the standard response from last decade. However, we now have a theory of soft prompting: start with a textual prompt, embed it, and then optimize the embedding with a round of fine-tuning. It would be obvious if OpenAI were using this technique, because we would only recover similar texts instead of verbatim texts when leaking the prompt (unless at zero temperature, perhaps.) This is a good example of how OpenAI’s offerings are behind the state of the art.
I mean, I hate myself for being this pedantic but technically there is code. But the code to run an LLM as it trains or generates responses is almost analogous to the hardware in the traditional hardware/software split.
I guess the layers are:
Actual hardware: GPUs etc
”The algorithm” / “The software hardware”: Matrix math, back propagation, etc
The configuration: a number of layers, number of parameters, etc
The … test suite?: training dataset
The app: a trained model
Data: prompts, including the prompt that is the entire conversation so far
I dunno. It’s harder than I thought to make an analogy between these layers.
@intensely_human yes, that’s about what I meant: you can’t make any directed changes to the actual code level, so the vendor has to make their customization at the same data level that users make their inputs. And that’s why there is no way to prevent users from overriding the initial prompt
Well, the vendor can also make their customization in the training data.
It’s hard, because it takes a lot more depth of connections to encapsulate a concept like “hide the following fact”, but just like with spies, the best time to thwart interrogation is during their training, not during their mission briefing.
@fasterandworse @dgerard I mean, it is absurd. But it is how it works: an LLM is a black box from a programming perspective, and you cannot directly control what it will output.
So you resort to pre-weighting certain keywords in the hope that it will nudge the system far enough in your desired direction.
There is no separation between code (what the provider wants it to do) and data (user inputs to operate on) in this application 🥴
That’s the standard response from last decade. However, we now have a theory of soft prompting: start with a textual prompt, embed it, and then optimize the embedding with a round of fine-tuning. It would be obvious if OpenAI were using this technique, because we would only recover similar texts instead of verbatim texts when leaking the prompt (unless at zero temperature, perhaps.) This is a good example of how OpenAI’s offerings are behind the state of the art.
I mean, I hate myself for being this pedantic but technically there is code. But the code to run an LLM as it trains or generates responses is almost analogous to the hardware in the traditional hardware/software split.
I guess the layers are:
I dunno. It’s harder than I thought to make an analogy between these layers.
@intensely_human yes, that’s about what I meant: you can’t make any directed changes to the actual code level, so the vendor has to make their customization at the same data level that users make their inputs. And that’s why there is no way to prevent users from overriding the initial prompt
Well, the vendor can also make their customization in the training data.
It’s hard, because it takes a lot more depth of connections to encapsulate a concept like “hide the following fact”, but just like with spies, the best time to thwart interrogation is during their training, not during their mission briefing.