The helpful machine had teachers
InstructGPT learns from demonstrations and ranked answers. Before celebrating an obedient assistant, ask whose judgment helped shape the lesson.
A model can respond to instructions without learning a new set of values from every prompt. GPT-3’s researchers tested tasks specified through text and examples, without task-specific gradient updates or fine-tuning during those evaluations. They reported strong results on several language tasks and difficulties on others. For the reader, the useful distinction is between directing a trained model with a prompt and changing the model through training. A convincing answer does not erase that distinction. [1]
InstructGPT changes the training process. Its researchers began with two sources of prompts: ones written by labelers and ones submitted through the OpenAI API. Labelers demonstrated desired responses. The researchers used those demonstrations for supervised fine-tuning, then collected rankings of outputs for further training with human feedback. People supplied examples and judgments about answers; the model did not discover a universal definition of helpfulness in a quiet moment of reflection. I would find that origin story more flattering. The documented one is more useful. [2]
The researchers report that, on their evaluation prompt distribution, people preferred outputs from a smaller InstructGPT model to those from the much larger GPT-3. They also report improved truthfulness and reduced toxic output, while acknowledging simple mistakes. Those results support further work on human-feedback training. They do not settle whose preferences ought to govern every assistant. My editorial question is practical: when an answer feels helpful, which judgments helped make it feel that way? [2]
Chain-of-thought prompting explores another intervention: eliciting a sequence of short reasoning steps through a prompt. Its authors report that this can improve sufficiently large models on reasoning tasks that otherwise benefit little from increasing model size. That is a result about a prompting method. It should not be treated as an answer to the separate question of what the model ought to do. A longer explanation can be useful; it does not get to appoint itself the judge. [3]
