What is reinforcement learning from human feedback (RLHF)?

Definition and Overview

Reinforcement Learning from Human Feedback (RLHF) is a machine learning technique that fine-tunes AI models based on human-provided feedback. At its core, it involves three main components:

• A pre-trained AI model (often an LLM)
• Human feedback collected on the model's outputs
• A reinforcement learning algorithm that updates the model based on this feedback

The fundamental idea is to train a model such as an LLM not just to perform a task (like predicting the next word) but to perform it in a way that humans find high-quality and preferable. This often involves training a separate “reward model” that learns to predict which outputs humans would rate higher.

This trained reward model then acts as the reward function within a standard reinforcement learning loop, guiding the original AI model to generate outputs that maximise the predicted human preference score.

It’s an approach in work that allows deep learning models to learn more human and complex, subjective qualities like tone, safety, and helpfulness that are difficult to capture with traditional metrics.

The Role of Human Feedback

Human feedback is the cornerstone of the RLHF process. Its primary role is to inject nuanced human judgment into the AI neural networks dataset training loop, guiding the model beyond simple task completion towards qualitative alignment with desired behaviours.

Defining qualities like “helpfulness,” “harmlessness,” or “truthfulness” programmatically is highly challenging. Humans, however, can intuitively assess these attributes in AI-generated outputs.

In a typical RLHF workflow, humans don't necessarily write perfect text answers or provide detailed corrections. Instead, they often compare different outputs generated by the AI for the same prompt and indicate which one they prefer (e.g., ranking responses from best to worst).

This comparative feedback is generally easier and more scalable for humans to provide to a model than detailed critiques, writing ideal responses from scratch, or developing a data policy.

RLHF Models and Training

Once the foundational concepts of RLHF and the importance of human feedback for a model like an LLM are understood, it’s worth delving into the specific models and training procedures involved.

This typically involves a high-quality multi-stage process in which human preferences are first captured in a specific model, which is then used to guide the fine-tuning of the main AI model using reinforcement learning algorithms.

Training Algorithms for RLHF

The core of RLHF lies in fine-tuning the Generative AI model (e.g., an LLM) using reinforcement learning guided by the signal derived from human feedback.

While various RL algorithms could be used, the most common and successful approach employed in practice, especially for large language models, is Proximal Policy Optimization (PPO). PPO is favoured for several reasons:

• Stability and reliability: Compared to simpler policy gradient methods, PPO incorporates mechanisms (like clipping the objective function) that prevent substantial updates to the model's policy (its strategy for generating text) in a single step. This leads to more stable and reliable dataset training.
   
• Sample efficiency: It generally balances the amount of data used in the dataset responses and data lakehouse (sample efficiency) and the ease of task implementation and tuning compared to some other complex RL algorithms.
   
• Maintaining capabilities: A crucial aspect of fine-tuning large pre-trained models is ensuring they don't “forget” their original capabilities or start generating nonsensical text while optimising for the new reward.

PPO often includes a penalty term (typically based on KL divergence) that discourages the fine-tuned model from deviating too drastically from its original, pre-trained behaviour.

This works to maintain language fluency and general knowledge while adapting to human preferences. The RL training loop using PPO in RLHF generally works as follows:

• A language prompt is sampled and fed into the current version of the AI model (the policy).
• The model generates a response.
• The human reward function (detailed below) evaluates the generated language response to get a scalar reward score.

The PPO algorithm uses this reward score and the KL divergence penalty to calculate an update for the AI model's parameters, aiming to increase the likelihood of generating responses that receive higher reward scores in the future.

Human Reward Function in RLHF Models

A potential point of confusion is the term “human reward function.” In RLHF, humans don't directly provide a feedback reward score during the main RL-supervised training loop.

Doing so with dataset feedback rewarding every step for every task would be incredibly slow and impractical when training an LLM or another model.

Instead, the human responses collected earlier (e.g., comparisons, rankings) is used to train a separate model known as the reward model (RM). This reward model acts as the reward function during the RL fine-tuning phase. Here’s how the reward model is typically built and used:

• Data collection: Humans provide preference quality feedback on pairs (or sets) of model outputs for various prompts, indicating their preferences (e.g., "Response A is better than Response B”).
   
• Reward model training: A separate model (often initialised using the same base pre-trained model as the one being fine-tuned but with a different output head) is trained on this preference language data. Its goal is to predict the quality preference score or rating a human would likely give to any given model output.
   
• Proxy for human judgment: Once the function is trained, the reward model is an automated, scalable text proxy for human judgment. During the PPO fine-tuning stage, when the main AI model generates a response, that response is fed into the trained reward model. The scalar output from the reward model is then used as the reward signal that the PPO algorithm tries to maximise when performing tasks.

So, the trained reward model uses feedback to internalize human preferences from the collected dataset and provides the necessary signal to guide the RL algorithm, allowing the main AI model to be optimized efficiently to generate outputs that align with those learned preferences.

The quality and robustness of this reward model are critical to the overall success of the RLHF process, including for a GPT LLM.

Application of RLHF in Language Models

While the principles of Reinforcement Learning from Human Feedback have broader alignment, its most significant impact has been realised within the domain of large language models (LLMs) and the quality of output produced.

RLHF has become a cornerstone language technique for refining these robust AI systems' capabilities, quality, and behaviours as RLHF moves beyond mere text prediction towards more sophisticated and aligned language interactions.

RLHF in Language Model Training

The training task of modern large language models often involves multiple stages, depending on the data policy. Initial pre-training on vast text corpora endows models with grammatical understanding, factual knowledge, and pattern recognition.

This is frequently followed by supervised fine-tuning (SFT), where the language model learns to follow specific instructions or adopt particular response styles, such as behaving like a helpful assistant, based on curated examples.

However, SFT and data and dataset policy alone often struggle to fully capture the subtleties of human preferences regarding qualities like preference for helpfulness, harmlessness, tone, or factual honesty, especially when desired trained outcomes are complex or subjective.

Alignment of RLHF with Natural Language Processing

The application of RLHF in LLM and GPT training is deeply connected to the broader challenge of AI alignment within Natural Language Processing (NLP) and GPT use.

Alignment, in this context, refers to ensuring that AI systems, particularly LLMs with vast capabilities, understand and act following human intentions, goals, and ethical values.

A high level of misalignment can manifest in various ways, from generating subtly biased or untruthful content to failing to follow trained instructions faithfully or producing harmful outputs. Given the complexity of language and human values, specifying desirable behaviour comprehensively through code or explicit rules is often intractable.

RLHF offers a practical, quality approach to tackling this alignment problem directly within NLP systems and with the right data policy. Rather than attempting to pre-define every aspect of desired behaviour, RLHF learns these preferences implicitly from human feedback.

By training a reward model to recognise the characteristics of quality responses that humans deem “good” (helpful, honest, harmless, etc.), RLHF creates a functional proxy for human values that can be integrated into the supervised training process.

The subsequent reinforcement learning phase then optimises the LLM and GPT responses and policy to respond to tasks and produce text that scores highly according to this learned proxy, effectively steering the model towards better alignment with human preferences.

This results in the use of large language models that are better aligned and more useful and safer across a range of NLP applications, including dialogue systems that converse more appropriately, summarisation tools that produce more relevant summaries, and content generation systems that better align with safety data and policy and used user and GPT intent.

Challenges and Future of RLHF

Despite its success in improving language models, Reinforcement Learning from Human Feedback has challenges in outcomes and quality.

Ongoing research and development continue to explore ways to mitigate its limitations and understand its broader impact on AI-supervised training methodologies. Key areas of focus include the quality of human feedback and the interplay between RLHF and established supervised learning techniques.

Overcoming Annotation Bias in RLHF

The effectiveness of GPT responses achieved through RLHF depends on the human feedback, data, and policy used to train the reward model. This dependency introduces a significant challenge: annotation bias.

The preferences, encoded into the reward model and subsequently into the fine-tuned LLM and GPT, directly reflect the judgments of the specific group of human annotators who provided the feedback during the learning task.

If this group is not sufficiently diverse or the annotation process introduces biases, the resulting AI model may exhibit skewed perspectives, unfair biases, or fail to align with the values of a broader user base.

Sources of model and trained dataset quality bias can range from the demographic makeup of the annotators to the specific instructions they are given, depending on the data policy, which might inadvertently steer their preferences.

High annotator fatigue, varying levels of effort, or differing interpretations of subjective criteria like “helpfulness” can also introduce noise and inconsistency in high quality. There's also the risk of converging on easily agreeable or majority viewpoints, potentially penalising valid but less common perspectives.

Impacts on Supervised Learning with RLHF

Reinforcement Learning from Human Feedback does not operate in isolation; it has a complex and synergistic relationship with supervised learning (SL), particularly supervised fine-tuning (SFT), within the typical LLM and GPT supervised training pipeline.

RLHF data and policy should not be seen as a replacement for SFT tasks but rather as a complementary refinement stage. SFT plays the crucial role of initially teaching the model foundational instruction-following capabilities, specific response formats, and core skills based on curated examples of desired outputs. This provides a necessary baseline of competence.