soft-prompting-gradient-descent-continuous-vectors

IN premise — summaries/2026/08/24/wiki-In-context_learning_natural_language_processing-chunk-1.md

Created 2026-08-24T17:11:12+00:00

Prefix-tuning (Li & Liang 2021) and prompt tuning (Lester et al. 2021) learn continuous floating-point prefix vectors via gradient descent to maximize output log-likelihood (arg max_X̃ Σᵢ log Pr[Yⁱ | X̃ * Xⁱ]), as a parameter-efficient alternative to discrete token-level prompting.

Summary

Rather than hand-writing a specific text prompt to guide a language model's output, these methods let gradient descent search for an optimal set of continuous numerical values to insert before the input. This gives a way to steer a large model's behavior by training only a handful of learned parameters, avoiding the cost of updating the full network or searching over discrete words.