genread-inference-cost-70s

Summary

Using the GenRead elicitive prompting technique on GPT-NeoX 20B adds roughly 70 seconds of processing time per individual query. In practice, this sets a hard latency floor for any workflow built on that pairing, so systems relying on it must be designed around slow turn-around rather than real-time responsiveness, and batch workloads need to budget for that per-query overhead.