DSPy: Transforming Language Model Calls into Smart Pipelines // Omar Khattab // MLOps Podcast

Join us at our first in-person conference on June 25 all about AI Quality: www.aiqualityconference.com/
MLOps podcast #194 with Omar Khattab, PhD Candidate at Stanford, DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines brought to us by ‪@WeightsBiases‬.
// Abstract
The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks. Unfortunately, existing LM pipelines are typically implemented using hard-coded "prompt templates", i.e. lengthy strings discovered via trial and error. Toward a more systematic approach for developing and optimizing LM pipelines, we introduce DSPy, a programming model that abstracts LM pipelines as text transformation graphs, i.e. imperative computational graphs where LMs are invoked through declarative modules.
DSPy modules are parameterized, meaning they can learn (by creating and collecting demonstrations) how to apply compositions of prompting, finetuning, augmentation, and reasoning techniques. We design a compiler that will optimize any DSPy pipeline to maximize a given metric. We conduct two case studies, showing that succinct DSPy programs can express and optimize sophisticated LM pipelines that reason about math word problems, tackle multi-hop retrieval, answer complex questions, and control agent loops. Within minutes of compiling, a few lines of DSPy allow GPT-3.5 and llama2-13b-chat to self-bootstrap pipelines that outperform standard few-shot prompting and pipelines with expert-created demonstrations. On top of that, DSPy programs compiled to open and relatively small LMs like 770M-parameter T5 and llama2-13b-chat are competitive with approaches that rely on expert-written prompt chains for proprietary GPT-3.5. DSPy is available as open source at github.com/stanfordnlp/dspy
// Bio
Omar Khattab is a PhD candidate at Stanford and an Apple PhD Scholar in AI/ML. He builds retrieval models as well as retrieval-based NLP systems, which can leverage large text collections to craft knowledgeable responses efficiently and transparently. Omar is the author of the ColBERT retrieval model, which has been central to the development of the field of neural retrieval, and author of several of its derivate NLP systems like ColBERT-QA and Baleen.
His recent work includes the DSPy framework for solving advanced tasks with language models (LMs) and retrieval models (RMs).
// MLOps Jobs board
mlops.pallet.xyz/jobs
// MLOps Swag/Merch
mlops-community.myshopify.com/
// Related Links
Website: omarkhattab.com/
DSPy: github.com/stanfordnlp/dspy
wandb.me/mlops_traces
⁠
-------------- ✌️Connect With Us ✌️ ------------
Join our slack community: go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: go.mlops.community/register
Catch all episodes, blogs, newsletters, and more: mlops.community/
Connect with Demetrios on LinkedIn: / dpbrinkm
Connect with Omar on Twitter: / lateinteraction
Timestamps:
[00:00] Omar's preferred coffee
[00:26] Takeaways
[06:40] Weight & Biases Ad
[09:00] Omar's tech background
[13:35] Evolution of RAG
[16:33] Complex retrievals
[21:32] Vector Encoding for Databases
[23:50] BERT vs New Models
[28:00] Resilient Pipelines: Design Principles
[33:37] MLOps Workflow Challenges
[36:15] Guiding LLMs for Tasks
[37:40] Large Language Models: Usage and Costs
[41:32] DSPy Breakdown
[51:05] AI Compliance Roundtable
[55:40] Fine-Tuning Frustrations and Solutions
[57:27] Fine-Tuning Challenges in ML
[1:00:55] Versatile GPT-3 in Agents
[1:03:53] AI Focus: DSP and Retrieval
[1:04:55] Commercialization plans
[1:05:27] Wrap up

[@_tnk_]

Great insights from Omar! Thanks for having him on.

[@rembautimes8808]

Hi Omar, I was a DSP engineer when I started my career 20 years ago. But I like the DSP connections of being fast , powerful and highly optimised

[@mysticaltech]

Love the face of Demetrios when the prompt generation phase is explained "call the model potentially with high temperature successively" 🤣🤣 Really good interview, keep up the good work folks!

[@MLOps]

You are too kind. i was soooo lost and just hanging on by the heel of my seat when doing this one!

[@matten_zero]

5:57 I like the mysticism and psychology of trying to craft prompts but this is a good point. As an AI Engineer I work at a level of abstraction that prompt engineering is a necessary evil. It works but we don't really know why. My hope is that this DSPy technique can formalize a lot of this

[@MLOps]

1000% agree. hopefully its something we look at in 10 years as an artifact of a different age

[@kevon217]

Great discussion!

[@billykotsos4642]

This sounds like a proper LLM optimization software.
Langchain is just something that is stitched together in a very brittle way...
Whereas DSP looks to ground itself on very concrete foundations!

[@RaviKumar-cn7pt]

Colbert is Awesome!!

[@MLOps]

Indeed!

[@NadaaTaiyab]

Great interview! What a brilliant guy!

[@MLOps]

100%

[@jakobkristensen2390]

Super informative!

[@MLOps]

Join us at our first in-person conference on June 25 all about AI Quality: www.aiqualityconference.com/

[@not_a_human_being]

all nice and dandy, except it doesn't work.