8000
Skip to content

Add documentation for provider-side prompt caching with Anthropic and OpenAI - #8970

Merged
TomeHirata merged 7 commits into
mainfrom
copilot/enable-prompt-caching
Oct 25, 2025
Merged

Add documentation for provider-side prompt caching with Anthropic and OpenAI#8970
TomeHirata merged 7 commits into
mainfrom
copilot/enable-prompt-caching

Conversation

Copilot AI commented Oct 25, 2025 •
edited
Loading
Copy link
Copy Markdown
Contributor
  • Explore repository structure and understand caching documentation
  • Understand how dspy.LM passes kwargs to litellm
  • Add prompt caching documentation section to the cache tutorial
  • Address review feedback - remove unnecessary paragraph
  • Simplify documentation by consolidating Anthropic and OpenAI sections
  • Remove additional configuration options section
  • Remove duplicated OpenAI example and redundant explanation
  • Add reference to LiteLLM documentation
  • Run code review and security checks

Summary

Added comprehensive documentation for using provider-side prompt caching with DSPy. The documentation covers:

  1. Provider Configuration: Shows how to use cache_control_injection_points with supported providers like Anthropic and OpenAI
  2. Use Cases: Highlights benefits for modules like dspy.ReAct() that send similar prompts repeatedly
  3. External Reference: Links to LiteLLM prompt caching documentation for additional details
Original prompt

This section details on the original issue you should resolve

<issue_title>[Feature] Enable Prompt caching for anthropic / openai</issue_title>
<issue_description>### What feature would you like to see?

I couldn't find anything in the docs / code regarding a way to enable prompt caching with a model provider.
Is this possible with dspy?

Sending cache=True to dspy.LM enables local caching but not prompt caching.

This is extremely useful when using dspy.ReAct() with the same prompt being sent multiple times

Would you like to contribute?

  • Yes, I'd like to help implement this.
  • No, I just want to request it.

Additional Context

Using dspy 2.6</issue_description>

<agent_instructions>Can you add a guide for how to use prompt cache? See my previous comment below for how to use it and where to refer to.

It seems LiteLLM supports cache_control_injection_points option. Can you all try passing the parameter to dspy.LM?

lm = dspy.LM(
    "anthropic/claude-3-5-sonnet-20240620",
    cache_control_injection_points=[
        {
            "location": "message",
            "role": "system",
        }
    ],
)
dspy.configure(lm=lm)
```</agent_instructions>

## Comments on the Issue (you are @copilot in this section)

<comments>
<comment_new><author>@TomeHirata</author><body>
Are you looking for a way to enable provider cache through the `cache_control ` parameter in the message? https://docs.litellm.ai/docs/completion/prompt_caching</body></comment_new>
<comment_new><author>@TomeHirata</author><body>
No, it's not supported yet. You can still customize LM and pass it to dspy.settings.configure.</body></comment_new>
<comment_new><author>@TomeHirata</author><body>
Thanks, let's keep this issue open. We are able to officially support it if there are high demands.</body></comment_new>
<comment_new><author>@TomeHirata</author><body>
It seems LiteLLM supports [cache_control_injection_points](https://docs.litellm.ai/docs/tutorials/prompt_caching#configuration) option. Can you all try passing the parameter to dspy.LM?

```python
lm = dspy.LM(
    "anthropic/claude-3-5-sonnet-20240620",
    cache_control_injection_points=[
        {
            "location": "message",
            "role": "system",
        }
    ],
)
dspy.configure(lm=lm)
```</body></comment_new>
<comment_new><author>@TomeHirata</author><body>
Thanks, I believe my previous comment is stanfordnlp/dspy#2. Can you check if that works for you?</body></comment_new>
<comment_new><author>@TomeHirata</author><body>
LiteLLM automatically injects the cashe control item when cache_control_injection_points is passed. Can you take a look at https://docs.litellm.ai/docs/tutorials/prompt_caching#configuration?</body></comment_new>
<comment_new><author>@TomeHirata</author><body>
Great! I think it's the clearer solution than DSPy automatically manages the partial prompt cache. Let me update the cache document later and close this issue.</body></comment_new>
</comments>

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

@TomeHirata
Add documentation for provider-side prompt caching
78b0c00 
Co-authored-by: TomeHirata <33407409+TomeHirata@users.noreply.github.com>
Copilot AI changed the title [WIP] Add support for prompt caching in dspy Add documentation for provider-side prompt caching with Anthropic and OpenAI Oct 25, 2025
Copilot AI requested a review from TomeHirata October 25, 2025 08:04
TomeHirata
TomeHirata reviewed Oct 25, 2025
View reviewed changes
Comment thread docs/docs/tutorials/cache/index.md Outdated

In addition to DSPy's built-in caching mechanism, you can leverage provider-side prompt caching offered by LLM providers like Anthropic and OpenAI. This feature is particularly useful when working with modules like `dspy.ReAct()` that send similar prompts repeatedly, as it reduces both latency and costs by caching prompt prefixes on the provider's servers.

DSPy seamlessly passes configuration parameters to LiteLLM, which in turn supports various provider-specific caching mechanisms. You can enable prompt caching by passing the appropriate parameters directly to `dspy.LM()`.
@TomeHirata TomeHirata Oct 25, 2025
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot This paragraph is unnecessary

Copilot AI Oct 25, 2025
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the unnecessary paragraph in commit 71adefc.

@TomeHirata
Remove unnecessary paragraph from prompt caching documentation
71adefc 
Co-authored-by: TomeHirata <33407409+TomeHirata@users.noreply.github.com>
Copilot AI requested a review from TomeHirata October 25, 2025 08:12
TomeHirata
TomeHirata reviewed Oct 25, 2025
View reviewed changes
Comment thread docs/docs/tutorials/cache/index.md Outdated

DSPy seamlessly passes configuration parameters to LiteLLM, which in turn supports various provider-specific caching mechanisms. You can enable prompt caching by passing the appropriate parameters directly to `dspy.LM()`.

### Anthropic Prompt Caching
@TomeHirata TomeHirata Oct 25, 2025
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot What's the difference between Anthropic and OpenAI? If the configuration strategy on DSPy is the same between providers, can we simplify this subsection?

Copilot AI Oct 25, 2025
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the configuration is identical between providers, I've consolidated the separate Anthropic and OpenAI subsections into a single unified section showing both examples. This simplifies the documentation while making it clear that the same approach works for both providers. Changes in commit dc73d6d.

@TomeHirata
Simplify prompt caching documentation by consolidating provider sections
dc73d6d 
Co-authored-by: TomeHirata <33407409+TomeHirata@users.noreply.github.com>
Copilot AI requested a review from TomeHirata October 25, 2025 08:15
TomeHirata
TomeHirata reviewed Oct 25, 2025
View reviewed changes
Comment thread docs/docs/tutorials/cache/index.md Outdated
- Working with long system prompts that remain constant
- Making multiple requests with similar context

### Additional Configuration Options
@TomeHirata TomeHirata Oct 25, 2025
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot let's remove this section

Copilot AI Oct 25, 2025
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the "Additional Configuration Options" section in commit cc39d9b.

TomeHirata
TomeHirata reviewed Oct 25, 2025
View reviewed changes
Comment thread docs/docs/tutorials/cache/index.md Outdated
],
)

# For OpenAI
@TomeHirata TomeHirata Oct 25, 2025
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot we don't need this duplicated example

Copilot AI Oct 25, 2025
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the duplicated OpenAI example in commit 3446ef4.

TomeHirata
TomeHirata reviewed Oct 25, 2025
View reviewed changes
Comment thread docs/docs/tutorials/cache/index.md

This configuration tells LiteLLM to automatically inject cache control markers at system messages, allowing the provider to cache the system prompt across multiple requests. This is especially beneficial when:

- Using `dspy.ReAct()` with the same instructions
@TomeHirata TomeHirata Oct 25, 2025
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot this is already explained above

Copilot AI Oct 25, 2025
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the redundant explanation in commit 3446ef4.

@TomeHirata
Remove additional configuration options section
cc39d9b 
Co-authored-by: TomeHirata <33407409+TomeHirata@users.noreply.github.com>
Copilot AI requested a review from TomeHirata October 25, 2025 08:25
@TomeHirata
Remove duplicated example and redundant explanation
3446ef4 
Co-authored-by: TomeHirata <33407409+TomeHirata@users.noreply.github.com>
TomeHirata
TomeHirata reviewed Oct 25, 2025
View reviewed changes
Comment thread docs/docs/tutorials/cache/index.md Outdated

In addition to DSPy's built-in caching mechanism, you can leverage provider-side prompt caching offered by LLM providers like Anthropic and OpenAI. This feature is particularly useful when working with modules like `dspy.ReAct()` that send similar prompts repeatedly, as it reduces both latency and costs by caching prompt prefixes on the provider's servers.

You can enable prompt caching by passing the `cache_control_injection_points` parameter to `dspy.LM()`. This works with supported providers like Anthropic and OpenAI:
@TomeHirata TomeHirata Oct 25, 2025
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot can you add a reference to LiteLLM page?

Copilo 992E t AI Oct 25, 2025
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a reference to the LiteLLM prompt caching documentation in commit 667b675.

@TomeHirata
Add reference to LiteLLM prompt caching documentation
667b675 
Co-authored-by: TomeHirata <33407409+TomeHirata@users.noreply.github.com>
Copilot AI requested a review from TomeHirata October 25, 2025 08:41
@TomeHirata
TomeHirata marked this pull request as ready for review October 25, 2025 08:59
@TomeHirata
TomeHirata merged commit 53c29f5 into main Oct 25, 2025
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@TomeHirata TomeHirata TomeHirata approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@TomeHirata TomeHirata

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature] Enable Prompt caching for anthropic / openai

2 participants

@TomeHirata
0