# Semantic Cache for LLM responses

# **Prerequisites**

* API Management instance with an Azure OpenAI model deployment as an API
    
* Deployment for the following APIs:
    
    * Chat Completion API
        
    * Embeddings API
        
* ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1767557003875/ca43291f-5117-413c-b8d7-e2d61141074a.png align="center")
    
* Configured API Management instance to use managed identity authentication to the Azure OpenAI service
    
* An Azure Managed Redis instance with the `RediSearch` module enabled
    
* ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1767557449758/e780876e-9063-48a5-b0a5-16edff317bfc.png align="center")
    

---

# **Authenticate with managed identity**

Let’s quickly explain how to use Managed Identity to authenticate to Azure OpenAI. Make sure that the managed identity is enabled on your API Management instance.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1767556855571/586023c6-b4d7-47fe-a2f0-a47357a447df.png align="center")

Assign the `Cognitive Services OpenAI User` role to the managed identity. Add the following inbound policy section to authenticate requests to the API by using the managed identity.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1767558274371/855717c0-6821-40ce-93d1-995b07475fdf.png align="center")

```xml
<authentication-managed-identity resource="https://cognitiveservices.azure.com" output-token-variable-name="managed-id-access-token" ignore-error="false" /> 
<set-header name="Authorization" exists-action="override"> 
    <value>@("Bearer " + (string)context.Variables["managed-id-access-token"])</value> 
</set-header>
```

---

# **Import an Azure OpenAI API**

If you have imported an Azure OpenAI instance as an API into APIM you should see something like this:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1767558312983/b1fd529b-d1ee-4651-a392-3585330623c9.png align="center")

If not, you can either import an Azure OpenAI API directly from a deployment in Microsoft Foundry or download and edit the OpenAPI specification.

---

# Azure Managed Redis Setup

We will setup an external cache using the Azure Managed Redis instance.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1767560600011/b2ba07b4-038a-4516-b093-730fb3e1eff8.png align="center")

<mark>Azure API Management uses a Redis connection string to connect to the cache. Because we are using Azure Managed Redis, we need to </mark> **<mark>enable </mark>** <mark>access key authentication for the cache and use the key as password in the connection string. Currently, we can't use Microsoft Entra authentication to connect Azure API Management to Azure Managed Redis.</mark>

The connection string will look like:

```plaintext
<cache-name>:10000,password=<cache-access-key>,ssl=True,abortConnect=False
```

---

# Semantic Cache Configuration

Start by creating a backend for the embeddings API.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1767558683062/db63ee10-6bec-451d-98fc-a39c16cac3e5.png align="center")

Next, let’s configure the semantic caching policies. In the Inbound section, add the `azure-openai-semantic-cache-lookup` policy. In the embeddings-backend-id attribute, specify the Embeddings API backend you created. In my case, the value is `Embedding`.

```xml
<azure-openai-semantic-cache-lookup
    score-threshold="0.15"
    embeddings-backend-id="Embedding"
    embeddings-backend-auth="system-assigned"
    ignore-system-messages="true"
    max-message-count="10">
    <vary-by>@(context.Subscription.Id)</vary-by>
</azure-openai-semantic-cache-lookup>
```

In the Outbound processing section for the API, add the `azure-openai-semantic-cache-store` policy.

```xml
<azure-openai-semantic-cache-store duration="60" />
```

To confirm the semantic cache is working, we can trace a test Completion or Chat Completion operation by using the test console in the portal.

Example from my trace operation:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1767560288292/7c3c2e90-7c53-40cd-83c9-e733134ae8dd.png align="center")

---