Create prompts

Create and manage prompts with expected answers and references. You can create a repository of prompts and use it to benchmark your knowledge base with various configurations.

Procedure

Navigate to Benchmarking > Prompts.
The Prompts page appears.
Select New.

The New prompt dialog box is displayed.
Enter a unique Name for to identify your prompt.
In the Prompt box, enter your prompt question in detail that will be used for benchmarking. Prompts should be comprehensive, specific, and significant in length.
Select the Category to organize your prompts.

By default, the category is set to your working category. You can select a different category from the Category list.
In the Tags box, enter a tag for your prompt and press Enter.
Add multiple tags as needed.

For Expected results, do either of the following:

Option	Description
Manually enter expected answers and references if you know the document name stored in the search index.	In the Expected answer box, enter the expected answer that should be returned for your prompt. For Expected references, specify the Document name and Page number. Select Add. Specify Expected tokens for executing this prompt.
Automatically generate the answers and references.	The answers are automatically generated only if the information you requested is available in the provided knowledge base documents. Select Auto populate. The Auto Populate configuration dialog box appears. In the AI Knowledge Base list, select the AI provider configuration to use for generating expected results. In the Azure AI Search Index list, select the search index that contains the documents to query. Configure the advanced settings. Select Run Auto populate. The system does the following: Queries only the knowledge base you selected and fetches the most relevant documents based on the retrieval engine your knowledge base uses (usually vector search + keyword search). Runs your prompt against the selected AI Knowledge Base and Azure AI Search Index. It uses the AI response to automatically populate the expected answer and reference documents, saving you time in creating comprehensive benchmark data; however, you can still edit or remove them manually afterward. The results are auto-populated in the following boxes: Expected answer Generated from retrieved content. This information is used as the reference answer for comparison during benchmarking. If the knowledge base documents are not available, a message appears stating "The information you requested is not available in the retrieved documents." Expected references: Retrieved from ranked retrieval results. The list of references only appears if the knowledge base contains the correct documents; otherwise, the auto-populated references also appear empty. The Auto-Populate option only mirrors what the knowledge base retrieval returns. Expected tokens: Estimated from auto-generated expected answer and not pulled from documents.

Option

Description

Manually enter expected answers and references if you know the document name stored in the search index.

In the Expected answer box, enter the expected answer that should be returned for your prompt.
For Expected references, specify the Document name and Page number.
Select Add.
Specify Expected tokens for executing this prompt.

Automatically generate the answers and references.

The answers are automatically generated only if the information you requested is available in the provided knowledge base documents.

Select Auto populate.

The Auto Populate configuration dialog box appears.
In the AI Knowledge Base list, select the AI provider configuration to use for generating expected results.
In the Azure AI Search Index list, select the search index that contains the documents to query.
Configure the advanced settings.
Select Run Auto populate.

The system does the following:
- Queries only the knowledge base you selected and fetches the most relevant documents based on the retrieval engine your knowledge base uses (usually vector search + keyword search).
- Runs your prompt against the selected AI Knowledge Base and Azure AI Search Index. It uses the AI response to automatically populate the expected answer and reference documents, saving you time in creating comprehensive benchmark data; however, you can still edit or remove them manually afterward. The results are auto-populated in the following boxes:
  - Expected answer Generated from retrieved content.
    
    This information is used as the reference answer for comparison during benchmarking.
    
    If the knowledge base documents are not available, a message appears stating "The information you requested is not available in the retrieved documents."
  - Expected references: Retrieved from ranked retrieval results.
    
    The list of references only appears if the knowledge base contains the correct documents; otherwise, the auto-populated references also appear empty. The Auto-Populate option only mirrors what the knowledge base retrieval returns.
  - Expected tokens: Estimated from auto-generated expected answer and not pulled from documents.

Select either of the following

Save	Saves the prompt and displays it on the Prompts page.
Save and add another	Saves the prompt and the New prompt dialog box is displayed for creating another prompt.

The prompt appears on the Prompts page with the following details:

A summary card with the following metrics.
- Total prompts: Number of prompts in the system.
- With references: Prompts that include a linked knowledge source.
- Tagged: Prompts that have metadata tags assigned.
A Search box where you can search for a specific prompt by entering the form name and then selecting . You can filter the prompt by category and tags.
A table that displays the prompt name, prompt text, the tags assigned to the prompt, the number of associated references, the user who last modified it, the date it was last updated, and actions to edit and delete the prompt.

More info

Maintain prompts