Count the number of Tokens in Your Prompt Azure Open AI .NET

In this Microsoft Azure Open AI blog post, we will show you how to count the number of tokens your prompt uses before or after it is sent to the API.

When working with an Azure OpenAI resource or with the OpenAI API, the service has a token limit that measures the size of your prompt in tokens.

In some cases, knowing the size of your prompt in tokens before sending it to the API is crucial, and in this post, we will show how to use the Microsoft ML tokenizers library to count the number of tokens before or after you send it.

Install Library

Installing the following library to count the number of tokens in your prompt.

dotnet add package Microsoft.ML.Tokenizers --version 0.21.1

Once installed, add the following code to your application.

//  Microsoft.ML.Tokenizers library 
using Microsoft.ML.Tokenizers;

Download the following vocabulary files and copy them to the root directory of your application.

  • vocab.json
  • merges.txt

Declare Variables

Add the following variables.

// vocabulary files used to encode the text into tokens
var vocabFilePath = @"vocab.json";
var mergeFilePath = @"merges.txt";
var tokenizer = new Tokenizer(new Bpe(vocabFilePath, mergeFilePath));

Count and print token count

Use the following code to count and print the number of tokens used in the prompt.

// The user's input is taken and encoded using the tokenizer.
var input = userPrompt; // can also be a string for testing
// The encoded result is stored in 'tokenizerEncodedResult'.
var tokenizerEncodedResult = tokenizer.Encode(input);
// The number of tokens used in the encoding is counted.
// The console's foreground color is set to Magenta.
Console.ForegroundColor = ConsoleColor.Magenta;
// The number of tokens used is printed to the console in Magenta color.
Console.WriteLine($"Number of tokens used: {tokenizerEncodedResult.Tokens.Count()}");
// The console's color settings are reset to default.

In my case, I’m printing the tokens count after and the output looks like this.

To get started with Azure Open AI, visit the following articles:

More Articles

Success! You're on the list.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.