The slide deck from the API Management in AI Era session from API Days Australia. https://www.apidays.global/events/australia

1. Nilesh Gule @nileshgule
API Management
in
AI Era

2. $whoami
{
“name” : “Nilesh Gule”,
“role” : “Senior Cloud Solutions Architect at Avanade”
“website” : “https://www.HandsOnArchitect.com",
“github” : “https://GitHub.com/NileshGule"
“twitter” : “@nileshgule”,
“linkedin” : “https://www.linkedin.com/in/nileshgule”,
“YouTube” : “https://www.YouTube.com/@nilesh-gule”
“likes” : “Technical Evangelism, Cricket”,
}

3. Mistral Prompt and Response Tokens

4. Deepseek - Prompt and Response Tokens

5. API Management - GenAI Gateway
Azure-Samples/AI-Gateway: APIM

6. Challenges in using GenAI APIs
• Track Token usage across multiple applications
• Ensure single app doesn’t consume whole TPM quota
• Distribute load across multiple endpoints
• Ensure committed capacity in PTUs is exhausted before falling back to PAYG instance

7. Monitor utilization of models
• Sends Token Merics usage to Applications Insights
• Provides overview of utilization of models across
multiple applications or API consumers
GenAI Gateway Capabilities in Azure API Management

8. Token Metrics Emitting – Token Metric Policy
GenAI Gateway Capabilities in Azure API Management

9. Token Metrics Emitting
GenAI Gateway Capabilities in Azure API Management

10. Enforce limits per consumer
• Manage and enforce limits per API consumer based on the
usage of API Tokens
GenAI Gateway Capabilities in Azure API Management

11. Token Rate Limiting – Token Limit Policy
GenAI Gateway Capabilities in Azure API Management

12. Token Rate Limiting
GenAI Gateway Capabilities in Azure API Management

13. Provisioned Throughput Units (PTU)
• Allows to specify the amount of throughput required in a model deployment.
• Granted to subscription as quota
• Quota is specific to region and defines the maximum number of PTUs that can be assigned to deployments in the
subscription and region
• PTU provides
• Predictable performance
• Allocated processing capacity
• Cost savings
Understanding costs associated with provisioned throughput units (PTU)

14. HA - Load Balanced Pool and Circuit Breaker
• Helps to spread load across multiple Azure OpenAI endpoints
• Round-robin, weighted or priority based load distribution
strategy
GenAI Gateway Capabilities in Azure API Management

15. Load Balanced Pool and Circuit Breaker
GenAI Gateway Capabilities in Azure API Management

16. Load Balanced Pool and Circuit Breaker
GenAI Gateway Capabilities in Azure API Management

17. AI Gateway capabilities of Azure API Management
AI Gateway
Security & safety
• Keyless managed identities
• AI Apps & Agents Authorizations -New
• Content Safety -GA
• Credential Manager
Resiliency
• Weight load balancing
• Priority routing to provisioned capacity models
• Backend pools with circuit breaker
• Session aware load balancing -GA
Scalability
• Token rate limits and token quotas
• Semantic Caching -GA
• Model load balancing
• Multi-regional deployments
Traffic mediation & control
• Azure AI Foundry & Azure OpenAI
• OpenAI compatible models -GA
• Responses API -GA
• WebSocket’s for Realtime APIs
• MCP server pass-trough - Soon
• Expose APIs as built-in MCP server - Preview
Developer velocity
• Wizard policy configuration experience
• Self-service with the Developer Portal
• API Center Copilot Studio connector - Preview
• Policy Toolkit
Observability
• Token counting per consumer
• Prompts and completions logging -GA
• Built-in reporting dashboard -GA
Governance
• Policy engine with custom expressions
• API Center MCP server registry - Preview
• Federated API Management
GA
GA
GA Soon
GA
GA
GA
GA
Preview
Preview
GA
Preview
New

18. Summary
• Track Token usage across multiple applications
• Emit Token Metrics policy
• Ensure single app doesn’t consume whole TPM quota
• Token Limit Policy
• Distribute load across multiple endpoints
• Backend pool load balancing and circuit breaker

19. Resources
• Azure OpenAI Gateway topologies
• Azure OpenAI Token Limit Policy
• LLM Token Limit Policy
• Azure OpenAI Emit Token Metric Policy
• LLM Emit Token Metric Policy
• Houssem Dellai Youtube videos
• GenAI Labs
• Designing and implementing GenAI gateway solution

20. Nilesh Gule
ARCHITECT | MICROSOFT MVP
“Code with Passion and
Strive for Excellence”
nileshgule
@nileshgule Nilesh Gule
NileshGule
www.handsonarchitect.com
https://www.youtube.com/@nilesh-gule

21. Source Code & slide deck
Nilesh Gule fork - GenAI Labs
https://github.com/NileshGule/AI-Gateway
GenAI Labs
https://aka.ms/apim/genai/labs
https://speakerdeck.com/nileshgule/
https://www.slideshare.net/nileshgule/

22. Q&A