Batch-GPT Documentation

A jump-server to convert OpenAI chat completion API requests to batched requests

Get Started View on GitHub

Overview

Batch-GPT enables seamless integration with OpenAI’s Batch API by acting as a drop-in replacement for standard OpenAI API endpoints. It intelligently collects and batches requests, providing significant cost savings while maintaining compatibility with existing OpenAI clients.

Quick Integration

It really is as simple as:

from openai import OpenAI
- client = OpenAI(api_key="sk-...")
+ client = OpenAI(api_key="dummy_openai_api_key", base_url="http://batch-gpt")

Highlights

  • Cost Savings: Up to 50% reduction using OpenAI’s Batch API
  • Zero Config: Drop-in replacement for OpenAI API clients
  • Reliability: Automatic recovery of interrupted batches
  • Monitoring: Real-time terminal-based batch status tracking
  • Flexibility: Sync/Async/Cache operation modes

Read More About Features