How to Use GPT-4o Audio Preview With LangChain and ChatOpenAI

This tutorial demonstrates how to leverage OpenAI's gpt-4o-audio-preview model with LangChain for seamless audio processing in voice-enabled applications. We'll cover model setup, audio handling, text and audio response generation, and building advanced applications.

Advanced gpt-4o-audio-preview Use Cases

This section details advanced techniques, including tool binding and multi-step workflows for creating sophisticated AI solutions. Imagine a voice assistant that transcribes audio and accesses external data sources – this section shows you how.

Tool Calling

Tool calling enhances AI capabilities by integrating external tools or functions. Instead of solely processing audio/text, the model can interact with APIs, perform calculations, or access information like weather data.

LangChain's bind_tools method seamlessly integrates external tools with the gpt-4o-audio-preview model. The model determines when and how to utilize these tools.

Here's a practical example of binding a weather-fetching tool:

import requests
from pydantic import BaseModel, Field

class GetWeather(BaseModel):
   """Fetches current weather for a given location."""
   location: str = Field(..., description="City and state, e.g., London, UK")
   def fetch_weather(self):
       API_KEY = "YOUR_API_KEY_HERE"  # Replace with your OpenWeatherMap API key
       url = f"http://api.openweathermap.org/data/2.5/weather?q={self.location}&appid={API_KEY}&units=metric"
       response = requests.get(url)
       if response.status_code == 200:
           data = response.json()
           return f"Weather in {self.location}: {data['weather'][0]['description']}, {data['main']['temp']}°C"
       else:
           return f"Could not fetch weather for {self.location}."

weather_tool = GetWeather(location="London, UK")
print(weather_tool.fetch_weather())

This code defines a GetWeather tool using the OpenWeatherMap API. It takes a location, fetches weather data, and returns a formatted string.

Chaining Tasks: Multi-Step Workflows

Chaining tasks allows for complex, multi-step processes combining multiple tools and model calls. For instance, an assistant could transcribe audio and then perform an action based on the transcribed location. Let's chain audio transcription with a weather lookup:

import base64
import requests
from pydantic import BaseModel, Field
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

# (GetWeather class remains the same as above)

llm = ChatOpenAI(model="gpt-4o-audio-preview")

def audio_to_text(audio_b64):
    messages = [("human", [{"type": "text", "text": "Transcribe:"}, {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}}])]
    return llm.invoke(messages).content

prompt = ChatPromptTemplate.from_messages([("system", "Transcribe audio and get weather."), ("human", "{text}")])

llm_with_tools = llm.bind_tools([GetWeather])
chain = prompt | llm_with_tools

audio_file = "audio.wav" # Replace with your audio file
with open(audio_file, "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode('utf-8')

result = chain.run(text=audio_to_text(audio_b64))
print(result)

This code transcribes audio, extracts the location, and uses the GetWeather tool to fetch the weather for that location.

Fine-tuning gpt-4o-audio-preview

Fine-tuning allows customization for specific tasks. For example, a medical transcription application could benefit from a model trained on medical terminology. OpenAI allows fine-tuning with custom datasets. (Code example omitted for brevity, but the concept involves using a fine-tuned model ID in the ChatOpenAI instantiation.)

Practical Example: Voice-Enabled Assistant

Let's build a voice assistant that takes audio input, generates a response, and provides an audio output.

Workflow

1. Audio capture from microphone.
2. Model transcribes audio.
3. Transcription processed to generate a response.
4. Model generates an audio response.

Implementation

import base64
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-audio-preview", temperature=0, model_kwargs={"modalities": ["text", "audio"], "audio": {"voice": "alloy", "format": "wav"}})

audio_file = "input.wav" # Replace with your audio file
with open(audio_file, "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode('utf-8')

messages = [("human", [{"type": "text", "text": "Answer this question:"}, {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}}])]

result = llm.invoke(messages)
audio_response = result.additional_kwargs.get('audio', {}).get('data')

if audio_response:
    audio_bytes = base64.b64decode(audio_response)
    with open("response.wav", "wb") as f:
        f.write(audio_bytes)
    print("Audio response saved as response.wav")
else:
    print("No audio response.")

This code captures audio, transcribes it, generates a response, and saves the audio response to a .wav file.

Conclusion

This tutorial showcased OpenAI's gpt-4o-audio-preview model and its integration with LangChain for building robust audio-enabled applications. The model offers a strong foundation for creating various voice-based solutions. (Links to additional LangChain tutorials omitted as requested.)

The above is the detailed content of How to Use GPT-4o Audio Preview With LangChain and ChatOpenAI. For more information, please follow other related articles on the PHP Chinese website!