Multimodal Message
Text · Image · Audio · Content List Format

1What is multimodal messaging? Detailed explanation of content list format

In the previous course,HumanMessage(content="...")ofcontentis a string. But LangChain’s messaging system supports a more powerful format:content list- Combine multiple media types in the same message.

Each element of the content list is a dictionary containingtypeField used to distinguish content types:

2Text input: the most basic content list format

Even plain text messages can be written in an explicit list format - this is useful when they need to be combined with images or audio:

from langchain.messages import HumanMessage
from langchain.agents import create_agent

agent = create_agent(
    model=deepseek_model(),
    system_prompt="You are a science fiction writer, create a capital city at the users request.",
)

# 显式的 content 列表格式（等价于 content="What is the capital of The Moon?"）
question = HumanMessage(content=[
    {"type": "text", "text": "What is the capital of The Moon?"}
])

response = agent.invoke({"messages": [question]})
print(response['messages'][-1].content)

Text list format and string format have exactly the same effect, but the writing method is different. When you need to add pictures to the same message, you need to append picture elements to this list.

3Image input: Base64 encoding + image_url type

The standard way to send images to multimodal LLM is to convert the image toBase64 encoding, then useData URIFormat embedded message:

import base64

# 假设 img_bytes 是图片的 bytes 数据
img_b64 = base64.b64encode(img_bytes).decode("utf-8")

question = HumanMessage(content=[
    {
        "type": "text",
        "text": "What is in this image?"    # 文字问题
    },
    {
        "type": "image_url",
        "image_url": {
            "url": f"data:image/png;base64,{img_b64}"  # Data URI 格式
        }
    }
])

Data URI format parsing

data:image/png;base64,{img_b64}The structure of this string is:

4Image upload process: FileUpload → bytes → base64

In Jupyter Notebook, you can useipywidgets.FileUploadProvide an image upload UI, and then read the uploaded file data:

from ipywidgets import FileUpload
from IPython.display import display
import base64

# 显示上传按钮
uploader = FileUpload(accept='.png', multiple=False)
display(uploader)

# 上传完成后（用户选择文件后）：
uploaded_file = uploader.value[0]  # 第一个上传的文件

# 关键：uploaded_file["content"] 是 memoryview，不是 bytes
content_mv = uploaded_file["content"]
img_bytes = bytes(content_mv)   # memoryview → bytes

# Base64 编码
img_b64 = base64.b64encode(img_bytes).decode("utf-8")

Uploaded file information structure

print(uploader.value[0])

5Audio input: recording + WAV encoding + base64

The same idea - record the microphone audio, convert it to base64, and send it to an LLM that supports audio input:

import sounddevice as sd
from scipy.io.wavfile import write
import base64
import io

# 录制 5 秒音频
duration = 5
sample_rate = 44100

print("Recording...")
audio = sd.rec(int(duration * sample_rate), samplerate=sample_rate, channels=1)
sd.wait()  # 等待录音完成
print("Done.")

# 写入内存 WAV 缓冲区
buf = io.BytesIO()
write(buf, sample_rate, audio)  # scipy 写 WAV 格式
wav_bytes = buf.getvalue()

# Base64 编码
aud_b64 = base64.b64encode(wav_bytes).decode("utf-8")

Audio message format (for models that support audio):

question = HumanMessage(content=[
    {
        "type": "input_audio",
        "data": aud_b64,
        "format": "wav"
    }
])

6Current status of model multimodal support: DeepSeek vs dedicated multimodal models

Not all models support image or audio input. This is clearly reminded in the notebook:

Multimodal support matrix

7Multimodal message format quick check

HumanMessage(content=[
    {"type": "text", "text": "你的问题"}
])
# 简写等价：HumanMessage(content="你的问题")

HumanMessage(content=[
    {"type": "text", "text": "这张图里有什么？"},
    {
        "type": "image_url",
        "image_url": {
            # 方式 1：base64 Data URI
            "url": f"data:image/png;base64,{img_b64}"
            # 方式 2：公开 URL
            # "url": "https://example.com/image.png"
        }
    }
])

HumanMessage(content=[
    {
        "type": "input_audio",
        "data": aud_b64,    # base64 编码的音频
        "format": "wav"    # 音频格式：wav / mp3
    }
])
# 注意：需要使用支持音频输入的模型（如 GPT-4o-audio）