How does pi agent read images?

how does pi agent read images?

Pi doesn’t “read” images by doing local OCR/vision itself. It turns image files into image attachments and sends those attachments to the selected model, if the model supports vision.

Main flow in the source:

1. read tool resolves and checks the file

   • packages/coding-agent/src/core/tools/read.ts
   • The read tool description says it supports jpg, png, gif, webp, bmp.
   • It calls detectSupportedImageMimeTypeFromFile() before deciding whether to treat the file as text or image.

2. Image detection is by file bytes / magic headers, not extension

   • packages/coding-agent/src/utils/mime.ts
   • It sniffs the first 4100 bytes and detects:
     • JPEG
     • PNG, excluding animated PNG
     • GIF
     • WEBP
     • BMP
   • This is why a PNG named image.txt can still be read as an image.

3. If it’s an image, Pi reads it as binary

   • In read.ts, when mimeType is truthy:

     const buffer = await ops.readFile(absolutePath);
     const processed = await processImage(buffer, mimeType, { autoResizeImages });
     

   • It then returns:

     [
       { type: "text", text: "Read image file [image/png]" },
       { type: "image", data: processed.data, mimeType: processed.mimeType }
     ]
     

   • The data field is base64.

4. Images are normalized / converted if needed

   • packages/coding-agent/src/utils/image-process.ts
   • Native supported inline formats are:
     • image/png
     • image/jpeg
     • image/gif
     • image/webp
   • Unsupported detected formats, like BMP, are converted to PNG if possible:

     return {
       bytes: pngBytes,
       mimeType: "image/png",
       convertedFrom: baseMimeType(mimeType),
     };
     

5. Images are resized by default

   • packages/coding-agent/src/utils/image-resize-core.ts
   • Default limits:
     • max width: 2000
     • max height: 2000
     • max base64 payload: 4.5MB
   • It uses Photon/WASM to resize, preserve EXIF orientation, and try PNG/JPEG encodings to fit provider limits.

6. The model only sees the image if it supports vision

   • packages/ai/src/types.ts defines:

     export interface ImageContent {
       type: "image";
       data: string;
       mimeType: string;
     }
     

   • If the model does not support images, packages/ai/src/api/transform-messages.ts replaces image blocks with placeholders like:

     "(image omitted: model does not support images)"
     

   • read.ts also adds a note:

     [Current model does not support images. The image will be omitted from this request.]
     

7. Provider adapters encode the attachment appropriately

   • Anthropic: packages/ai/src/api/anthropic-messages.ts converts image blocks to Anthropic base64 image sources.
   • OpenAI-compatible chat: packages/ai/src/api/openai-completions.ts sends images as data URLs:

     url: `data:${block.mimeType};base64,${block.data}`
     

Also, initial CLI @file arguments use nearly the same path:

• packages/coding-agent/src/cli/file-processor.ts
• It detects images, processes them, adds them to images: ImageContent[], and includes a <file name="...">...</file> text reference.

So in short: Pi reads image bytes, detects the real image type, optionally converts/resizes, base64-encodes it as an ImageContent block, and passes that block to a vision-capable model.