DeepSeek Harness: Multimodal Support

┌─ Chat Area ──────────────────────────────┐
│                                           │
│ 👤 Alice:                                 │
│ 📎 [screenshot.png] [ui-mockup.jpg]      │
│ Which of these two UI designs is better?  │
│                                           │
│ [Send]                                    │
└───────────────────────────────────────────┘

# Start CLI with attached image
dsh cli --attach ./screenshot.png

# Use /attach command during conversation
> /attach ./ui-mockup.jpg
> What's wrong with this UI?

# Attach multiple images
> /attach ./before.png ./after.png
> What's the difference between these two versions?

from dsh import DSHClient

client = DSHClient(base_url="http://127.0.0.1:3080")
session = client.create_session(workspace="/home/alice/project")

# Attach a single image
response = session.send(
    "What's wrong with this UI screenshot?",
    images=["./screenshot.png"]
)

print(response.content)

# Attach multiple images for comparison
response = session.send(
    "Compare these two designs, which is better?",
    images=["./design-a.png", "./design-b.png"]
)

👤 Alice: [attaches error-page-screenshot.png]
What's wrong with this error page? Help me fix it

🤖 Agent:
🔍 Analyzing image...

I see the following issues:
1. Error message text overflows the container boundary
2. The "Retry" button is truncated, only showing half
3. Background color is inconsistent with the design spec (should be #f5f5f5, not white)

📝 Using tool: file_edit (edit)
  → Path: src/pages/ErrorPage.tsx
  → Fix 1: Add text-overflow: ellipsis
  → Fix 2: Adjust button margin
  → Fix 3: Change background color

⚠️ Approval Required: Edit file src/pages/ErrorPage.tsx

# Send an audio file
response = session.send(
    "Transcribe this audio and summarize the key points",
    audio=["./meeting-recording.mp3"]
)

print(response.content)
# Audio transcription results and summary...

# Send a video file
response = session.send(
    "Analyze this demo video, list the features shown",
    video=["./demo.mp4"]
)

print(response.content)
# 1. User login feature (0:00-0:15)
# 2. Dashboard display (0:15-0:45)
# ...

# Analyze a PDF document
response = session.send(
    "Summarize the core points of this technical document",
    files=["./api-specification.pdf"]
)

print(response.content)

# Analyze CSV data
response = session.send(
    "Analyze this sales data, find trends and outliers",
    files=["./sales-2026.csv"]
)

print(response.content)
# Data analysis results...

🤖 Agent:
📊 Analyzing CSV data...
  → Rows: 12,450
  → Columns: 8
  → Date range: 2026-01 to 2026-08

Key findings:
1. Monthly average growth 12%, but March dropped 8% (suspected seasonality)
2. Top 3 products account for 65% of total revenue
3. Customer retention rate dropped from 72% to 68%

🔧 Using tool: shell
  → Command: python3 -c "import pandas; ..."
  → Generating visualization...

👤 Bob: [attaches api-response-screenshot.png]
This API response format is wrong, help me check the code

🤖 Agent:
🔍 Analyzing screenshot...
  → Expected format: { data: [...], meta: {...} }
  → Actual format: { results: [...] }

🔍 Using tool: search
  → Pattern: "results"
  → Type: content
  → Found in: src/api/handlers.ts:45

📝 Using tool: file_edit (read)
  → Reading src/api/handlers.ts

📝 Using tool: file_edit (edit)
  → Fixing response format at line 45-52
  → Changing "results" to "data", adding "meta" field

import glob

screenshots = glob.glob("./screenshots/*.png")

response = session.send(
    "Review all screenshots, find pages with UI issues, and fix them one by one",
    images=screenshots[:5]  # Max 5 images
)

👤 Alice: [attaches architecture-diagram.png]
Generate a Docker Compose configuration based on this architecture diagram

🤖 Agent:
🔍 Analyzing architecture diagram...
  → Identified 4 services: API, Worker, Database, Cache
  → Connections: API → Database, API → Cache, Worker → Database

📝 Using tool: file_edit (create)
  → Creating docker-compose.yml
  → 4 services with proper networking

✅ Docker Compose configuration generated, including:
- api-service (Express.js, port 3000)
- worker-service (Node.js, background jobs)
- postgres (PostgreSQL 16, volume mounted)
- redis (Redis 7, cache layer)

# dsh.config.yaml
multimodal:
  # Image settings
  images:
    max_size_mb: 20
    max_per_message: 5
    resize_large: true          # Auto-resize oversized images
    max_resolution: [4096, 4096]
  
  # Audio settings
  audio:
    max_duration_minutes: 10
    max_size_mb: 50
    transcription_model: "whisper-1"
  
  # Video settings
  video:
    max_duration_minutes: 5
    max_size_mb: 100
    frame_rate: 1               # Frames per second to extract
    max_frames: 300
  
  # Document settings
  documents:
    max_size_mb: 50
    pdf_ocr_enabled: true
    csv_max_rows: 100000