Ollama Migration Guide

Monthly cost: $450

Annual cost: $5,400

3-year cost: $16,200

Self-Hosted Ollama Costs

Hardware (one-time):

Operating costs (annual):

3-year total: $3,600 + ($600 × 3) = $5,400

Savings vs Claude: $16,200 - $5,400 = $10,800 (67% savings)

Break-even point: 12 months

Cost per 1M Tokens

At scale (1B tokens/year):

• Claude: $18,000/year
• Ollama: $600/year (electricity + maintenance)
• Savings: $17,400/year (97%)

Privacy & Compliance Benefits

Data Privacy

Cloud APIs (Anthropic, OpenAI):

• Data sent over internet
• Stored on provider servers (30-90 days)
• Subject to subpoenas, data breaches
• Provider terms can change

Ollama Self-Hosted:

• All processing on-premises
• Zero data transmission
• Full audit trails
• Complete control

Compliance Advantages

Real-World Example: Healthcare Company

// Before: Send patient data to Claude API (HIPAA violation!)
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const response = await anthropic.messages.create({
  model: 'claude-3-5-sonnet-20241022',
  messages: [{
    role: 'user',
    content: `Analyze patient record: ${patientData}` // ❌ HIPAA violation!
  }]
});

// After: Process locally with Ollama (HIPAA compliant)
const response = await fetch('http://localhost:11434/api/generate', {
method: 'POST',
body: JSON.stringify({
model: 'llama3.3:70b',
prompt: `Analyze patient record: ${patientData}` // ✅ Never leaves network
})
});

Result: Company saves $8,000/month in API costs + eliminates HIPAA compliance risk.

Migration Strategies

Strategy 1: Gradual Rollout (Recommended)

Week 1-2: Pilot (5% traffic)

// Route 5% of requests to Ollama, 95% to Claude
async function routeRequest(prompt: string): Promise<string> {
  const useOllama = Math.random() < 0.05; // 5% to Ollama

if (useOllama) {
return await callOllama(prompt, 'llama3.3:70b');
} else {
return await callClaude(prompt);
}
}

Week 3-4: Expand (25% traffic)

• Monitor quality metrics
• Compare latency, error rates
• Collect user feedback

Week 5-6: Majority (75% traffic)

• Ramp up if metrics acceptable
• Keep Claude as fallback

Week 7: Full Migration (100% Ollama)

• Keep Claude API key for emergencies
• Monitor for regressions

Strategy 2: Feature-Based Migration

Phase 1: Simple tasks to Ollama

const taskRouting = {
  'code-completion': 'ollama',      // Qwen 2.5 Coder
  'summarization': 'ollama',        // Mistral 7B
  'chat': 'ollama',                 // Llama 3.1 8B
  'complex-reasoning': 'claude',    // Keep Claude for hard tasks
  'creative-writing': 'claude'      // Keep Claude for creative work
};

async function route(task: string, prompt: string): Promise<string> {
const provider = taskRouting[task] || 'claude';
return provider === 'ollama'
? await callOllama(prompt, selectBestModel(task))
: await callClaude(prompt);
}

Phase 2: Migrate complex tasks when confident

Phase 3: Decommission Claude API

Strategy 3: Hybrid Architecture

// Use Ollama for cost-sensitive workloads, Claude for quality-critical
class HybridLLMRouter {
  async execute(prompt: string, options: { priority: 'cost' | 'quality' }): Promise<string> {
    if (options.priority === 'cost') {
      try {
        return await this.callOllama(prompt);
      } catch (error) {
        console.warn('Ollama failed, falling back to Claude');
        return await this.callClaude(prompt);
      }
    } else {
      return await this.callClaude(prompt);
    }
  }

private async callOllama(prompt: string): Promise<string> {
const response = await fetch('http://localhost:11434/api/generate', {
method: 'POST',
body: JSON.stringify({
model: 'llama3.3:70b',
prompt,
stream: false
})
});
const data = await response.json();
return data.response;
}

private async callClaude(prompt: string): Promise<string> {
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const response = await anthropic.messages.create({
model: 'claude-3-5-sonnet-20241022',
max_tokens: 4096,
messages: [{ role: 'user', content: prompt }]
});
return response.content[0].text;
}
}

Use Cases for Hybrid:

• Development: Ollama (cheap iterations)
• Production: Claude (high stakes)
• Batch jobs: Ollama (cost optimization)
• Real-time chat: Claude (low latency from edge servers)

Production Deployment

Docker Deployment

# Dockerfile for Ollama production deployment
FROM nvidia/cuda:12.1.0-base-ubuntu22.04

# Install Ollama
RUN curl -fsSL https://ollama.com/install.sh | sh

# Download models at build time
RUN ollama serve & sleep 10 && ollama pull llama3.3:70b && ollama pull qwen2.5-coder:32b && ollama pull mistral:7b

# Expose Ollama API
EXPOSE 11434

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 CMD curl -f http://localhost:11434/api/tags || exit 1

# Run Ollama server
CMD ["ollama", "serve"]

Deploy with Docker Compose:

# docker-compose.yml
version: '3.8'
services:
  ollama:
    build: .
    ports:
      - "11434:11434"
    volumes:
      - ollama_models:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

volumes:
ollama_models:

Kubernetes Deployment

# ollama-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
spec:
  replicas: 3  # Scale horizontally with multiple GPUs
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - containerPort: 11434
        resources:
          limits:
            nvidia.com/gpu: 1  # 1 GPU per pod
        livenessProbe:
          httpGet:
            path: /api/tags
            port: 11434
          initialDelaySeconds: 60
          periodSeconds: 30

apiVersion: v1
kind: Service
metadata:
name: ollama-service
spec:
selector:
app: ollama
ports:
- protocol: TCP
port: 80
targetPort: 11434
type: LoadBalancer

Load Balancing Multiple GPUs

// Round-robin load balancer for multiple Ollama instances
class OllamaLoadBalancer {
  private instances = [
    'http://gpu-1.local:11434',
    'http://gpu-2.local:11434',
    'http://gpu-3.local:11434'
  ];
  private currentIndex = 0;

async generate(prompt: string, model: string): Promise<string> {
const instance = this.instances[this.currentIndex];
this.currentIndex = (this.currentIndex + 1) % this.instances.length;

const response = await fetch(`${instance}/api/generate`, {
method: 'POST',
body: JSON.stringify({ model, prompt, stream: false })
});

if (!response.ok) {
// Retry on next instance
return this.generate(prompt, model);
}

const data = await response.json();
return data.response;
}
}

Best Practices

DO ✅

• Start with pilot testing

// Test Ollama on non-critical workloads first
   const isPilotUser = ['user-123', 'user-456'].includes(userId);
   const provider = isPilotUser ? 'ollama' : 'claude';

• Use appropriate models for tasks

// Match model size to task complexity
   const modelSelection = {
     'simple-chat': 'mistral:7b',          // Fast
     'code-completion': 'qwen2.5-coder:32b', // Specialized
     'complex-reasoning': 'llama3.3:70b'    // High quality
   };

• Implement fallback to cloud

async function withFallback(prompt: string): Promise<string> {
     try {
       return await callOllama(prompt);
     } catch (error) {
       console.warn('Ollama failed, using Claude');
       return await callClaude(prompt);
     }
   }

• Monitor GPU utilization

# Track GPU usage
   nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1

• Pre-download models

# Download models during deployment, not runtime
   ollama pull llama3.3:70b
   ollama pull qwen2.5-coder:32b

• Use quantized models for budget hardware

# Q4 quantization fits in 8GB GPU
   ollama pull llama3.1:8b-instruct-q4_0

DON'T ❌

• Don't migrate without testing

// ❌ Instant switch - risky
   const response = await callOllama(prompt);

// ✅ Gradual rollout with monitoring
const response = canaryPercentage > Math.random()
? await callOllama(prompt)
: await callClaude(prompt);

• Don't use CPU-only in production

# ❌ CPU inference is 50-100x slower
   ollama run llama3.3:70b  # On CPU: 2-5 tokens/sec

# ✅ Use GPU
ollama run llama3.3:70b  # On RTX 4090: 50-60 tokens/sec

• Don't expect identical quality

// ❌ Expecting Claude-level reasoning
   const analysis = await callOllama('Solve complex logic puzzle');

// ✅ Set realistic expectations
const analysis = await callOllama('Summarize this text'); // Better fit

• Don't skip monitoring

// ❌ No visibility
   await callOllama(prompt);

// ✅ Track metrics
const start = Date.now();
const response = await callOllama(prompt);
metrics.recordLatency('ollama', Date.now() - start);

• Don't ignore hardware limits

// ❌ Run 70B model on 8GB GPU
   ollama run llama3.3:70b  // Out of memory!

// ✅ Use appropriate model size
ollama run llama3.1:8b   // Fits comfortably

Tools & Resources

Ollama Installation

macOS/Linux:

curl -fsSL https://ollama.com/install.sh | sh

# Download from https://ollama.com/download/windows
ollama-windows-amd64.exe

Verify installation:

ollama --version
ollama serve  # Start server
ollama pull llama3.3:70b  # Download model

Model Management

# List downloaded models
ollama list

# Pull specific model version
ollama pull llama3.3:70b-instruct-q4_K_M  # Quantized version

# Remove unused models
ollama rm mistral:7b

# Show model info
ollama show llama3.3:70b

API Usage

// JavaScript/TypeScript
const response = await fetch('http://localhost:11434/api/generate', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    model: 'llama3.3:70b',
    prompt: 'Write a hello world function',
    stream: false
  })
});

const data = await response.json();
console.log(data.response);

# Python
import requests

response = requests.post('http://localhost:11434/api/generate', json={
'model': 'llama3.3:70b',
'prompt': 'Write a hello world function',
'stream': False
})

print(response.json()['response'])

Claude Code Plugins with Ollama

From this marketplace (258 plugins):