---
name: evernote-incident-runbook
description: |
  Incident response runbook for Evernote integration issues.
  Use when troubleshooting production incidents, handling outages,
  or responding to Evernote service issues.
  Trigger with phrases like "evernote incident", "evernote outage",
  "evernote emergency", "troubleshoot evernote production".
allowed-tools: Read, Write, Edit, Bash(curl:*), Grep
version: 1.0.0
license: MIT
author: Jeremy Longshore <jeremy@intentsolutions.io>
---

# Evernote Incident Runbook

## Overview

Step-by-step procedures for responding to Evernote integration incidents including outages, rate limits, authentication failures, and data issues.

## Prerequisites

- Access to monitoring dashboards
- Production logs access
- Evernote API credentials
- Communication channels for escalation

## Incident Classification

| Severity | Impact | Response Time | Example |
|----------|--------|---------------|---------|
| P1 - Critical | All users affected | 15 min | Complete API outage |
| P2 - High | Major feature broken | 30 min | OAuth failures |
| P3 - Medium | Partial degradation | 2 hours | High error rate |
| P4 - Low | Minor issues | 1 day | Slow response times |

## Incident Response Procedures

### INC-01: Complete API Outage

**Symptoms:**
- All Evernote API calls failing
- 5xx errors from Evernote
- Connection timeouts

**Investigation:**

```bash
# Step 1: Check Evernote service status
curl -I https://www.evernote.com/
curl -I https://sandbox.evernote.com/

# Step 2: Check status page
# https://status.evernote.com/
```

```javascript
// Step 3: Run diagnostic
async function diagnoseOutage() {
  const results = {
    timestamp: new Date().toISOString(),
    checks: []
  };

  // DNS resolution
  try {
    const dns = require('dns').promises;
    const addresses = await dns.resolve4('www.evernote.com');
    results.checks.push({ name: 'DNS', status: 'ok', addresses });
  } catch (error) {
    results.checks.push({ name: 'DNS', status: 'failed', error: error.message });
  }

  // TCP connectivity
  try {
    const net = require('net');
    await new Promise((resolve, reject) => {
      const socket = net.connect(443, 'www.evernote.com');
      socket.setTimeout(5000);
      socket.on('connect', () => { socket.destroy(); resolve(); });
      socket.on('error', reject);
      socket.on('timeout', () => { socket.destroy(); reject(new Error('Timeout')); });
    });
    results.checks.push({ name: 'TCP', status: 'ok' });
  } catch (error) {
    results.checks.push({ name: 'TCP', status: 'failed', error: error.message });
  }

  // HTTPS request
  try {
    const https = require('https');
    await new Promise((resolve, reject) => {
      const req = https.get('https://www.evernote.com/', { timeout: 10000 }, (res) => {
        results.checks.push({ name: 'HTTPS', status: 'ok', statusCode: res.statusCode });
        resolve();
      });
      req.on('error', reject);
    });
  } catch (error) {
    results.checks.push({ name: 'HTTPS', status: 'failed', error: error.message });
  }

  return results;
}
```

**Mitigation:**

1. Activate circuit breaker to prevent cascading failures
2. Enable graceful degradation (show cached data)
3. Display user-friendly error message
4. Monitor Evernote status page

```javascript
// Activate graceful degradation
const degradationMode = {
  enabled: true,
  reason: 'Evernote service unavailable',
  startTime: Date.now(),

  shouldServeFromCache: true,
  shouldBlockWrites: true,
  userMessage: 'Note syncing is temporarily unavailable. Your changes will sync when service is restored.'
};

// Apply to endpoints
app.use((req, res, next) => {
  if (degradationMode.enabled) {
    res.locals.degradationMode = degradationMode;
  }
  next();
});
```

**Resolution:**
1. Monitor Evernote status for resolution
2. Gradually re-enable API calls
3. Trigger sync for affected users
4. Document incident timeline

---

### INC-02: Rate Limit Crisis

**Symptoms:**
- Frequent RATE_LIMIT_REACHED errors
- rateLimitDuration > 300 seconds
- Multiple users affected

**Investigation:**

```javascript
// Check rate limit metrics
async function analyzeRateLimits() {
  const metrics = await prometheusQuery(`
    sum(increase(evernote_rate_limits_total[1h])) by (api_key)
  `);

  const apiCallRate = await prometheusQuery(`
    sum(rate(evernote_api_calls_total[5m])) by (operation)
  `);

  return {
    rateLimitsLastHour: metrics,
    currentCallRate: apiCallRate,
    suspectedCauses: identifyCauses(apiCallRate)
  };
}

function identifyCauses(callRate) {
  const causes = [];

  // Check for polling abuse
  if (callRate['NoteStore.getSyncState'] > 1) {
    causes.push('Excessive sync state polling');
  }

  // Check for inefficient requests
  if (callRate['NoteStore.getResource'] > 10) {
    causes.push('Individual resource fetching (should batch)');
  }

  return causes;
}
```

**Mitigation:**

```javascript
// Emergency rate limiting
class EmergencyRateLimiter {
  constructor() {
    this.globalPause = false;
    this.pauseUntil = 0;
  }

  async activateEmergencyPause(durationSeconds) {
    this.globalPause = true;
    this.pauseUntil = Date.now() + (durationSeconds * 1000);

    console.warn(`Emergency rate limit pause activated for ${durationSeconds}s`);

    // Notify operations
    await alertOps('Emergency rate limit pause', {
      duration: durationSeconds,
      reason: 'Excessive rate limits detected'
    });

    // Auto-deactivate
    setTimeout(() => {
      this.globalPause = false;
      console.info('Emergency rate limit pause deactivated');
    }, durationSeconds * 1000);
  }

  async checkBeforeRequest() {
    if (this.globalPause) {
      const waitTime = this.pauseUntil - Date.now();
      if (waitTime > 0) {
        throw new Error(`Rate limit emergency: retry in ${Math.ceil(waitTime / 1000)}s`);
      }
    }
  }
}
```

**Resolution:**
1. Identify and fix inefficient API usage
2. Increase cache TTLs
3. Implement request coalescing
4. Request rate limit boost from Evernote

---

### INC-03: Authentication Failures

**Symptoms:**
- Users receiving auth errors
- OAuth flow failing
- Token rejections

**Investigation:**

```javascript
// Auth diagnostic
async function diagnoseAuthIssue(userId) {
  const user = await db.users.findById(userId);
  const token = await db.tokens.findByUserId(userId);

  const diagnosis = {
    userId,
    hasToken: !!token,
    tokenExpired: token ? (Date.now() > token.expiresAt) : null,
    tokenExpiresIn: token ? Math.floor((token.expiresAt - Date.now()) / 1000 / 60 / 60) + ' hours' : null
  };

  // Test token validity
  if (token && !diagnosis.tokenExpired) {
    try {
      const client = new Evernote.Client({ token: token.accessToken, sandbox: false });
      const userStore = client.getUserStore();
      await userStore.getUser();
      diagnosis.tokenValid = true;
    } catch (error) {
      diagnosis.tokenValid = false;
      diagnosis.tokenError = {
        code: error.errorCode,
        parameter: error.parameter
      };
    }
  }

  return diagnosis;
}
```

**Common Causes & Fixes:**

| Error Code | Cause | Fix |
|------------|-------|-----|
| 4 (INVALID_AUTH) | Token revoked | Re-authenticate user |
| 5 (AUTH_EXPIRED) | Token expired | Re-authenticate user |
| 3 (PERMISSION_DENIED) | Insufficient permissions | Check API key permissions |

**Resolution:**
```javascript
// Batch re-auth notification
async function notifyUsersToReauth(userIds) {
  for (const userId of userIds) {
    await sendNotification(userId, {
      type: 'REAUTH_REQUIRED',
      message: 'Please reconnect your Evernote account to continue syncing.',
      action: { type: 'REDIRECT', url: '/auth/evernote' }
    });

    await db.tokens.markInvalid(userId);
  }
}
```

---

### INC-04: Data Sync Issues

**Symptoms:**
- Notes not appearing
- Sync state stuck
- Missing changes

**Investigation:**

```javascript
// Sync state diagnostic
async function diagnoseSyncIssue(userId) {
  const syncState = await db.syncState.findByUserId(userId);
  const client = await getClientForUser(userId);

  const remoteSyncState = await client.noteStore.getSyncState();

  return {
    localUSN: syncState.lastUpdateCount,
    remoteUSN: remoteSyncState.updateCount,
    behind: remoteSyncState.updateCount - syncState.lastUpdateCount,
    lastSyncAt: syncState.lastSyncAt,
    needsSync: remoteSyncState.updateCount > syncState.lastUpdateCount,
    fullSyncRequired: syncState.fullSyncRequired
  };
}

// Force resync
async function forceResync(userId) {
  await db.syncState.update(userId, {
    lastUpdateCount: 0,
    fullSyncRequired: true,
    lastSyncAt: null
  });

  // Queue sync job
  await syncQueue.add('full-sync', { userId, priority: 'high' });

  return { status: 'queued', message: 'Full resync initiated' };
}
```

---

## Incident Communication Templates

### Status Page Update

```markdown
## [Investigating] Evernote Integration Issue

**Time:** [TIMESTAMP]
**Status:** Investigating

We are currently investigating issues with Evernote note synchronization.
Some users may experience delays in note updates.

We will provide updates as we learn more.
```

### User Notification

```markdown
## Temporary Sync Delay

Hi [USER],

We're experiencing a temporary delay in syncing notes with Evernote.
Your local changes are saved and will sync automatically when the
issue is resolved.

No action is needed on your part.

Expected resolution: Within 2 hours

Thank you for your patience.
```

### Resolution Update

```markdown
## [Resolved] Evernote Integration Issue

**Time:** [TIMESTAMP]
**Status:** Resolved

The Evernote synchronization issue has been resolved.
All notes should now be syncing normally.

**Root Cause:** [BRIEF DESCRIPTION]
**Duration:** [X hours Y minutes]
**Impact:** [NUMBER] users affected

We apologize for any inconvenience.
```

## Post-Incident Checklist

```markdown
## Post-Incident Review

### Timeline
- [ ] Document incident timeline
- [ ] Record all actions taken
- [ ] Note what worked/didn't work

### Root Cause
- [ ] Identify root cause
- [ ] Determine contributing factors
- [ ] Assess detection time

### Prevention
- [ ] Define preventive measures
- [ ] Update monitoring/alerts
- [ ] Improve runbooks

### Follow-up
- [ ] Schedule post-mortem meeting
- [ ] Assign action items
- [ ] Update documentation
```

## Output

- Incident classification guide
- Step-by-step response procedures
- Diagnostic scripts
- Mitigation strategies
- Communication templates
- Post-incident checklist

## Resources

- [Evernote Status Page](https://status.evernote.com/)
- [Evernote Developer Support](https://dev.evernote.com/support/)
- [Error Handling](https://dev.evernote.com/doc/articles/error_handling.php)

## Next Steps

For data handling best practices, see `evernote-data-handling`.
