instantly-observability

# Instantly Observability ## Overview Set up comprehensive observability for Instantly integrations. ## Prerequisites - Prometheus or compatible metrics backend - OpenTelemetry SDK installed - Grafana or similar dashboarding tool - AlertManager configured ## Metrics Collection ### Key Metrics | Metric | Type | Description | |--------|------|-------------| | `instantly_requests_total` | Counter | Total API requests | | `instantly_request_duration_seconds` | Histogram | Request latency | | `instantly_errors_total` | Counter | Error count by type | | `instantly_rate_limit_remaining` | Gauge | Rate limit headroom | ### Prometheus Metrics ```typescript import { Registry, Counter, Histogram, Gauge } from 'prom-client'; const registry = new Registry(); const requestCounter = new Counter({ name: 'instantly_requests_total', help: 'Total Instantly API requests', labelNames: ['method', 'status'], registers: [registry], }); const requestDuration = new Histogram({ name: 'instantly_request_duration_seconds', help: 'Instantly request duration', labelNames: ['method'], buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5], registers: [registry], }); const errorCounter = new Counter({ name: 'instantly_errors_total', help: 'Instantly errors by type', labelNames: ['error_type'], registers: [registry], }); ``` ### Instrumented Client ```typescript async function instrumentedRequest( method: string, operation: () => Promise ): Promise { const timer = requestDuration.startTimer({ method }); try { const result = await operation(); requestCounter.inc({ method, status: 'success' }); return result; } catch (error: any) { requestCounter.inc({ method, status: 'error' }); errorCounter.inc({ error_type: error.code || 'unknown' }); throw error; } finally { timer(); } } ``` ## Distributed Tracing ### OpenTelemetry Setup ```typescript import { trace, SpanStatusCode } from '@opentelemetry/api'; const tracer = trace.getTracer('instantly-client'); async function tracedInstantlyCall( operationName: string, operation: () => Promise ): Promise { return tracer.startActiveSpan(`instantly.${operationName}`, async (span) => { try { const result = await operation(); span.setStatus({ code: SpanStatusCode.OK }); return result; } catch (error: any) { span.setStatus({ code: SpanStatusCode.ERROR, message: error.message }); span.recordException(error); throw error; } finally { span.end(); } }); } ``` ## Logging Strategy ### Structured Logging ```typescript import pino from 'pino'; const logger = pino({ name: 'instantly', level: process.env.LOG_LEVEL || 'info', }); function logInstantlyOperation( operation: string, data: Record, duration: number ) { logger.info({ service: 'instantly', operation, duration_ms: duration, ...data, }); } ``` ## Alert Configuration ### Prometheus AlertManager Rules ```yaml # instantly_alerts.yaml groups: - name: instantly_alerts rules: - alert: InstantlyHighErrorRate expr: | rate(instantly_errors_total[5m]) / rate(instantly_requests_total[5m]) > 0.05 for: 5m labels: severity: warning annotations: summary: "Instantly error rate > 5%" - alert: InstantlyHighLatency expr: | histogram_quantile(0.95, rate(instantly_request_duration_seconds_bucket[5m]) ) > 2 for: 5m labels: severity: warning annotations: summary: "Instantly P95 latency > 2s" - alert: InstantlyDown expr: up{job="instantly"} == 0 for: 1m labels: severity: critical annotations: summary: "Instantly integration is down" ``` ## Dashboard ### Grafana Panel Queries ```json { "panels": [ { "title": "Instantly Request Rate", "targets": [{ "expr": "rate(instantly_requests_total[5m])" }] }, { "title": "Instantly Latency P50/P95/P99", "targets": [{ "expr": "histogram_quantile(0.5, rate(instantly_request_duration_seconds_bucket[5m]))" }] } ] } ``` ## Instructions ### Step 1: Set Up Metrics Collection Implement Prometheus counters, histograms, and gauges for key operations. ### Step 2: Add Distributed Tracing Integrate OpenTelemetry for end-to-end request tracing. ### Step 3: Configure Structured Logging Set up JSON logging with consistent field names. ### Step 4: Create Alert Rules Define Prometheus alerting rules for error rates and latency. ## Output - Metrics collection enabled - Distributed tracing configured - Structured logging implemented - Alert rules deployed ## Error Handling | Issue | Cause | Solution | |-------|-------|----------| | Missing metrics | No instrumentation | Wrap client calls | | Trace gaps | Missing propagation | Check context headers | | Alert storms | Wrong thresholds | Tune alert rules | | High cardinality | Too many labels | Reduce label values | ## Examples ### Quick Metrics Endpoint ```typescript app.get('/metrics', async (req, res) => { res.set('Content-Type', registry.contentType); res.send(await registry.metrics()); }); ``` ## Resources - [Prometheus Best Practices](https://prometheus.io/docs/practices/naming/) - [OpenTelemetry Documentation](https://opentelemetry.io/docs/) - [Instantly Observability Guide](https://docs.instantly.com/observability) ## Next Steps For incident response, see `instantly-incident-runbook`.