Telemetry & Pattern Analysis

Cloud Feature: Telemetry collection, pattern analysis, and cost analytics are available on orchex cloud (Pro, Team, Enterprise tiers). The local npm package does not collect or store telemetry data. Local learning (adaptive thresholds via file-based events) works offline without cloud connectivity.

Note (2026-02-23): The Telemetry API described in this document is planned for P2 (Observability & Trust). Current telemetry consists of local adaptive thresholds only. Cloud telemetry, webhooks, and the programmatic API are not yet available.

Orchex collects execution telemetry to improve orchestration intelligence, optimize performance, and provide insights into usage patterns. This guide explains what data is collected, how it's used, and privacy controls.

Overview

Orchex telemetry serves three primary purposes:

Performance Optimization — Identify bottlenecks and optimize execution strategies
Pattern Recognition — Learn from successful orchestrations to improve future plans
Usage Analytics — Track resource consumption for billing and capacity planning

All telemetry collection respects user privacy and provides opt-out controls for sensitive data.

What Data is Collected

Execution Metrics

Collected for every orchestration:

{
  executionId: 'exec_1234567890',
  userId: 'user_abc123',
  teamId: 'team_xyz789',
  status: 'completed',
  
  timing: {
    startedAt: '2026-02-05T10:00:00Z',
    completedAt: '2026-02-05T10:05:32Z',
    duration: 332000,  // milliseconds
    queueTime: 150     // time waiting in queue
  },
  
  resources: {
    tokensUsed: 12500,
    apiCalls: 8,
    stepCount: 6,
    contextSize: 45000,  // characters
    artifactsGenerated: 4
  },
  
  outcome: {
    success: true,
    errorCode: null,
    retryCount: 0
  }
}

Step-Level Telemetry

Collected for each orchestration step:

{
  stepId: 'step_5',
  type: 'plan' | 'execute' | 'validate',
  duration: 8500,
  tokensUsed: 2100,
  
  operation: {
    action: 'file_edit',
    target: 'src/server.ts',
    changeSize: 145  // lines added/modified
  },
  
  result: {
    success: true,
    validationPassed: true,
    commandExitCode: 0
  }
}

Pattern Data (Anonymized)

Used for improving orchestration intelligence:

{
  patternId: 'pattern_auth_implementation',
  category: 'feature_addition',
  
  // Hashed/anonymized identifiers
  taskHash: 'sha256:abc123...',
  filePatternHash: 'sha256:def456...',
  
  strategy: {
    planningApproach: 'file_tree_first',
    contextDepth: 'moderate',
    stepSequence: ['analyze', 'plan', 'implement', 'test', 'validate'],
    successRate: 0.95
  },
  
  metadata: {
    projectType: 'typescript_node',
    complexity: 'moderate',
    similarPatternsCount: 42
  }
}

Error Telemetry

Collected when orchestrations fail:

{
  errorId: 'err_1234567890',
  errorType: 'validation_failure',
  errorMessage: 'Test command failed',
  
  context: {
    stepNumber: 4,
    commandRun: 'npm test',
    exitCode: 1
  },
  
  recovery: {
    recovered: false,
    retryAttempts: 2,
    fallbackUsed: false
  },
  
  // Stack traces are anonymized
  stackHash: 'sha256:xyz789...'
}

Pattern Recognition

How Patterns Are Identified

Orchex analyzes execution telemetry to identify recurring patterns:

1. Task Clustering

Similar tasks are grouped using semantic hashing:

// Similar tasks cluster together
cluster_auth_features = [
  'Add user authentication',
  'Implement login system',
  'Create auth middleware',
  'Add JWT token validation'
]

2. Success Pattern Extraction

Successful orchestrations contribute to pattern libraries:

pattern = {
  name: 'api_authentication_implementation',
  successRate: 0.92,
  avgDuration: 180000,
  avgTokens: 8500,
  
  commonSteps: [
    'Review existing auth structure',
    'Create auth middleware',
    'Add token validation',
    'Update routes with auth',
    'Add auth tests',
    'Update API documentation'
  ],
  
  commonFiles: [
    'src/middleware/auth.ts',
    'src/routes/*.ts',
    'tests/auth.test.ts'
  ],
  
  recommendations: [
    'Include existing auth patterns in context',
    'Test both success and failure cases',
    'Update OpenAPI spec if present'
  ]
}

3. Failure Pattern Analysis

Recurring failures inform prevention strategies:

failurePattern = {
  trigger: 'missing_test_coverage',
  frequency: 0.15,  // 15% of similar tasks
  
  commonCauses: [
    'Tests not updated after implementation',
    'Missing test fixtures',
    'Import paths not updated'
  ],
  
  preventionStrategy: [
    'Add explicit test update step',
    'Verify imports before validation',
    'Include test file patterns in context'
  ]
}

Using Pattern Recommendations

Orchex applies learned patterns to new orchestrations:

// When you execute a task
const result = await orchex.execute({
  task: 'Add rate limiting to API endpoints'
});

// Orchex automatically:
// 1. Identifies similar pattern: "api_middleware_addition"
// 2. Applies learned context strategy
// 3. Uses proven step sequence
// 4. Includes recommended files
// 5. Applies prevention strategies for common failures

Viewing Applied Patterns:

// Check what patterns were used
console.log(result.intelligence.patternsApplied);

// Output:
[
  {
    patternId: 'pattern_api_middleware',
    confidence: 0.89,
    adjustments: [
      'Added test file patterns to context',
      'Included existing middleware for reference'
    ]
  }
]

Privacy Controls

Data Anonymization

Sensitive data is automatically anonymized:

File Paths:

// Original
path: '/Users/john/projects/my-secret-app/src/auth.ts'

// Anonymized for pattern storage
pattern: 'src/auth.ts'
projectHash: 'sha256:abc123...'

Task Descriptions:

// Original
task: 'Fix login bug for Acme Corp client portal'

// Anonymized for pattern analysis
taskPattern: 'fix authentication issue'
taskHash: 'sha256:def456...'

Code Content:

// Never stored in pattern database
// Only structural information retained:
{
  fileType: 'typescript',
  linesChanged: 42,
  changeType: 'function_addition',
  testCoverage: true
}

Opt-Out Controls

Disable Pattern Learning

Per Execution:

const result = await orchex.execute({
  task: 'Implement sensitive feature',
  privacy: {
    disablePatternLearning: true  // Don't contribute to pattern database
  }
});

Team-Wide Setting:

orchex config set telemetry.pattern-learning false

Environment Variable:

export ORCHEX_DISABLE_PATTERN_LEARNING=true

Disable All Telemetry (Self-Hosted Only)

# Disable all telemetry collection
export ORCHEX_TELEMETRY=false

# Or in configuration
orchex config set telemetry.enabled false

Note: Cloud-hosted Orchex requires basic usage telemetry for billing and rate limiting. Self-hosted deployments can disable all telemetry.

Data Retention Controls

// Request data deletion
await orchex.privacy.deleteExecutionData({
  executionIds: ['exec_123', 'exec_456'],
  includePatterns: true  // Also remove pattern contributions
});

// View your data
const myData = await orchex.privacy.exportData();

// Auto-deletion policy
await orchex.privacy.setRetentionPolicy({
  executionData: '90d',    // Delete after 90 days
  patternData: 'never',    // Never contribute to patterns
  errorLogs: '30d'         // Delete error logs after 30 days
});

Telemetry API

Querying Your Telemetry

Get Execution Metrics:

const metrics = await orchex.telemetry.getMetrics({
  executionId: 'exec_1234567890'
});

console.log(metrics);
// {
//   duration: 332000,
//   tokensUsed: 12500,
//   stepCount: 6,
//   efficiency: 0.87,  // vs similar tasks
//   costEstimate: 0.25  // USD
// }

Aggregate Team Metrics:

const teamMetrics = await orchex.telemetry.getTeamMetrics({
  period: '30d',
  groupBy: 'day'
});

console.log(teamMetrics);
// {
//   totalExecutions: 247,
//   successRate: 0.94,
//   avgDuration: 245000,
//   totalTokens: 1250000,
//   topPatterns: [...]
// }

Pattern Insights:

const insights = await orchex.telemetry.getPatternInsights({
  taskDescription: 'Add API endpoint'
});

console.log(insights);
// {
//   matchingPatterns: [
//     {
//       name: 'api_endpoint_addition',
//       confidence: 0.92,
//       recommendations: [
//         'Include existing API routes for consistency',
//         'Add OpenAPI documentation',
//         'Include integration tests'
//       ],
//       estimatedDuration: 180000,
//       estimatedTokens: 8500
//     }
//   ]
// }

Webhook Events

Receive telemetry events in real-time:

// Register webhook
await orchex.webhooks.create({
  url: 'https://yourapp.com/orchex-events',
  events: [
    'execution.completed',
    'pattern.discovered',
    'usage.threshold_reached'
  ]
});

// Webhook payload
{
  event: 'pattern.discovered',
  timestamp: '2026-02-05T10:30:00Z',
  data: {
    patternId: 'pattern_new_123',
    category: 'refactoring',
    confidence: 0.88,
    occurrences: 5,
    description: 'TypeScript class to functional component conversion'
  }
}

Performance Insights

Efficiency Scoring

Orchex calculates efficiency scores by comparing executions to similar patterns:

const execution = await orchex.execute({
  task: 'Refactor authentication module'
});

console.log(execution.metrics.efficiency);
// {
//   score: 0.87,  // 87% efficiency vs similar tasks
//   comparison: {
//     avgDuration: 240000,    // similar tasks
//     yourDuration: 195000,   // your execution (faster!)
//     avgTokens: 9500,
//     yourTokens: 8200,       // more efficient
//     avgSteps: 7,
//     yourSteps: 6
//   },
//   insights: [
//     'More efficient than 78% of similar orchestrations',
//     'Used 14% fewer tokens than average',
//     'Context selection was optimal'
//   ]
// }

Optimization Recommendations

const recommendations = await orchex.telemetry.getOptimizationRecommendations();

console.log(recommendations);
// [
//   {
//     type: 'context_optimization',
//     impact: 'high',
//     description: 'Your orchestrations include 30% more files than necessary',
//     suggestion: 'Use depth: "moderate" instead of "comprehensive"',
//     estimatedSavings: {
//       tokens: 2500,
//       cost: 0.05,
//       duration: 15000
//     }
//   },
//   {
//     type: 'manifest_reuse',
//     impact: 'medium',
//     description: 'Task "add API endpoint" executed 12 times',
//     suggestion: 'Create reusable manifest for API endpoint additions',
//     estimatedSavings: {
//       setupTime: 120000,
//       tokens: 8000,
//       cost: 0.16
//     }
//   }
// ]

Cost Analysis

const costAnalysis = await orchex.telemetry.getCostAnalysis({
  period: '30d'
});

console.log(costAnalysis);
// {
//   total: 24.50,  // USD
//   breakdown: {
//     apiCalls: 18.20,
//     tokensUsed: 6.30
//   },
//   byPattern: [
//     {
//       pattern: 'feature_implementation',
//       cost: 12.30,
//       executions: 15,
//       avgCost: 0.82
//     },
//     {
//       pattern: 'bug_fix',
//       cost: 8.20,
//       executions: 28,
//       avgCost: 0.29
//     }
//   ],
//   projectedMonthly: 30.00,
//   recommendations: [
//     'Consider Team plan for 18% savings at current usage',
//     'Use manifests to reduce per-execution costs by ~25%'
//   ]
// }

Self-Hosted Telemetry

Local Telemetry Storage

Self-hosted Orchex can store telemetry locally:

# config/telemetry.yml
telemetry:
  enabled: true
  storage:
    type: postgres  # or sqlite, file
    connectionString: postgresql://localhost/orchex_telemetry
  
  retention:
    executionData: 90d
    patternData: 365d
    errorLogs: 30d
  
  privacy:
    anonymizeFilePaths: true
    anonymizeTaskDescriptions: true
    storeCodeContent: false
  
  export:
    format: json  # or csv, parquet
    schedule: weekly
    destination: s3://your-bucket/orchex-telemetry/

Pattern Database Export

Export learned patterns for backup or sharing:

# Export patterns
orchex telemetry export-patterns --output patterns.json

# Import patterns (e.g., for new deployment)
orchex telemetry import-patterns --input patterns.json

# Share patterns with team
orchex telemetry share-patterns \
  --team team_xyz789 \
  --categories feature_implementation,bug_fix

Custom Telemetry Integrations

// Send telemetry to your own systems
orchex.telemetry.on('execution.completed', (event) => {
  // Send to your monitoring system
  datadog.increment('orchex.executions', 1, {
    status: event.status,
    pattern: event.patternId
  });
  
  // Log to your analytics
  analytics.track('Orchex Execution', {
    duration: event.duration,
    tokens: event.tokensUsed,
    success: event.success
  });
});

Privacy & Security

Data Security

Encryption at Rest: All telemetry data encrypted using AES-256
Encryption in Transit: TLS 1.3 for all data transmission
Access Controls: Role-based access to telemetry data
Audit Logs: All telemetry access logged

Compliance

GDPR Compliance:

Right to access: Export your telemetry data
Right to deletion: Delete your execution history
Right to portability: Export in standard formats
Right to opt-out: Disable pattern learning

Security practices:

Encryption at rest for sensitive data (AES-256-GCM)
Sensitive field redaction in logs
Incident response procedures

Orchex never shares:

Your code content
Project names or file paths
Task descriptions
Team member information
Execution results

Orchex may share (anonymized):

Aggregated usage statistics
Pattern success rates
Performance benchmarks
Error frequency statistics

All sharing is opt-in for Enterprise plans.

Telemetry Best Practices

Maximize Pattern Learning Benefits

1. Consistent Task Descriptions

// Good - consistent, clear
await orchex.execute({ task: 'Add user authentication' });
await orchex.execute({ task: 'Add admin authentication' });

// Less useful - inconsistent patterns
await orchex.execute({ task: 'auth stuff' });
await orchex.execute({ task: 'make login work' });

2. Use Manifest Streams for Repeated Tasks

# Builds better patterns through consistent execution
streams:
  add-api-endpoint:
    description: Add new REST API endpoint
    plan: |
      1. Review existing endpoints for patterns
      2. Create route handler
      3. Add validation schema
      4. Write integration tests
      5. Update OpenAPI documentation

3. Provide Feedback on Executions

// Mark successful patterns
await orchex.feedback.rate(executionId, {
  rating: 5,
  notes: 'Perfect approach for API endpoints'
});

// Report issues to improve future executions
await orchex.feedback.report(executionId, {
  issue: 'Missed updating related documentation',
  suggestion: 'Include docs/ in context for API changes'
});

Optimize for Cost

Monitor Token Usage:

const usage = await orchex.telemetry.getTokenUsage({ period: '7d' });

if (usage.avgTokensPerExecution > 15000) {
  console.warn('High token usage detected');
  const recs = await orchex.telemetry.getOptimizationRecommendations();
  console.log(recs);
}

Set Usage Alerts:

await orchex.alerts.create({
  type: 'monthly_token_threshold',
  threshold: 1000000,
  action: 'email',
  recipients: ['team@company.com']
});

Troubleshooting

Telemetry Not Appearing

Check Configuration:

orchex config get telemetry
# Should show: enabled: true

Verify Network Connectivity:

orchex telemetry test
# Tests connection to telemetry endpoints

Check Opt-Out Settings:

echo $ORCHEX_TELEMETRY
echo $ORCHEX_DISABLE_PATTERN_LEARNING
# Should be empty or 'true'

Pattern Recommendations Not Working

Insufficient Data:

Patterns require 5+ similar successful executions
May take 1-2 weeks to build pattern library

Check Pattern Status:

const patterns = await orchex.telemetry.getPatterns({ minConfidence: 0.7 });
console.log(`Available patterns: ${patterns.length}`);

High Token Usage

Analyze Token Distribution:

const analysis = await orchex.telemetry.analyzeTokenUsage({
  executionId: 'exec_high_usage'
});

console.log(analysis);
// {
//   contextLoading: 8500,    // 68% - context too large!
//   planning: 2000,
//   execution: 1500,
//   validation: 500,
//   recommendations: [
//     'Reduce context depth from "comprehensive" to "moderate"',
//     'Use more specific file patterns'
//   ]
// }

Additional Resources

Privacy Policy — Complete privacy policy
Security — Security practices and certifications
API Reference — Telemetry API endpoints
Cloud Execution — Cloud-specific telemetry features
Error Handling — Error telemetry and debugging

Support

Questions about telemetry, privacy, or pattern analysis?

Documentation: orchex.dev/docs
Support: support@orchex.dev
Privacy Inquiries: privacy@orchex.dev
Security Issues: security@orchex.dev

Last updated: February 2026