Integration Broker Architecture

System Overview

Core Components

Broker Workers

The primary processing engine that:

• Consumes messages from Redis queues
• Executes field mappings and transformations
• Dispatches API calls to target systems
• Handles retries and error recovery

Message Queues

Redis-based queue system providing:

• Inbound Queue: Webhook triggers from external systems
• Processing Queue: Active operations being executed
• Retry Queue: Failed operations with exponential backoff
• Dead Letter Queue (DLQ): Operations exceeding max retries

Field Mapping Engine

Dynamic mapping system that:

• Translates data schemas between systems
• Applies transformation actions (sync, static, date convert, etc.)
• Supports conditional logic and complex mappings
• Handles bidirectional field mapping

Custom Action Processor

JSONNET-based transformation engine for:

• Complex data manipulation logic
• Conditional field population
• Custom business rules
• Advanced array and object transformations

Learn more about JSONNET →

Data Flow

Inbound: External System → Field Nation

Outbound: Field Nation → External System

Message Processing

Queue Processing Strategy

The broker uses a poll-and-acknowledge pattern:

1. Worker retrieves message from queue
2. Worker processes message completely
3. Worker acknowledges successful processing
4. If worker crashes, message remains unacknowledged and retries after timeout

Rate Limiting

Each integration respects external API quotas:

• Maximum requests per minute
• Concurrent connection limits
• Burst allowances
• Per-platform rate limit configuration

Retry Logic

Failed operations retry with exponential backoff:

After maximum retries, operation moves to Dead Letter Queue for manual review.

Authentication & Security

Credential Management

• Encryption at Rest: AES-256 encryption for all credentials
• In-Memory Only: Decrypted only during active use
• No Logging: Credentials never logged or exposed in errors
• Token Refresh: Automatic OAuth token refresh

Authentication Methods Supported

Authorization: Basic base64(username:password)

Authorization: Bearer {access_token}

X-API-Key: {api_key}
Authorization: Bearer {token}

<soap:Header>
  <TokenPassport>
    <account>{account_id}</account>
    <consumerKey>{consumer_key}</consumerKey>
    <token>{token}</token>
  </TokenPassport>
</soap:Header>

Request Authentication

Each trigger URL includes unique client token:

https://api.fieldnation.com/integrations/trigger/{client_token}

Broker validates token before processing any data.

Error Handling

Error Categories

Authentication Errors

• Invalid/expired credentials
• Insufficient permissions
• Token refresh failures
• Action: Manual intervention required

Validation Errors

• Missing required fields
• Invalid data types
• Schema validation failures
• Action: Fix field mappings or source data

API Errors

• Rate limit exceeded
• Network timeouts
• Service unavailable
• Action: Automatic retry with backoff

Transformation Errors

• JSONNET execution failures
• Field mapping exceptions
• Data type conversions
• Action: Review custom actions

Error Notifications

Configure email alerts for:

• Failed operations
• Dead letter queue entries
• Authentication failures
• High error rates

Performance Characteristics

Processing Capacity

• Throughput: Thousands of work orders per minute
• Scaling: Horizontal scaling via additional worker instances
• Concurrency: Multiple workers process queue in parallel

Latency

• Typical: 2-5 seconds end-to-end (trigger → work order created)
• Factors: External API response time, field mapping complexity, queue depth

Reliability

• Uptime: 99.9% availability
• Failover: Automatic worker recovery
• Circuit Breakers: Isolate failing external systems
• Health Checks: Continuous monitoring

Monitoring & Observability

Integration Logs

All operations logged with:

• Timestamps
• User identifiers
• Operation types
• Outcomes (success/failure)
• Error details
• API request/response

Health Dashboards

Track key metrics:

• Queue depths by type
• Processing rates
• Error frequencies by category
• API response times
• Sync success rates

Audit Trail

Comprehensive audit records for:

• Compliance requirements
• Incident investigation
• Performance analysis
• Capacity planning

Configuration

Field Nation Settings

Integration Broker UI: app.fieldnation.com/integrations

Configure:

• Authentication credentials (encrypted)
• Field mappings (source → target)
• Event triggers (which events sync)
• Operation types (create only vs bidirectional)
• Email notifications

External Platform Setup

Required Configuration:

• Webhook endpoints pointing to Field Nation trigger URL
• Automation rules (Flows, Business Rules, Workflows)
• API user with appropriate permissions
• Firewall/IP whitelist configuration (if applicable)

Platform-specific guides →

Data Privacy

Data Handling

• In-Transit Processing: Data processed in memory, not stored
• Logging: Error logs scrubbed of sensitive information
• Retention: Audit logs retained per compliance requirements
• Encryption: All API calls use TLS 1.2+

Compliance

The Integration Broker supports:

• SOC 2 compliance
• GDPR data handling
• Audit trail requirements
• Data residency considerations

Best Practices

Configuration

• ✅ Test in sandbox before production
• ✅ Use service accounts (not personal credentials)
• ✅ Set up error notifications
• ✅ Monitor queue depths
• ✅ Document field mappings

Security

• ✅ Rotate credentials periodically
• ✅ Use least-privilege API access
• ✅ Enable IP whitelisting where possible
• ✅ Review audit logs regularly
• ✅ Encrypt credentials in your documentation

Performance

• ✅ Keep field mappings simple when possible
• ✅ Use custom actions only when necessary
• ✅ Batch updates where supported
• ✅ Monitor API rate limits
• ✅ Scale workers for high volume

Troubleshooting

Common Issues

Complete troubleshooting guide →