Realtime Compliance Aggregator

Situation

Compliance systems that rely on periodic batch jobs miss regulatory changes between runs. When a rule conflict is detected hours or days after a source update, the window for a clean response has already closed. Most implementations also couple the ingestion logic directly to the detection logic, making it hard to add new sources or change detection rules without rewriting core code.

Task

Build a service that ingests compliance rules from multiple sources continuously, detects conflicts as soon as they appear, and emits structured change events downstream. Use OTP's supervision model so the system recovers from partial failures without manual intervention.

Action

• Built a GenServer-based poller per compliance source. Each source runs on its own supervised process so a failing source does not affect others.
• Used Task.async_stream capped at 10 concurrent fetchers to respect source rate limits without external queue infrastructure.
• Conflict detection runs as parsed rules flow in. Each new rule is checked against the existing rule set and conflicts are emitted as structured events immediately.
• OTP supervision tree automatically restarts any crashed poller process. The system is self-healing by design.
• Phoenix JSON API exposes current rule state and conflict history for downstream consumers.

Result

• Regulatory conflicts surface in real time instead of on a batch schedule.
• The system self-recovers from source failures automatically with no manual restart or alerting required.
• New compliance sources can be added as independent processes without modifying detection logic.
• Structured downstream events make it straightforward for consumers to react to specific conflict types.