WeSpeak - Service disruption — Virtual Assistant & CRM Panel (third-party provider) – Incident details

All systems operational

Service disruption — Virtual Assistant & CRM Panel (third-party provider)

Resolved
Major outage
Started 4 days agoLasted about 6 hours

Affected

Asistente Virtual

Major outage from 4:16 PM to 4:40 PM, Operational from 4:40 PM to 5:51 PM, Major outage from 5:51 PM to 6:09 PM, Operational from 6:09 PM to 10:06 PM

Panel CRM

Major outage from 4:16 PM to 4:40 PM, Operational from 4:40 PM to 5:51 PM, Major outage from 5:51 PM to 6:09 PM, Operational from 6:09 PM to 10:06 PM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved. Our infrastructure provider confirmed at 21:17 UTC that the underlying issue was fixed on their end, and our messaging pipeline has been operating normally since 18:09 UTC (15:09 Argentina time): incoming messages are reaching the platform immediately, the Virtual Assistant is replying, and agents can send replies from the CRM panel. All messages delayed during the two disruption windows (16:16–16:40 UTC and 17:51–18:09 UTC) were processed. We apologize for the inconvenience and thank you for your patience.
  • Update
    UTC
    Update
    Our infrastructure provider reports that a fix has been deployed on their side and that they are monitoring to confirm the system is fully operational. Since 18:09 UTC (15:09 Argentina time) our messaging pipeline has been operating normally, with no further delays: incoming messages are reaching the platform immediately, the Virtual Assistant is replying, and agents can send replies from the CRM panel. We will keep this incident in monitoring until the provider confirms full resolution, and will post a final update then.
  • Monitoring
    UTC
    Monitoring
    Service has recovered again and all messages delayed during the disruption have been processed. Today the messaging pipeline was interrupted during two windows: 16:16–16:40 UTC and 17:51–18:09 UTC (13:16–13:40 and 14:51–15:09 Argentina time). Incoming messages received during those windows reached the platform late, and the Virtual Assistant replied late. Our infrastructure provider has not yet confirmed full resolution on their side, so we are monitoring closely and will post a final update once they do.
  • Investigating
    UTC
    Investigating
    The disruption at our infrastructure provider has recurred. Incoming messages are again delayed before reaching the platform and the Virtual Assistant is not replying while the delay lasts. Messages are queued, not lost. We are monitoring the provider's incident and will post an update as soon as service recovers.
  • Monitoring
    UTC
    Monitoring
    Service has recovered: incoming messages are reaching the platform again, the Virtual Assistant is replying, and the messages delayed during the disruption have been processed. Our infrastructure provider has not yet closed its incident, so we continue to monitor closely.
  • Investigating
    UTC
    Investigating
    One of our infrastructure providers is experiencing degraded function execution. As a result, incoming messages are being delayed before they reach the platform, the Virtual Assistant is not replying while the delay lasts, and replies sent by agents from the CRM panel are also delayed. No messages are lost: they are queued and will be processed once the provider recovers. Our team is monitoring the provider's incident closely. We apologize for the inconvenience.