Ivy Billing temporarily halted
Incident Report for Teliax
Postmortem

The Ivy software performs regular processing of recurring and non-recurring billing items. When a customers balance falls below a certain threshold the software will attempt to email a notification to the affected customer. Since the check runs frequently throughout the day we do not want spam the customer with notifications, so a check is performed to see if we have already notified an affected customer. The ability to write to the db cache was impaired (writes failed) for the process that performs the email notifications. The thread to the redis cache was lost and did not re-establish itself properly so the writes could not be executed. As a result the software was sending errant emails to certain affected customers. To resolve this we temporarily halted the billing processes (we could have simply blocked notifications but we took a more conservative approach to investigating the issue not yet knowing if the issue was wider spread). Our team quickly identified this issue but took time to determine the cause and the best way to log, detect and prevent the issue in the future. That software patch was released overnight as a hotfix and billing has been resumed as of this morning.

Posted 9 days ago. Sep 11, 2019 - 13:57 MDT

Resolved
This incident has been resolved.
Posted 9 days ago. Sep 11, 2019 - 13:56 MDT
Monitoring
This issue has been resolved with a software update from our engineering team. Billing operations have resumed.
Posted 9 days ago. Sep 11, 2019 - 06:52 MDT
Update
We are continuing to investigate this issue.
Posted 10 days ago. Sep 10, 2019 - 11:26 MDT
Investigating
We are currently investigating a db aching issue on the Ivy Partner Platform that has caused errant email notifications to be sent to some customers. Backend billing services have been temporarily halted. This will not affect call completion however the call records will not appear in the user portal until billing services have been re-established.
Posted 10 days ago. Sep 10, 2019 - 11:26 MDT
This incident affected: Ivy (Ivy Billing Processes).