The Ivy software performs regular processing of recurring and non-recurring billing items. When a customers balance falls below a certain threshold the software will attempt to email a notification to the affected customer. Since the check runs frequently throughout the day we do not want spam the customer with notifications, so a check is performed to see if we have already notified an affected customer. The ability to write to the db cache was impaired (writes failed) for the process that performs the email notifications. The thread to the redis cache was lost and did not re-establish itself properly so the writes could not be executed. As a result the software was sending errant emails to certain affected customers. To resolve this we temporarily halted the billing processes (we could have simply blocked notifications but we took a more conservative approach to investigating the issue not yet knowing if the issue was wider spread). Our team quickly identified this issue but took time to determine the cause and the best way to log, detect and prevent the issue in the future. That software patch was released overnight as a hotfix and billing has been resumed as of this morning.