orderBilly - Platform performance degradation – Incident details

Platform performance degradation

ResolvedPartial outage47 minutes

Affected

External - Payment providers - MultiSafepay

Degraded performance from 1:49 PM to 1:56 PM, Partial outage from 1:56 PM to 2:37 PM

MultiSafepay → API

Degraded performance from 1:49 PM to 1:56 PM, Partial outage from 1:56 PM to 2:37 PM

MultiSafepay → Payments

Degraded performance from 1:49 PM to 1:56 PM, Partial outage from 1:56 PM to 2:37 PM

MultiSafepay → MultiSafepay payment pages

Degraded performance from 1:49 PM to 1:56 PM, Partial outage from 1:56 PM to 2:37 PM

  • UpdateUTC

    Incident Post-Mortem: Transaction Processing Disruption — 27 May 2026

    Post-incident update
    On 27 May 2026, MultiSafepay experienced a service disruption that significantly affected transaction processing.
    The first alerts were triggered at approximately 15:28 CEST, when our monitoring detected an increase in system errors. Our engineering team started investigating immediately and identified the issue in a database table used for asynchronous transaction actions.

    15:28–16:07 — Partial impact  
    An internal capacity limit was reached, preventing certain transaction flows from being completed successfully. As a result, a large number of online transactions failed. POS transactions continued to process normally throughout this period.

    16:07–16:23 — Processing outage  
    The work needed to resolve the underlying issue caused a processing outage. No transactions were processed during this window.
    Transaction processing resumed at 16:23 CEST. After the main service was restored, our team continued recovery work on the affected follow-up processes. Key recovery steps were completed during the evening, with additional validation continuing afterwards to confirm completeness.

    Root cause
    The root cause was a database issue that occurred during a capacity expansion operation on a long-standing database table supporting offline and asynchronous transaction actions. The integer identifier range on this table was approaching its upper limit due to sustained growth in processing volumes. During the expansion activity to extend this range, the database issue prevented new records from being inserted where this table was required, disrupting affected transaction processing.

    What we’re doing
    We are taking the following actions to reduce the risk of this happening again:

    • reviewing any similar configuration limits and increasing capacity thresholds where needed, to ensure our platform can accommodate peak loads without disruption.
    • Improving monitoring and increasing internal alerts so we can act well before any limit is reached.
    • Using these findings to further strengthen the reliability of our platform.

    We are sorry for the disruption this caused to our merchants and their customers. We take the trust our merchants place in us seriously. If you have questions, please reach out to our team.

  • ResolvedUTC

    The incident has been resolved, and all services are operating normally.

  • MonitoringUTC

    The incident has been resolved and performance has returned to normal levels. We are continuing to monitor the situation closely.

  • UpdateUTC

    Our team has identified the root cause and is working on the resolution.

  • IdentifiedUTC

    Our platform is experiencing degraded performance. We are aware of the issue(s) and our team is investigating, we will update as soon as possible.