Degraded database performance

Started at Mar 12, 2024 09:30 (UTC)

Resolved
Dashboard & API

resolved

Mar 12, 2024 11:39 (UTC)

Our database has recovered and runs are now processing normally again.

identified

Mar 12, 2024 09:30 (UTC)

A very large spike in runs caused degraded the database to have high CPU. This made the dashboard very slow to load and reduced the speed that runs were being processed.

We have fixed the issue but it will take a couple of hours to get back to normal run processing times.