The final few thousand missing CalloutLogEntries have been backfilled. Thank you for your patience. We are going to resolve this incident. Contact support@tanklogix.com with any questions or concerns.
Posted Mar 10, 2023 - 14:35 CST
Update
The small amount of remaining alarm event data has been backfilled.
Posted Mar 10, 2023 - 12:38 CST
Update
The system is still functioning well, and has been been since early the morning of 03/09. We have more scale and performance improvements coming. We are still working on a very small amount of historical callout log and alarm event data that needs to be backfilled but should have that resolved in a matter of hours.
We will leave this incident open for visibility but the system continues to be fully operational.
Posted Mar 10, 2023 - 11:54 CST
Update
Calculated flow data has been backfilled. We still have a small amount of callout log and alarm event data to backfill and will post a final update when complete.
Posted Mar 09, 2023 - 17:45 CST
Update
The main service has recovered.
Posted Mar 09, 2023 - 15:07 CST
Update
As part of our updates we are restarting the main service. You may see values go in and out for the next 5-10 minutes.
Posted Mar 09, 2023 - 15:04 CST
Update
Historical alarm event data is now current, without lag. There is still a gap in the last 24 hours we are backfilling, all historical data prior is still loading as it has been.
Posted Mar 09, 2023 - 12:36 CST
Update
Historical data has been backfilled for a few hours. We are still working on some calculated flow, callout log, and historical alarm event data.
Posted Mar 09, 2023 - 11:34 CST
Monitoring
We continue to backfill some calculated flow, callout log, alarm event, and historical data. The system is stable. Disruption has been lessened by our ability to keep realtime data, alarming, and many other aspects of the system up.
Most of the data left to backfill is used in caching/performance and is not critical to operations. We will post when all data has been restored.
We thank you for your patience.
Posted Mar 09, 2023 - 06:10 CST
Update
We have bypassed the read only database for now while we work on a permanent fix. We have to manually fill some data and perform more configuration but it should help us resolve the issue much faster. Realtime data, alarms and notifications remain fully operational.
Posted Mar 09, 2023 - 05:07 CST
Identified
We are reconfiguring much of the read only database. You may see lag in historical data for the current month, callout log, and alarm events on the desktop and mobile clients.
All data is flowing into our primary write database and realtime data and alarms are operating normally. The read only database exists to reduce load on the primary database but is having issues. Our team is working hard on the issue.
Posted Mar 09, 2023 - 03:48 CST
Monitoring
We have implemented a fix and data is catching up again. We are working to identify the root cause and will post as soon as we have a permanent solution.
Posted Mar 08, 2023 - 23:55 CST
Identified
We are seeing some history lag on our read only database again. All data is caught up on our primary write database, just desktop and mobile apps will show a lag. Working on making more configuration changes to help the issue.