TracCloudStatusHistory: Difference between revisions
From Redrock Wiki
No edit summary |
No edit summary |
||
| Line 2: | Line 2: | ||
<i>History:</i><br> | <i>History:</i><br> | ||
<b>2024-09-16</b> - TracCloud SAML logins failing with 504/502 error beginning at 1:08pm, TracCloud down beginning at 1:40pm, both resolved at 2:18pm MST. | <hr> | ||
<div style="float: left; margin-top: 0em; margin-bottom: 1em"><b>2024-09-16</b> - TracCloud SAML logins failing with 504/502 error beginning at 1:08pm, TracCloud down beginning at 1:40pm, both resolved at 2:18pm MST.</div><div class="mw-collapsible mw-collapsed"> | |||
<br><br> | |||
We wanted to inform you that the system issue experienced today has been successfully resolved. While Amazon Web Services (AWS) does not always provide direct notifications for hardware-related issues, we have identified that the root cause was likely a hardware fault affecting systems in one of AWS's availability zones in the region TracCloud resides in (Zone A). This issue impacted several unrelated systems that were all using resources in that zone. | |||
Upon realizing this, to resolve the problem, we manually redirected operations to a different zone (Zone B), and both TracCloud and other associated systems are now functioning as expected. We are also in contact with AWS for further clarification, but at this point, the issue has been fully addressed. | |||
</div> | |||
<hr> | |||
<b>2024-09-12</b> - TracCloud periodically running slowly between August 29th and September 12th. | <b>2024-09-12</b> - TracCloud periodically running slowly between August 29th and September 12th. | ||
Revision as of 14:48, 16 September 2024
History:
We wanted to inform you that the system issue experienced today has been successfully resolved. While Amazon Web Services (AWS) does not always provide direct notifications for hardware-related issues, we have identified that the root cause was likely a hardware fault affecting systems in one of AWS's availability zones in the region TracCloud resides in (Zone A). This issue impacted several unrelated systems that were all using resources in that zone.
Upon realizing this, to resolve the problem, we manually redirected operations to a different zone (Zone B), and both TracCloud and other associated systems are now functioning as expected. We are also in contact with AWS for further clarification, but at this point, the issue has been fully addressed.
2024-09-12 - TracCloud periodically running slowly between August 29th and September 12th.
2024-09-11 - TracCloud 502/504 errors or slowness, resolved at 9:00am MST.
2024-08-15 - TracCloud slowness or timeout errors between 3:00pm and 3:30pm and 4:20pm to 4:34pm MST.
2024-08-09 - TracCloud sporadically inaccessible with 503 and 504 errors between 10:48am and 11:20am MST.
2024-07-11 - Reason drop-down on search availability widget not working on some TracCloud instances between 10:49am and 11:31am MST.
2024-05-31 - TracCloud job server down, causing automated emails and processes to run late.
2024-05-22 - 502 Bad Gateway error from 11:45am to 11:51am MST.
2024-04-23 - AWS errors preventing many features from working as expected. Resolved at 7:45am MST.
2024-04-22 - Appointment statuses not displaying in appointments over the weekend. Resolved on Monday at 8:29am MST.
2024-04-11 - Reminder emails not sending for some instances. Resolved at 5:32pm MST.
2024-02-15 - SAML error 500 preventing SSO login between 2:00pm and 2:30pm MST.
2024-02-13 - SAML errors preventing SSO login, resolved at 8:00am MST.
2024-02-12 - Registration lists displayed when booking appointments, logging in, etc not functioning correctly starting at 2:57pm. Most issues resolved at 3:13pm MST, quick visit section selection and badge section selection resolved at 3:57pm.
2024-02-02 - SAML SSO logins failing from 2:35pm to 2:49pm MST.
2024-01-25 - TracCloud displaying a 503 error between 1:51pm and 2:07pm MST.
2024-01-16 - 403 Forbidden error on certain records, resolved 7:02am MST.
2024-01-04 - 403 Forbidden error on Center, Group, and Staff records from 7:40am to 8:40am MST.
2023-12-13 - Users/kiosks being frequently logged out during system use. Resolved at 8:50am MST.
2023-12-12 - TracCloud error 500 from 11:15 to 11:20 MST due to PHP 8.2 issue.
2023-12-07 - TracCloud update not functioning as expected, reverted to previous version at 6:44am MST.
2023-11-27 - Palo Alto Networks incorrectly blocked TracCloud, clients using their firewall service lost access to TracCloud until PAN whitelisted traccloud.go-redrock.com.
2023-11-02 - TracCloud error 500 from 1:10pm to 1:50pm PST.
2023-11-01 - TracCloud job server limitations, email address errors, and time zone issues. Resolved at 8:40am.
2023-10-31 - TracCloud error 500 from 2:10pm to 2:15pm PST.
2023-10-31 - TracCloud error 500 from 11:37am to 11:48am PST.
2023-10-27 - TracCloud job server limitations, causing automated emails and processes to run late.
2023-10-26 - TracCloud error 500 from 4:55pm to 5:15pm PST.
2023-10-06 - TracCloud errors and slowness from 9:50 to 10:20am PST.
2023-10-05 - TracCloud 500, 502, 503, 504 errors from 3:40 to 4:15pm PST.
2023-09-26 - TracCloud bad gateway error from 2:45 to 3:00pm PST.
2023-06-13 - TracCloud instances with custom URLs received error 500 from 3:15pm to 3:35pm PST.
2023-05-09 - TracCloud 503 error from 2:00pm to 2:10pm.
2023-05-08 - TracCloud slowness/errors due to failed maintenance over weekend, resolved at 1:30pm.
2022-12-05 - TracCloud connection timing out for certain ISPs due to AWS outage from 11:34 AM to 12:51 PM PST. https://health.aws.amazon.com/health/status
2022-10-18 - TracCloud running slowly due to AWS limitations. Resolved at 2:30pm MST.
2022-10-03 - TracCloud Appointment Scheduling limitations, resolved at 9:19am MST
2022-07-28 - TracCloud Down due to AWS outage from 10:11am to 11:25am MST. https://health.aws.amazon.com/health/status
2022-07-25 - TracCloud SSO limitations, resolved at 8:00am MST.