Issues with the Automatic WebLink Download for Hosted Systems
Incident Report for EZ-CARE2
Postmortem
What Happened:

Upon our deployment of the TLS 1.2 upgrade (which was actually disabling TLS 1.0 and 1.1 communication across our environments) we discovered a problem with our EZ-CARE2 hosted clients downloading transactions from WebLink. Our team traced the issue to a mismatch between our testing environment and our production hosting environment for EZ-CARE2 hosting with respect to how these protocols are handled and how they communicate to WebLink where this particular communication actually terminates. Stand-alone and self-installed EZ-CARE2 clients were not affected as this mismatch was solely in our EZ-CARE2 hosted environment.

What we're doing about it:

We have re-enabled TLS 1.0 and 1.1 on WebLink to allow downloads to resume while we realign our production and test environments and confirm the issue has been resolved. Once we confirm that the environments are realigned, we will resolve the download issue in the test, verify the fix, and then schedule a time to apply any necessary fixes to the EZ-CARE2 production hosting environment. We will notify clients via statuspage when we re-enable the change and will do our best to minimize any downtime when we do.

We thank you for your support and flexibility with this process.

Posted over 1 year ago. Feb 14, 2018 - 10:50 EST

Resolved
This issue has been resolved through the rollback of the TLS security update.
Posted over 1 year ago. Feb 14, 2018 - 09:13 EST
Monitoring
In order to restore the WebLink automatic download capability for all hosted systems, our team has rolled back the security update that was performed this morning. Your automatic downloading will function properly now. We still plan to implement the TLS 1.2 security protocol, and we will notify you when we plan to attempt the security update again.
Posted over 1 year ago. Feb 13, 2018 - 17:09 EST
Investigating
Our team is still working on a solution to the automatic download issue for hosted systems. We implemented a potential solution earlier this afternoon, however, we continue to see disruption in the automatic WebLink downloads for hosted systems. Thank you for your patience as we resolve this issue.
Posted over 1 year ago. Feb 13, 2018 - 14:21 EST
Identified
We believe we have identified the issue causing a disruption with the automatic WebLink download. Our team is currently making changes and will be gradually rebooting the servers to implement the solution. Thank you for your patience, and we'll continue to keep you updated here.
Posted over 1 year ago. Feb 13, 2018 - 12:42 EST
Investigating
Our team is currently investigating an issue affecting our hosted clients that is causing a disruption with the automatic downloads from WebLink into the EZ system. We believe the issue is related to the TLS security update that was performed today. We apologize for any inconvenience this might cause you.

Our team is working hard to resolve the issue as quickly as possible. In the meantime, you can download WebLink data by logging into WebLink and manually downloading the data. Thank you for your patience, and we'll keep you updated here.
Posted over 1 year ago. Feb 13, 2018 - 12:17 EST