5

We have an Azure Application Gateway V2 setup with WAF feature enabled. We ran into trouble when suddenly, the health of all our backend pools (3 in our case) started showing unhealthy and the error was being shown when we checked backend health:

Backend server certificate expired. Please upload a valid certificate.

Now, the same certificate is applied on application gateway and on the backend pool servers/VMs. We had to switch to non-SSL port for our backend servers to make the problem go away.

I am not able to pinpoint the problem as if any certificate is expired when applied to backend pool VM should give the same error once applied to the application gateway itself. Can someone help to identify the actual root cause of the problem and why it is showing error only at the backend pool level?

Very important thing is that the certificates being shown in browser when the backend pool URL is opened is showing still more than 2 months before it expires.

Thanks.

Improve this question

5 Answers 5

Reset to default
4

I can't comment as I have not enough reputation point but this might be connected to the fact that Sectigo's legacy AddTrust External CA Root certificate expires on May 30, 2020 https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020 In my case, the browser was also showing that everything is ok, but HttpClient responsible for doing the healtheck was not able to connect to the endpoint. What is interesting is the fact that I wasn't able to hit the endpoint with HttpClient running on dotnet core 2.1 but on dotnetcore 3.1 everything was fine

Improve this answer
1
  • on my nginx I had to modify the certificate chain to get it to work again with the waf
    – Nikos Tsagkas
    Jun 10, 2020 at 22:51
1

Same thing happened with my web app (roughly at the same time as reported by you) hosted on Azure which is also behind an Application Gateway V2 setup with WAF enabled. Taking a restart of the web app resolved the error.

Improve this answer
3
  • Strange, but how come app gateway, when acting as a server is working fine but when working as client (sending request to backend pool server), showing error.
    – Hammad Dar
    Jun 1, 2020 at 10:54
  • Are you still experiencing the issue? Did you take a restart of your backend servers and did that help? Microsoft is looking into the issue and has not come back with an answer. My issue had got resolved as soon as I took a restart of my web app. I am wondering if there was some problem with Azure infra/services and thats why my issue resolved on its own.
    – user13656474
    Jun 1, 2020 at 11:17
  • I had to change the SSL Certificate as an intermediate certificate got expired. I guess you faced the same issue at the same time due to commonality in our CA.
    – Hammad Dar
    Jun 26, 2020 at 11:48
1

I'm not sure what the underlying issue is with Application Gateway at the moment, but a restart of the webapp worked for one of my apps, however an other one stayed unhealthy. I could successfully connect to the webapp from the local server however, just not through the app gateway.

What worked for me in the end was removing the server from the backend pool and re-adding it. Once it was added back, the Backend health changed to 'Healthy'.

Improve this answer
0

After checking via openssl, it was identified that the intermediate certificate was revoked by the CA and it was causing the issue.

To check whether certificate is valid or not, Online Certificate Status Protocol can be used or CRI file can be downloaded and the certificate can be checked in that file. The second method has little drawback that CRI files take a little time at the CA end to be updated.

Improve this answer
0

In my case, the webapps published in the backend pool got 2 ways to get the certificate. The expired certificates were in Webapp - Certificates - Managed Certificates and in Webapp - Certificates - Bring your own certificate (pfx). After removing the expired certificates in Webapp - Certificates - Manage Certificates and updating the binding in WebApp - Custom Domains to the correct certificate, it resolved the issue.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.