Backup failing due to failing to read the license ID

jim · August 27, 2019, 12:50pm

My backups which have been running fine out of cron for several months now have suddenly started failing, indicating it was failing to read the license ID. I logged into the server and hand-ran the backup and got the following output:

[root@landroval:/var/spool/cron/crontabs] /vmfs/volumes/5d0cc0a3-b22da97c-5c0d-   b083fed617c2/verticalbackup/vertical backup \* --email
2019-08-26 16:58:24.905564 INFO PROGRAM_VERSION Vertical Backup 1.3.2
Failed to login: HTTP error response: Service Unavailable
2019-08-26 17:00:11.878195 ERROR LICENSE_FAILURE Failed to read the license ID
2019-08-26 17:00:15.020410 INFO EMAIL_SEND A notification email has been sent to jim@XXX.com

I assumed it was an issue with the license, so did a verification:

[root@landroval:/var/spool/cron/crontabs] /vmfs/volumes/5d0cc0a3-b22da97c-5c0d-b083fed617c2/verticalbackup/vertical license show
Vertical Backup 1.3.2
Failed to login: HTTP error response: Service Unavailable
Failed to read the license ID

So this looks like an availability or reachability issue towards the license server.

Are there any known issues? If not, is it possible to get details (address) for the server so I can look into potential reachability issues? (Yes, I could tcpdump the network connection or get a system trace of the vertical process to get the address, but I’m being lazy and asking )

Thanks!

Jim

gchen · August 29, 2019, 5:08pm

This looks like something wrong with vim-cmd. Can you run this command manually, such as vim-cmd vmsvc/getallvms?

jim · September 28, 2019, 8:38pm

Sorry for the delay getting back to this! Here’s the output from vim-cmd … looks to be fine.

[root@landroval:/var/spool/cron/crontabs] vim-cmd vmsvc/getallvms
Vmid              Name                                              File                                     Guest OS       Version                                                                                                                                                                                                                                                                                                                                                                                                                                 Annotation
1      colovpn.daedelus.com          [Data] colovpn.daedelus.com/colovpn.daedelus.com.vmx                 ubuntu64Guest     vmx-07    OpenVPN Access Server 2.0.1
Ubuntu Server 13.10
Release Date: October 22, 2013
http://openvpn.net/index.php/access-server/overview.html
10     24x7-collector.daedelus.com   [Data] 24x7-collector.daedelus.com/24x7-collector.daedelus.com.vmx   centos7_64Guest   vmx-13
11     opennms-minion.daedelus.com   [Data] opennms-minion.daedelus.com/opennms-minion.daedelus.com.vmx   centos7_64Guest   vmx-13
2      efa.daedelus.com              [Data] efa.daedelus.com/efa.daedelus.com.vmx                         centos64Guest     vmx-07    Email Filter Appliance Version 3.0.0.5

http://www.efa-project.org

E.F.A stands for Email Filter Appliance. E.F.A is born out of a need for a cost-effective email virus & spam scanning solution after the ESVA project died.

We try to create a complete package using existing open-source anti-spam projects and combine them to a single easy to use (virtual) appliance.

For more information go to http://www.efa-project.org

E.F.A V3 is a rebuild of the previous ESVA; the same components are used whenever possible but are all updated to the latest version.

All scripts and programs created for E.F.A. are licensed by GNU GPL-3.0. All used open-source project tools that are combined to form E.F.A. are licensed by their respective owners.

For more information, please visit:

https://raw.githubusercontent.com/E-F-A/v3/3.0.0.5/LICENSE
3      uillean.fuaim.com             [Data] uillean.fuaim.com/uillean.fuaim.com_1.vmx                     centos64Guest     vmx-07
4      imladris.daedelus.com         [Data] imladris.daedelus.com/imladris.daedelus.com.vmx               rhel5Guest        vmx-07
5      clairseach.fuaim.com          [Data] clairseach.fuaim.com/clairseach.fuaim.com.vmx                 rhel5Guest        vmx-07
6      minecraft.ameliagail.com      [Data] minecraft.ameliagail.com/minecraft.ameliagail.com.vmx         ubuntu64Guest     vmx-13
7      groundwork.daedelus.com       [Data] groundwork.daedelus.com/groundwork.daedelus.com.vmx           ubuntu64Guest     vmx-13
8      lm-collector.daedelus.com     [Data] lm-collector.daedelus.com/lm-collector.daedelus.com.vmx       centos7_64Guest   vmx-13
9      nagios.daedelus.com           [Data] nagios.daedelus.com/nagios.daedelus.com.vmx                   centos7_64Guest   vmx-13

However, there does seem to be another smoking gun. I updated to the latest version of vertical (1.3.3) and it was now outputting this:

[root@landroval:/var/spool/cron/crontabs] /vmfs/volumes/5d0cc0a3-b22da97c-5c0d-b083fed617c2/verticalbackup/vertical backup \* --email
2019-09-28 20:09:38.319841 INFO PROGRAM_VERSION Vertical Backup 1.3.3
2019-09-28 20:10:54.628254 WARNING LICENSE_SERVER Failed to connect to the license server: HTTPSConnectionPool(host='vertical-portal.appspot.com', port=443): Max retries exceeded with url: /request_license (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0xb87d48e150>: Failed to establish a new connection: [Errno 113] No route to host',))
2019-09-28 20:10:54.628652 ERROR BACKUP_UPDATE Failed to update the license

Pinging the v4 address of vertical-portal.appspot.com worked fine, and the “no route to host” made me think my v6 config was the issue. And indeed it was! (at least in part). I was missing a default v6 route. I fixed that and was now able to ping vertical-portal over both v4 and v6.

[root@landroval:/var/spool/cron/crontabs] ping vertical-portal.appspot.com
PING vertical-portal.appspot.com (216.58.211.116): 56 data bytes
64 bytes from 216.58.211.116: icmp_seq=0 ttl=54 time=137.602 ms
64 bytes from 216.58.211.116: icmp_seq=1 ttl=54 time=137.307 ms

--- vertical-portal.appspot.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 137.307/137.454/137.602 ms
[root@landroval:/var/spool/cron/crontabs] ping6 vertical-portal.appspot.com
PING vertical-portal.appspot.com (2a00:1450:400e:809::2014): 56 data bytes
64 bytes from 2a00:1450:400e:809::2014: icmp_seq=0 time=137.256 ms
64 bytes from 2a00:1450:400e:809::2014: icmp_seq=1 time=137.481 ms

--- vertical-portal.appspot.com ping statistics ---
3 packets transmitted, 2 packets received, 33% packet loss
round-trip min/avg/max = 137.256/137.369/137.481 ms

I thought, “Cool! We’re done!”. Alas, now I just get connection timeouts:

[root@landroval:/var/spool/cron/crontabs] /vmfs/volumes/5d0cc0a3-b22da97c-5c0d-b083fed617c2/verticalbackup/vertical backup \* --email
2019-09-28 20:26:27.523001 INFO PROGRAM_VERSION Vertical Backup 1.3.3
2019-09-28 20:28:58.798324 WARNING LICENSE_SERVER Failed to connect to the license server: HTTPSConnectionPool(host='vertical-portal.appspot.com', port=443): Max retries exceeded with url: /request_license (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x2a47630150>: Failed to establish a new connection: [Errno 110] Connection timed out',))
2019-09-28 20:28:58.798718 ERROR BACKUP_UPDATE Failed to update the license

Any known issues on your side with retrieving the license over v6?

Thanks!

gchen · September 29, 2019, 12:20am

This may be a firewall issue now. Remove the file /etc/vmware/firewall/verticalbackuphttps.xml and try the backup again.