I have a project that rsync’s data from an RPM repository for a local version of this repo. The issue I was faced with was the remote mirror would sometimes stop the rsync due to overloaded network or other unforeseen issues. I wanted to use rsyncs hashing algorithm to have it start right where it left off so I wrote a function to do this. If 900 seconds was hit it usually meant there was an issue with the transfer. I also want to state here that I observed the rsync stop serving issue on many mirrors so it was not just an issue with the TCP network. I use this in production and it logs each iteration or restart. The function below will also kill the current rsync so multiple copies are not running at the same time. I also only wanted to perform 5 iterations of rsync upon error or timeout so I use a while loop here. <\/p>\n
Here are the individual rsync commands in the INI configuration. <\/p><\/blockquote>\n
\r\n[rsync_cmds]\r\nrsync01 = \/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/os\/x86_64\/ 7\/x86_64\/\r\nrsync02 = \/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/updates\/x86_64\/ 7\/updates\/x86_64\/\r\nrsync03 = \/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/centosplus\/x86_64\/ 7\/centosplus\/x86_64\/\r\nrsync04 = \/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/extras\/x86_64\/ 7\/extras\/x86_64\r\n<\/pre>\n\nHere is how I call the execute_jobs_timeout() function:<\/p><\/blockquote>\n
\r\nrsync_commands = dict(config.items('rsync_cmds'))\r\ndef rsync_data():\r\n for name, cmds in sorted(rsync_commands.items()):\r\n execute_jobs_timeout(cmds)\r\n<\/pre>\nThe function:<\/p><\/blockquote>\n
\r\ndef execute_jobs_timeout(cmd):\r\n iteration = 0\r\n while iteration < 5:\r\n proc = subprocess.Popen(shlex.split(cmd),\r\n start_new_session=True)\r\n try:\r\n logger.info('Start Command: [%s]' % sanitize(cmd))\r\n stdout_data, stderr_data = proc.communicate(timeout=900)\r\n if proc.returncode != 0:\r\n logger.critical(\r\n \"%r failed, status code %s stdout %r stderr %r\" % (\r\n sanitize(cmd), proc.returncode,\r\n stdout_data, stderr_data))\r\n iteration += 1\r\n if iteration == 5:\r\n logger.critical('Execute Jobs Failed After 5 Iterations.')\r\n break\r\n continue\r\n logger.info('Success: [%s]' % sanitize(cmd))\r\n break\r\n except (subprocess.TimeoutExpired, subprocess.SubprocessError) as e:\r\n os.killpg(os.getpgid(proc.pid), signal.SIGKILL)\r\n logger.warning('[%s]' % e)\r\n logger.info('Restarting [%s]' % sanitize(cmd))\r\n iteration += 1\r\n if iteration == 5:\r\n logger.critical('Execute Jobs Failed After 5 Iterations.')\r\n break\r\n continue\r\n\r\n<\/pre>\nLog Snippet showing each command executing:<\/p><\/blockquote>\n
\r\n2019-05-25 03:15:03,872 - __main__ - INFO - Restarting [\/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/os\/x86_64\/ 7\/x86_64\/] - devdbadmin\r\n2019-05-25 03:15:03,875 - __main__ - INFO - Start Command: [\/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/os\/x86_64\/ 7\/x86_64\/] - devdbadmin\r\n2019-05-25 03:27:53,801 - __main__ - INFO - Success: [\/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/os\/x86_64\/ 7\/x86_64\/] - devdbadmin\r\n2019-05-25 03:27:53,821 - __main__ - INFO - Start Command: [\/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/updates\/x86_64\/ 7\/updates\/x86_64\/] - devdbadmin\r\n2019-05-25 03:42:53,821 - __main__ - WARNING - [Command '['\/usr\/local\/bin\/rsync', '-a', 'rsync:\/\/mirror.cogentco.com\/CentOS\/7\/updates\/x86_64\/', '7\/updates\/x86_64\/']' timed out after 899.9999316609465 seconds] - devdbadmin\r\n2019-05-25 03:42:53,822 - __main__ - INFO - Restarting [\/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/updates\/x86_64\/ 7\/updates\/x86_64\/] - devdbadmin\r\n2019-05-25 03:42:53,850 - __main__ - INFO - Start Command: [\/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/updates\/x86_64\/ 7\/updates\/x86_64\/] - devdbadmin\r\n2019-05-25 03:57:53,851 - __main__ - WARNING - [Command '['\/usr\/local\/bin\/rsync', '-a', 'rsync:\/\/mirror.cogentco.com\/CentOS\/7\/updates\/x86_64\/', '7\/updates\/x86_64\/']' timed out after 899.9999369028956 seconds] - devdbadmin\r\n2019-05-25 03:57:53,852 - __main__ - INFO - Restarting [\/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/updates\/x86_64\/ 7\/updates\/x86_64\/] - devdbadmin\r\n2019-05-25 03:57:53,854 - __main__ - INFO - Start Command: [\/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/updates\/x86_64\/ 7\/updates\/x86_64\/] - devdbadmin\r\n2019-05-25 04:01:28,522 - __main__ - INFO - Success: [\/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/updates\/x86_64\/ 7\/updates\/x86_64\/] - devdbadmin\r\n2019-05-25 04:01:28,524 - __main__ - INFO - Start Command: [\/usr\/local\/bin\/rsync -a rsync:\/\/mirror.cogentco.com\/CentOS\/7\/centosplus\/x86_64\/ 7\/centosplus\/x86_64\/] - devdbadmin\r\n2019-05-25 04:16:28,527 - __main__ - WARNING - [Command '['\/usr\/local\/bin\/rsync', '-a', 'rsync:\/\/mirror.cogentco.com\/CentOS\/7\/centosplus\/x86_64\/', '7\/centosplus\/x86_64\/']' timed out after 899.9999288369436 seconds] - devdbadmin\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"I have a project that rsync’s data from an RPM repository for a local version of this repo. The issue I was faced with was the remote mirror would sometimes stop the rsync due to overloaded network or other unforeseen issues. I wanted to use rsyncs hashing algorithm to have it start right where it […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,4],"tags":[24,25,60,59],"class_list":["post-722","post","type-post","status-publish","format-standard","hentry","category-general-code","category-python","tag-python-2","tag-rsync","tag-subprocess","tag-timeout"],"_links":{"self":[{"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/posts\/722","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jasonralph.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=722"}],"version-history":[{"count":8,"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/posts\/722\/revisions"}],"predecessor-version":[{"id":856,"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/posts\/722\/revisions\/856"}],"wp:attachment":[{"href":"https:\/\/jasonralph.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=722"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jasonralph.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=722"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jasonralph.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=722"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}