{"id":144,"date":"2017-05-20T11:01:27","date_gmt":"2017-05-20T09:01:27","guid":{"rendered":"http:\/\/www.gerjon.com\/?p=144"},"modified":"2017-06-01T11:37:07","modified_gmt":"2017-06-01T09:37:07","slug":"netscaler-12-0-losing-network-connectivity","status":"publish","type":"post","link":"https:\/\/www.gerjon.com\/?p=144","title":{"rendered":"Netscaler 12.0 VPX Losing network connectivity"},"content":{"rendered":"<p>Recently I\u00a0had a project at a customer who had an Private cloud environment based on VMWare.<br \/>\nThis environment was running in a datacenter and we had no access to the Hypervisor.<br \/>\nAfter installing a fresh Netscaler 12.0 VPX from the OVF template we encountered a strange phenomenon: the Netscaler was running fine but after a while all connections to the Netscaler where dropped and the Netscaler would not be reachable over the network<\/p>\n<p><a href=\"https:\/\/i0.wp.com\/www.gerjon.com\/wp-content\/uploads\/2017\/05\/page-not-found.jpg\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-147\" src=\"https:\/\/i0.wp.com\/www.gerjon.com\/wp-content\/uploads\/2017\/05\/page-not-found.jpg?resize=300%2C113\" alt=\"\" width=\"300\" height=\"113\" srcset=\"https:\/\/i0.wp.com\/www.gerjon.com\/wp-content\/uploads\/2017\/05\/page-not-found.jpg?resize=300%2C113&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.gerjon.com\/wp-content\/uploads\/2017\/05\/page-not-found.jpg?resize=768%2C290&amp;ssl=1 768w, https:\/\/i0.wp.com\/www.gerjon.com\/wp-content\/uploads\/2017\/05\/page-not-found.jpg?w=839&amp;ssl=1 839w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>We had to log on to the console to reboot the Netscaler. After the reboot it would work for a while but then it would stop functioning again. After some troubleshooting\u00a0I\u00a0found drops on the network interface of the Netscaler. So it seems it cannot respond of send data over the network.<\/p>\n<p><a href=\"https:\/\/i0.wp.com\/www.gerjon.com\/wp-content\/uploads\/2017\/05\/origingeel.jpg\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-145\" src=\"https:\/\/i0.wp.com\/www.gerjon.com\/wp-content\/uploads\/2017\/05\/origingeel.jpg?resize=300%2C73\" alt=\"\" width=\"300\" height=\"73\" srcset=\"https:\/\/i0.wp.com\/www.gerjon.com\/wp-content\/uploads\/2017\/05\/origingeel.jpg?resize=300%2C73&amp;ssl=1 300w, https:\/\/i0.wp.com\/www.gerjon.com\/wp-content\/uploads\/2017\/05\/origingeel.jpg?w=750&amp;ssl=1 750w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>After some digging i found a <a href=\"https:\/\/kb.vmware.com\/selfservice\/microsites\/search.do?language=en_US&amp;cmd=displayKC&amp;externalId=2092809\" target=\"_blank\" rel=\"noopener noreferrer\">article<\/a> from VMWare stating this is an Citrix Netscaler bug. But the article was for older versions of Netscaler. Citrix also has an <a href=\"https:\/\/support.citrix.com\/article\/CTX200278\" target=\"_blank\" rel=\"noopener noreferrer\">article<\/a> stating this issue. We tried different solutions (redeploy, re-configuring, etc.) but after a while I\u00a0thought what the heck and tried the workaround offered in the Citrix article, and it worked!<\/p>\n<p>Reason for losing network connectivity is caused by &#8220;tx_ring_length&#8221; mismatch, which causes TX stalls and that causes drops on your network cards.<br \/>\nThe workaround will add this line:\u00a0<strong>&#8220;hw.em.txd=512&#8221;<\/strong> to a new file called l<strong>oader.conf.local<\/strong>. This wil set the TX value to 512, please do not set it any lower as this will cause core dumps on the Netscaler.<\/p>\n<p><span style=\"text-decoration: underline;\"><strong>Here&#8217;s how i got it working<\/strong><\/span><br \/>\nLog on to the console of the Netscaler<br \/>\nrun this line and reboot<\/p>\n<p><strong>echo &#8220;hw.em.txd=512&#8221; &gt;&gt; \/flash\/boot\/loader.conf.local<\/strong><\/p>\n<p>Or you can use the Citrix solution:<br \/>\nSSH and log on to Citrix NetScaler VPX appliance as nsroot.<\/p>\n<p>Type <strong>shell<\/strong>.<\/p>\n<p>Change directory (cd) to \/flash\/boot.<\/p>\n<p>Create file \/flash\/boot\/loader.conf.local (if not present) with same permissions as \/flash\/boot\/loader.conf. Add the following line and reboot:<br \/>\n<strong>hw.em.txd=512<\/strong><br \/>\nNote: To create the file, use command <strong>touch loader.conf.local.<\/strong><\/p>\n<p>vi Commands<br \/>\nThe following are the vi commands to edit the document:<\/p>\n<p>From NetScaler shell type:<br \/>\n<strong>vi<\/strong><\/p>\n<p>Move the cursor to the last character of text in the file, type &#8220;<strong>a<\/strong>&#8221; and click Enter.<\/p>\n<p>Type the line:<br \/>\n<strong>hw.em.txd=512<\/strong><\/p>\n<p>Press the <strong>ESC<\/strong> key and then &#8220;:&#8221; key. The cursor will move to the bottom of the page, then type <strong>wq!<\/strong>.<\/p>\n<p>So although the bug seemed to be fixed in the newer Netscaler builds it somehow seems to resurfaced in a fresh install of version 12.0.<\/p>\n<p><strong>UPDATE:<\/strong><\/p>\n<p>After a while under load the Netscaler stopped working again. I opened a case at Citrix and it seems to be related to the combination of E1000 NIC interfaces selected on the ESX in and DSV (VMWare Distirbuted Virtual Switch) enabled on the underlying Hypervisor. After changing the NIC to a VMXNet 3 the issue dissapeared.<\/p>\n<p>Beware that in my case the license and config where lost so please backup your config (and export it from the Netscaler to a server) and have your MyCitrix credentials at hand to regenerate te license.<\/p>\n<p>Hope this helps\u00a0\ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recently I\u00a0had a project at a customer who had an Private cloud environment based on VMWare. This environment was running in a datacenter and we had no access to the Hypervisor. After installing a fresh &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[7,24],"tags":[],"class_list":["post-144","post","type-post","status-publish","format-standard","hentry","category-citrix","category-netscaler"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p59CpB-2k","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.gerjon.com\/index.php?rest_route=\/wp\/v2\/posts\/144","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.gerjon.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gerjon.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gerjon.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gerjon.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=144"}],"version-history":[{"count":0,"href":"https:\/\/www.gerjon.com\/index.php?rest_route=\/wp\/v2\/posts\/144\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.gerjon.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=144"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gerjon.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=144"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gerjon.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=144"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}