tag:blogger.com,1999:blog-18181005454907851152024-03-19T01:15:54.838-07:00Cyber ExplorerMy exploration in cyberspace - software, hardware and anything in betweenCyber Explorerhttp://www.blogger.com/profile/15872243741791530580noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-1818100545490785115.post-22962679788290464422014-01-21T02:04:00.002-08:002014-01-21T17:05:08.550-08:00Sniffing and decoding NRF24L01+ and Bluetooth LE packets for under $30<h4>
</h4>
In this long post I am going to describe my journey to sniff and decode popular digital wireless protocols off the air for very cheap. So cheap practicality anyone can obtain the equipment quickly.<br />
<br />
<b>I was able to decode NRF24L01+ and Bluetooth Low Energy protocols using RTL-SDR. </b><br />
As far as I can see, this is the first time the NRF24L01+ is being decoded, especially considering the low entry price for the hardware. Given the extreme popularity of this transceiver, we are likely to see a wave of hackers attacking the security of many wireless gadgets, and they are likely to succeed as security is usually the last priority for hardware designers of such cheap gadgets.<br />
<br />
A lot of work have been done to decode bluetooth using dedicated hardware and I am sure this software can be adapted to output the right format as input to the existing Bluetooth decoders such as Wireshark.<br />
As far as I can see, this is also the first time BTLE can be decoded using a very cheap generic device.<br />
<br />
The main software repository for this project is at <a href="https://github.com/omriiluz/NRF24-BTLE-Decoder">https://github.com/omriiluz/NRF24-BTLE-Decoder</a><br />
<br />
<h4>
Developing a wireless mesh network challenges</h4>
Recently I've been working on a project to create a super cheap (<$5) sensor node that can be flexible and power efficient so I can just leave sensors everywhere and need absolutely no maintenance.<br />
I decided to use the extremely popular <a href="http://www.nordicsemi.com/eng/Products/2.4GHz-RF/nRF24L01P" target="_blank">NRF24L01+</a> transceiver from Nordics Semiconductor due to a balance of performance, power and price - and you'll be surprised how many hardware designers have taken the exact same decision once you start sniffing the air for packets. In my home alone I can see 15 addresses - wireless keyboards and mouse, remote controls, toys and appliances all use this tiny transceiver for wireless communication.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgjIjPeyxhhFsJXrf32xoeDT_81LPomVpQc74Yiv1Nq90-iIUPYMT_3gAYmwOwa8FAu3dIOzcJgAJQPzoszqxXVzyjSr5FVHqIYtNUQKEvvhKoZHymGhTrFeoIZezCI5Z5PvYxBW7zmRI/s1600/sensor.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgjIjPeyxhhFsJXrf32xoeDT_81LPomVpQc74Yiv1Nq90-iIUPYMT_3gAYmwOwa8FAu3dIOzcJgAJQPzoszqxXVzyjSr5FVHqIYtNUQKEvvhKoZHymGhTrFeoIZezCI5Z5PvYxBW7zmRI/s1600/sensor.jpg" height="187" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small; text-align: start;">The sensor node with NRF24L01+ </span></td></tr>
</tbody></table>
While working on the mesh network code, my progress slowed to nearly a halt. The code is extremely complex and depends on external conditions like signal strength, noise, etc. But worst of all? I was completely blind on what really happens once packets leave the safety of my micro controller using SPI to the transceiver. For normal (i.e. non-wireless) projects I'm used to being able to connect my scope or logic analyzer and "see" what happens on the wire. This makes debugging a breeze. Unfortunately this is not the case for wireless projects and I had absolutely no idea what happens between the transceivers. To make things worse, these transceivers work in the ISM band of 2.4Ghz. this is fast. much faster than any equipment I have available.<br />
<br />
<br />
<br />
<h4>
Enter Software Defined Radio (SDR) </h4>
<div>
<a href="http://ecx.images-amazon.com/images/I/51A9W6Ac0QL._SL1500_.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://ecx.images-amazon.com/images/I/51A9W6Ac0QL._SL1500_.jpg" height="200" width="175" /></a>I assume you all know about the magic of SDR and specifically the cheap RTL-SDR. If not, take a break and head to <a href="http://sdr.osmocom.org/trac/wiki/rtl-sdr">http://sdr.osmocom.org/trac/wiki/rtl-sdr</a> to read about it. For $13-18 (<a href="http://www.amazon.com/gp/product/B00EM7C5PU/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B00EM7C5PU&linkCode=as2&tag=cybeblog-20" target="_blank">Amazon 1</a>, <a href="http://www.amazon.com/gp/product/B00EM7HCKS/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B00EM7HCKS&linkCode=as2&tag=cybeblog-20" target="_blank">Amazon 2</a>) you open a world of possibilities that stretches far beyond analog radio into the 2.4Ghz digital space, as you'll read on this post.</div>
<div>
<br /></div>
<div>
Back to the problem of debugging - having experience with rtl-sdr, I immediately started thinking how can I use it to sniff packets off the air. This is impossible using any version of the rtl-sdr as the highest you can buy reach 2.2Ghz. just shy of the 2.4Ghz we need.</div>
<div>
<br /></div>
<div>
I started looking for a way to convert the signal down to a frequency usable by the rtl-sdr. Building one was a possibility, but I had no idea how and all the commercial/DIY products costs hundreds of dollars.</div>
<div>
<br /></div>
<h4>
China to the rescue</h4>
<div>
Another option was to try and find an existing, mass produced and cheap product with my required specification. A quick search on Aliexpress.com found exactly that - <a href="http://en.wikipedia.org/wiki/Multichannel_Multipoint_Distribution_Service" target="_blank">MMDS </a>LNB.</div>
<div>
MMDS is a digital broadcast system used in some countries for digital TV, and the LNB is part of the antenna. The MMDS LNB can be found for a variety of frequencies and LO frequencies.</div>
<div>
<br /></div>
<div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSdlnIXZpe9can8ItUjMJVZGZoDIuwl0perR7oqoZtueLs2hGlcyoEhtXsvPfpDXJNFFL1wNXMDwDhJchSEPM9wqeCMIFyWGqmpSCDpj_d7qrcz7ppFdu_F64fK_Jnnm2kaQP6wNJhXeA/s1600/IMG_20140121_020724220_HDR.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSdlnIXZpe9can8ItUjMJVZGZoDIuwl0perR7oqoZtueLs2hGlcyoEhtXsvPfpDXJNFFL1wNXMDwDhJchSEPM9wqeCMIFyWGqmpSCDpj_d7qrcz7ppFdu_F64fK_Jnnm2kaQP6wNJhXeA/s1600/IMG_20140121_020724220_HDR.jpg" height="180" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Complete setup</td></tr>
</tbody></table>
The base frequency defines the filters on the device and the LO frequency defines by how much will it reduce the input frequency. </div>
<div>
<br /></div>
<div>
Based on the specification, it would do EXACTLY what we need - take 2.2-2.4Ghz signal and down convert it to around 400Mhz. Then we can use the rtl-sdr and some code to decode packets off the air.</div>
<div>
<br /></div>
<div>
As it was very cheap ($12+shipping at <a href="http://www.aliexpress.com/item/2012-best-selling-L-O1998MHZ-MMDS-down-converter/670265064.html" target="_blank">Aliexpress</a>) I took the chances and ordered one. About 10 days later it popped in my mail. I quickly hooked everything up and to my extreme surprise, after minutes -</div>
<div>
<br />
<br /></div>
<h4>
Success!</h4>
I used <a href="http://sdrsharp.com/" target="_blank">SDR#</a> with the new radio setup to see if I can find signals where I expect them to. The easiest one to find was my Logitech wireless mouse (which uses nrf of course). Tuning to 2,405Mhz (or 407Mhz after down conversion using LO of 1998Mhz) clearly show a strong signal when I move my mouse.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7JVyMAhJctOE3XMR3JWHeUZZHfoyuPqDqyLQsk4LRu_4DS_GHrvNsvyO0nxTHm_vQzc1e7vqAIAeWda-eCTARkSLsCU4i77VDFCU0KVH2Xja8C7vx7kUE7kCqLABai4M_insR41MQl0M/s1600/sdrs.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7JVyMAhJctOE3XMR3JWHeUZZHfoyuPqDqyLQsk4LRu_4DS_GHrvNsvyO0nxTHm_vQzc1e7vqAIAeWda-eCTARkSLsCU4i77VDFCU0KVH2Xja8C7vx7kUE7kCqLABai4M_insR41MQl0M/s1600/sdrs.jpg" height="498" width="640" /></a></div>
<br />
<br />
Developing the software to decode the packets was a bit of a headache, but once it started shaping up it was very easy to use it to capture and decode the packets.<br />
<br />
<h4>
So what do you need to make it work?</h4>
<ul>
<li>RTL-SDR dongle - ~$15. Easiest to buy on Amazon - (<a href="http://www.amazon.com/gp/product/B00EM7C5PU/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B00EM7C5PU&linkCode=as2&tag=cybeblog-20" target="_blank">Amazon 1</a>, <a href="http://www.amazon.com/gp/product/B00EM7HCKS/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B00EM7HCKS&linkCode=as2&tag=cybeblog-20" target="_blank">Amazon 2</a>), but you can find it everywhere. I have both an E4000 and R820T dongles and they both work perfectly.</li>
<li>MMDS down converter - $12+shipping at <a href="http://www.aliexpress.com/item/2012-best-selling-L-O1998MHZ-MMDS-down-converter/670265064.html" target="_blank">Aliexpress</a>. Buy one for 2.2-2.4Ghz with L.O. of 1998Mhz. If you buy from the link here, the seller will ask you for these details after you purchase.</li>
<li>(optional) Cables - Different rtl-sdr sticks have different input plug, you need to find a way to connect it to your down converter. This is really optional as I simply hacked some wire and it worked fine.</li>
<li>(optional) Power Injector - the down converter is an active component and requires 14-24V. I started by simply connecting my power supply to the wire and it worked fine. later I purchased a commercial power injector for less than $10. you can find one at <a href="http://www.amazon.com/gp/product/B005AME7Y8/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=B005AME7Y8&linkCode=as2&tag=cybeblog-20" target="_blank">Amazon </a>or from the same <a href="http://www.aliexpress.com/item/high-quality-Microwave-MMDS-down-converter-power-supply-with-18V-0-2A/670968406.html" target="_blank">Aliexpress seller</a>.</li>
</ul>
<br />
<h4>
Back to the comfort of software</h4>
From now one, all we need in order to get the packets is some clever software and the comfort of our computer (whether it's windows, linux, mac or RPi).<br />
<br />
The NRF24L01+ (nrf from now on) uses <a href="http://en.wikipedia.org/wiki/Gaussian_frequency-shift_keying" target="_blank">GFSK </a>modulation for the data. FSK (and it's derivatives like GFSK) is the digital cousin of FM. the modulator takes a bit stream, and emits one frequency to represent "1" and another frequency to represent "0".<br />
Luckily for us, there is already a <a href="http://sdr.osmocom.org/trac/wiki/rtl-sdr" target="_blank">great library</a> that does the basic rtl-sdr work and includes a software FM demodulator, rtl_fm.<br />
<br />
Using the rtl_fm code on the incoming stream, I exported to excel a raw demodulated feed and filtered to find interesting results -<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNW3Z4MV7h3xGtMznnPr2wpot5npztPYSamSRCaZ8Omu_WDtYM4SgJjV6QsX4VlV8YDuJFWYvqm4mHhRAnbOpk5f9367POAHsTIZsWu8Tn4GwzagSZfpMAvgx4fUIDmWwRglSHygACMX8/s1600/excel.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNW3Z4MV7h3xGtMznnPr2wpot5npztPYSamSRCaZ8Omu_WDtYM4SgJjV6QsX4VlV8YDuJFWYvqm4mHhRAnbOpk5f9367POAHsTIZsWu8Tn4GwzagSZfpMAvgx4fUIDmWwRglSHygACMX8/s1600/excel.jpg" height="219" width="640" /></a></div>
<br />
<br />
This is without a doubt an nrf packet. You can see:<br />
<br />
<ul>
<li>Noise before (<80) and after (>395) the packet</li>
<li>Radio turn on time (85 to 125)</li>
<li>Preamble sequence of alternating bits (01010101 here) for the demodulator to detect a packet start and sync clocks (125 to 160)</li>
<li>Packet data (160 to 395)</li>
</ul>
<br />
In my code, I detect the preamble and calculate a Threshold number - anything above that number is considered a binary "1" and anything below is a "0". This provides a bit stream which represent the packet.<br />
My code takes this bit stream, and manipulate it to recover the packet.<br />
The last step is to take the packet, apply CRC and compare to the CRC in the last two bytes to verify that this is a valid packet. if the CRC match, we print a decoded packet.<br />
<br />
For a detailed description of how I turn this bitstream into a decoded packet, I suggest you open my code over at <a href="https://github.com/omriiluz/NRF24-BTLE-Decoder" target="_blank">https://github.com/omriiluz/NRF24-BTLE-Decoder</a>, it is documented and should be relatively simple to understand.<br />
<br />
<h4>
Getting rtl_fm to output the right signal</h4>
Once you install the librtlsdr and have it working with your dongle, the basic command line for rtl_fm to product the bitstream we need as input is:<br />
<span style="font-family: Courier New, Courier, monospace;"><b>rtl_fm -f 402m -s 2000k -g 0 -p 239</b></span><br />
<br />
The parameters are:<br />
<br />
<ul>
<li>-f - frequency. remember to take the nrf channel frequency and reduce your down converter LO frequency. in the case here it's 2400-1998=402.</li>
<li>-s 2000k - mandatory. my code expects a 2M samples per seconds stream</li>
<li>-g 0 - to avoid noise, it's important to disable the auto gain control and reduce the gain as much as possible. I use 0 when everything is on my table and 10-15 when it's in my house.</li>
<li>-p defines the rtl-sdr permanent frequency drift. As a cheap device, the rtl-sdr is not calibrated. it's easy to calibrate it out of cellular signals using <a href="https://github.com/steve-m/kalibrate-rtl" target="_blank">kalibrate-sdr</a></li>
</ul>
<br />
<h4>
Sniffing NRF24L01+ packets</h4>
<div>
Once rtl_fm works, simply pipe the output into my software to see packets decoded -</div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>rtl_fm -f 402m -s 2000k -g 0 -p 239 | nrf24-btle-decoder -d 1</b></span></div>
<div>
<br /></div>
<div>
2 simple parameters:</div>
<div>
<ul>
<li>-t nrf | btle - should we decode nrf or bluetooth LE packets (more on this later)</li>
<li>-d 1 | 2 | 8 - select packet speed - the nrf can do 2mbps, 1mbps or 256kbps. you need to pick the right one.</li>
</ul>
<div>
Having my sensor node send one byte of data (an increment counter) with an acknowledgment from another node, the output would look like:</div>
</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmcnh5_7FmwytooWxwpeW7v-9ls_7x5UMrY0tkjcFs_nrTV7bXeZ2hI-Nx8J-2e-BL1dH6ZLxKPC1LhRC3Tn9YvorxrJniiNAvNE_hr4T8DJ4-jn_LDkBZPESHr0LykOx8nc2XK_kUQ9A/s1600/software.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmcnh5_7FmwytooWxwpeW7v-9ls_7x5UMrY0tkjcFs_nrTV7bXeZ2hI-Nx8J-2e-BL1dH6ZLxKPC1LhRC3Tn9YvorxrJniiNAvNE_hr4T8DJ4-jn_LDkBZPESHr0LykOx8nc2XK_kUQ9A/s1600/software.jpg" height="241" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Sniffing 2Mbps NRF24L01+ traffic on channel 0 (2,400Mhz)</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
And now I'm not blind anymore when debugging!<br />
<br />
<h4>
Taking it further</h4>
</div>
<div>
As one <a href="http://dmitry.gr/index.php?r=05.Projects&proj=11.%20Bluetooth%20LE%20fakery" target="_blank">smart blogger explained</a>, the physical radio of the NRF24L01+ and Bluetooth Low Energy (btle from now one) are quite similar. This allowed me to quickly adapt my code to sniff btle packets as well.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPNunltyHs9fdA0haND1eS9ryzCgg5kA1F_rnsiCRKKMLtjS7Lqkwm12VmI5OjnqHryutMrUHv1ymQk-Nhl_W6lrjr7jXoAto1sgPzTSOtnlgkNT9iQ5hyphenhyphenuMyRVtMQnd9OHLEZp1DokcY/s1600/software2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPNunltyHs9fdA0haND1eS9ryzCgg5kA1F_rnsiCRKKMLtjS7Lqkwm12VmI5OjnqHryutMrUHv1ymQk-Nhl_W6lrjr7jXoAto1sgPzTSOtnlgkNT9iQ5hyphenhyphenuMyRVtMQnd9OHLEZp1DokcY/s1600/software2.jpg" height="158" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Sniffing Bluetooth Low Energy advertisement channel 38 (2,426Mhz)</td></tr>
</tbody></table>
The code for sniffing btle is complete for the advertisement channels, but not for the data channels, it would be my next step to add it. The main issue is frequency hopping as required by btle, which I'm not sure our lowly rtl-sdr can do fast enough.<br />
<br />
<br />
<br />
<br />
<br />Cyber Explorerhttp://www.blogger.com/profile/15872243741791530580noreply@blogger.com236tag:blogger.com,1999:blog-1818100545490785115.post-90254638499184884382013-03-28T03:42:00.001-07:002013-03-29T00:51:05.543-07:00Improving VM to VM network throughput on an ESXi platform<span style="color: #20124d;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Recently I virtualized most of the servers I had at home into an ESXi 5.1 platform. This post would follow my journey to achieve better network performance between the VMs.</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">I am quite happy with the setup as it allowed me to eliminate 5-6 physical boxes in favor of one (very strong) machine. I was also able to achieve some performance improvements but not to the degree I hoped to see.</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">I have a variety of machines running in a virtualized form:</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">1. Windows 8 as my primary desktop, passing dedicated GPU and USB card.from the host to the VM using VMDirectPath</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">2. Multiple Linux servers</span><br />
<span style="color: #20124d;"><span style="font-family: Arial, Helvetica, sans-serif;">3. Solaris 11.1 as NAS, running the great napp-it software (<a href="http://www.napp-it.org/)" target="_blank">http://www.napp-it.org/)</a></span><span style="font-family: Arial, Helvetica, sans-serif;"> </span></span><br />
<span style="color: #20124d;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">All the machines have the latest VMware Tools installed and running paravirtualized drivers where possible.</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">VM to VM network performance has been great between the Windows/Linux boxes once I enabled Jumbo Frames. </span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Throughout this post I'll use iperf to measure network performance. It's a great and easy to use tool and you can find precompiled version for almost any operating system. <a href="http://iperf.fr/" target="_blank">http://iperf.fr/</a></span><br />
<span style="color: #20124d;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Let's start with an example of network throughput performance from the Windows 8 Machine to Linux:</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibUZvY-nKqntsFoQbiRD0VuA7YnEfF6L40jtuZNQHm-ETbOnG55YJMgFDo9SMbHPKV40VIOl_OYsPhziapA3o7bQUNyhTYolodHNet2OpmI2nCy8W-t_I0r1dhGvJHR2IwhnyftajKgwo/s1600/windows.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibUZvY-nKqntsFoQbiRD0VuA7YnEfF6L40jtuZNQHm-ETbOnG55YJMgFDo9SMbHPKV40VIOl_OYsPhziapA3o7bQUNyhTYolodHNet2OpmI2nCy8W-t_I0r1dhGvJHR2IwhnyftajKgwo/s1600/windows.png" /></a></div>
<code></code><br />
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></div>
<div style="text-align: left;">
<code><span style="color: #20124d;"><span style="font-family: Arial, Helvetica, sans-serif;">11.3 Gbps, not bad. CPU utilization was around 25% on the windows box throughout the test.</span></span></code></div>
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Network performance between the Solaris VM and any other machine on the host was relatively bad. </span></code><br />
<code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">I started by using the E1000G virtual adapter, as recommended by VMware for Solaris 11 (<a href="http://kb.vmware.com/kb/2032669" target="_blank">http://kb.vmware.com/kb/2032669</a>). We'll use one of my Linux VMs (at 192.168.1.202) as a server for these tests. using iperf to test:</span></code><br />
<code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code> </code><br />
<div class="separator" style="clear: both; font-family: 'Courier New', Courier, monospace; font-size: small; text-align: center;">
<code><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1hel4BlQnYY_7sIWmU0N_izpgI5RsBv1h30ipvQUo8a6ERbIbLbJmhoBRU66SLagoLDj6zx2oe5EpSM_qvuHWn4FtouyoVnnL0Gxo_pq91UOLDF5FL2cC-Tw7997_yiNpR2oBNtXa4oM/s1600/default+e1000.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1hel4BlQnYY_7sIWmU0N_izpgI5RsBv1h30ipvQUo8a6ERbIbLbJmhoBRU66SLagoLDj6zx2oe5EpSM_qvuHWn4FtouyoVnnL0Gxo_pq91UOLDF5FL2cC-Tw7997_yiNpR2oBNtXa4oM/s1600/default+e1000.png" /></a></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">1.36 Gbps. Not bad between physical servers, but unacceptable between VMs on the same host. also notice the very high CPU utilization during the test - around 80% system time.</span></code><br />
<code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code> <code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">My immediate instinct was to enable jumbo frames. Although the adapter driver is supposed to support jumbo frames, I was unable to enable it no matter how hard I fought it. </span></code><br />
</code><br />
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br />
</span></code></code></div>
<code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">root@solaris-lab:/kernel/drv# dladm set-linkprop -p mtu=9000 net0</span></code><br />
<code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> dladm: warning: cannot set link property 'mtu' on 'net0': link busy</span></code><br />
<br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">I gave up on getting better performance from the E1000G adapter and switched to VMXNET3. I immediately saw improvement:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span> <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRtrNTCPBbSnxHbOM008npJZziqoVilyZ5M-VbzLZvFsOmyldSThmRnL8BYhni8NZUh_4xwSoyq3iuRQK0LqbKwOnPzJgtQTn22nugVQ1-AowGNkCuEPp12mZYpH4-urmKpisQmx3Elec/s1600/default+vmxnet.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRtrNTCPBbSnxHbOM008npJZziqoVilyZ5M-VbzLZvFsOmyldSThmRnL8BYhni8NZUh_4xwSoyq3iuRQK0LqbKwOnPzJgtQTn22nugVQ1-AowGNkCuEPp12mZYpH4-urmKpisQmx3Elec/s1600/default+vmxnet.png" /></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">2.31 Gbps. but more importantly, the cpu utilization was much lower.</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Now let's try to enable jumbo frames for the vmxnet3 adapter - followed the steps in <a href="http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2012445" target="_blank">http://kb.vmware.com/kb/2012445</a> and <a href="http://kb.vmware.com/kb/2032669" target="_blank">http://kb.vmware.com/kb/2032669</a> without success. The commands work, but jumbo frames were not really enabled. we can test with 9000 byte ping -</span><br />
<br />
<div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">root@solaris-lab:~# ping -s 192.168.1.202 9000 4</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">PING 192.168.1.202: 9000 data bytes</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">----192.168.1.202 PING Statistics----</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">4 packets transmitted, 0 packets received, 100% packet loss</span></code></code></div>
</div>
<div>
<code><code><br />
</code></code><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">As my next step I was planning on running some dtrace commands, and accidentally noticed that the drivers I have installed are the Solaris 10 version and not the Solaris 11 version.</span><br />
<span style="font-size: x-small;"><code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="font-family: inherit;"></span></code></code></span><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">root@solaris-lab:~/vmware-tools-distrib# find /kernel/drv/ -ls |grep vmxnet3</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">78669 2 -rw-r--r-- 1 root root 1071 Mar 27 01:42 /kernel/drv/vmxnet3s.conf</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">78671 34 -rw-r--r-- 1 root root <span style="background-color: yellow;">34104 </span>Mar 27 01:42 /kernel/drv/amd64/vmxnet3s</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">78670 25 -rw-r--r-- 1 root root <span style="background-color: cyan;">24440 </span>Mar 27 01:42 /kernel/drv/vmxnet3s</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">root@solaris-lab:~/vmware-tools-distrib# find . -ls |grep vmxnet3</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 231 25 -rw-r--r-- 1 root root 24528 Nov 17 07:55 ./lib/modules/binary/2009.06/vmxnet3s</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 234 2 -rw-r--r-- 1 root root 1071 Nov 17 07:55 ./lib/modules/binary/2009.06/vmxnet3s.conf</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 250 2 -rw-r--r-- 1 root root 1071 Nov 17 07:55 ./lib/modules/binary/10/vmxnet3s.conf</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 244 25 -rw-r--r-- 1 root root <span style="background-color: cyan;">24440 </span>Nov 17 07:55 ./lib/modules/binary/10/vmxnet3s</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 262 34 -rw-r--r-- 1 root root <span style="background-color: yellow;">34104 </span>Nov 17 07:55 ./lib/modules/binary/10_64/vmxnet3s</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 237 35 -rw-r--r-- 1 root root 35240 Nov 17 07:55 ./lib/modules/binary/11_64/vmxnet3s</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 227 34 -rw-r--r-- 1 root root 34256 Nov 17 07:55 ./lib/modules/binary/2009.06_64/vmxnet3s</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 253 25 -rw-r--r-- 1 root root 24672 Nov 17 07:55 ./lib/modules/binary/11/vmxnet3s</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 259 2 -rw-r--r-- 1 root root 1071 Nov 17 07:55 ./lib/modules/binary/11/vmxnet3s.conf</span></code></code><br />
<div>
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code><br />
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">This is very strange as installation of the Tools is a straightforward procedure with no room for user error.</span></code></code><br />
<span style="color: #20124d;"><code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code><code><code><span style="font-family: Arial, Helvetica, sans-serif;">So I decided to open the Tools installation script (perl) and found an interesting bug -</span></code></code></span><br />
<br />
<pre class="prettyprint"><code><code><span style="font-family: inherit;">...
sub configure_module_solaris {
my $module = shift;
my %patch;
my $dir = db_get_answer('LIBDIR') . '/modules/binary/';
my ($major, $minor) = solaris_os_version();
my $os_name = solaris_os_name();
my $osDir;
my $osFlavorDir;
my $currentMinor = 10; # The most recent version we build the drivers for
if (solaris_10_or_greater() ne "yes") {
print "VMware Tools for Solaris is only available for Solaris 10 and later.\n";
return 'no';
}
if ($minor < $currentMinor) {
$osDir = $minor;
} else {
$osDir = $currentMinor;
}
</span></code></code></pre>
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">For Solaris 11.1, $minor is 11, which forces $osDir to be Solaris 10. Bug ?</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Either way it's very easy to fix - just change "<" to ">":</span><br />
<br />
<pre class="prettyprint">if ($minor > $currentMinor) {</pre>
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><code><code><br />
</code></code></span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Re-install Tools using the modified script and reboot. </span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Let's check the installed driver now:</span><br />
<br />
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <br />
<div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">root@solaris-lab:~/vmware-tools-distrib# find /kernel/drv/ -ls |grep vmxnet3</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">79085 2 -rw-r--r-- 1 root root 1071 Mar 27 02:00 /kernel/drv/vmxnet3s.conf</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">79087 35 -rw-r--r-- 1 root root <span style="background-color: yellow;">35240 </span>Mar 27 02:00 /kernel/drv/amd64/vmxnet3s</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">79086 25 -rw-r--r-- 1 root root <span style="background-color: cyan;">24672 </span>Mar 27 02:00 /kernel/drv/vmxnet3s</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">root@solaris-lab:~/vmware-tools-distrib# find . -ls |grep vmxnet3</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 231 25 -rw-r--r-- 1 root root 24528 Nov 17 07:55 ./lib/modules/binary/2009.06/vmxnet3s</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 234 2 -rw-r--r-- 1 root root 1071 Nov 17 07:55 ./lib/modules/binary/2009.06/vmxnet3s.conf</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 250 2 -rw-r--r-- 1 root root 1071 Nov 17 07:55 ./lib/modules/binary/10/vmxnet3s.conf</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 244 25 -rw-r--r-- 1 root root 24440 Nov 17 07:55 ./lib/modules/binary/10/vmxnet3s</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 262 34 -rw-r--r-- 1 root root 34104 Nov 17 07:55 ./lib/modules/binary/10_64/vmxnet3s</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 237 35 -rw-r--r-- 1 root root <span style="background-color: yellow;">35240 </span>Nov 17 07:55 ./lib/modules/binary/11_64/vmxnet3s</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 227 34 -rw-r--r-- 1 root root 34256 Nov 17 07:55 ./lib/modules/binary/2009.06_64/vmxnet3s</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 253 25 -rw-r--r-- 1 root root <span style="background-color: cyan;">24672 </span>Nov 17 07:55 ./lib/modules/binary/11/vmxnet3s</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 259 2 -rw-r--r-- 1 root root 1071 Nov 17 07:55 ./lib/modules/binary/11/vmxnet3s.conf</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></code></code></div>
</div>
<div>
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><code><code><br />
</code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;">Now we have the correct version installed. </span></code></code></span><br />
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Let's enable jumbo-frames as before and check if it made any difference:</span></code></code><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><code><code><br />
</code></code></span></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">root@solaris-lab:~# ping -s 192.168.1.202 9000 4</span></code></code></div>
<div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">PING 192.168.1.202: 9000 data bytes</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">9008 bytes from 192.168.1.202: icmp_seq=0. time=0.338 ms</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">9008 bytes from 192.168.1.202: icmp_seq=1. time=0.230 ms</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">9008 bytes from 192.168.1.202: icmp_seq=2. time=0.289 ms</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">9008 bytes from 192.168.1.202: icmp_seq=3. time=0.294 ms</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">----192.168.1.202 PING Statistics----</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">4 packets transmitted, 4 packets received, 0% packet loss</span></code></code></div>
<div>
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">round-trip (ms) min/avg/max/stddev = 0.230/0.288/0.338/0.04</span><span style="font-family: inherit;">4</span></code></code></div>
</div>
<div>
<code><code><span style="font-family: inherit;"> </span></code></code></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><code><code><br />
</code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><b>Success</b>! jumbo-frames are working.</span></code></code></span><br />
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code></div>
<div>
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Let's test throughput with iperf:</span></code></code><br />
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiV95fXRIlGkdOttiQ_qYUbMHwM9RTkglDihuFkxk9HdYShwdB4WtNUU4OUZMi17nVqXcRBkrbuhOninRuwdzotIRNtNUiInUDSdJCof2Ke4wc4ZZdts8U1qu7f9WyTuldo45WtJ3gVP8Y/s1600/vmxnet+jumbo.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiV95fXRIlGkdOttiQ_qYUbMHwM9RTkglDihuFkxk9HdYShwdB4WtNUU4OUZMi17nVqXcRBkrbuhOninRuwdzotIRNtNUiInUDSdJCof2Ke4wc4ZZdts8U1qu7f9WyTuldo45WtJ3gVP8Y/s1600/vmxnet+jumbo.png" /></a><code><code></code></code></div>
<div>
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"> </span></code></code></div>
<div>
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"> </span></code></code></div>
<div>
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br /></span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Less than 1Mb/s, not what we expected at all!</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Need to take a deeper look at the packets being sent. Let's use tcpdump to create a trace file:</span><br />
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">root@solaris-lab:~# tcpdump -w pkts.pcap -s 100 -inet1 & PID=$! ; sleep 1s ; ./iperf -t1 -c192.168.1.202; kill $PID</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[1] 1726</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">tcpdump: listening on net1, link-type EN10MB (Ethernet), capture size 100 bytes</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">------------------------------------------------------------</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Client connecting to 192.168.1.202, TCP port 5001</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">TCP window size: 48.0 KByte (default)</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">------------------------------------------------------------</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[ 3] local 192.168.1.206 port 35084 connected with 192.168.1.202 port 5001</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[ ID] Interval Transfer Bandwidth</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">[ 3] 0.0- 1.3 sec 168 KBytes 1.02 Mbits/sec</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">70 packets captured</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">70 packets received by filter</span></code></code><br />
<code><code><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">0 packets dropped by kernel</span></code></code><br />
<div style="font-family: Arial, Helvetica, sans-serif;">
<code><code><br />
</code></code></div>
<div style="font-family: Arial, Helvetica, sans-serif;">
<span style="color: #20124d;">and open it in Wireshark for easier analysis:</span><br />
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<code><code><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRmVIsJwlmjBAACod9t3LnBNS3ERavsBHyfKmYE644TRupQrkIkIutZ6ZQrITUWcsSZo78RDpAU5Hfq8wfS4CXFFqXGlCjrCq9UF8s1J6iPZO35BnL9GAWKqTjEIN_a8M79RuGxkj5sLc/s1600/tcpdump+with+lro.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="171" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRmVIsJwlmjBAACod9t3LnBNS3ERavsBHyfKmYE644TRupQrkIkIutZ6ZQrITUWcsSZo78RDpAU5Hfq8wfS4CXFFqXGlCjrCq9UF8s1J6iPZO35BnL9GAWKqTjEIN_a8M79RuGxkj5sLc/s640/tcpdump+with+lro.png" width="640" /></a></code></code></div>
<div style="font-family: Arial, Helvetica, sans-serif;">
<code><code><br />
</code></code></div>
<br />
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code></div>
<div>
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"> </span></code></code></div>
<div>
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">The problem is clear with packet 7 - the driver is trying to send a 16KB packet, above our 9K MTU jumbo frame. This packet is not received</span><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"> outside of the VM and after a timeout it is being fragmented and retransmitted. This happens again for every packet generating a massive delay and causes throughput to be very low.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Reviewing the vmxnet3 driver source (open source at <a href="http://sourceforge.net/projects/open-vm-tools/">http://sourceforge.net/projects/open-vm-tools/</a>) it seems the only way a packet larger than the MTU to be sent is if the LSO feature is enabled. </span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">To learn more about LSO (Large Segment Offload) read <a href="http://en.wikipedia.org/wiki/Large_segment_offload">http://en.wikipedia.org/wiki/Large_segment_offload</a>.</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Essentially, the kernel is sending large packets (16K in the capture) and the NIC (or virtual NIC) is supposed to fragment the packet and transmit valid-size packets. On a real hardware NIC, at high speeds, this saves considerable amounts of CPU. in a virtualized environment I don't see the benefit. And it seems to be badly broken.</span><br />
<span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
Let's disable LSO:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ndd -set /dev/ip ip_lso_outbound 0</span><br />
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">And try to run iperf again:</span></code></code><br />
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code></div>
<div class="separator" style="clear: both; text-align: center;">
<code><code><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEje_pMi4Pa60bqH4jjVfoeE2nYKwEEsJKVTam6R37ezkxyBRIfQs5uLT-rsQuF2q3m_gm8OuFpjYdxr3JAcdsd5x_BU9KCXNuL5oYvNaK24_7ARAA3kSTB07yTE0jLJdmontre3Sq-Rzlk/s1600/vmxnet+jumbo+no+lro.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEje_pMi4Pa60bqH4jjVfoeE2nYKwEEsJKVTam6R37ezkxyBRIfQs5uLT-rsQuF2q3m_gm8OuFpjYdxr3JAcdsd5x_BU9KCXNuL5oYvNaK24_7ARAA3kSTB07yTE0jLJdmontre3Sq-Rzlk/s1600/vmxnet+jumbo+no+lro.png" /></a></code></code></div>
<div>
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"> </span></code></code></div>
<div>
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"> </span></code></code></div>
<div>
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <b style="font-family: Arial, Helvetica, sans-serif;"><br />
</b> <b style="font-family: Arial, Helvetica, sans-serif;"><br /></b><br />
<b style="font-family: Arial, Helvetica, sans-serif;">12.1 Gbps, SUCCESS!</b><br />
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">Now that that we are able to transmit from Solaris out in decent rates, let's check the performance of connections into the Solaris VM:</span></code></code><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaOTkM8gE66Cl6lQEqBCBygKFF3korYnkh0mv5AFgM1Pt7ocEtH56MvIVPVIjCb0CiO8H01xLtjez6BEun8B9_nzmCE0_iZGg-8CPrjbJk1ZK4dr2_KaHYgarN-Dz3YggR1YrnwolKl1s/s1600/server+default.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaOTkM8gE66Cl6lQEqBCBygKFF3korYnkh0mv5AFgM1Pt7ocEtH56MvIVPVIjCb0CiO8H01xLtjez6BEun8B9_nzmCE0_iZGg-8CPrjbJk1ZK4dr2_KaHYgarN-Dz3YggR1YrnwolKl1s/s1600/server+default.png" /></a></div>
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <span style="color: #20124d;"><code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code></span><br />
<span style="color: #20124d;"><code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code></span> <span style="color: #20124d;"><code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code></span> <span style="color: #20124d;"><code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code></span> <span style="color: #20124d;"><code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code></span> <span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">3.74 Gbps, not bad, but we can do better - let's at least get to 10Gbps.</span><br />
<span style="color: #20124d;"><code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code></span> <span style="color: #20124d;"><code><code><span style="font-family: Arial, Helvetica, sans-serif;">Next step is to tune the TCP parameters to accommodate the higher speed needed - the buffers are simply too small for the amount of data in flight -</span></code></code></span><br />
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <code><code></code></code><br />
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code>root@solaris-lab:~# ipadm set-prop -p max_buf=4194304 tcp</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code>root@solaris-lab:~# ipadm set-prop -p recv_buf=1048576 tcp</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code>root@solaris-lab:~# ipadm set-prop -p send_buf=1048576 tcp</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><br />
</code></code></div>
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">And run iperf again:</span></code></code><br />
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><br />
</span></code></code> <br />
<div class="separator" style="clear: both; text-align: center;">
<code><code><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxhmQOecojClDFddpsqCSDVDaM2wYIfXV5JNc9vlkFTyj_A4lmfpU2HWP9WtRb49KZT6Bm_0_xCp6QyIO4cfVY2wYk85rIVMBDPacg0ux2uTjhiAp5CBpf9L7sGG_GOEZyvo-u3Qu8-l4/s1600/server+after+tune.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxhmQOecojClDFddpsqCSDVDaM2wYIfXV5JNc9vlkFTyj_A4lmfpU2HWP9WtRb49KZT6Bm_0_xCp6QyIO4cfVY2wYk85rIVMBDPacg0ux2uTjhiAp5CBpf9L7sGG_GOEZyvo-u3Qu8-l4/s1600/server+after+tune.png" /></a></code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><br />
</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><br />
</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><br />
</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><br />
</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><br />
</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><br />
</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><br />
</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><br />
</code></code></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<code><code><br />
</code></code></div>
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> <br />
<b style="color: #20124d; font-family: Arial, Helvetica, sans-serif;">18.3 Gbps, Not bad!</b><br />
<code><code><span style="color: #20124d; font-family: Arial, Helvetica, sans-serif;"><b><br />
</b></span></code></code> </div>
<div class="separator" style="clear: both; text-align: center;">
<code><code> </code></code></div>
<div>
<code><code><span style="font-family: Arial, Helvetica, sans-serif;"> </span></code></code></div>
<div>
<code><code> </code></code></div>
<div>
<code><code> </code></code></div>
</div>
</div>
<div>
</div>
Cyber Explorerhttp://www.blogger.com/profile/15872243741791530580noreply@blogger.com119tag:blogger.com,1999:blog-1818100545490785115.post-9958323657182127502013-03-26T02:46:00.000-07:002013-03-26T02:46:10.533-07:00First postFirst post! testing blogger.com functionality.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTDdUCvjg2R7ssEGD2nZR26TK1QtsueWQ7WdOJALaivdhS73W_z3g" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTDdUCvjg2R7ssEGD2nZR26TK1QtsueWQ7WdOJALaivdhS73W_z3g" /></a></div>
Cyber Explorerhttp://www.blogger.com/profile/15872243741791530580noreply@blogger.com25