We might be what you would call an early adopter of 10GbE at my company. We've now been on it a little over a year. We were quick to abandon any Fiber-Channel or block-level sort of storage connectivity because....who needs the headache?!
10GbE NFS is here, and it is here to stay. And with folks like Oracle literally BUILDING-IN a client in their database software to completely bypass the kernel...that's staying power. And it ain't goin' nowhere.
I wanted to take a minute to summarize how 10GbE has "changed our life," and how we hope to take advantage of it in the future.
First and foremost....Oracle. We recently migrated from an 18U Sun refrigerator to a 2U HP server. Now, most of that is just advancement in x86 hardware, but what we were testing alongside all of that was remarkable. We were running VMware VM's with Oracle Enterprise Linux, and piggybacking VMXNET3 vnic's over the same 10GbE connection the hosts' use to connect their datastores, to mount storage directly to the VM.
I want you to think about that for a minute, and how frickin' cool that is, and where we were as systems/network admins even as recent as five years ago. Pretty remarkable isn't it?
Now, I've heard some naysayers coming across the line, specifically about pv-vmxnet3 adapters/software/whatever, and most of that is because it doesn't support FT. Well, that's not a hard club to be in, considering almost everything and the kitchen sink doesn't support FT.
What I'm here to say is, we drank the kool-aid.
All of my hosts connect all of their datastores with 10GbE NFS. (2 clusters, 7 hosts, 89 VM's, including PRODUCTION SQL, Oracle, & Exchange)
Many of my VM's are running 10GbE NFS directly mounted to the VM
My Exchange Mailbox servers are running 10GbE iSCSI (created 1500 MB's in :58 seconds)
Our PRODUCTION Oracle 10g Servers are running 10GbE NFS, and running circles around the v880's and Fiber-Channel gear we used to have.
Call me an evangelist, but I love me some 10GbE.
So go find yourself some Nexus switches, sit 'em at the top of your racks and hold on to your hats.
Nothing I've ever architected runs as fast as my VMware virtual environment on 10GbE. And I'm looking forward to future deployments where more arguments for 10GbE NFS can be made.
#NAS4LIFE 
-Nick