iSCSI and Virtualization

I was looking into this area, and was curious - does anyone have experience with this? I am looking at a HyperVisor Virtualization solution (such as VMWare, XEN or HyperV) that could support the guests via iSCSI. What solution/setup would you recommend, and the pros/cons to it?

exactly. I think hosts who do not get on this train will be left at the station crying. this is where it is all going and you dont want to be the last guy on. I would suggest sooner than later. I would also not suggest jumping in a big shared cloud - we have seen the up and down issues there. Do your homework and demand the best.

Wow, seriously? Get back on topic?

About right. Dell wants to charge $20K for 8x1 TB SATA drives (comes about to $2500/drive) for the EqualLogic arrays. Note that these are not off-the-shelf drives either. They do something funky to the drives (removing the built in cache, different format, etc).

Thanks! I've used NetApp at previous jobs for just about everything except iSCSI. I'm still a big fan and would love to acquire some of their filers in our environment. The one thing I HATE about NetApp is how they charge you to use each different protocol. If they would quit jacking around with customers that way they'd pretty much own the market, and nobody would even look at most of their competitors. For virtualization, I know that NetApp owns the performance crown for NFS datastores. I've tried many times to tune a Linux NFS server to get performance that would be good enough, but have never been pleased with the results. And as I said earlier, Lefthand VSAs are just so cheap that it's not even worth the time for someone in my group to spend messing around with freebie solutions. However, something political happened before I came on-board at this job and NetApp won't step foot back in our place.

Hypervisors could care less about iSCSI as long as a LUN is exposed from the target to the initiator. The iSCSI LUNs are block devices that can be filesystem formatted then assigned to guests. There's complexity involved especially when setting logical volumes but the benefits are great. Check out Openfiler as it helps with configuring iSCSI on the target devices. But before you put this configuration in production, be sure to simulate path failure and learn how to recover from it. Regards Joe

Isn't TOE only available in windows? And it's not just tcp. Whenever the hypervisor has to supervise the VM, there's a huge performance hit. Anandtech has some good articles on this. I do agree that virtualization will become a bigger force in the future even moreso than now. This is inevitable as servers become more powerful, fewer users will need dedicated servers and more will be able to get by with cheap VMs, while power and space won't be getting any cheaper. With disks and VMs, however, you're not usually constrained by space so much as performance. Even if you leave your disks half full, it's entirely possible that disk performance will be your bottleneck.

Annoying that this turned into a VPS.net thread, but I've got to say that eming has a much stronger case than the guy with the "reliable source"

What's the rpm on that SATA? Talking about 100% random IOs, no 7200 rpm SATA drives can push 160 IOPS, more likely to be around 80 IOPS or so. The ratio of IOPS between 7200 rpm SATA, 10K rpm SAS, and 15K SAS is about 1:2:3 (all 100% random IOPS). That's why the EqualLogic PS5500E/PS6500E (48 x 7200 rpms SATA drives) is rated as having the same amount of IOPS as EqualLogic PS5500XV/PS6500XV (16 x 15K rpms SATA drives).

As others have said, it depends. How big are your i/o's? Sequential or random? If you're serving 4k thumbnails at 160 i/o/s you'll be doing just over half a megabyte per second. If each i/o is a full megabyte, you'll be able to go at the drives maximum sequential rate (or close to it). One of the big ways that google has innovated is by making their workload as sequential as possible, even if it means wasting disk space. This way they get the most performance for the least money.

There is another thread going on here: http://www.webhostingtalk.com/showthread.php?t=915005 , regarding SAN's that I am watching. This is a tough topic since we provide VPS hosting and storage is one of your biggest concerns especially with Hyper-V hosting. However, there has to be some middle ground between the big "SAN" solutions and a project like the iscsi project for providing SAN or ISCSI storage without the pricetag. The big factors here are performance vs reliability and/or redundancy.

It sounds as though you're coming from an HPC perspective. We do some of that also, but do not use virtualization there. We go with lots of relatively cheap, non-redundant systems with few moving parts and small SSD for those situations. Some are single 1U chassis with two independent systems inside. This segment of the business is completely different with different requirements and goals. I disagree with your assessment of the CPU hit from the hypervisor or the storage network. I hear and read about this stuff all the time, but quite frankly I don't know where these people get their info. I'm not seeing those kinds of results. In practice, we see a small performance hit from virtualization in the area of 2-10% (depending on what we virtualize), and less than 5% CPU on the storage side. In my experience, if you're running CPU margins that thin then you have much bigger problems that need to be addressed, and you definitely should not consider virtualization.

IMO UK2 Group pretty much defines "reliable source"...

Very interesting and useful figures. Thanks for sharing. I've not researched this in the past, but the things you learn casually browsing a forum here or there... --Chris

Hi bennice, I am not certain what has gone on in your facility.. though, I am quite aware of the dynamics you have described - and this is not unique to any vendor amongst its customer base... However, regarding their pricing in particular.. As far as I know, there is no charge for different protocol.. We are running NFS and iSCSI in our environment. We have it split out across controllers and trays.. the concept of pricing for protocol has never even been mentioned.. I am more then a little suprised to hear this - mind you though, our relationship with NetApp is around a year old.. so, I am not sure how they were run or priced previously.. I can tell you though that our pricing came in at 50% more then a comparable equallogics unit.. except with NetApp, its easy to see an ROI 6-12 months out as their disk utilization is far superior (in my opinion anyway) - and their ability to better utilize and target specific bottlenecks - thereby not throwing your money away on new units prematurely - made it completely worth the investment.. we are already realizing this ROI and we believe it will be much more obvious once we start to significantly scale (not too mention we paid in CAD vs USD ) Anyway, hope this is helpful to you - either now or in the future... eitherway, thank you for your comments.. Our team have greatly appreciated them - if nothing else, you have validated our expectations with real life production scenarios.. absolutely wonderful value and wanted you to know we greatly appreciate it...

I think we are going to play around with some SAN/iSCSI configurations with our Virtual environments. I am curious to learn of the experiences of others, if they are willing to share That thread looks good. I am mainly concerned about the performance issues of putting live VMs on iSCSI. Everything I know tells me this is a bad idea, but I've seen many providers doing this.

Agreed. A SAN with single points of failure isn't really a SAN at all. It's just JBOD that is able to serve blocks using one of the standard protocols (iSCSI, FC, etc.). We take single nodes of our Lefthand SAN clusters offline all the time to perform maintenance. Like all commodity-grade equipment, components fail and need to be replaced from time to time. The good news is that nobody outside our SAN administrator group ever notices when this happens, which is the goal.

I'll admit the last few posts are a bit far off topic, but as a whole I don't think the discussion has been completely useless to the topic at hand. It brings up some important considerations, and vps.net is perhaps one of the best examples of 'large scale' in this business - so it's definitely a relevant subject. It would for an example have been interesting to hear why they (apparently) think thin provisioning is a bad idea, when they state themselves that they run on a RAID 10 SAN Backend and use LVM . But I guess that won't happen, so let's move on.

My understanding is they just have altered the firmware. No actual hardware alterations.

I took a look at lefthand. It looks cool but does it only support esx? I plan to use xen and would consider something like lefthand that combines the cost effectiveness of on node storage with the features of a san

Yes, this makes sense. I was really asking about how are providers setting up their network/virtualization to run live VMs over iSCSI. This seems like a bad idea to me, but I am curious to see if any providers have gotten this right? What are the main performance constraints, and what hardware/software eases those constraints?

It's more of a profit margin issue than a cpu margin issue. If you're losing 5% of your cpu time because you've virtualized your storage, that's 5% less cpu time you can sell on your VPS nodes, period. Now, if you've got your own servers for your own projects, and you want to consolidate, that's a different story entirely. Of course on my servers I want to have a margin for extra cpu time. But if I target a peak usage of 75% cpu usage, and storage is taking up 5%, that means I get 5% less out of the server to hit my target 75% usage. And if you have to use high end nics at $200 apiece, on a server that normally would cost $2000, in order to get the cpu hit down to only 5%, when otherwise it might be 10 or 15 or 20%, well that adds 10% to your server costs so the whole thing is a wash there anyway. If you want to offer vps hosting, you want to be able to offer the best solution at the best price. The best solution means the best performance and the best uptime. Intentionally creating performance bottlenecks and inefficiencies doesn't give you better performance per dollar, so you either pass those costs along, you give people less performance, or you have lower margins. Really all there is to that. As for uptime, yes, there are ways to make your SAN more redundant or more robust, but it will be difficult to make it truly bulletproof; the cost of doing so will greatly increase the costs to offer your service. The only upside is that if a vps node fails you can move people to another node. The downside is, if your entire SAN fails, all your nodes go down. I'd rather have a few hours downtime on one server than who knows how long of downtime on all my servers. And engineering for 0% failure rate on the SAN is going to be prohibitively expensive. Now, I do plan to set up a CDN based on mogilefs to do cloud storage. This is a special use case in my opinion. For this, customers don't have to commit to a particular amount of disk space, they want to have everything stored redundantly on more than one server, and quite possibly in more than one location, and it needs to scale up to storing 100TB or down to 1gb seamlessly. I don't know of a good way to do that without some kind of networked storage, so that's what I'll use there. But if you're offering clients xGB of storage for each of their VPSs, and X number of i/o/s, you can plan your VPS nodes to have the necessary storage capacity for the quantity of VPSs you plan to put on the system. If you put the thing in a Raid 10, you get decent redundancy and performance at a reasonable price. And to be honest, I think that's what most people are looking for. Again, for enterprise customers, or consolidating your own servers, you can certainly use a SAN. Also, for cloud storage, where people need to scale up to storage levels beyond a single server, and don't want to code anything, SAN can be a good choice. But I'm thinking more from the perspective of a VPS hosting provider, you want the maximum disk i/o per user, maximum disk space per user, maximum cpu time per user, maximum ram per user, and maximum average uptime, for the minimum up front costs, rack space, and power, and SAN storage fails this test. At the very least, the storage nodes are going to eat up 50-60 watts more by virtue of having their own cpu in them, versus putting disks in your servers. If you have the same number of disks, you have the same costs there. If you have less disks, then it's not a fair comparison anyway, you could always just put less disks in your VPS nodes. And then you have to buy a lot of networking equipment and engineer it not to fail, while pushing obscene amounts of traffic that otherwise would not be necessary. If all your VPS nodes are 1u anyway, you don't save any space or power by taking out the 4 disks that would fit into the server and putting them into a different server, a server you would not need if you had the disks in the VPS server. And then the servers doing the storage are going to need high end battery backed raid cards, because to be cost effective you're going to want 24 disks to a server, and you're going to need a raid card to accomplish that. With disks in the servers themselves, onboard raid 10 with 4 disks works fine and costs nothing extra. I just don't see myself increasing my costs by at least 20% while reducing my performance by an equal amount being a good business decision.

Well now, add me to the list of people curious about your hardware infrastructure because what Shoel described is exactly what I assumed you were doing at vps.net too

The 1TB caviar black is a 7200rpm drive. Doesn't have TLER enabled either, so these drives can and DO fall out of raid arrays. However, in some benchmarks the drives do pull pretty decent IO's, I believe it's due to the way benchmarks work and the 32mb cache on the drive. Not sure what effect this has on real world performance though.

The VSA appliance only comes in .ovf or .vmdk right now as far as I know, but I have heard that they were working on official support for other hypervisors. Unless you really hate VMware though, there's really no reason not to use it though since it runs fine on the free versions of ESXi.

You have as much I/O bandwidth as you have copper/fiber running between the storage and the compute node (of course your only going to get as fast as the harddisk hardware allows for). Which is why trunking is a great technology for a project like that. For example I run a small-ish setup right now that has a couple compute nodes, hooked to an iSCSI ZFS Solaris setup. I use 2x 1000mbit copper to my storage, and 1000mbit to my compute nodes. I have seen no issues with the setup at all. I'm currently working on advancing the system with a network boot mechanism to mount the entire root fs via network. The benefits for a setup like this is obviously power, you can use a small cluster of HD's in one machine, and have your compute nodes external. If each of your compute nodes was Raid 10 minimum you would have 8 drives, or you can setup a ZFS share with 3 spinning drives and an SSD. Cutting your HD power requirements in just a bit over half. Also if you use iSCSI, there is no need to go with expensive fiber channel, your existing copper network will work just fine (assuming its CAT6).

Ahh...I see. You are definitely coming from a much different business model than me. I have no experience in the VPS space, and don't really understand how that business works. But from what little I know about that space it seems that virtualization would be a perfect fit there with the right design.

Yes, I agree. That's why I was saying this is not an apples to apples comparison, as using fewer disks will of course be cheaper. I have a business partner I've worked with on and off for a while who has been doing VPS hosting for several years now, and it's both of our opinion that disk performance is more of a bottleneck than most other things in the VPS server, as far as how many customers you can put on a node. Basically on a server with 16gb ram and 4 disks, each 1gb ram vps gets 1/4 of a disk. Do they need 1/4 of a disk worth of space? Maybe not, but you're actually trying to give them 1/4 of a disk worth of performance, and the space just happens to come along with it. I could be wrong, the VPS system could be limited more by space than by speed, and it may make sense to oversubscribe space. If I'm using 1tb disks in my servers, I'll need at least two to do raid 1. If I really only need 20% of the disk space or disk speed, then yes, having a san with 2 disks per 5 servers in raid 1, may end up being cheaper than having 2 disks in each server, as you're cutting out 4/5 of your hard drive count. In that case, however, you're also cutting out 4/5 of your hard disk performance, so it's hardly a fair comparison. I suppose the bottom line is, the SAN would allow you to grow as you needed to, for both space and performance, and not waste resources (other than network i/o and network protocol processing), whereas putting the disks on the servers will save you money on a per gb and per performance basis, but only if you appropriately size the number of disks per server to the workload you'll see on those servers.

One thing that should be clear by now is that there is a large price range depending on how someone wants to implement things. For a very simple iSCSI SAN where all LUNs are mirrored on two different SAN units (and thus can withstand the complete failure of a storage chassis), the price I came up with for 100TB of usable capacity is $30K, plus network gear. There are obviously tradeoffs involved if you do it that way compared to more expensive SANs, so it really depends on what it is being used for.

I heard the free version of esx does not have nearly the same performance / features of the paid version. How does free esx compare to xen?

we tested quite a few and in fact run openfiler on home grown stuff for our backup san which works quite well. However, after testing we decided to go with hp storageworks for many reasons - but mainly performance. In fact we even found the need for 2 different systems here - one for sql transactions and one for regular file service. it can be complicated but the beauty is most will let you test and find soemthing for your budget. most providers these days will sell you a slice of their enterprise level san if you can not afford it as was recommended andrew on the other thread referenced on this thread. good luck!

funkywizard, you should look into virtualization and SAN. it costs more in the short-run but in the long-run it makes life easy, retains customers, and lets you grow as fast as you want. the only place for a bunch of 1U servers without a SAN, in my opinion, is a company like google that has 100,000 servers that are all identical (and when one dies they throw it away)

Ditlev already stated that there's an IOPS benefit. By doing a 1:1 allocation, there's less IO per unit of storage. As far as calling Ditlev a liar, I don't think that was appropriate. He very specifically stated that they don't oversubscribe the storage for their VPS.NET brand. If you don't believe him, keep that to yourself unless you have clear evidence that you're right.

Per unit of space, yes. Consider a situation where you have two VMs that do 100 IOPS each, and each have 100GB allocated to them. Now say that under a thin provisioning configuration, each "use" 50GB. Without thin provisioning, you're doing 100 IOPS over 100GB of SAN capacity. With thin provisioning, you're now doing 200 IOPS to the same 100GB. There's nothing wrong with thin provisioning, but there are situations where it's easier and cheaper not to utilize it.

I decided to go ahead and get RE drives instead of blacks for my raided servers. They cost 160 each instead of 100 which hurts a little, but not as much as losing an array due to a simple timeout. Still going to use blacks in my jbod systems

The main performance constraints will be: -CPU overhead from generating/processing a vast amount of TCP/IP packets -- This concern can be eliminated with the use of an iSCSI HBA, and can be offset with the use of TOE capable NICs -Network congestion on the host -- If you're going to do ip-based storage it's almost a must that you're multipathed, coming from physically separate NICs, and that you implement some type of round-robin load balancing scheme. -Network congestion on the SAN -- same principles apply. If you're going to be buying an actual SAN from netapp, dell and the like these considerations have generally been taken care of for you. If you're building a whitebox SAN you might want to seriously consider some 10GBit nic's. -Network configuration on the switch -- there are varying schools of thought here. Some people think you should have a dedicated storage network w/ dedicated switches, and others think it's okay to simply segment the networks with VLANs. =========== As unixy aptly noted, provisioning storage can be a pain with ip-based storage. For that reason I recommend Citrix XenServer. It will automatically provision the storage for your VM's, and you won't have to worry about it. In general I think networked storage is a very good thing, and is the future. There are certainly some complications inherent to it, but I believe net-net, and with the right hardware, ip-based storage can be more efficient and more cost-effective than host-based storage. Give it a go!

It depends very much on what kind of scale you plan to do business on. If you are running a small number of VMs it might not be the most affordable course of action, especially not if you are fine with running onboard software raid (prone to data corruption) and cheap S-ATA disks (slow & prone to failures). What you are not taking into account is that VMs can be thin-provisioned with virtual storage that'll grow according to the real usage of the VM instead of setting aside a given chunk of disk space for each VM. That means you can use storage *a lot* more efficiently, essentially having less GB's of storage for a given amount of VM's. To give you a simple example: you do not need 4TB of storage to sell 40 x VPS' with 100GB storage if the average usage of all the VM's (thus average virtual disk size) in reality is.. say.. 10GB each. With consolidated storage you could get away with 10% of the physical storage in that scenario (+ a little extra margin to accommodate growth). If you do this on a larger scale, you can for an example run the hypervisors off a flash disk on each node (in other words run completely without internal spindle disks), and consolidate all your spindle storage into larger arrays where the storage is used a lot more efficiently, shared by a higher number of nodes, and where you can scale your storage according to actual usage and not some unrealistic max usage based on how much space you are actually 'selling'. (There's nothing wrong in doing this as long as you operate with enough margin to actually deliver the promised space should a client require it all.) In addition (when doing this on a serious level) you save on not needing expensive RAID controllers in the nodes, cutting the number of spindles greatly (which can save a lot of money when we are talking enterprise grade storage, SAS disks), cutting power costs due to having less spindles and controllers, using much cheaper chassis for the servers due to not needing 4 HS HDD bays per node, reducing time spent on maintenance of nodes, storage and backups, and so on. As a bonus you gain advantages such as high availability with fail-over of VM's to available nodes in case of hardware failure, snapshots, storage replication, and overall better redundancy/less downtime for customers (if designed properly). This can be really beneficial to building a reputable brand, and can be exploited for marketing purposes. This is how for an example vps.net does it, and if you knew how little storage they use compared to the number of nodes and VPSs - you'd quickly start to understand how this can pay off.

That may be correct, but there are many ways to do a highly available solution. Your method is to make your storage system highly available. Raid 10 local storage I suppose makes things "reasonably well available" but not highly available as you said. Reasonably available SAN storage is harder to do than reasonably available local storage, but I do agree, highly available local storage is harder to do than highly available SAN storage. But then again, there are many ways to do HA, if you need it. You could for example, get two VPS's on two different nodes, and use mysql master-master replication. This would be even more robust for that use case than simply having redundant storage, and it would give you double the mysql read performance as having a single vps, while still being as highly available (or more highly available) as the SAN model. I do agree that this would not be totally transparent to the customer, so there is definitely some value in having HA that is more transparent and easy to use, which can be done with a SAN. There are a lot of ways to do everything, with different tradeoffs and benefits. Raid 10 is a great way to get good availability, better performance, and lower cost. A highly engineered san is a good way to get high availability on your storage, without requiring the customer to program high availability into their application, but it comes at a price. And a poorly engineered san is a good way to introduce many single points of failure, while still increasing your costs compared to local storage of the same level of performance / disk space.

The blacks are a 7k rpm drive. I believe this was on the iometer database transaction benchmark. It's pretty popular on tomshardware.com disk benchmarks. The database benchmark pattern is a mix of reads and writes, mostly random. http://www.tomshardware.com/charts/2...7.27,1034.html Most of the sata drives in the roundup can do around 100 i/o/s on that benchmark. (the slowest does under 70 i/o/s, and the fastest that's not a 10k drive does 170 or so). The caviar black can do 160 i/o/s for $100 and has 1tb space. The only other sata drives that can do more than 160 i/o/s are susbstantially more expensive than the 1tb black, and aren't amazingly faster, so the black is the drive of choice. If you want raw disk space and don't care about performance, you could even go with 1.5tb drives, but I want the best i/o/s/$ as well as good disk space, so the 1tb black fits there. Unfortunately, the enterprise hard drive charts haven't been updated to the 2006 version of i/o meter, so their results are not directly comparable to the 2009 3.5" desktop hard drive charts. To compare them, you can look at the old 3.5" drive charts, which use a version of iometer that reports dramatically higher i/o/s counts (not sure why exactly, maybe it's on a different scale entirely), and then compare the 3.5" on the 2003 test to the enterprise disks on the 2003 test. This leaves something to be desired though because the old 3.5" test doesn't include the newer sata drives you'd be considering. Sometimes you can look at other reviews on tomshardware that are not the huge "chart" roundups, and those will be directly comparable as they'll throw in a couple SATA drives, some SSDs and some 15k drives, all using the new version of the iometer benchmark. So, using the "Database I/O Benchmark Pattern IOMeter 2003.05.10" (which is on a different scale than the 2006 test), ( http://www.tomshardware.com/charts/3...ttern,661.html ) the 1tb caviar black clocks in with 864 i/o/s. On the new benchmark version, it only does 160, which seems more realistic. So, we can assume that the old test produces scores that are 5.4 times higher than the new version of the test. On the old version of the test, 15k SAS drives can do 1900 - 2200 i/o/s. Since the old version of the test reports 5.4 times as many i/o/s than the new version, we normalize this to say 15k drives can do 350 - 400 i/o/s on the new version. The fastest 300gb SAS drive on the chart is the Fujitsu MBA3 RC, which costs around $350. You could save money going with 147gb or 73gb drives, but if you're getting a pile of 15k SAS drives in order to get enough i/o/s and you only need 147gb or 73gb of space per drive, you're much better off getting SSD's anyway, as the performance is an order of magnitude better. So the only fair comparison is with 300gb drives. What do you get for the $350 Fujitsu MBA3 RC? Well, you get towards the higher end of the SAS benchmarks, around 2.5 times the performance of the 1tb WD Caviar black. But for 2.5 times the performance, you're paying 3.5 times the price and getting 1/3 the space. Am I the only one doing the math on this one? Sure, you could get 74gb SAS drives which are not a lot more expensive than the 1tb blacks, and will be 2.5 times as fast. But as I said, if you only need 74gb of space because you're getting a pile of these drives in order to get your i/o/s up, you're a great candidate for an SSD rather than a SAS drive, as a good SSD will give you 10x the performance of the SAS drive in random i/o. http://www.tomshardware.com/charts/e...ttern,704.html To those who say $250k is about right and I'm being short sighted on $20k for the disk cost, yes, if you're buying from an overpriced vendor, you can certainly pay any amount of money. I didn't know we were pricing components based on vendor wish list pricing, I thought we were going by the actual cost of the parts. Building my own storage boxes (I guess you guys would call them archival grade quality), is going to cost such that the disks themselves make up about 50% of the cost of the entire system. Obviously you can spend an unlimited amount of money on hardware if you'd like, but the least you're going to pay is $20k for the disks, and another $20k or so for the host hardware. Throw in some networking equipment, and other goodies, and you're looking at maybe $50k for 200tb (100tb in raid 1). Now, some might say this is an unrealistically low price, but I plan to use mogilefs to handle redundancy and failover so it should be a good solution with good uptime, at minimum cost. Mogilefs is certainly not a place you stick VPS images, databases, or other stuff, but it should work well for CDN type applications where the data is write once read many. In this space, I care about cost per gb, cost per i/o/s, and making sure the data is always available. By being able to handle a node or drive failure without having to engineer massively redundant everything, I can get the uptime I need without having to pay enterprise SAN prices. Obviously that won't work for everyone. If you get such a setup from a storage vendor, you're going to pay dearly, but if you care about price you're not doing that anyway. So I would say you could do it for $50k, rather than $250k.

ESXi indeed does not have the features unless you buy a license. Ideally, your ESXi box will run on a dedicated piece of hardware (no other VMs on the same box), and none of the additional features really matter in this scenario. All you're providing is an abstraction layer for the guest (VSA) to run on. Performance wise, I don't know of any differences until you get to large amounts of memory and more than 6 CPU cores per socket...all things that are not necessary to run a VSA. As for Xen, I think on the free end it's probably the better product, but they are a couple steps behind VMware for the paid stuff...but they're gaining. However, every time Xen makes one step VMware usually makes 2 or 3.

I am interested to see if there are any specific setting changes that need to happen for iscsi to be effective in a Windows VPS environment. The Windows vs Linux VPS environment in terms of files that would be "served" using iscsi would seem to be best fit for a linux environment where the mount points can actually be stored on disk as files. Whereas in the Windows environment you would be dealing with 10-20GB+ .vhd files. I am not sure how this would work performance wise even with trunking enabled with TOE offloading NIC's. What are your thoughts? Anyone ever use HyperScsi?

a properly setup vps system with fast spindles 99% of the time you will still run out of IO before CPU is even a issue. We have nodes with 72g of ram with 60 or so vms on them so far and we are only using 25% of the total CPU usage. With the proper setup, thin provisioning etc a san storage solution is the optimal way to do things. However this requires some capital which most people do not want to do. There is a huge difference between mentioned above in my post and the average vps host on wht. However ultimately if someones spending 3-4k on a proper vps node to handle the local storage "appropriately" than in the long run someone who uses a storage solution from say 3par, or compellent will usually end up spending MORE in the long run while deoploying dozens/hundreds of nodes.

Well now, that link says ataoe, so it's not very relevant to the thread titled iscsi & virtualization I guess... Same thing, different way to get there.

simply love your posts in this thread. Particularly your network comments. For now, we have separated off everything onto its own gig and 10G network for hostnodes/storage networks. But, we are very highly considering routing everything through 6509 cores with dedicated 10G cards. This is our intention, but, real live metrics on rapidly growing production environments will tell us the real story over time I guess... Also, I do not want to start a massive storage debate with anyone here. There are multiple approaches to this and everyone will choose a different product for a different reason. @bennice - based on reading some of your posts here, reading whatever you have been gracious enough to share about your environment, etc.. I must strongly recommend you look at something like the NetApp 3140 for your environment. You can configure different trays or volumes for specific purposes, add whatever specific resources you need to address your specific bottleneck concerns for that particular purpose, etc... can cluster across head units and trays - heck, you can even dedicate specific cards within the unit for specific purposes and assign redundancy to those cards which have specialized roles.. anyway, appreciate all of the info you shared.. after reading your posts and comments, first thing I thought was that NetApp would be a perfect fit in your environment and would probably save you guys a whack of cash as far as TCO goes. cheers for now..

Unless things have changed since the last filers I bought in 2006-2007, Netapp charges you for an additional license to use iSCSI, CIFS, NFS, NDMP, use snap mirroring and any of their other features. The hardware isn't actually terribly expensive - it's the licensing and support that drive the cost up. It could very well be that this has changed by now though. I only wish I could find a good enough reason and a way to get them back in the door to talk to us.

You're lucky in that Windows (especially 2008) has a good MPIO implementation for ISCSI. From a host / node perspective it's really not too difficult and in most cases doesn't require any elaborate settings. Use high quality NICs with TCP/IP offloading -- I prefer Intel. Present redundant and physically separate ISCSI network paths to the host / node and implement jumbo frames throughout the entire switching fabric. I don't use trunking, but MPIO with multiple links and give each host 4 1GB uplinks dedicated to ISCSI (two separate switch fabrics). What you use will vary depending on the SAN behind it (we use Dell / Equallogic). Our VHD's range in size from 30GB to 100GB and consist of both fixed and dynamic disk depending on the application.

just to set the record straight - that is NOT how we (vps.net) are doing it. We do not oversell storage (or anything else), and we fully allocate every gigabyte reserved for clients. So at VPS.NET one sold gigabyte is equal to one allocated gigabyte. Storage is relatively low cost, large drives are easy to come by, the problem is not the GB's utilized, but the IOPS generated. D

another thing we overlooked is storage and file management on an external distributed san vs internal drives. when you need to add more disk - just slap another array on the iscsi network and your volumes will expand to it. internal drives? not so fast - you are looking at downtime and serious databackup and restore to handle this. need more disk and have it on the san? just point and click and expand the lun. it makes life so much simpler vs internal drives.

I also feel there's a lot of good information in this thread, have learned quite a bit. Sorry if I've come off combative. I feel we've all had a good debate here

Such as? We use XenServer Platinum and we pretty much have everything we need.

I have looking into MPIO, and it looks very interesting. Thank you for your insight. I have been considering QLOGIC's iSCSI HBA adapters, they look interesting too. I am debating whether we'd need HBA adapters or high-end NICs with TCP off-loading - but I guess only real world testing would answer that question for us. Any insight into this, since you have chosen a high-end NIC? Which model has given you the best value from Intel?

Hey Ditlev, I know there is no way of getting how you run things at VPS.net out of you, however you wanna let us know who your preferred hardware manufacturers are? I'm interested in who you use for motherboards and cases - because if its good enough for VPS.net - then its something I want to play with at work.

I have to think that vps.net does NOT use thin provisioning. Think about it. Now my example here is not of vps.net but of a typical VPS provider: If they run 1TB SATA spindles (which I assume they would from a cost standpoint) there is no way in heck they are going to get 100 VPS on them (I am guesstimating that 100 10GB VPS would thin provision to about 500GB or 5GB each -- and 500GB is what a 1TB spindle would provide with RAID10). The only way thin provisioning for a vps provider is going to make sense is if they run 146GB 15k SAS disks -- or maybe 300GB 15k SAS (possibly) because then they get a lot more IOPS. But I am sure from a cost standpoint you run 1TB SATAs and you just run a lot of them. The end result (performance of VPS) is the same, but the VPS provider saves money, and the VPS provider has no reason to thin provision -- you could do it, but why?? bottom line: if the customers opt for a 25GB package vs a 10GB it probably costs the provider nothing extra. I am not sure how vps.net does it but i have a lot of real world experience and that is how a smart company would do it. lots of cheap spindles, extremely reliable, and no need for thin provisioning. and when i say cheap, believe me this setup is expensive -- probably 100TB usable would cost about $250k++ -- if you opt'd for 15k SAS disks maybe $1M++ or more. It makes no sense to do the 15k SAS and thin provision.

IOPS/GB? I wonder how useful this metric is. I have not read any white papers that discuss IOPS/GB. IOPS has nothing to do with your sotrage utilization. You can have 100 TB of data, and have monthly average of 0 IOPS (backup data that's only access when needed). Or you can have 1 GB of data and max out your SAN IOPS (multiple simulataneous video streaming, etc). When your SAN hits its IOPS limitation, time to add more enclosure into the array and spread the load around, no matter how much space you have used, how much more space you have, how efficient your space usage is, etc. IMHO, the only useful metrics to monitor is your overall SAN IOPS. This alone is hard to measure (and different vendors will mesaure it different way). The best thing to do (to make sure fair comparison) is to measure what your SAN is capable doing for 100% random IOs. That's your worst case scenario.

Such as distributed vSwitch, thin provisioning at the hypervisor layer (this may have changed recently though), CPU and memory capacity limits, memory oversubscription (taboo in some circles, but effective in certain situations if managed properly), hot-add CPU and RAM, and fault tolerance - not the HA feature you get from clustering, but the ability to have a lock-step copy of a VM on another host. And then there's support - not only technical support resources, but 3rd party applications, integration with storage vendors, etc. Granted, Xen has most everything most people need and use but there's no question that VMware is a couple steps ahead in most categories. Now pricing is where VMware may lose, depending on what features you want and how hungry your reseller is.

The only thing about HBA's and TOE capable NICs is that they can disrupt routing and may not play nicely with your other software -- be sure to do a very thorough compatibility check before making a purchase.

ups - wrong thread

So I wonder if there is any worth into looking into the Linux open source project for iscsi vs. the dell 3000i storage units or even openfiler. I know they are not in the same class but I believe that each type has a specific niche. I see many of the folks here in this thread have mentioned Equal Logic SAN's but for environments where you have under 5 or 10 servers that may benefit from accessing a iscsi san of some sort, an example would be a vps hosting environment. Is the open source iscsi project worth it? Would you come close in performance with a RAID 10 setup? Obviously separate network for the SAN traffic is required and this can be made redundant as well. Would a setup with 2XgigE Ethernet uplinks to the SAN host or even iscsi toe cards work well? Hard Disk wise I am guessing that you would max out a 4-8 SATA array so SAS 15k RPM disks may be an alternative. It would be good to see some example configs without going into too much detail. It's really about what may work given the factors.

IOPS/gb is a very useful metric, along with IOPS/$ and $/gb. If you know how many IOPS/gb you need, you can more easily make good business decisions. SSD offers fantastic i/o/s/gb, as well as fantastic i/o/s/$, but terrible $/gb, so SSD makes sense if your i/o/s/gb needs are very very high, or your gb needs are very very low. On the other end of the spectrum, if your i/o/s/gb needs are 0, your best bet are 1.5tb or 2tb sata drives. They perform poorly on a per gb basis, but the price per gb can't be beat. 1TB caviar blacks offer much higher i/o/s/gb than the 1.5tb or 2tb drives, while offering almost as low a cost per gb, so this is the solution I've chosen for my servers. And then, SAS drives offer a much higher i/o/s/GB than SATA, because they have much fewer GB as well as being faster. Unfortunately, their i/o/s/$ is roughly the same as SATA, while the $/gb is dramatically higher. Before intel SSD, SAS was your only option if you needed high i/o/s/gb, and with 73gb 15k SAS being reasonably priced, they were a good option for high i/o/s/gb. These days however, SSD is king for i/o/s/gb and i/o/s/$, if you are not concerned about $/gb. If thin provisioning allows you to go from 300gb SAS drives to 147gb SAS drives, or from 147gb SAS to 73GB sas drives, that's helpful as the smaller drives cost less and perform the same. You'll still need the same number of disks, but those disks will cost less. If, however, you're using 1tb SATA drives, you probably can use all the i/o/s/gb you can get. Thin provisioning will just mean your disks will be emptier, you'll still need just as many of them to get the i/o/s that you need, as the drives smaller than 1TB are not dramatically cheaper and don't perform as well. The i/o/s/gb on a 500gb SATA drive is about the same as the i/o/s/GB on a 1TB drive, since the 1tb caviar black is 50-100% faster than smaller SATA drives. This means that the i/o/s/$ on the 500gb is about the same as the 1tb as well. So thin provisioning doesn't help you, as you're still better off using the 1TB drives regardless if they're 1/3 full, 2/3 full, or completely full. For archival storage / backups, well, you could probably get away with thin provisioning 2tb drives if you wanted to, as sequential read / write rates are what matter for doing backups. If your i/o/s/gb needs are very low, this makes sense. Overall, its an essential metric.

Ideally you would have a separate switch for each SAN, and then connect each server to both switches. In the inexpensive setup I was talking about, each physical server would see two separate LUNs (one on SAN A, and one on SAN B), and then mirror them with something as simple as LVM or DRBD. A single block device is then presented to the virtual machines. No data would be lost, since the data is written to both SANs. If there is a failure, the bad side would drop out of the mirror after a short timeout (say 500ms). Whitepapers are usually written by someone that has something to sell

It all depends. If using a QLogic, which I suggest, in an ISCS/virt layout, I would look into Infiniband. http://www.qlogic.com/Products/HPCNe...ptersHome.aspx They have the highest message rate, and I have sold them into many universities and hosting facilities that require low MPI latency and highest effective bandwidth, that enables MPI and TCP applications to scale to thousands of nodes.

Hehe, if you say so. I've heard otherwise from a pretty reliable source, and quite frankly it would not really make sense to do it any differently at the scale you guys are operating at. There's nothing wrong in scaling storage according to actual needs instead of a completely unrealistic theoretical max. It makes sense from a business perspective and ultimately benefits the customers as long as you are always able to deliver what you have sold IF the customer wishes to utilize it all. In other words, it was not meant in a negative manner at all. Similarly, a large consumer ISP cannot possibly handle max utilization from all their customers at once - because that kind of usage is completely unrealistic, and there is no sense in scaling for it. Is it wrong of them to do it this way? Of course not. Not as long as John Doe, the customer, is able to utilize his line to it's full capacity when he needs it. There's a huge difference between scaling resources to be cost effective while still delivering what you are promising, and 'overselling' with the consequence of customers suffering.

Like I said earlier in this thread, I don't do VPS so my perspective may be a bit skewed. But I really don't see any reason to go with Openfiler or a Linux-based iSCSI target unless it's for archive grade storage where you can afford outages. In my opinion the things that make iSCSI attractive are the features such as thin provisioning, remote replication (for DR purposes), snapshots (this does work in Openfiler), some degree of automatic storage tiering, LUN migration capabilities, redundant controllers, large caches and ease of management and ease of scaling. These are why I prefer EqualLogic for most scenarios. To me, if you're going to use Openfiler or open source you may as well just use JBOD with a run of the mill array controller and as many NICs as you can throw in with NFS or CIFS. And on the iSCSI initiator side, I do not recommend anything less than 2x1Gb NICs w/TOE or iSCSI HBAs, and they should either be in a bond or MPIO configuration. Again, these are just my opinions based on my experience. Others may disagree and your mileage may vary.

For that, it can be useful. But I don't see what it has anything to do with thin provisioning.

For memory over provisioning, doesn't that actually hurt performance? Reading over some papers on anandtech about how memory works in VM systems, it has to take a virtual memory request (as they're de-duplicating memory and oversubscribing it), map that onto the VM's memory space, and then map that again onto the host physical memory space. Because of the oversubscribing and de-duplicating, a given virtual memory page could be associated with any "VM physical" memory page, and any "VM physical" memory page could be associated with any "host physical" memory location, so it has to keep track of all these associations. I would think this would be unnecessary if you use a hypervisor that physically allocates memory instead of faking it, as you can allot 0MB-1024MB for VPS 1 = 1024MB-2048MB on the host system, as a static allocation, not having to do two stages of lookups for each memory access. My understanding might be lacking here, but it seems like you trade a lot of performance in order to oversubscribe ram this way.

I think the model we primarily use is the Intel PT 1000 NICs in dual and quad configs depending on how many NICs the server already has built in (newer Dells have 4). I've been told by Dell (guys in the labs and not sales) that dedicated ISCSI HBA's will get you ~ 2-5% better performance than the best off-load ethernet NICs. I take this on word and don't know it to be fact since I've never used them myself. For our environment it's not needed, especially since we are already presenting 4 GB NICs to each host. Our virtual clients are still mostly "web" environments with some SQL. If running highly loaded or dedicated SQL servers off ISCSI it might be worth the expense, but I think for 95% of implementations that users of this forum would have high quality ethernet NICs would be just fine. It would be cheap to buy a decent Intel or Broadcom and test your workload before investing in HBA devices. Good luck - Wayne

you calling me a lier? Seriously though...we do not oversell storage in VPS.NET. UK2Group obviously does it on some of the shared hosting brands like midphase and anhosting. But in VPS.NET there is no overselling going on. VPS.NET was made as a reaction to all the "bursting" going on in the normal VZ vps environments, and we decided from day one that we would never oversell CPU/RAM or Storage. Funny thing with a setup like VPS.NET is that we are actually forced to undersell by at least 30% to allow clients to scale up and down (and leave room for failovers) when needed without other clients being influenced by it. who ever that source is, he shouldn't be on your "reliable source" list anymore, because he is not D

We seem run fine without either of those. Run it in bonded connection. Even in one of our busiest VM (a mail server, serving about 200-300K mails/day), CPU load is still low. I am sure few % of the CPU is taken away because we don't use TOE, but as you said, if you struggle with few percent CPU, then you have a bigger problem to worry about. But out of curiousity, how much different will the TOE NIC give? How does it compare to iSCSI HBA?

It has everything to do with thin provisioning. As was said earlier, if you offer 10gb space, and you thick provision it, and your user needs 100 i/o/s with their 10gb space, you need 10 i/o/s/gb. If you do thin provisioning to offer the same 10gb space, but it only takes up 5gb of physical space on your disks, and still needs the same 100 i/o/s, you're now needing 20 i/o/s/gb. So, to get the same i/o/s/gb, you can buy smaller disks (if you're using SAS), or leave your existing disks half empty (if you're using SATA), or you could even go with SSD instead of SAS, if thin provisioning brings up your i/o/s/gb needs high enough.

I'll go with the common answer I've seen a lot in my short time here...it depends. For example, a virtual desktop type of implementation I have here. 40-50 virtual desktops on a host, each with 1-2GB memory allocated, and a total of 32GB available on the host. Well only about half of the virtual workstations are actually being accessed and doing something at the same time, but any combination of said systems may be active. Another example - web servers. On a host, half of them may be busy during US market hours, the other half during the EU's market day. All need to run 24x7, but all are not busy at the same time so it makes sense.

Good stuff here. I can only add my sentiments and share success and failure stories from my experience. We run several farms of ESX/vSphere servers, and use various iSCSI technologies for different things. On the low end, Openfiler works fine and we mainly use it for development and archive-grade storage (backups, templates, etc.). We use Lefthand (now HP) VSA's for applications that do not require high IO rates. Things like virtual desktop machines and static file servers work fine here. The beauty here is that you can use your old servers with JBOD and create decent sized pools of storage for relatively small amounts of money. We also use Lefthand/HP NSMs (I think they call them the P4500 SANs now). These have performed pretty well for our virtual environment. Today we have somewhere around 120 VMs (about half are web servers) running off a three node cluster in one of our datacenters. We connect the VMware servers to the SAN via 2-port Intel gig NICs. The servers are all in HA clusters on the VMware side. For our higher IO requirements we use EqualLogic SANs. These have worked extremely well for us, and are by far some of the easiest to implement and manage. We run some virtual machines on these as well as some SQL LUNs. One of the biggest lessons we've learned is to design your network right the first time around, and don't skimp on your design efforts. Your storage network should definitely be a dedicated VLAN, and unless you're running everything back to a pretty beefy core switch (Cisco 6500 or something along those lines), try to keep your initiators as close to your targets as possible. If you introduce a lot of hops between them performance will degrade, and you can easily saturate uplinks with high packet rates.

From my point of view (relating to our VPS product) you're missing being able to have a clustered, highly available VPS. It's just not possible when all the storage is local to one node. I see two basic VPS sales models and I'm over simplifying, but those who are highly available and those who are not. Local storage only goes so far and that's fine if you're targeting a specific VPS market. If you're not simply competing on price and offering a higher end feature set infrastructure needs can greatly shift and a SAN/NAS becomes a requirement. Wayne

So I agree with bennice .. but my question to you is what do you want to accomplish? How many money is it worth it to you to save 5 minutes or 2 days of downtime? I have installations with as little as 6 or 8 servers with a $100,000 SAN behind it. Because the servers virtualize and there are 100 VMs. These VMs run business critical processes and if the systems went down for three minutes it would cost money. The most important thing for these companies is their data and access to the data. They could care less if one of the 6 servers died. Are you running 5 servers that arent really making you much money and it is more of a hobby? Will you easily pay $10,000 to me right now if it meant I wouldnt flip the switch and your whole system would avoid 36 hours of downtime? Or would you rather keep the $10k in your pocket and spend the next 36 hours rebuilding your data from backups? The reason I ask is you state 5-10 servers but what are those servers doing and how important is it? 10 years ago I knew what it meant, but with virtualization today, for all I know you running the equivalent of 200 servers.

Right, you increase your IOPS by adding more spindles. But a side effect of adding more spindles is that you're adding more space, whether you want it or not. That means that depending on your I/O requirements, thin provisioning may not save space. It's not something that people think about much, but it does matter for many workloads. Funkywizard did a good job of saying other things that I wanted to say, so I'll leave it at this.

I'm not really talking about memory use, I'm talking about memory management overhead. Assuming that you have plenty of memory available with either method, doesn't oversubscribing memory have a lot of memory management overhead to eat up lots of cpu? With systems taking up to 72gb of reasonably priced ram, I'd rather not lose tons of cpu time.

I just don't see the point of building in this much complexity unless you have unpredictable per-node storage requirements, or unless you need more storage or i/o on a single node than you can easily provide on a single node. Having two 7k drives in raid 1 on a system is hardly expensive, so you stand no chance of saving money with a san unless your san allows you to use less than 2 dedicated disks per node anyway. Ethernet ports also aren't free. If you're doing 4x gbit ports per server, that's easily $100-200 in up front costs and 10 watts of power for each node just in switching gear, assuming a 1 tier network architecture. 2 or 3 tier network layouts would use more power and costs. It just doesn't seem to make sense to add a bunch of tcp processing overhead, network overhead, and additional costs, to end up with something with less performance than simply having the disks be on the server itself, along with creating several critical failure points. Am I missing something?

Hehe, no - I wouldn't go as far as calling you a liar (I'm sure you are a honest and decent person), but you are in a position where you have a strong interest in protecting a companys reputation. Thus I have to take what you say with a pinch of salt, I hope you understand. It would be rather naive of me to merely take your word for it. I fully support that a client should get what they pay for at all times, that is of course a very important principle. One does however not have to actually allocate space in real/set amounts to achieve this. I can't see how anyone can think that grossly over-scaling a hosts resource pool, to a level that is utterly unrealistic compared to real-world usage, will be beneficial to the clients. Higher costs are ultimately passed down to the clients in one way or the other, and massively underselling does not necessarily mean they get a better service. The *only* reason over-selling (or rather over-allocation) is a negative bound word with many, is because of hosts who do it without the appropriate resources and planning to maintain a well-performing service and delivering what's being promised. I would never expect this to apply to vps.net. Therefore I have to ask; Why on earth would you choose to use a bloated virtual disk format (allocating a set amount) instead of thin provisioning? What does that achieve / how does it make any sense for you to do this? Your smallest VPS comes with 10GB storage, and your signature claims 10,000 deployed in the past 6 months. That's close to a 100 terrabytes of storage as a very minimum in the same amount of time, so we are talking pretty large scale here. You'll have to forgive me for being slightly skeptical to the idea of provisioning enterprise grade storage at these levels without being reasonably conservative on space. For the record I think you guys have done a fantastic job on vps.net, and you absolutely have something to be very proud of. It seems like a rock solid product, and you have a lot of happy customers as a testimony. I'm not in any way trying to bash this. At the moment I'm stil inclined to believe that over what you've said, but if you care to shed some light on how you do things and why you do it that way - I'll be open to changing opinion.

TOE gives about 5%. HBA is significantly better (but costly). Let the CPU do the TOE is how I typically install. There is a price point and business case for everything and obviously HBA is better -- buys you improved performance, but no TOE is preferred in virtualization (Xen, in particular) ...you can run into some problems with TOE

Funkywizard, I'd caution you against using the wd 1tb blacks in raid arrays now. You can no longer set the TLER on like you could in the past. We've already seen a few colo'd boxes with these drop out of raid arrays and cause data loss. 3 drive raid 5 array with 1tb blacks... ugh. No problems with the RE models.

Again, in the capacity that we use virtualization the overhead is not an issue, as we're not so thin on resources that there is any noticeable impact on performance. The trade-off for potential, (but unnoticed) overhead vs. cramming as many virtual hosts on a physical piece of hardware is well worth it for us in this situation. And we're a firm that sees billions (with a B) of hits on our sites each month resulting in lots of happy customers with deep, deep pockets.

very nice setup are you using the 4500's for dedicated sql? welcome to wht btw - who are you?

oh, so you are calling me a lier ... pretty bold statement Mister... In any case, my initial point was that SAN/NAS overselling simply does not work in a sustainable cloud environment with persistent storage. Storage is not the bottleneck, IOPS are. And - in large scale - you would meet the IOPS limits far before you even think of overselling.

you let the CPU do the TOE? Why have TOE NIC in the first place then? And TOE NIC is to be avoided on XEN? Interesting. Never heard of it.

Gotcha. I was planning to use raid10 for some machines with these, and JBOD for others. Obviously it's no issue with Jbod, but since raid 10 can withstand multiple drives failing (depending which drives fail), would it still be a huge problem? The same way the RE's can do TLER (time limited error recovery) to make the raid not drop them from the raid set, isn't there a way to configure the raid controller to simply be more patient?

Well, if you're more constrained on memory use than cpu time, that makes sense. But looking at things, intel and vmware have had to create some fancy technology to speed up these memory lookups, which, if you get chips that support it, makes a huge performance difference. Which just leads me to wonder, if you're using a hypervisor that allocates memory 1:1, do you still see this performance hit on memory lookups? It's important to me as I plan to have more active VMs that are using more cpu, versus a bunch of idle VMs that benefit from memory de-duplication and oversubscribed ram.

Depending on your situation, you may be missing lots. For us, it was a no brainer. Our environment started off similar to what you describe, but once the strengths of virtualization were realized there was a huge push to convert physical servers. Today about 40% of our infrastructure runs virtualized (or roughly around 700 hosts) on around 50 VMware servers and 13 SAN nodes. The cost savings for rack space, power and cooling alone were plenty enough to justify this, but there are also benefits that go well beyond that. For database servers, you can attach lots more SAN storage than you ever could with direct attached storage (this goes for iSCSI as well as FC SAN), and if you need clustered systems shared storage is a prerequisite. It's also nice to be able to decouple your compute power from your storage subsystems. This way, you can use smaller form factor servers to drive your disks, which again, saves datacenter resources. This opens up all kinds of possibilities. As far as points of failure - well, this is up to the architect to educate him/herself about how to eliminate them. Every best practice document I've ever seen tells us to create redundancy at every point. In my experience, we've suffered host failures, switch failures and power failures but have never suffered an outage due to the infrastructure.

I think it's pretty ridiculous of you to expect me to just blindly accept your word for it. Who do you think you are? It's somehow fine for you to call my source unreliable, but it's not acceptable for me to raise reasonable questions regarding what you are claiming? Please. Talk about bold statement. I've never said storage size is the bottle neck. I've said I do not believe that having X number of spindle disks (typically SAS, which is very expensive) and a controller in each and every node is cost effective compared to thin provisioning on centralized storage when running large scale. That using centralized storage can allow more efficient storage utilization, as well as numerous other benefits (including potential performance gains with more spindles, better caching, beefier controllers, etc). Seeing as you have automatic fail-over of your virtual machines I assume you do utilize centralized storage, so why would you not use thin provisioning?

I would never even imagine taking a SAN and filling it with 1TB SATA drives, and I highly doubt vps.net is. Not to diverge onto vps.net again though. It was already well discussed earlier in the thread that with a SAN, you will almost always run into the IOPS bottleneck before the storage bottleneck. A massive SAN with 1TB is pointless. You will never use all that space because you'd end up only being able to have a couple VPS host nodes pointing to it. Using 15k SAS drives and not using thin provisioning is still completely practical. Sure SAS costs more, but storage in general is still cheap in the long run. At scale, a large SAS SAN would still be more cost effective than local storage. If I was looking to offer a high performance solution, I wouldn't even consider using SATA for local storage, yet alone a SAN.

I've not come across a method to accomplish this, perhaps because I've not dug too deeply - we use the RE drives for our servers, only a few clients use the blacks. I do know that the drives will drop from both mdraid and adaptec controllers.

Thanks! I've spent the better part of the past two years designing and implementing these things, and we are quite pleased with the results so far. I'm nobody really. I'm just another guy working in an under-appreciated IT dept. for a web hosting firm. I just found this place while Googling to find some info about what people are using for iSCSI switch fabric setups. My environment is just exploding with growth, and I'm always looking for the most efficient ways to keep it scaling out nicely. Oh, for SQL I mostly use the EqualLogic SANs. The Lefthand stuff is mostly for VMware right now, but this may change at some point depending on which vendor is offering what. But any SAN we use for SQL will always be dedicated. That is, any particular SAN node will run virtual machines or database files - not both.

So do we. We donot oversell our storage and no restriction on how clients choose to use their storage. But we do thin-provision our servers. Thin provisioning has nothing to do with overselling. Overselling is promising something you cannot physically deliver. Thin provisioning is simply better management of your storage space.

Nah, you do it right and you can get 200TB (100tb usable) for a lot less than $250k. The disks themselves only cost $20k.

Thanks for the information.

I'm curious what hardware configuration (specifically disk spindle count and raid level) you are using under your VSA nodes. We've been experimenting with various configurations to produce the 10TB maximal capacity the VSA's can handle. Be it 8,16,24 spindles in 0,50,10. As well as testing the impact of the Adaptec MaxIQ cache accelerators. Curious as to see what others findings may be in regards to how the underlying raid set impacts your iops.

Exactly.

It depends on your business model. If you're running an ad supported website, chances are the hosting is running you 10-80% of your revenue. Doubling (or more) your hosting costs in that situation would be very painful, and you're not going with a host that charges 50% more, let alone double or more, even if the quality is better. My website is not a hobby by any means, but that doesn't mean I can afford to double my hosting costs. It just so happens that hosting costs make up 3/4 of my entire budget. Show me a business who can afford to double the cost of an item that takes up 3/4 of their budget and I'll show you a business that is about to see a lot of price lowering competition. For enterprise applications, the IT costs are a small part of the operating expenses, and the IT is critical to making sure all your expensive employees are able to do their job. And in other cases, you're doing ecommerce. Again, hosting costs should be a very small part of your cost structure. Having the website down while you're paying for advertising to drive traffic to your site would be a calamity. So you have to know your audience. The enterprise / ecommerce people are going to come up with their own solutions, or be willing to pay what it costs. Most people in the ad supported website space view hosting as a commodity, not unlike oil drilling. If the price of oil is $50 and it costs you $40 to drill, you keep drilling. If the price goes down to $30, or the cost goes up to $60, you cease your operations. Doubling your costs would put you out of business just as fast as a server outage, so you put up with the chance of downtime here and there. If you've got a sufficient budget, you pay for backups, or get raid, or both, but you're still not deploying a $100k san for 5 servers.

Very interesting thread, even though i don't know much about SAN's; What does IOPS mean in real world? Ie, can you change 100 IOPS to MB/s? As i'm thinking of getting a MD3000i, but i'm wondering how much MB/s i can get for read/write with 15x1TB 7.2k Drives or 15*1TB 7.2k SAS drives, even a average will suffice.

It varies, and depends on the purpose. A typical VSA we have here is driven by something along the lines of a Dell 1950 server with pretty much any CPU, 4GB RAM (good place to put all those 512MB sticks that are useless elsewhere), a PERC card of some flavor with some BBC, and MD1000 arrays full of drives of the same speed and capacity. Another common config we have, which looks a lot like a P4500, is an HP DL320s or DL180/185 full of drives and a P400 controller with 512MB BBC. We've been playing with different array configurations to see what yields the best performance, but typically we'll carve up RAID 5 or 6 virtual disks at the array controller level across all disks in the array. Inside the VSA appliance itself, you're fine as long as each slice is 2TB or less, and there are a maximum of 5 slices attached to the VSA virtual machine. Some interesting things about the VSA - you don't get jumbo frame support (yet), and there is no way that I know of for the VSA to utilize more than one vNIC. For jumbo frames to work, Lefthand needs to update the VSA to use the VMXNET drivers. The single vNIC limitation isn't a big deal if you configure LAGs on your ESX/vSphere host and at the switch. I have no experience with the Adaptec accelerators at this point, but may look into something like that. The big thing is to assess whether or not it's cost effective since those will add cost to the overall configuration.

Sorry - I may be overreacting, I am just not used to be called a lier on a public forum. It may not be a big deal to you, but to me it is quite a serious attack to my business integrity. You went on a public forum and said we were thin provisioning our storage, I am saying we are not (I pay the bills, so I should know), and you say you do not believe me. I am not sure how I should react?! Sit back and say OK? not saying that your source is a unreliable, he might just not know better. Who knows, perhaps his source is unreliable? My source is sort of first hand, not sure how yours could be better I am not going into the whole "is overselling good or bad" discussion here - all I am saying is that on VPS.NET we are not doing it - and then leave it up to potential clients to decide if they appreciate that or not D

I would disagree. The performance and price is best on the 1tb caviar blacks. You can do about 160 i/o/s for around $100, and get 1tb of space. Certainly for many use cases, you want more speed per gb, but even then, the 15k sas drives that give you double the performance cost 2-4 times as much, use twice the power, and have very little storage space. So worst case, you leave a bunch of 1TB drives nearly empty, but it's still better for dollars per performance than SAS, and you get extra storage capacity thrown in for free. That gives you the flexibility to duplicate all your data for backups as well as offer a good per-gb price for massive amounts of data that is primarily accessed sequentially (like video). The only downside is you double your drive count, which requires more ports on your disk controller and more space in your rack. If you really need tons of i/o/s per gb of data, you should go with SSD anyway. If you store all your small files on SSD and all your large files on 1tb SATA, you get the best streaming performance (sata) along with great random performance (ssd), while maximizing both performance per dollar and gb per dollar. I believe ZFS can do this transparently for you if you have some SSDs available on the system.

So how reliable is the mirrored SANs and what is the availability like? In other words, your mirrored SANs, can you multipath to both SANs for the same LUN? and what is the risk of data loss? certainly i would think there is some data loss if you happen to be writing to the failed SAN at the nanosecond of its failure? Are there any whitepapers or documentation on how this all works? I obviously can see the benefit of someone building an off-the-shelf SAN if you can achieve full redundancy and reliability ..

I could see if you needed something tuned for databases, vs storage tuned for something else, on the same VPS server, it would be easier to have a portion of a SAN tuned for one, and another portion of the SAN tuned for another, as that way you don't have to worry about where you run your mysql vps vs your everything else vps. That makes sense. I was thinking along the lines of having VPS servers, so virtualization makes sense to me here too. But even in 1u, I can get a 2.8ghz quad xeon, 4x 3.5" disks and 1x 2.5" ssd. I could even go with dual socket processors if I wanted to, but in my research the e3460 xeon had similar performance clock for clock as the e5520 xeon, while having much greater clock speed for the price. Single socket motherboards are also cheaper, so overall for the same processing power you pay less money on the single socket. Also, unless you can cram more than 40amps into a 40u rack, 1 cpu socket and 4 disks per u makes the most sense. I guess if I had no disks in my VPS server, that would drop off 30-40 watts of usage, and then I could add (half of) another processor socket, but then I'll just have some 4u servers with tons of disks and only one cpu socket to even things out. Either way you can still only get the same number of disks and cpus into 40amps of power, regardless of which servers they're in. I will admit that using 1u cases that support 1 drive will save you like $200 versus getting cases that support 4 drives, but that cost savings goes away once you need a hard disk controller card in each SAN box, a dedicated storage network for your SAN servers, and more expensive / higher port count network cards in all your servers. In a virtualized environment, network i/o takes up even more cpu time than it does on dedicated servers, so the last thing I'd want to do is create several gbit / s of needless network traffic between my vps server and my vps server's storage pool. The performance hit to your cpu is pretty significant there without virtualization, I can only imagine how much of your compute nodes are going to waste when they spend all their time doing tcp calculations in the hypervisor.

agreed - good explanation

So you are saying thin provisioning generates significantly more I/O than using a 1:1 allocation? I might be missing something, but I've yet to see this to be the case.. Own testing as well as for an example this paper by vmware (pdf) shows very similar performance between thin/thick provisioning. Please do correct me if I am wrong. In addition you have the aspect of replicating and backing up data, where less is obviously better.

distributed architecture sans are incredibly reliable especially the new ones that are virtualized like the left hand VSA's that virtualize the SAN OS and run across many boxes. You can architect them so that they are distributed physically through different switching fabrics / power feeds and geographic locations in your data center for complete redundancy in your facility and then they network raid throughout all the units. This allows you to lose an entire unit or lose individual disks without data loss. there may be some performance degredation depending on how many units you have deployed but its a really nice setup. I personally think its the only way to go and very congruent with the coming virtualization trend. I hear the new VSA's fuly support SSD's currently but there is a question as to whether or not they will perform better atm based on current limitations than a properly configured 15k sas raid - I will let you know in a month - I am testing them. Not sure if the expense will be worth it though since its double the price.

I think it's a bad idea too but possibly for a different reason. If one VM store goes down, however seamless the redundancy, you'll have many many angry customers. Especially if the filer is shared amongst many hardware nodes. Yes, that's a trade off but not a fair one because the beneficiary is the hosting firm, due to much needed cost consolidation, whereas the disadvantage is to the customer(s) who experiences disruption. Regards Joe

Asking questions by what people claim is only healthy, and I have provided reasonable explanation to why I chose to do so. If you still choose to see past the compliments I have given you, and my reasoning behind asking these questions, and instead take offense just because I do not blindly accept your words as fact.. That is indeed being oversensitive / overreacting, if you ask me. There's no reason to why your word, a person I do not know, should be better than someone elses just because you sign the bills for this company. On the contrary, it gives you a lot of potential alternate motives to paint a favorable image of the company. I'm not saying or suggesting that this is what you are trying to do here, but it gives others all the more reason to ask for further explanation if you make a statement that conflicts with their own information on a matter concerning the company. That is what you need to understand, and accept. Avoiding questions and getting caught up in the fact that they are being asked will do nothing but raise more. That's absolutely what you said, but I do not care. I merely used it to highlight the hypocrisy of your statement there, and how silly it is to get all sensitive over having some pretty reasonable questions asked about the technical aspect of your business. But anyway, if my questions and belief in my own sources and knowledge (in lack of decent information/reasoning suggesting otherwise from yourself) have somehow deeply offended you and your integrity - I do sincerely apologize. I thought this was a place where people could discuss matters such as these at a mature level without taking it as a personal insult. I did not intend to be rude.

Yes, CPU does TOE. Disable TOE on NIC. Specifically there are issues with Citrix XenServer. I do not know if it is "XEN" or Citrix. But if you surf their forums you will see many problems that people have encountered exporting VMs or moving massive amounts of data and the fix was to disable off-loading. Since TOE does not give much performance benefit, I say skip the headache and disable it (or better yet, never configure the hardware with it). Problem has existed in version 4.0.1,4.1,5.0, and now 5.5 of xenserver.

You'll love my answer... "it depends". 1. What are you planning to do with the SAN (virtualization)? 2. What's the planned workload (web servers, SQL servers, mail)? 2. What's the typical data pattern (random, seq. read, etc)? 3. What RAID level do you wish to run? I've made a few other posts about the MD3000i -- I think it's a great entry level SAN as long as you fully understand its limitations. We still are very happy with the purchase. We use it for an 8 node Hyper-V cluster running ~ 120 machines in production on 450GB SAS disks in RAID-6. Performance is respectable and never have had any client issues or complaints about it. Our typical workload is a web server along with some light SQL and mail. The typical client is someone who'd have purchased a $100-150/month dedicated server and that's the level of performance they expect. There are some real pain points with the unit, but it is an entry level SAN, so you get what you pay for. My personal issues are: 1. Management console is very easy to use, but has next to nothing in terms of performance reporting. You have to use CLI which exports out a CSV file in which you can bring it into Excel and create your own graphs. Absolutely nothing "real time" and just the slice of time for which you ran the manual query. 2. Resizing LUNs (volumes) is painful. Done through CLI only and operations can take 24-48 hours in our case. Plan your volumes ahead as much as possible to avoid this. Thankfully I don't have to do it much. 3. Each volume is bound to a single controller interface, so although you have multiple interfaces you'll never have more than 1GB/s to a single volume regardless of RAID or aggregate drive performance. Considering how we are using the SAN and presenting it to each node (2x 1GB NICs -- each VM has it's own volume) this hasn't been an issue. We're at space capacity with our current number of machines and where I feel comfortable with performance. Although up to two MD1000's can be chained for further capacity I wouldn't do that in my situation. 4. Deleted volumes don't automatically free that space into the larger available unused storage pool. Assume you have 300GB free as a whole. If you delete a 50GB volume that sits in between two 30GB volumes, that 50GB doesn't go back into 300GB available, but rather sits there as a 50GB available block of space. You can use it, but only up to 50GB. You can't combine it with other free space available. There is a method to pool all the available space together, but it's another CLI operation and it can take days to run depending on your capacity and I/O load. I've had one issue in the almost year we've had it where I'm confident we just pushed the box too hard under abnormal circumstances. We had a situation where one controller missed a heatbeat request from the other (due to being overloaded) which cause all the volumes to "flip" over to the other controller. All of these machines subsequently rebooted, but no data loss occurred and about 60 seconds of outage. I would consider this rare due to the circumstances and it's never occurred again since. Overall I have no regrets about the purchase and it was an excellent entry into the SAN world. I expect it to have a long life with us and will be certainly re-purposed into other areas as time progresses. We've recently made the jump into the Equallogic line, but would purchase another MD3000i if it fit the purpose. I've been told by a Dell SE that an upgraded MD3000i is being released this year, so you might want to wait it out a bit. Nothing was confirmed for features, but I think thin provisioning is high on their list along some of the other typical "enterprise" level features. Good luck - Wayne

I think what you're saying has merit in terms of providers who slap openfiler on a node and call it a SAN, but you're over-generalizing. If the provider has a redundant HSRP storage network with multiple switches, and a SAN with multiple power supplies and multiple controllers, I really think it's laughable to say that that's a riskier configuration than host-based storage.

I see no other option than to react like I did. It's just like publicly accusing Coco Cola of actually putting sugar in their Diet Coke. And if they say "no, we do not" then your answer would be "ok, then give us your secret recipe"? ... and if Coke refuses to hand you their business secrets, it's proof to you that they actually do fill their Diet Coke's with sugar? But alright, yes - fair enough - let's not discuss this anymore. It did take this good thread off-topic, and I will stay out of it D

1TB SATA drives in a SAN are the best IOPS for the dollar. Now SAS drives are expensive. But I guess are we talking off the shelf SAS or enterprise SAN SAS? The list price on a 450GB SAS 15k from EMC is $2499 for one drive. You cant use off the shelf drives with EMC. Even if you can work out a 40% discount you are still talking big money for those iops. 1TB SATA are best bang for the dollar no matter how you slice it. You get half the IOPS of 15k SAS for a fraction of the money. But you are right, maybe we dont need 1TB, we might get away with 500GB drives or even smaller ones.

Thanks for the great answer. It really does help! I think i'll wait for the upgraded MD3000i and see if i can find any more information on it later on, or i might just jump to a Equallogic. btw: I'd be hosting live Xen images on the SAN, nothing which eats up too much I/O but still want something reasonable. I'd be looking to run RAID 10 or maybe RAID 5/6 Depending on how much I/O the SAN can give. As for the typical data pattern, I'm not too sure, prob a bit of this and that! Thanks again for the great information!

Virtualization (server and storage) is not for everybody. This hosting industry will move towards virtualization more and more in the coming years. And SAN will become more common. The benefit of virtualization and SAN far exceeds what you described above. Why would you tie your servers down to your local drives with non-redundant raid controller? What happens if your server is having hardware problem. How soon can you replace your server and how much downtime will be involved? Besides, what do you do with the all the waste of the unused space on your local drives? Welcome to virtualization. Offer much better uptime to your customers. Thin provision your storage. Protect your data with snapshots. Do live migration of your servers. Replicate your data to another SAN. Scale your storage throughput. Simply offer better service to your custome. BTW, those who are serious about SAN will not put the iSCSI traffic with network traffic together. And, btw, I don't see any issue about CPU doing TCP calculation. If you have concer about this get TOE enabled NIC cards or a iSCSI HBA.

Hehe, I hardly accused vps.net of anything obscene or asked you to reveal anything on the scale of a secret recipe - so I don't think that's a particularly good analogy. It's fair enough that you choose not to answer my questions, that's entirely up to you, but you have absolutely no reason to take offense just because I ask you to back up or explain your statements.

Still not sure what space has anything to do with IOPS. Just because you host 1 TB of data does not mean the all bits and bytes of that 1 TB data is being accessed all the time. We have EqualLogic PS5500E with 48x1 TB SATA drives (in just 4U!). The massive number of spindles makes these SATA array comparable to sixteen 15K SAS drives at the worst case scneario (100% random IOs), giving us the best balance of space and IOPS.

All questions I've been wondering about Open source iSCSI projects. DRBD doesn't even do all of that, and 2 years ago it was a royal PITA to set up. And back then it was a crapshoot as to whether or not your failover node was actually working. Granted, I haven't tried doing HA with Openfiler or DRBD since the 2.2 release. From what I've been told, DRBD and HA work better in the 2.3 release, but it's really more trouble than its worth with Lefthand VSA licenses being as cheap as they are these days.