Storage demands for personal and professional use have grown quickly in recent years. Take a video editing workstation, for example, where large quantities of raw footage must be saved, or a home media server storing high-resolution movies and shows for on-demand entertainment.
SAS expanders are a cost-effective plug-and-play solution for your home server or PC, whether you want to add many disks to a server or SAS drives to a consumer motherboard.
In this article, we’ll be going over what a SAS expander is, how it works, and how you can use it to add more storage to your system.
What is a SAS expander?
You can think of a SAS expander as a simple unmanaged network switch for storage devices. Like a network switch that takes a connection and distributes its bandwidth downstream, a SAS expander takes a SAS port and expands it to multiple SAS connections.
In our article comparing SAS expanders with HBAs, a key takeaway was that a SAS expander multiplies the available ports on an HBA, letting you populate it with the maximum number of drives the HBA can support.
The main benefit of a SAS expander is its ability to expand a server or PC’s storage capacity without needing to invest in expensive HBAs that take up PCIe expansion slots. Note that an HBA+SAS Expander setup might not always be ideal if you can find an HBA with enough connectors that are both cheaper and easier to install.
How do SAS expanders work?
SAS expanders present your drives directly to the upstream adapter, which may be an HBA or motherboard’s SAS port. A single cable connects the SAS expander to an HBA, but additional connections can increase bandwidth.
SAS expanders also can be daisy-chained to each other up to a theoretical maximum of SAS devices specified by the manufacturer of the SAS expander and its upstream device. However, as you keep adding expanders, you reduce the per-drive bandwidth, which can be inadequate for faster storage media like SSDs.
Do SAS expanders reduce drive bandwidth?
A SAS-2 interface supports a maximum theoretical bandwidth of 6 Gbps. As you add more expanders downstream, this bandwidth is divided by the number of simultaneous connections.
We’ve stressed simultaneous connections here because even if you have 128 drives connected to your motherboard through a network of HBAs and SAS expanders if only a single drive is being actively written to, the drive should have access to the complete 6Gbps bandwidth.
If a series of operations are being actively performed on multiple drives, you will see a reduction in bandwidth. With the fastest hard disks only capable of 524 MBps (around 4Gbps), you won’t run into issues with bandwidth while using SAS expanders with HDDs. SSDs have much faster transfer rates that completely populate the 6Gbps SAS-2 bandwidth, so we do not recommend using them with expanders.
Internal vs. External SAS Expanders
Based on the connection to the HBA, SAS Expanders can be classified as either external or internal. Apart from a difference in connection, both types are identical in form and function.
An internal SAS expander is helpful in scenarios where it resides with the HBA in the same chassis. It features only a single type of SFF-8087 or similar internal port, one or two of which needs to be wired to the HBA.
If you plan on adding drives to another chassis and wiring it to your primary server, an external SAS expander is what you need. It has an SFF-8088 or equivalent external port that can receive an external SAS cable from the HBA in another chassis.
External SAS Expanders are also helpful in wiring redundant SAS arrays.
Internal SAS expanders can also be used in an external chassis, provided an adapter can convert the external SFF-8088 cable to an internal SFF-8087 one compatible with the SAS expander.
How to power a SAS Expander?
Most SAS expanders are powered through PCIe, usually through the PCIe slots in your server or PC’s motherboard. Their PCIe pins are for power only and do not transfer data. Some SAS expanders also come with a Molex connector allowing you to run them off your power supply.
If your motherboard lacks sufficient PCIe slots for a SAS expander or you want to run one-off in a separate chassis, we have an entire guide dedicated to using a SAS expander without a motherboard.
SAS Expander pricing and availability
SAS expanders can be found pretty cheap in the used or refurbished market. The Intel RES2SV240 is a popular Internal SAS-2 Expander with six SFF-8087 connectors supporting up to 24 devices. This external HP SAS Expander is also a good choice if you run your SAS expander in a separate chassis.
SAS Expander Installation
Installation of an expander is pretty straightforward, but each setup is different, especially if you are planning on utilizing hardware RAID on your HBA. So, it is good practice to consult the manufacturer’s instructions before installation.
The general steps for installation can be summarized as follows:
- Identify where you need to install your SAS expander. Most expanders have PCIe pins for power, so you may have to install them on your motherboard.
- Once installed, connect the expander to the upstream device, which may be an HBA or your motherboard’s inbuilt SAS ports. Use the appropriate cable with suitable connectors on both ends. Generally, a single SAS cable is needed, but you may need to connect two for higher bandwidth.
- Connect the downstream devices. According to your requirements, it may be a drive, another SAS expander, DAS, etc. Ensure suitable cables with the correct connectors are used. Consult manufacturer documentation for the maximum number of devices that can be connected.
- When you start your server or PC, check if your operating system has detected the devices connected to the expander.
- If you face any issues, double-check the connections and try with only a single device downstream of the SAS expander. Keep adding devices to isolate the problematic ones and check them for issues and compatibility with your expander and upstream devices.
SAS Expanders are plug-and-play devices that work with any server or PC, irrespective of their manufacturer. By saving you from purchasing multiple HBAs that can take up expansion slots and cost extra, they are a must-have for those looking to build a high-capacity storage server or PC.
If you found this article helpful, feel free to share it around. Please let us know in the comments how you plan to use a SAS expander in your storage project. We will guide you through any issues, so post them below.
Comments
Hi again. Just a quick question – What would happen if you have 2 HBA cards connected to one expander card, say one 4-port cable from each, and the remaining ports of the expander connected to drives?
I’m just curious about what would happen.
Still not updated on the homepage btw, still says “0 COMMENTS”
A SAS expander can be connected to two HBAs if you seek redunancy so that one HBA remains up if it fails. I’ve refreshed the cache so the comments should be visible…
Ok, I take it there’s some heavy configuration behind setting that up.
I’m not very interested in using RAID in itself much, as it requires spare drives and you should basically RAID your RAID for extra redundancy to prevent data loss.
JBOD is more my melody and using StableBit DrivePool to make it into one drive. It has its drawbacks alright. Data loss can occur, and data can be stored uneven on disks, and of course no redundancy unless set up, but then you can just as well use RAID in my opinion =0/
But, who knows – If I ever become a millionaire, I’ll buy hundreds of drives, controllers, expanders, cables of every kind & number and experiment until I drop.
But for now, I’ll just start with what I ordered and do some experiments and I suspect that I will buy more of these. At least that expander seem very interesting with its Plug ‘n Play configuration ability.
Thanks for the info.
Hi again. I have another almost random question:
What happens if you connect two different computer’s RAID-cards to the same expander?
Btw, turns out that my Windows-installation was the culprit for the homepage not working properly.
Also, bought another RAID controller off eBay (Adaptec ASR-8885) that I’m eager to try out, also still waiting for some other stuff so that I can build a computer and dedicate the disks to my cabinet. Right now I have everything in it and it’s a bit crowded and is having a bit of a heat-problem at times.
Hopefully by next week things will start coming by mail.
Hey there! The configuration you’re describing is known as a multi-host system. SAS expanders will support this feature, but compatibility with SATA drives may not be guaranteed due to their half-duplex link. You can use a SAS interposer to make SATA drives appear as SAS ones.
Ok, so with these SAS imposers, any disks will appear on both computers then?
I have noticed their use in the Direct Attach Storage arrays as mentioned before. Each expander will have two SAS expander modules corresponding to chanel A and B that sepeately can connect to different hosts. So to use two DAS arrays between twi hosts, you would need to set up cabling as shown here: https://docs.oracle.com/cd/E19295-01/820-3223-14/SASMultipath.html
For a regular SAS expander the problem is connecting a SAS drive or SATA + Interposer to two SAS expanders. You would need a specialized bacplane that offers two SAS ports per drive to do thar.
As for a Single SAS expander to two hosts, I am unsure if any support this feature and even if they do, the OS needs to support it too.
Wow… Remind me never to use any of Sun’s products, or Linux on ditto for that matter. Mildly confusing and from zero to complex in 10 seconds, then it leveled out when the part with Windows was explained.
Interesting. It would be fun to explore such stuff if I had spare HBAs, cables and disks lying around, but I’ll bookmark this page and perhaps in the future I’ll get back and ponder on it a bit or two. Thanks for the link
Hi again. Still waiting for the last of the stuff to arrive so I can separate my drives from the computer, so while I’m waiting, I was fiddling around with Eagle CAD a bit and made a schematic with own symbols so that I could connect wires to make a diagram over what I was asking about earlier; Topology for how to connect multiple disks in the best way possible
The image shows red blocks which i supposed to be the Adaptec AEC-82885T/36-Port expanders, and it should be self-explanatory. “A” and “B” refers to the connection to the RAID card with two 4-ports connectors for maximum bandwidth which is the whole idea with this concept to ensure that it’s even throughout the ‘network’.
As the end-goal is to make it 320 drives in total, this schematic is only showing half of it, meaning that this is for one SATA RAID controller since it can only handle 256 devices in total, so 160 drives, and I’m guessing that expanders counts as a device too, just like an USB device and how they’re counted and taking up placeholders, so to speak.
This topology sure uses a lot of cables, so I wonder if this ‘constellation’ would be beneficial at all? After all, cables aren’t cheap. Just paid about $30 for 4 60 cm/2 ft cables, and the external cables took the cake with almost 50 bucks for 2 m/6.6 ft.
Luckily, the latter doesn’t require a lot of these so it might be worth the cost.
The SAS to SATA cables aren’t cheap either, but they can be found at resonable prices at eBay, Amazon or Ali Express to a fair price for decent quality.
Cabling can be expensive, especially at the scale you’re working at. I would suggest taking advantage of volume pricing with any vendor you can find…
Not sure if you were able to see the image, so I’ll put a plain link here:
https://postimg.cc/sBHX7NqG
I added it as an embedded link in the text “Topology for how to connect multiple disks in the best way possible” but this page is a bit strange and still doesn’t update contents properly (Visiting this URL doesn’t show the latest posts still)
The illustration is excellent! From the looks of it, you are pulling four connections out of each SAS expander connected to each drive. It’s different from the usual daisy-chaining process I’ve seen elsewhere, and I am curious to see how it plays out.
I feel this setup is pretty redundant because it will work even if two of your cables fail. I’ve cleared the cache so the comments should be visible now…
Yes, you and me both.
First I must accumulate some funds so that I can start purchasing more of these.
As for my “schematic”: It does look like it should work, if indeed these expanders are truly plug & play and thus finding/negotiating paths automatically (I’m still baffled how these work, like Windows finding these as a such device, yet they’re transparent, have no configurations that can be made either via drivers/utility nor on-board), so I think that this multi-pathway setup should ensure sufficient bandwidth throughout.
These two leftover 4-ports that are unused can maybe be used for further redundancy with either another RAID-card or computer. But I don’t think (nor do I actually know) that using additionally two ports on the same RAID-card would do much. This with 8 PCIe lanes in mind and the ditto X8 slot and thinking that theoretically, each lane could be occupied by one disk, so 8 simultaneous transfers at the same time, so using additional 8 lanes would probably do more harm, as I suspect that they would share the bandwidth.
Meaning, 8 concurrent transfers + another would then share bandwidth with one of the lanes or how many others that are initiated.
That’s about how I imagine it’s working, and I lack the resources to confirm this “statement” of mine, so I leave it as a ‘theory’.
Although, in my case, 6Gbit/sec is quite fast, and I intend to use SSDs in my build, I suspect that I’d, perhaps, find that somewhat of a bottleneck if a running disk-to-disk operation suddenly were impeded by another disk-to-disk transfer.
Not that I plan on doing any such multi-transfers, but you never know.
So, with this wiring and it being half of 360 drives, and 8 PCIe lanes, it would mean 16 lanes in total at the same time. This of course means that the board would have to be able to handle it too.
Don’t think my current setup would, considering that it’s only PCIe v2.0 and quite old – But, at least it runs with 152GB of DDR3 ECC RAM, and dual 16-core CPU at 2.4GHz (Max 3.1 GHz) so at least it runs Windows 10 without any problems, or, almost that is…
I’ll upgrade somewhere along the way to something better I’m sure, but I’m hooked on dual-CPU, so that’s most likely what I’ll continue using since it means more lanes, thus better bandwidth =0P
But, for now, I’ll use my existing 2 expanders, my “new” Adaptec RAID 8885 with 10 x 4-port SAS to SATA cables, 4 x 4-port SAS internal cables; External to internal port adapter (which will be mounted in the locker. Two for the adapter, two between the expanders.
Then the 2 x 2m/6.6ft to the server around the corner and hopefully it’ll be a bit more quiet and faster, and I won’t have to use regular cheap SATA controllers ever again.
A small discovery, of sorts that I made:
I spent probably 10-12 hours yesterday moving PSUs, assembling the new chassis for the PC. At first nothing happened, and I was worried that I had broken something by reversing the cables (surprisingly easy to make the external plug fit the wrong way, even without force!) – Turns out the design of the chassis is a bit, well – Bad. There’s not enough room for the connector to go all the way.
The slots are recessed from the back and there’s not enough space between the last slot and the backside. Perhaps the connector itself could’ve been straight instead of beveled.
Anywhoo, I had to move the RAID-card to slot #6 and suddenly I saw drives.
I shut it down and started connecting the way I wanted it.
Naturally, when it didn’t find any first, I tried ripping out all cables, and connecting in several different ways.
Now, I used the two external connectors in one expander, then two internal to external on the second expander, just like I illustrated.
When I fired up the PC, I got some weird error. Something about firmware error, and kernel panic, and I’m guessing that either it simply doesn’t work with two cables at the same time – Or, one of the expanders have some type of patched or modified firmware.
What was weird, is that, cited from memory, of the ports on the PCB, left to right; First and last are dedicated for expanders, and not for drives. The last one for sure.
This I read from the manual that said that up to 24 disks can be connected, in total.
This was from a revised manual.
First I downloaded said up to 20 and the rest of the ports were for expanders only.
However, the last expander I bought came with 7 x SAS to SATA-cables. I asked the seller about this, also mentioning what I read from the manual. They didn’t respond.
What I was going with this was that the last port does indeed work with drives.
At first I didn’t get all drives, and since I’m using a custom back plane and not all slots are populated, I was a bit confused. I did color code the cables but I didn’t care about this as first, but I can easily sort these out, but I need to know the exact order of these ports if I was to make sense out of it.
Ports 1 and 2 are unused, and seem to be the ones that are used for cascading expanders – Makes sense too, that they’d be located in front since most regular builds have disks in that direction and needs accessibility and so on.
I haven’t tested all ports as I’m currently in need of my disks, and right now I’m just glad that it works.
I will try more later on.
At least I would hope that one more internal cable between them should work, but who knows..?
If it doesn’t my schematic is out the window.
And another update, so disregard what I just wrote above:
I decided to upgrade the firmware, some version “b052” was installed on both of them, and I’m guessing that they’ve never been flashed, which would almost explain why some ports aren’t working as they’re supposed to.
However, after flashing I decided to try out my topology again, so I just hooked up two cables in the vacant ports, and this time it didn’t give me any firmware error followed by a Kernel panik.
It listed a whole list of odd numbers, briefly, then I was presented with the Ctrl+A and upon approving the list (Stupid that you have to approve it before you can clearly see if it’s what you wanted!), and I saw a bunch of drives, but three were missing, so I powered down and this time I moved the connector I had in the last receptacle from the left – The very same that was described in the manual as for expanders only.
Earlier I checked the manual again, and it clearly states that up to 28 drives are supported, so someone over at Adaptec or Microchip clearly didn’t check with the other editors of the manual.
I powered it back up again, and all drives were found, but I had to do the same with the other expander too.
So, now it works exactly like I had hoped it would, now I only hope that they utilize the entire 8 lanes.
As you mentioned, the internal SAS port labeled I on the 82885T doesn’t support direct attach storage drives and can only connect to another expander or controller. The official documentation online states that 24 drives are supported, so the manual may have been incorrect.
Sorry, been having some issues, but not with this setup. It’s been running next to perfectly.
No disks that are falling out like before.
But, this problem’s been with my other computer.
Too long to explain what’s not relevant to this topic, so I leave it alone.
Wow… Just checked that page – They really know how to confuse their users.
On that page you linked, it says (but you have to click on “Read more” to see it: “*Use of 8 internal ports for connectivity to RAID Adapter or HBA reduces the maximum number of ports available for drive connectivity to 20.
Can’t say that I saw this in the manual though, or, it’s hidden somewhere.
Just found something else in the manual that is equally confusing, section “3.2 Major Components”:
• 36-Port 12 Gb/s SAS-3 Expander Chip
• Provides 36 PHYs
Any PHYs may be combined into wide port(s) ••
Any PHY can be SAS or SATA attached.
These guys are all over the manual tossing numbers left & right, and neither makes sense, until someone outside (the manual) explains them further. In this case, that little info about using 8 ports could perhaps toss some light on that weird port config that I experienced, but it doesn’t explain why that firmware enabled the 8 ports in the first place…
I’ve always liked Adaptec’s stuff, but they really suck at support stuff.
I recently tried contacting them, but ended up using some form.
Only… The next day, I got an email from Google (Using gmail), saying that they’re unable to contact webfeedback1@adaptec.com.
Naturally, I figured perhaps there’s someone making a change without checking with his coworkers, so I sent a manual one to webfeedback@adaptec.com instead – Nope, took 2 days until I got the same crap.
Apparently, they either are oblivious to this issue, or they like the silence.
Incompetent morons…
I sent another email through the FB-page, but doubtful they’ll respond though.
I did actually get a reply, but it ended up in the Spam folder, yet Google claims it never was sent o_O
Oh, well.
They re-iterated what I already knew, that in order for T.R.I.M. to work, HBA must be enabled.
It is, fstrim says it’s on, some apps says it’s off and I don’t know what to believe. Adapted also said “current SSDs do their own garbage collection very effectively that you should not see any issues. ” – I sure hope so.
Was looking forward to seeing “The TRIM feature of the SSD is supported and enabled for optimal performance.” in Hard Disk Sentinel when I bought the 8885-card
Ok. Thank you so much for your reply.
But I was amazed that neither my post is visible here on the web page after it was approved, but strangely when clicking the link in the mail sent from you showed the contents, and as number 47 – I tried, out of curiosity a lower number, but that didn’t work, neither opening a link (Run-dialog box).
When clicking on that link now I get my post and your reply, but no one else’s…
Even there it says “0 Comments” and “Be the first to comment” is visible, so there’s something wrong with the page I believe.
Anyway, I do look forward to experimenting with these, and if works well I probably will buy more such hardware since it might be just what I need. First of all I need it to be stable and not as flimsy as these multiplier cards have been. But, time will tell.
First things first – I need to get this first batch of fun stuff firstly.
This was actually the first listed post on the search (DuckDuckGo) that had content that didn’t veer off to Mars or whatever – In other words, it was more or less exactly what I was looking for. Except of course the additional questions.
Thats great to hear Micke. Please let us know your final setup and any issues you faced along the way. As we are not using a third-party for comments it is possible that the site’s latest cache may have not loaded your comments. We’ll update the article with the questions you asked in the FAQ section. Best of luck with your new hardware…
Ok. Judging from the number, I’m inclined to think I’m not the only one commenting here. Could’ve been spam or someone asking for directions to Area 51…
No matter. I will report my successes and failures once I get my stuff delivered from China.
Cheers!
So, first off, there’s something wrong with this site and cookies management. I managed to slip by that manage-function by incidentally pressing the scroll wheel (Which I’ve added as Ctrl+Click to open i new tab, normally). The reason I mention this is because when opening the developer’s tool (What it translates to from Swedish) shows multiple errors. Not sure if my VPN’s got something to do with it (Might have ad-servers filter active), but it refuses to enter:https://bshr.ezodn.com/?did=231201&bf=30000&dc=%7C1254144
among other things. You should look into this.
Anywhoo. Got my stuff now, and it’s working beautifully, but I do have some quirks and I’m not sure what is causing it. I had some memory problems after I shifted my GPU and I removed one pair of DIMMs (Out of 16), ran MemTest for a while (Only about maybe 20 minutes) no errors, and Windows (10) started as normal.
The problem is that some programs aren’t working the way they’re supposed to. For eg, Hard Disk Sentinel crashes after a few minutes, and looking at Process Monitor I can see that an app called C:\Windows\SysWOW64\WerFault.exe starts, and immediately after it shuts down Hard Disk Sentinel.
Not sure if it’s my Windows installation that is swamped with crap, but it runs otherwise fine.
Another thing that is weird with the whole thing is that since the ASR-71605 doesn’t support T.R.I.M. I decided as a temporary solution (I’m thinking of ordering an ASR-8885 which apparently is supposed to support it) that I’d use a normal SATA controller for my SSDs.
At first I used two of these (9 x 2TB SSDs), one with 6 ports, and another with 4 to accomodate the disks.
The short while Hard Disk Sentinel’s alive, it says that T.R.I.M. is installed and activated, and only seconds later it says it’s been deactivated.
The app HD Tune Pro says “TRIM” but doesn’t indicate which state that is…
I sort of feel Windows is handling things differently since there’s a RAID card in it. I have yet to actually test without it.
There are a few things that could be the cause of this.
One being that I nicked something, possibly one of the CPUs causing it to glitch slightly, and thereby causing that memory problem.
Another can be since the idiot that I bought 10 x 16 GB ECC DIMMs from shipped them in a Non-ESD bag (Simple Zip bags if you can belive it!!? More frustrating that it was a company! Think they went out of business though – Wonder why..?), so there’s a possibility that it was a latent fault that just surfaced when I was moving the different controllers around. Who knows.
The sad part is that I’ve been lazy, sort of, meaning that my chassis isn’t a normal one, but rather an old fire alarm box, a locker of sorts, so I’ve Jerry-rigged things inside it, which includes the fans who are strung up with wires (Don’t ask . Wire wrapping wire was what I had around).
I’m thinking of moving the motherboard to a normal chassis as I’m contemplating buying long SAS cables along with the controller card, as well as another expander card and tons of SAS->SATA cables so that I can rid myself of the airflow problem I’ve been having lately.
I currently have 4 x 120 mm 2000 R.P.M. fans in the top and bottom of the chassis, a total of 8, so the flow is nominal, but uneven thanks to the motherboard in the middle, and now with these two furnaces (RAID + expander) it’s not any better.
I also think I need to replace the thermal mayonnaise on one of the CPUs since it’s slightly higher than the other one.
First thing I’ll do next few days is to try installing Windows anew to see if that remedies anything. I’ve got a “spare” SSD that I can use for that purpose.
Hopefully it’ll work. If it does, I’ll re-do the installation to my NVMe SSD on PCIe to use with Clover UEFI Bootloader (Say that 10 times fast..!)
Computers… Can’t use them, can’t afford them, can’t be without them – Yet here we are experienced users, useless without them and is our most priced possessions that we keep forking out dough for…
I forgot to ask something too:
I’ve been searching the web, but gets unrelated and simplified results.
I’m curious since the specs for these RAID cards says up to 128 to 1024 drives, there’s very little info about topologies to use.
When it come to LAN/WAN etc there are a few different ways, token ring, star net, cascade and so on.
There are some mentioning here and there, but I didn’t see any image other than daisy chaining SAS.
What I’m curious about, since my ultimate build would have 320 drives. and I’m thinking of using these 36-ports expanders, with 2 x 4-ports between of these – This would mean 20 + 20 disks in one, “Array” – Note, not RAID array, just using that word.
And having 8 of these “arrays” means quite a few – 16 to be exact.
I am thinking of using two RAID controllers to get so that not being dependent on just one controller, but at least 4 x 4-port cables,
I’m curious, since the AEC-82885T can use any ports for communication, I was thinking of using a such as a distributor rather than daisy chaining them.
I haven’t really sat down and played around with pen & paper about it (hence the googling).
I’m curious about redundancy. bandwidth and to make it as efficient as possible.
Connect one expander to 8 x arrays to one controller, then connect another expander to the other controller with some cables between them..?
There are a lot of different combos possible so any tips & ideas are most welcome
Hey Mickie,
Nice to hear back from you. I’m sorry to hear the difficulties your facing and hope that they get resolved soon.
In the case of your SSDs not showing up, I would recommend you change the ASR-71605 from IR to IT mode by pressing Ctrl + C during boot. IT mode will present connected drives directly to the OS in which you can then make a RAID array.
As for your CPU issues, try booting with a single stick of RAM and attach the SAS controller with your drives. To troubleshoot the memory issues you can then add one stick at a time and test for instabilities till a problem is observed. If you notice a problem, swap the stick of RAM with one that you’re sure is working to make sure the memory channel corresponding to the RAM slot is not damaged on your CPU.
For better cooling, you can zip tie a Noctua fan to the haetsink of your controller.
I’m impressed with your plans for a 320-drive setup. Some suggestions I would make is looking into Direct Attched Storage arrays like the NetApp DS4243/6 (you may need to look into newer models for SSDs). These have in-built SAS expanders and directly attach to your SAS controller/expander. Managing the mess of cables is going to be difficult with so many drives.
If you want to use expanders then your approach of using the AEC-82885T as a distributor is great idea. In fact the technical brief for the AEC-82885T mentions it in Fig 3.
Hope you found this helpful. Do get back with any queries and let us know how your build is progressing…
First off, not currently using the SSDs on the RAID-controller, but on a separate SATA-controller since it has T.R.I.M. support, and the RAID-controller does not.
As for the fan I’m currently using a 120mm fan hot-glued to the chassis aimed towards and in very close proximity to the RAID-controller and it seem to work just fine.
The expander is placed directly above the 4 x 120 mm fans with one of those PCIe extender modules for power, so it too gets adequate cooling.
I’m also not interested in buying any bulky chassis for a small amount of drives, but my goal is to get all of the drives into a single chassis.
As for the expanders, there are numerous ways to connect these, and I was thinking of using two expanders for distributors to each “array” as I mentioned containing each two expanders in daisy chain config.
These “arrays” would then be daisy chained top to bottom,for eg, From distributor1 port 1 to “array 1” ext, connector 1. From ext. connector 2 to next “array” and repeat.
It gets complicated and very messy, but the idea is to have each “distributor” communicate with each “array” while having one connection between both distributors.
I felt it prudent to use 8 lanes in total instead of 4 for the sake of bandwidth, so with 4 for each distributor and all networked it feels like it would work.
At one moment it makes perfect sense, the next I feel like my brain is in the blender. Tried jotting it down on paper, but my brain started to hurt after a few lines were drawn…
It hurts even more if I wanted to introduce another 8-lane RAID controller, and thoughts were not only for redundancy, but also, again, for sake of bandwidth.
I doubt very much I’ll ever do anything like this since the cables are very expensive, and the above configuration would requite a ton of them… Meh…
Not to mention finding short ones would be very difficult, and probably very very expensive.
Also doubt that I can find and purchase connectors to make my own cables, but that would be an option in that case.
Oh ok, does it show the connected SSDs in a list during boot up?
As for your plan, I think I have an idea of where you’re heading. A useful tip would be to start from the top of the topology and then add expanders or drives as you go down in levels.
Level 1 would be your controllers, then Level 2 would be the SAS expanders with a direct connection to the controller, which in turn connect to Level 3 “arrays” made up of daisy-chained expanders.
For redunancy you can cross-connect each level so if an expander or controller fails it can manage to load by itself. I don’t think by connecting the Level 2 connectors you will get a boost in bandwidth.
Let me know if this is what you’re aiming for…
The SSDs aren’t connected to the RAID-card due to lack of T.R.I.M. support, so I’m using an ASM1166 6-port SATA and 2 x 5-port SATA port multiplier.
This is temporary as I’m about to order another RAID card, cables and another expander card.
However, before I put the RAID card in this machine, I tried it on an old Dell Optiplex 740-something. Has Core 2 Duo or something like that -Old crap useful for testing things like this, to the extent of its limitations.
Anywhoo, it did find the smaller KingSpec 180GB SSDs (Don’t ask, my first ones I bought a little over 5 years ago from Ali Express after years of hesitation over price & wether it was worth it and so on).
Useless drives, and I didn’t try my new 9x2TB PNY-SSDs I recently bought, just for the sake of no T.R.I.M-support.
With the redundancy we’ll see about. It’s a costly build, and I have a somewhat limited budget so it’s a slow build.
It’s just now during the summer where I earn a little more money that I can afford spending some on this project of mine, and the cables are expensive so I don’t plan on buying that many, and buying more just for testing different topologies I don’t know since my little idea would involve darn many of them.
If I ever get around to it, I think I’ll draw a schematic of how I’m picturing this since it gets messy as I pointed out several times.
I’m not aiming for any “boost” as you put it, just so that these 8 lanes are evenly used, if say (random number 4 to 7) disk are in full use on the same array and only 4 lanes are being used, it would then use up all the bandwidth, or at least that’s how I’m imagining it to be like, so using 2×4-ports between all expanders is what I’m curious about if it would solve the bandwidth problem, should it arise.
Hi!
I have questions:
I’ve just ordered a few items, an Adaptec RAID 71605 16-port controller and a 36-port expander of the same brand, plus various cables (2 for external since the expander has such).
Now, I understand that you can connect these cables to pretty much any port on either controller or expander, but it’s apparently recommended to use the lower port numbers.
What I’m curious about, and I haven’t found any info about it, is that if I use 2 ports, say 1 and 2 connected to the ext. port of the expander, then if I get another expander can I connect that to port 3 and 4 without any performance drop along the line?
Also curious, since there’s no mentioning of that either, but what happens if I only use one (4-port) cable, and if so, connect those to four expanders?
What will happen if 2 ports are mixed, like 1 and 4 to one expander, 2 and 3 to another?
And finally, something that I will try, but if you’re reading this (and replying) before I get these;
Does these cheap SATA 1 to 5 port multiplier cards work on either the RAID controller or the expander?
The multiplier card demands FIS and NCQ and I can only find info about NCQ for the RAID controller.
Yes, I know these multiplier cards are garbage, but they’re somewhat cheap and simple to use and I’ve got several of these, but it looks like I might be going over to these expanders in the future.
Btw, I have multiple disks (+20), all 2½” SATA. and they’re getting phased out to SSDs, The importance isn’t speed, but it’s well welcome, but I do need it to be stable, something I’ve had numerous problems with due to shifty controllers and cables, with failing drives as a result.
I don’t care if the speed drops when more than one drive is being accesses (Which is why I was wondering, among other things, if it works with one 4-port cable between RAID and expander)
Hey Micke, that’s an excellent choice of RAID controller. I believe you have the Adaptec 82885T 12Gb/s SAS Expander Card per your description.
Addressing your first question, the performance is capped at the RAID controller’s PCIe Gen 3.0 x8 speed. That is a max theoretical bandwidth of 64 Gbps. A SAS Expander card will operate on two 4-port connectors at a max bandwidth of 48Gbps, so adding another Expander shouldn’t slow it down.
Using one port also works, but your SAS expanders will only benefit from half of the bandwidth. The 82885T supports 24 direct attached drives, so the bandwidth per drive will decrease from 2 Gbps to 1 Gbps.
Unless mixing ports is not allowed in the manual, it shouldn’t cause issues, but routing cables from adjacent ports between the controller and expander is easier.
The speeds mentioned are if all drives are being utilized simultaneously. If only a single drive is used, it can saturate the entire 6Gbps bandwidth.
SATA port multipliers require a SATA hub. Afaik SAS HBAs do not support such multipliers. I’ve also seen reports on single drive failure causing connected drives to fail, so it’s best to avoid it.