On January 4, I ordered and prepaid an RTX 3080 around MSRP from Saturn (which we have in Germany and is comparable to Microcenter in the USA) but had no luck. A week later, I called them, was told the card was out of stock and got reimbursement. The alternative of prepayment for being on a waiting list was a too high risk for my taste.

Meanwhile, US trade war has resulted in an 25% US import tariff for GPUs. I might not care but Asus has taken this chance to raise their GERMAN RTX 3000 prices by €100, apparently in an attempt to let Germans pay a good part for US customers. This is not the first OEM scalping incident.

Needless to say, currently RTX 3080 is unavailable in Germany even at scalping prices. Apparently, the mining boom eats them all. Because, you know, if we believed Nvidia, demand among gamers would be the only cause for scarcity.

I get it: build your own GPU!

Current market prices for RTX3000 are more than double the MSRPs because miners buy these GPUs and laptops nevertheless. Even old, used GPUs are similarly expensive. It is unknown whether Nvidia and AMD print money instead of selling most of their GPUs. With a new GPU, currently one can mine the equivalent of USD 5 to 9 per day so that a GPU pays for itself within 3 or 6 months, provided the cryptocurrency rates remain high long enough. Buying GPUs for the main purpose of mining is betting on the cryptocurrency rates.

For only-gamers, the entire GPU market is dead. Gamers without GPU yet need to either a) wait for the cryptocurrency boom to collapse or b) act as new miners themselves (paying more than twice the MSRP betting on the cryptocurrency rates).

If you already have a GPU, you might just use it most of the time for mining if you don't mind the impact on the PC components' durability.

For non-owners, the situation is very unfair because a) those lucky enough to get their GPUs early at reasonable prices have an advantage when needing less time to let the GPU pay for itself while the cryptocurrency boom has reached its peak (thus far), b) those living in countries with cheap energy have an advantage because their risk is lower, c) during Corona, not everybody can travel to the GPU factories to buy hundreds of GPUs directly (and possibly pay customs).

Hi guys,

I am looking to build a rig with 2, maybe even 3, 3080s or 3090s to run LeelaZero/KataGo. However, when I tried stacking a 3090 and 3080, the 3090 on top just choked on hot air so much it thermal throttled ridiculously.

Does anyone here run 2x FE edition cards with the blow through fans and much more aggressive blower fan pushing hot air out of the case? Are the thermals acceptable? Can a FE 3080 on top sustain ~1850mhz?

I'm having trouble finding people with temperature data with such configurations.

Thank you

If you aircool, the cards need at least 1cm (better more) space each below them, a case /setting with good airflow and sufficient fans in a meaningful configuration. For a closed case, this means e.g. 3 input fans, at least 1 output fan, straight airflow and more / equal input pressure than output pressure. However, for 3 RTX 3000 cards, there are no suitable mainboards for aircooling, unless you use riser cables. So use an open configuration with extra fans or water cooling. Also get a suitable mainboard with fast enough PCI slots, see earlier in this thread, unless you get special mining boards.

If you put the (especially aircooled) cards directly next to each other, you are doing it wrong!

For 2 aircooled Nvidia FE cards, put the RTX 3090 below onto a mainboard with 2 PCI 4.0 slots running at x8 speed at least 3 slots apart into a Meshify 2 XL (the only case with enough space below the lower (the 3090) card). If you put the 3090 above or put 2x3090, you need a mainboard with 2 PCI 4.0 slots running at x8 speed at least 4 (!) slots apart (see earlier in this thread).

EDITs

I had 3090 and 3080 next to each other. There was definitely a cm of space but the temp was just out of control. It was in a big Lian Li o11d-xl case too with 6 intake fans and 4 exhaust fans.

I am thinking my best bet is getting a MB that can allow me to get more space between two cards and give up on 3/4 cards. But before I do that, does NVLink help with performance?

6 intake fans and 4 exhaust fans? The number of fans (if enough minimum number and air volume) is of low importance. What matters is airflow. If you crossfire air, you reduce airflow. Lian Li has some cases with special fan positions, for which I have not seen enough reviews to know for sure. However, if you have, say, 3 intake on a side and 3 on the bottom, chances are great that you crossfire air.

Better have one straight direction of intake air, of which enough must go directly to the GPUs.

Ok, it is quite possible that 1cm space might not be enough and 2cm would be better for sure.

Another aspect: how about undervolting the GPUs? If have seen reports than one can reduce ca. 70 - 80W per GPU while keeping its speed constant.

Oh, and the intake fans must not be blocked by a radiator or disk chassis.

I'm using the Lian-Li O11D-XL. The orientations are kind of weird.

3 intakes at bottom blowing directly on the GPU (sadly only on bottom GPU)

3 intakes on side

3 exhausts on top

1 exhaust on the back

The orientations of the fans really aren't ideal since they aren't helping to push hot air out the back of the PCI slots.

I might just need a new case but as it stands, I'm probably looking at getting something like a MB with 4x 8+xPCI-e slots so I could space two cards far enough to breathe.

I just don't know if NVLink brings some performance benefit. If it does, I might just end up spending a little more and get someone to build a water cooled rig.

PS: I am not entirely insane. I do run fluid simulations for O&G wells, among other things, that require a lot of computational power.

Before buying a new case, you might try this: deactivate the bottom fans, slow down the 3 top exhaust fans (and / or possibly deactivate the right-most top exhaust fan) so that intake pressure is higher than output pressure. Verify that still a good amount of air flows below the lower GPU.

NVLink is superfluous for KataGo but I cannot know if NVLink helps for mining.

If you should go for 4 GPUs, they should be watercooled, which is a science / requires good assembly.

The 4 card dream is dead. I looked into it and, without resorting to a mining rig, custom water cooling loop is all but necessary and that drives the cost of the system to pretty close to 20k, if not more, if I wanted warranty of some kind from a custom builder.

I am thinking the most realistic setup is putting 2x 3090 in a MB with 4 PCIe slots... the idea is to use slot 1 and 3 so there is like an inch and half of space between the two cards. No NVlink but apparently that doesn't matter so okay.

Some of my work workloads would like NVlink but I'm spending my own money so whatever.

3090 cards are super hard to find. I think for now, I am just going to get a 4 slot MB and just run 3090 and 3080 in parallel. Eventually, depending on availability, I'm hoping 2x 3080ti (assuming KataGo doesn't need VRAM) will be the sweet spot.

This is assuming I could even find MBs with 4 slots. I understand miners are scooping those up too.

IIUC, your 3090 is 2 slots thick. So if you put two cards 3 normal PCIe slots distance, your free space is 2.08cm.

You mention a board with 4 PCIe slots using slots 1 and 3. Do you envision a board with ordinary layout? Then your free space is 0cm (non-existent). Or do you consider a workstation board for which slots 1 and 3 are 3 ordinary slot distances apart for 2.08cm free space? Why do you need a board with 4 PCIe slots?

If you do buy a new board, why not one with two fast PCIe slots with distance 4 slots? Too expensive board? Too little choice for cases with enough space below? Just curious. Since you have Nvidia FE, 2,08cm free space might be enough bit please report if indeed it will be:)

KataGo needs 3GB.

No. I mean MBs with 4 PCIe (at least) 8x slots with at least 2 that are at least 2 slots apart. For example, if I get a 7 PCI-E slot X299 board, I could put 3 cards on Top, 4th, and then 7th, and have quite a bit of clearance between the cards. One of the cards will probably be confined to PCI-e x 8 but I don't care.

The options for MBs like that are pretty thin on the AMD side. Something like X299 MB with 7 PCI-E slots, 1600w PSU, 3x3090 (prefer 3080ti, but it will depend on availability) I think is the best I can do without resorting to watercooling, risers, or some other very customized solution.

When you write 2 slots apart, you seem to mean 2 slots in between, so I call that 3 slots apart, which means 2.08cm free space between two cards each being 2 slots thick.

Yes, suitable mainboards are scarce and somewhat expensive.

As a native speaker, I call it 2 slots apart. In go we likewise have one-space jump, one-space pincer, two-space extension, etc where the numeral indicates the number of empty points between the two boundary posts. I haven't thought about it before, but that seems, off the top of my head, common to all the languages I know.

Is German really different in that regard? Or is it just your personal foible?

If the boundary posts are together. that is 0 slots apart. You can't possibly say that is 1 slot apart. The next step up is surely 1 slot apart, and so on ad infinitum.

If we only have the PCIe slots (without cards in them), there is a gap even between adjacent slots. The most useful measure of that gap is between centers of slots. From a slot to itself is 0. From a slot to an adjacent (not omitted) slot is 1 (multiply by 2.08cm if you like).

A card being said to be 2 slots thick has 2 (or 2 * 2.08cm).

If you put two cards each 2 slots thick into slots 1 and 3 (slot 2 is skipped), ideally they attach each other so the gap between the cards is 0 while their slot positions are 2 apart.

If you put two such cards in slots 1 and 4 (slots 2 and 3 skipped in ordinary ATX layout), the gap between the cards is 1 while their slot positions are 3 apart.

It is not the one-space jump gap counting but a measurement of the grid distance of its stones' centers.

The usage of »n spaces apart« does seem to differ even within native German and English speakers. I am more used to the definition that 1 and 2 are 1 apart, and I distinctly remember very explicit (because it is not self-explanatory) explanations that a »three space extension« in japanese usage is meant to be »three free grid points in the space between«.

One may think that these different views are equivalent, but I personally find it very useful when such numbers add up. So, if I have two things that are 2 units apart, and another thing in the same direction that is another 3 units from the second, then it should be 5 units from the first.

However, if you do two »ikken tobi« in a row, then if you take away the middle one, you end up not with a »niken tobi« (1 + 1 = 2), but with a »sanken tobi«, which, when seen from this perspective might as well be a 散見跳び.

Anyway, in Go, it seems to be this way, but in engineering, it is that way.

_________________
A good system naturally covers all corner cases without further effort.

FWIW, to me apart means to the side, not in between. So slots 1 and 2 are one slot apart, while slots 1 and 3 have one slot in between, and are two slots apart.

Now, if the distance is measured differently, the two converge. Seats that are 2 metres apart have 2 metres in between them.

_________________
The Adkins Principle:
At some point, doesn't thinking have to go on?
— Winona Adkins

Visualize whirled peas.

Everything with love. Stay safe.

For what it's worth, I tried the config with 3090s. They are so thick even with two slots in between, there is very little clearance and they still overheated. EVGA and Zotac 3080s also overheated on top. The founder's edition 3080 was the only one that didn't throttle a lot when put on top but even then it was running pretty close to throttling with fans blasting at 95%. I could maybe undervolt it and get 90% of the performance while keeping the card reasonably cool but I am just not going to do that.

I am at the point I think I have to get risers/rack/watercooling. Gonna have to see if I could convince my firm to subsidize this expense.

Thank you for your report! Although it might have to do with Lian Li's case, we are warned that a closed case with 2 thick high-TDP aircooled cards at constant workload can easily overheat even in winter.

For the last case I would say "one slot away".



And for that I would say "two yards apart"

RTX 3060 comes with BIOS (and driver) restrictions for (Ethereum) mining, whose hashrate shall fall by up to 50%. Possibly successors of RTX 3060 TI, 3070, 3070 TI, 3080 and 3090 might also get such a restriction. Now I wonder whether that only affects mining algorithms or whether other high workloads, such as go AI, will also be affected. Is the future good or bad?!