Libelektra: Replacing a7

[markus2330]

a7 is completely unreliable and at most likely two parts are involved (CPU causing segfaults and something unknown causing daily crashes, see #160). Our options are:

1. trying to search the faulty part while still using it as our main buildserver. (I think that trying to find the faulty hardware might not be worth the effort, especially because there are many people involved and any action is complicated and takes very long.)
2. simply not using a7 and migrating everything to other existing hardware. (I am afraid it is still not clear if v2 is now stable.)
3. buying a new hardware for a7. (I am open for suggestions for which hardware to buy.)
4. renting a new server. (I am open for suggestions where. Also at Hetzner would be one possibility.)
5. putting it into the cloud. (I am not a fan on being dependent on complicated prize calculations, not having control over the data etc. If the main load is on v2, it might be okay, though.)

What are your opinions? (Especially @MasterToney and @ingwinlu)

[ingwinlu]

The problems are 100% load related (as crashes were much less frequent before switching over to the new build system).

Let's exchange the CPU first before making any further steps.

[markus2330]

The problems are 100% load related (as crashes were much less frequent before switching over to the new build system).

It is a bit more complicated. Several months ago (when the hardware was bought) intensive load tests were done, without any errors. Also the gcc compilation was done for many hours. It might be that these tests were too synthetic or for some other reason the problems were not triggered.

Let's exchange the CPU first before making any further steps.

Let us see if AMD ships something, I did not hear from them yet.

[ingwinlu]

synthetic tests

Pretty sure the segfault problems are not even occurring if you use clang so that is entirely possible.
However I am surprised that no problems with the PSU got detected.

Has v2 crashed since replacing the PSU + CPU?

[markus2330]

v2 has uptime 20:41:59 up 2 days, 6:01, 1 user, load average: 0.09, 0.05, 0.01, so afaik there was no crash after replacing the power supply unit.

[markus2330]

A relative minor change would be to swap a7 and v2. Then crashes of a7 would not influence v2. (The assumption is that a7 crashes more often than v2.)

[markus2330]

I have a i7 system that we could also use in the same server room.

Any suggestions which SSD we should buy for a7?

[mpranj]

@markus2330 regarding the SSDs it depends what we can afford. I have some good experience with Samsung SSDs. Here are some examples of current SSDs. There are older models, but they are not cheaper 🙃.

For SATA:

• Samsung SSD 860 EVO 500GB (TBW: 300TB) (Geizhals: € 109,28)
• Samsung SSD 860 PRO 512GB (TBW: 600TB) (Geizhals: € 185,92)
  Both pretty much saturate the SATA (6Gb/s) throughput.

For M.2:

• Samsung SSD 970 EVO 500GB (TBW: 300TB) (Geizhals: € 169,25)
• Samsung SSD 970 PRO 512GB (TBW: 600TB) (Geizhals: € 220,88)
  The slower does: read3400MB/s, write: 2300MB/s

[markus2330]

@mpranj Thank you for the tip! I purchased a Samsung EVO. It should arrive within next week.

[markus2330]

Seems like we have to exchange the main boards, too. (v2 had a freeze without any output, so the power supply unit is unlikely to help for a7, too.)

Does anyone has experience with mainboards for Ryzen?

[mpranj]

I have no experience with them, but it seems that ASUS boards are quite popular judging from reviews.

One thing that comes to mind talking about this: I've read that Ryzen can be sensitive to RAM choice. They have a list of validated RAM. Sometimes the mainboard vendors also maintain such a list. It is of course possible that other modules are supported.

[markus2330]

Do you have a particular ASUS board in mind?

A wrong RAM choice should have been caught by RAM tests, shouldn't it?

[mpranj]

Do you have a particular ASUS board in mind?

Unfortunately, I don't really know the exact hardware of a7. Are we talking about these here? There would be newer boards with newer chipsets we could try. There are only older ASUS µ-ATX boards. Maybe Anton should take a look before we buy anything incompatible.

A wrong RAM choice should have been caught by RAM tests, shouldn't it?

Probably. I'm just throwing in suggestions. There are different reports of instability with ryzen. Someone who has a similar board (but different chipset) reports (sorry about the production quality) that he fixed it in the bios by changing some Pstate settings. It might be less work to just pick a new and stable mainboard in the first place, though.

[markus2330]

Are we talking about these here?

Yes, I did not know about that page. Good to have documentation :smile:

he fixed it in the bios by changing some Pstate settings.

We disabled the C6 state but in the latest version of the BIOS these settings do not exist anymore.

It might be less work to just pick a new and stable mainboard in the first place, though.

Yes, but which one?

[mpranj]

It seems we don't have much choice if we can only fit µ-ATX boards.

• ASRock AB350M Pro4 (most likely ECC support)
• ASUS Prime B350M-K
• Gigabyte GA-AX370M-DS3H (most recent board and feature rich chipset)

All above support overclocking and other features we supposedly don't need. The board which is already in a7 was a good choice judging by tech specs. I can't think of what is wrong there.
As I said, before you order anything, maybe let Anton or someone else take a look at it.

If we can fit larger boards, there might be better options.

[markus2330]

Some recent changes:

• We now received the power supply unit and SSD (we will install it once a7 goes down again).
• We ordered an ASUS board. @mpranj thanks for the tip!
• We disabled hyperthreading on a7+v2 in the hope that it helps.

[markus2330]

a7 is now reliable. We could add an SSD if someone has time to set it up. @MasterToney?