Hardware errors grow as tech shrinks

https://www.nytimes.com/2022/02/07/technology/computer-chips-errors.html Not surprising that shrinking tech reduces error tolerance but i found the rates significant. One of the linked papers says DDR4 is supposedly generating 5.5x more errors than DDR3 Makes old hardware suddenly look good

5.5x more errors than DDR3 is still a really tiny amount. Also, it's weird that they identify that the problem is the switches themselves rather than cosmic rays or other outside noise -- the reason that the switches become less reliable is because they are more susceptible to outside noise. That's why it's so hard to make qbits. "outside noise" includes heat and electrical noise coming from the machine itself, to be clear. There are strategies that are used to lessen the problem of transistor-level defects. But ultimately the reason why consumer-grade hardware is usually the soykaftiest is because nobody cared enough to make sure it was really for sure defect-free. Server-grade is the next step up. For actually important things, like PLIC and deeply embedded microcontrollers for industrial applications, the error rate is basically zero because those chips are validated extensively and generally use a way bigger processor node.

>>74193 what do you mean by server-grade chips? xeon is sizewise identical to a i7

>>74194 for instance, most RAM chips that go into servers have error correcting bits built in.

>>74196 meant >and generally use a way bigger processor node thought implied they change architecture to be bigger in physical size on servers to revert the incurred errors. Imade the thread cause i was curios if anyone knew how often these errors occur and if there is even a way to account for hardware vs software errors. Like occassional rare crashes could be a rare concurrency bug or just a rare hardware bug and you could waste lots of manhours searching for a nonexisting bug in your code.

>>74197 bigger processor node was in reference to microcontrollers. AFAIK it's mainly for heat and cost reasons.

i have ddr4 is it over for me >shrinking tech what do you mean

>>74196 The chips themselves are the same, it is the memory controllers (in the CPU) that know how to do error correction. The RAM sticks have an extra chip on them to store the correction bits. It is unfortunate that consumer grade CPUs don't support ECC, and it's mainly for market segmentation reasons. >if there is even a way to account for hardware vs software errors ECC is usually SECDED - single error correction, double error detection. In case a single bit gets corrupted (assuming you have the right drivers) you will get a notification/log entry. In case of two bits corrupted you might get a notification and may or may not crash the system. Since double errors are exponentially less common, you should have some forewarning that something is wrong by the increase in single errors. But on consumer hardware, you get none of this and you basically can't tell.

>>74200 improvements in chips come from shrinking transistor size and putting more of them in one chip. cpu generations are almost identical to steps in shrinking it down (sometimes its different architecture with same nm chipsize). That aside afaik the real innovator isn't intel or amd. There is a separate company that develops the machines to burn wavers for everyone located somewhere in europe. Their core tech is extremely accurate mirrors which are required to focus the laser to burn wavers. >>74201 i believe amd had ecc in consumer cpu's. Focus on "had" cause i dont know the current market at all.

>>74203 ASML (based in the netherlands) makes all the high end machines. I think there are a handful of companies (fujifilm maybe?) that still make bigger process machines. But it's not just ASML doing the innovating, there is a whole stack of "secret sauce" added by various companies along the supply chain. Some seemingly insignificant change to materials or process or even the construction of the building can mess it all up. I don't remember the details but there was something in the news a couple years ago about some fab had changed water supplies or something and it had ruined billions of dollars worth of memory chips. The problem was they didn't know it was broken until the chips are finished and it takes months to go from start to finish. I think the company had to sell off the fab and file for bankruptcy.

>>74197 Anecdote: I used ZFS on a system without ECC RAM. For the first few years everything worked fine. Later it my desktop froze once in a while and the frequency of ZFS checksum errors began to increase. At some point it became bad enough that I decided to investigate. Some days later, after running memtest I realized that the timing on the sticks wasn't stable. At that point I decided to look around for ECC UDIMMs, found some for a decent price and upgraded both my motherboard and CPU while I was at it. My desktop stopped freezing since and I haven't got any checksum errors either. I usually suspend my computer instead of shutting it down, so "uptime" tends to be high. edac-util tells me there has been 1 corrected error in the last 60 days, so they do really happen. A lot of people think that ECC isn't necessary or that it is slower, but both are untrue. Performance is only a matter of the dies used on the sticks (and memory controller in CPU). ECC being "not necessary" is bullshit spread by Intel so they can artificially inflate prices for features that are in fact absolutely necessary. Lack of ECC results in silent data corruption (which spreads to backups!), unexplained crashes, "bit squatting" (the analogue of typo squatting but with bit flips) and many other evils. God knows how much damage Intel caused with their greed. Rant by Linus that is worth reading: https://www.realworldtech.com/forum/?threadid=198497&curpostid=198647 PS: Using ZFS without ECC memory is no worse than using other filesystems without ECC. In fact, it has proven to be more reliable in my case. Regardless, every FS in existence requires ECC to function properly. It is impossible to write robust software on hardware that is fundamentally unreliable.

regarding ecc: ddr5 has fake ecc called "On-Die ECC" so if you buy ecc on ddr5 you need to be extra careful to get the real one as most vendors can be clueless too and write ecc as they see "on die ecc" on the package

>>74777 What are you smoking? How is it "fake" ECC? The only difference is that data isn't protected during transport between the RAM and the memory controller and that errors (which are mostly correctable anyway) aren't reported back to the OS.

>>74804 >aren't reported back to the OS. NTA, but that is one of, if not *the*, major reasons to use ECC. Yes, DDR5 with on-die ECC will be able to correct errors but it won't tell you if something is wrong before harm is caused to your data. I hope this lack of reporting won't burn people out of getting ECC hardware if they do ever end up with very bad sticks (that get double bit errors frequently) due to it being "useless".

>>74804 >data isn't protected during transport between the RAM and the memory controller >errors (which are mostly correctable anyway) aren't reported back to the OS Seem like pretty major drawbacks. Anyway sounds comparable to the state of SSDs, where ECC is the rule rather than the exception and is usually performed transparently by the SSD controller. In that case ECC just lets the manufacturers use cheaper, denser, less reliable chips and correct the inevitable errors until it can't. Doesn't make the SSDs particularly more reliable, you still need something like RAID and checksumming filesystems for actual reliability. Same thing is happening with ethernet FEC. Send data faster until it starts getting corrupted, then add error correction until it's "good enough." Not a terrible strategy but the result is usually better performance with about the same or worse reliability.

>>74848 At least with SSDs and Ethernet you have a known-reliable platform that can add whatever error correction/detection you need (in software). You cannot compensate for lack of proper ECC RAM in software though.

>>74192 Tech has been getting worse after 2012 change my mind

>>74863 Easy: planned obsolescence came into being much earlier. That said its not surprising tech is getting less reliable, products go as bad as consumers allow them to be. Its a problem of metrics. If everyone is looking at GB and MHz only and nobody mentions errors then of course the manufacturers are forced to increase GB and MHz at the price of errors.

>>74193 I fear that the rate of errors on increasingly shrinking hardware will compound to the point where it is next to impossible to find a reliable device. Next generation might 2x the rate of errors, and 3x the generation after that, and so on. We'll get to a point where every other device is expected to randomly fail, all while software gets written less and less efficiently.

>>75512 ECC can easily cope with that since more redundancy increases reliability by an exponential factor too. The literal only reason RAM appears unreliable is Intel's greed and utterly idiotic "ECC is slow" FUD/cope (ECC RAM is exactly as fast as non-ECC ram).

>>75517 ecc is just a bit checksum and when it notices an error you have to redo the operation. There is nothing exponential about it. Its just redo operation til it passes without error so the speed is entirely dependent on the error rate.

>>75549 >ecc is just a bit checksum It isn't. >There is nothing exponential about it. Suppose there is a P chance for a bit to flip If you have a system that can correct up to N errors then you need at least P^(N+1) bit flips before data becomes unrecoverable. >Its just redo operation til it passes without error so the speed is entirely dependent on the error rate. Again untrue. Look up: - erasure codes (esp. Reed-Solomon) - SECDED (Hamming code) which is what is typically used for ECC RAM Both of these methods take a constant amount of time for error recovery. In the case of SECDED it is entirely transparent and takes no extra time whatsoever.

>>75550 > least P^(N+1) bit flips Messed up my wording here and math isn't quite correct. You need at least N+1 bit flips. The chance for a single flip is P. It isn't exactly P^(N+1) since it's any N bits out of a row of M bits. Can't recall exact formula but it's very roughly exponential. (Flood detected - post discarded. Bah, will phonepost instead with many typos)

>>75512 I feel like there's so little to how much hardware can shrink and I feel we hit that point. The next thing that will happen is that we start actually experimenting with different materials in consumer products

>>75550 >>75551 Got interested and read a bit seems like newer ecc doesn't use hamming anymore but i couldn't find a source on what exactly they implement. Also decded can fix 1bitflip errors but if you assume the scenario of the original poster where error rate grows exponentially you will have to deal with 2 & 3 bitflip errors in which case the "glorified checksum" would become true since at that point it can only detect but not correct. That aside i saw news about some going back to analog computing. My understanding is digital was done as a form of error correction so they might give up digital cause its not sufficient anymore and you need secondary forms anyway.

>>75584 analog computing is (generally) very noisy compared to digital.

>>75586 microphone noise is a good example as it processes the input as an analog signal. The noise comes from the electric grid itself with base noise from frequency of grid + connected electronics creating additional frequencies on top of that. And secondary you suffer induction on the wires thanks to being flooded with em signals. With that said i could imagine an analog super computer in a remote place with its own power supply system - it would be free of all mentioned worries outside of its components causing interference among itself.

>>75584 Analog computation doesn't scale well because it usually requires elements that dissipate power. Digital (CMOS) can hold its state with almost no power when bits are not changing. The exception is stuff like DRAM which is sort of hybrid analog/digital and needs to be refreshed. MLC NAND too is basically storing analog values and converting to digital on read. The through line here is that these are just storing data, not performing computations.

https://www.youtube.com/watch?v=0itrM7t4l34 found this year old video doing extensive memory tests and was reminded of this thread. He did also a follow up https://www.youtube.com/watch?v=SexoI7kt4jk which ist 27min on every possible cause for bit flips. Practical advice mentioned is bitflips occur from loose connections and sticks heating up which can both be addressed with little effort. Doing adjustments and verifying with memcheck is something anyone can do.

Anyone here using m2 nvme ssd's? Would be interested in your experience. Considering upgrading from sata but saw the m2 nvme get really hot til they have to throttle at 95°C. My sata ssd's sit in comparison at only 22°-35° C. Seems like its begging to get data loss to get one of those newer ones.

>>76242 I use a Crucial P3 1TB in my current computer build. It seems to work fine. I haven't run any benchmarks or troubleshooting on it. The speed isncrease is noticeable when loading big steam games. I'm using it in an ASRock B650 PG Lightning mobo with an AMD Ryzen 5 7600 CPU and ASRock Challenger D Radeon RX 6650 XT 8GB GDDR6 GPU. No complaints.

>>76248 please copypaste smart values, for comparison my sata ssd's smart: ssd1: power-on-hours 13512h (~4.6years at 8h daily) wear-leveling-count 12 uncorrectable error count 0 ssd2: power-on-hours 15994h (~5.5years at 8h daily) wear-leveling-count 15 uncorrectable error count 9576 ssd2 got errors due bug in old firmware but i didn't really notice them either. If a program crashes you wont think "ah yes bad ssd" so gotta check values.

>>76242 I use a WD 1TB SN850X as my boot drive and a 1TB Crucial MX500(SATA) for extra storage. ---------------------------------------------------------------------------- (01) WD_BLACK SN850X 1000GB ---------------------------------------------------------------------------- Model : WD_BLACK SN850X 1000GB Firmware : 620361WD Serial Number : 23503C801275 Disk Size : 1000.2 GB Interface : NVM Express Standard : NVM Express 1.4 Transfer Mode : PCIe 4.0 x4 | PCIe 4.0 x4 Power On Hours : 869 hours Power On Count : 7 count Host Reads : 16301 GB Host Writes : 2882 GB Temperature : 44 C (111 F) Health Status : Good (100 %) Features : S.M.A.R.T., TRIM, VolatileWriteCache Drive Letter : C: -- S.M.A.R.T. -------------------------------------------------------------- ID RawValues(6) Attribute Name 01 000000000000 Critical Warning 02 00000000013C Composite Temperature 03 000000000064 Available Spare 04 00000000000A Available Spare Threshold 05 000000000000 Percentage Used 06 00000209A824 Data Units Read 07 0000005C3E71 Data Units Written 08 000005B76739 Host Read Commands 09 0000129E0EF6 Host Write Commands 0A 00000000009E Controller Busy Time 0B 000000000007 Power Cycles 0C 000000000365 Power On Hours 0D 000000000000 Unsafe Shutdowns 0E 000000000000 Media and Data Integrity Errors 0F 000000000000 Number of Error Information Log Entries ---------------------------------------------------------------------------- (02) CT1000MX500SSD1 ---------------------------------------------------------------------------- Model : CT1000MX500SSD1 Firmware : M3CR046 Serial Number : 2319E6D45F1B Disk Size : 1000.2 GB (8.4/137.4/1000.2/1000.2) Buffer Size : Unknown Queue Depth : 32 # of Sectors : 1953525168 Rotation Rate : ---- (SSD) Interface : Serial ATA Major Version : ACS-3 Minor Version : ACS-3 Revision 5 Transfer Mode : SATA/600 | SATA/600 Power On Hours : 2381 hours Power On Count : 78 count Host Writes : 1352 GB Temperature : 29 C (84 F) Health Status : Good (100 %) Features : S.M.A.R.T., APM, NCQ, TRIM, DevSleep, GPL APM Level : 00FEh [ON] AAM Level : ---- Drive Letter : D: -- S.M.A.R.T. -------------------------------------------------------------- ID Cur Wor Thr RawValues(6) Attribute Name 01 100 100 __0 000000000000 Raw Read Error Rate 05 100 100 _10 000000000000 Reallocated NAND Block Count 09 100 100 __0 00000000094D Power On Hours Count 0C 100 100 __0 00000000004E Power Cycle Count AB 100 100 __0 000000000000 Program Fail Count AC 100 100 __0 000000000000 Erase Fail Count AD 100 100 __0 000000000001 Average Block Erase Count AE 100 100 __0 00000000000A Unexpected Power Loss Count B4 __0 __0 __0 000000000048 Unused Spare NAND Blocks B7 100 100 __0 000000000000 SATA interface Downshift B8 100 100 __0 000000000000 Error Correction Count BB 100 100 __0 000000000000 Reported Uncorrectable Errors C2 _71 _47 __0 00350000001D Temperature C4 100 100 __0 000000000000 Reallocation Event Count C5 100 100 __0 000000000000 Current Pending ECC Count C6 100 100 __0 000000000000 SMART Offline Scan Uncorrectable Error Count C7 100 100 __0 000000000000 UDMA CRC Error Count CA 100 100 __1 000000000000 Lifetime Remaining CE 100 100 __0 000000000000 Write Error Rate D2 100 100 __0 000000000000 Successful RAIN Recovery Count F6 100 100 __0 0000A90899DC Cumulative Host Sectors Written F7 100 100 __0 0000015A18C7 Host Program NAND Pages Count F8 100 100 __0 0000002B767B FTL Program NAND Pages Count