Quote:Intel hardware is, generally speaking, extremely reliable. Every processor and chipset has some degree of bugs, but these issues rarely become critical flaws that jeopardize a product. Evidence is mounting, however, that Intel’s C2000 server platform — that’s the Avoton and Rangeley low-power server variants, all of which are based on Atom — have a serious flaw that’s bricking hardware.

Cisco has released an advisory warning informing its customers of the following: “In some units, we have seen the clock signal component degrade over time. Although the Cisco products with this component are currently performing normally, we expect product failures to increase over the years, beginning after the unit has been in operation for approximately 18 months. Once the component has failed, the system will stop functioning, will not boot, and is not recoverable. This component is also used by other companies.”

So how does this link back to Intel, given that Cisco declined to name a vendor? Sharp-eyed boffins at The Register spotted an errata notice Intel issued in January, 2017. Intel’s AVR54 reads:

Problem: The SoC LPC_CLKOUT0 and/or LPC_CLKOUT1 signals (Low Pin Count bus clock outputs) may stop functioning.

Implication: If the LPC clock(s) stop functioning the system will no longer be able to boot.

Workaround: A platform level change has been identified and may be implemented as a workaround for this erratum”

The problem appears to affect all current steppings of the C2000 product family. While a platform fix has been identified, many manufacturers are choosing to recall products altogether rather than apply it. Cisco has announced that it will proactively replace hardware that was still under warranty or any other type of valid service contract as of November 2016. Hardware will be replaced whether it has already failed or not. Cisco isn’t formally using the word “recall,” but that’s what this is.

Did Intel use NDAs to squash reporting?

ServeTheHome did some digging into the situation and found multiple vendors willing to acknowledge being affected by the issue: Cisco, Netgate, QCT, and Supermicro all acknowledged the flaw — but not the cause.

ServeTheHome states that it has spoken to eight separate vendors, none of whom were willing to acknowledge which supplier was responsible. Multiple vendors cited NDAs with their suppliers that prevented them from identifying the source. Not many companies have the clout to force multi-billion dollar corporations to kowtow to its desire for secrecy. Not many suppliers are in a position to dictate to their customers how they will and won’t address a problem or speak to its cause. And of course, there’s the fact that Intel just updated its own documentation to identify exactly the problem in the C2000 family that eight separate vendors (so far) have acknowledged in their own products.

If you have Avoton or Rangeley hardware deployed for any reason, including as a component in embedded systems or networking hardware, we recommend contacting the vendor to determine whether replacement hardware is required or available.

Quote:As detailed in a recent Product Change Notification (PCN) document, Intel is refreshing four of the company's Celeron and Pentium Apollo Lake processors due to degradation concerns. The problem stems from the same issue that initially cropped up with the C2000 Atom family, which had enough of an impact that Intel had to establish a reserve fund to cover the costs associated with replacing the processors.
...
It remains to be seen if this issue is as widespread as the C2000 issue, and given that this also impacted the E3800-series processors, we might see even more Intel processor families impacted. We're reaching out to Intel for comment for further details, and will update as necessary.

Quote:We've never seen a PCN pulled from Intel's site, so we followed up with the company to investigate. Intel tells us it posted the PCN in error, and the new version has no reference to an LPC bus degradation issue.

Apparently the Apollo Lake Pentium and Celeron chips do not suffer from LPC bus degradation.

Here you can see the new PCN (PDF) posted to Intel's site this morning. The new PCN clarifies the stepping change, with an obvious emphasis on assuring the chips are in fact reliable. But it doesn't answer the question of why the change is being made.

However, the change says "Customers that do not need Intel IOTG Long Life Product Availability" can continue to purchase the older chips. Instead, the new chips are positioned for long life availability.

Quote:According to this new document, customers who do not need “Intel IOTG Long Life Product Availability” can continue to rely on the B1 stepping parts without any worry. Only customers who need a guarantee under this program, it’s implied, may need to switch from B1 to F1 steppings. The F1 parts, meanwhile, are confirmed to still match all PC hardware requirements.

There are a few more pieces to this puzzle to consider. First, Intel dealt with a very similar problem to this back in 2017, when it announced it would set up a reserve fund for repair costs related to issues with the Atom C2000 family. These parts, codenamed Avoton, had exactly the same issue — Circuit degradation can lead to early product failure.

Also in 2017, Intel extended the expected platform life for its IoT products, from seven years to 15. It may be that this is the “Intel IOTG Long Life Product Availability” that the company referred to in the text above. What Intel is saying, we think, is that only customers who expect to need platform support for the full 15 years are going to have anything to worry about. Customers who are planning to use the hardware for a length of time that corresponds to the typical PC lifecycle, on the other hand, won’t need to replace equipment.

As for why Intel pulled this PDN and replaced it with a different one, the company is likely sensitive to the idea that any of its products have reliability issues. The fact that this problem seems to be repeating — Apollo Lake chips from 2016 are having problems in 2019, just as Avoton chips launched in 2013 started having issues in 2016 — is not a positive. Hopefully, the new F1 steppings resolve this issue, once and for all.

Quote:After an extended period of back and forth with Intel, we have confirmed that the existing B-1 stepping chips do suffer from the LPC bus degradation issue, which can be rectified via a firmware update. This issue is classified as Errata APL47, as listed here (PDF). When the B-1 Stepping is used in accordance to the published specifications and design guidelines, it meets PC Client Usage requirements.

The new F-1 stepping products have the LPC bus degradation issue fixed in the silicon and will be available for order for 15 years.

We've amended the article below to include the new information and a portion of the errata/fix description.