Silent Data Corruption: Test or Reliability Problem?

Abstract

Recently, companies such as Google, Meta (Facebook), and Microsoft reported in the mainstream press about seemingly random errors which, initially undetected ("silently"), had crept into their large cloud data centers. These reports mentioned that very specific instructions were intermittently incorrectly executed, propagated through the operating system, and would potentially manifest themselves as application-level errors. Are the root causes of these so-called silent data errors test escapes and/or reliability issues? Why are they only noticed now? Is that only the case because such large server farms bring together larger numbers of CPUs than ever seen before? And what counter measures can we take against them?