I'm biased here, but my colleague Edward Chuah has done some research on identifying and predicting these kinds of failures, though we have not taken full advantage of these predictions, yet, at TACC. Perhaps, also, there's not as much ML in this as you'd like to see, depending on your definition of ML.

Only top voted, non community-wiki answers of a minimum length are eligible