“An Artificial [General] Intelligence could rewrite its code from scratch — it could change the underlying dynamics of optimization.”
Optional extra (for ML researchers and techxecutives):
“Such an optimization process would wrap around much more strongly than either evolution accumulating adaptations, or humans accumulating knowledge. The key implication for our purposes is that an AI might make a huge jump in intelligence after reaching some threshold of criticality.”
“An Artificial [General] Intelligence could rewrite its code from scratch — it could change the underlying dynamics of optimization.”
Optional extra (for ML researchers and techxecutives):
“Such an optimization process would wrap around much more strongly than either evolution accumulating adaptations, or humans accumulating knowledge. The key implication for our purposes is that an AI might make a huge jump in intelligence after reaching some threshold of criticality.”
EY, AI as a pos neg factor, 2006ish