The insecure domains mainly work because people have charted known paths, and shown that if you follow those paths your loss probability is non-null but small.
I think this is a big part of it, humans have some kind of knack for working in dangerous domains successfully. I feel like an important question is: how far does this generalize? We can estimate the IQ gap between the dumbest person who successfully uses the internet (probably in the 80′s) and the smartest malware author (got to be at least 150+). Is that the limit somehow, or does this knack extend across even more orders of magnitude?
If imagine a world where 100 IQ humans are using an internet that contains malware written by 1000 IQ AGI, do humans just “avoid the bad parts”? What goes wrong exactly, and where?
I feel like an important question is: how far does this generalize? We can estimate the IQ gap between the dumbest person who successfully uses the internet (probably in the 80′s) and the smartest malware author (got to be at least 150+). Is that the limit somehow, or does this knack extend across even more orders of magnitude?
If imagine a world where 100 IQ humans are using an internet that contains malware written by 1000 IQ AGI, do humans just “avoid the bad parts”?
For reactive threats, the upper bound is probably at most “people capable of introspection who can detect they are not sure some action will be to net benefit, and therefore refuse to take it”. For active threatening factors, that’s an arms race (>=40% this race is not to infinity—basically, if more-cooperating DT strategies are any good).
Maybe the subject is researched more in biology? Example topic: eating unknown food (berries, nuts) in forest, and balance of lifetime adaptation vs evolutionary adaptation (which involves generations passing).
For almost everything, yeah, you just avoid the bad parts.
In order to predict the few exceptions, one needs a model of what functions will be available in society. For instance, police implies the need to violently suppress adversaries, and defense implies the need to do so with adversaries that have independent industrial capacity. This is an exception to the general principle of “just avoid the bad stuff” because while your computer can decline to process a TCP packet, your body can’t decline to process a bullet.
If someone is operating e.g. an online shop, then they also face difficulty because they have to physically react to untrusted information and can’t avoid that without winding down the shop. Lots of stuff like that.
I think this is a big part of it, humans have some kind of knack for working in dangerous domains successfully. I feel like an important question is: how far does this generalize? We can estimate the IQ gap between the dumbest person who successfully uses the internet (probably in the 80′s) and the smartest malware author (got to be at least 150+). Is that the limit somehow, or does this knack extend across even more orders of magnitude?
If imagine a world where 100 IQ humans are using an internet that contains malware written by 1000 IQ AGI, do humans just “avoid the bad parts”? What goes wrong exactly, and where?
For reactive threats, the upper bound is probably at most “people capable of introspection who can detect they are not sure some action will be to net benefit, and therefore refuse to take it”. For active threatening factors, that’s an arms race (>=40% this race is not to infinity—basically, if more-cooperating DT strategies are any good).
Maybe the subject is researched more in biology? Example topic: eating unknown food (berries, nuts) in forest, and balance of lifetime adaptation vs evolutionary adaptation (which involves generations passing).
For almost everything, yeah, you just avoid the bad parts.
In order to predict the few exceptions, one needs a model of what functions will be available in society. For instance, police implies the need to violently suppress adversaries, and defense implies the need to do so with adversaries that have independent industrial capacity. This is an exception to the general principle of “just avoid the bad stuff” because while your computer can decline to process a TCP packet, your body can’t decline to process a bullet.
If someone is operating e.g. an online shop, then they also face difficulty because they have to physically react to untrusted information and can’t avoid that without winding down the shop. Lots of stuff like that.