I think you are massively overestimating the ability of even a very strong narrow hacker AI to hide from literally everyone.
I seriously hope you are right, but from what I’ve learned, Reverse Code Engineering (RCE) is not done quickly or easily, but it’s a finite game. I know the goal; then, I believe you can define rules and train an AI. RCE is labor-intensive; AI could save me a lot of time. For an organization that hires many of the brightest IT minds, I m convinced they ask the right questions for the next iteration of Hacker-AI. I may overestimate how good Hacker-AI (already) is, but I believe you underestimate the motivation in organizations that could develop something like that. Personally, I believe, work on something like that started about 7 or 8 years ago (at least) - but I may be off by a few years (i.e. earlier).
Yes, Hacker-AI would need to hide from all detection setups – however, they are at most in the few K or 10K range (for all systems together) but not in the millions range. Additionally, there are a few shortcuts this Hacker-AI can take. One: make detectors “lie”—because hacker-AI has the assumed ability to modify malware detectors as well (Impossible? If a human can install/de-install it, so do a Hacker-AI). Also: operators do not know what is true; they accept data (usually) at face value. A ghost could run the detector app in a simulator is another scenario. And then there is the “blue screen of death”. Hacker-AI could trigger it before being discovered – and then who is being blamed for that? The malware detector app, of course …
Regarding military systems: I don’t know enough about them; what I read did not give me confidence that what they offer is sufficient – but I might have had a bias. From what I read, I assume: they have a main CPU (with von Neuman architecture), unified RAM, logical address space, and (a much more complex) access control system, all managed by a single OS—and yes: many standard tools are missing. Are the differences between commercial and military systems significant enough? How can I know? I am (simply) skeptical about claims like: “Optimized for Cyber-Defensibility” (sounds to me like marketing talk).
it doesn’t mean the world is safe; I think you are quite right that this weapon can be constructed. However, I don’t think it would be the same kind of catastrophic invisible-until-no-response-is-possible failure that a strongly superhuman planning-self-coherent-and-causal UFAI could use to take over the internet, and I don’t think a human creator could get away clean from trying to take over the world using it. the military systems are probably vulnerable, but not so vulnerable that it could completely evade detection, unless targeted specifically—effectively at that point you’re describing a stuxnet-like attack. it’s a real thing, but I think you’re overestimating maximum cloakability slightly. I do not intend to reassure.
Overestimating to which degree Hacker-AI could make itself undetectable? And do I potentially underestimate the effort of making a digital ghost undetectable for malware detection? I disagree because I have answered the following questions for myself.
(1) How conceptionally different are the various operating systems? In 1,000s of details, I believe they are different. But the concepts of how they are designed and written are similar among all multitasking/multithreading OS. Even multi-processing OS are built on similar but extended concepts.
(2) If we ask kernel developers: could they keep app(s) operational but disappear them? I believe he would have multiple ideas on how he could do that. Then we could ask what he could do to cover his tracks (i.e., that he made these changes to the kernel): Could they make changes they made to the OS disappear/undetectable for the OS? I believe yes. A detection tool run on a compromised system could be forced to jump into another memory area, in which these changes were not made, and then forced to jump back. Could a detector be deceived about that? Again, I believe: yes. These are instructions to the DMA (direct memory access) loading data from RAM into the different caches or processor kernels.
(3) Because OS and CPU were designed and optimized for improving performance, many low-level concepts are not done with simplicity in mind. How could a detector determine that a certain operation is due to resource optimization and which was done due to a ghost trying to make itself undetectable?
I don’t think this is productive because I don’t actually disagree with your core point as much as my reply seems to have made it appear. all three points are quite solid and I have no serious criticism of your questions; I’m not comfortable answering questions like these, though. Perhaps someone else in the security research community can comment.
I seriously hope you are right, but from what I’ve learned, Reverse Code Engineering (RCE) is not done quickly or easily, but it’s a finite game. I know the goal; then, I believe you can define rules and train an AI. RCE is labor-intensive; AI could save me a lot of time. For an organization that hires many of the brightest IT minds, I m convinced they ask the right questions for the next iteration of Hacker-AI. I may overestimate how good Hacker-AI (already) is, but I believe you underestimate the motivation in organizations that could develop something like that. Personally, I believe, work on something like that started about 7 or 8 years ago (at least) - but I may be off by a few years (i.e. earlier).
Yes, Hacker-AI would need to hide from all detection setups – however, they are at most in the few K or 10K range (for all systems together) but not in the millions range. Additionally, there are a few shortcuts this Hacker-AI can take. One: make detectors “lie”—because hacker-AI has the assumed ability to modify malware detectors as well (Impossible? If a human can install/de-install it, so do a Hacker-AI). Also: operators do not know what is true; they accept data (usually) at face value. A ghost could run the detector app in a simulator is another scenario. And then there is the “blue screen of death”. Hacker-AI could trigger it before being discovered – and then who is being blamed for that? The malware detector app, of course …
Regarding military systems: I don’t know enough about them; what I read did not give me confidence that what they offer is sufficient – but I might have had a bias. From what I read, I assume: they have a main CPU (with von Neuman architecture), unified RAM, logical address space, and (a much more complex) access control system, all managed by a single OS—and yes: many standard tools are missing. Are the differences between commercial and military systems significant enough? How can I know? I am (simply) skeptical about claims like: “Optimized for Cyber-Defensibility” (sounds to me like marketing talk).
it doesn’t mean the world is safe; I think you are quite right that this weapon can be constructed. However, I don’t think it would be the same kind of catastrophic invisible-until-no-response-is-possible failure that a strongly superhuman planning-self-coherent-and-causal UFAI could use to take over the internet, and I don’t think a human creator could get away clean from trying to take over the world using it. the military systems are probably vulnerable, but not so vulnerable that it could completely evade detection, unless targeted specifically—effectively at that point you’re describing a stuxnet-like attack. it’s a real thing, but I think you’re overestimating maximum cloakability slightly. I do not intend to reassure.
Overestimating to which degree Hacker-AI could make itself undetectable? And do I potentially underestimate the effort of making a digital ghost undetectable for malware detection? I disagree because I have answered the following questions for myself.
(1) How conceptionally different are the various operating systems? In 1,000s of details, I believe they are different. But the concepts of how they are designed and written are similar among all multitasking/multithreading OS. Even multi-processing OS are built on similar but extended concepts.
(2) If we ask kernel developers: could they keep app(s) operational but disappear them? I believe he would have multiple ideas on how he could do that. Then we could ask what he could do to cover his tracks (i.e., that he made these changes to the kernel): Could they make changes they made to the OS disappear/undetectable for the OS? I believe yes. A detection tool run on a compromised system could be forced to jump into another memory area, in which these changes were not made, and then forced to jump back. Could a detector be deceived about that? Again, I believe: yes. These are instructions to the DMA (direct memory access) loading data from RAM into the different caches or processor kernels.
(3) Because OS and CPU were designed and optimized for improving performance, many low-level concepts are not done with simplicity in mind. How could a detector determine that a certain operation is due to resource optimization and which was done due to a ghost trying to make itself undetectable?
I don’t think this is productive because I don’t actually disagree with your core point as much as my reply seems to have made it appear. all three points are quite solid and I have no serious criticism of your questions; I’m not comfortable answering questions like these, though. Perhaps someone else in the security research community can comment.