I wouldn't be very afraid, but I would be pretty worried about this becoming Skynet. (Image Source: DARPA)
The Defense Advanced Research Projects Agency (DARPA), the Defense Department agency that specializes in research and development, has been known for creating some of the more interesting aspects of military technology. They are behind the predecessor of the Internet known as ARPANet, modern radar, and drone warfare. The drone aspect has been a particular interest to DARPA in recent decades, especially the notion of not needing humans to run specific things on site. However, their latest project, takes things a bit too far: DARPA has announced a contest, the Cyber Grand Challenge to create an unmanned computer system that could fix errors, hacks, and exploits on its own without need human assistance. In other words, DARPA wants to create the first step toward Skynet.
The Cyber Grand Challenge is built on the premise of the successful DARPA Grand Challenge, a series of races intended to make self-driving cars a reality, in order to make the military more efficient. In the Cyber Grand Challenge, the issue at heart is minimizing the impact of hacking and hackers, as cyber attacks have increased over the years, and cyber "wars" may become the future of wars between nations. The idea is to run a "capture-the-flag" hacking event where unmanned systems would essentially attempt to infiltrate each other in real-time and steal data, all the while trying to protect themselves from hacker attacks, both human and automated. DARPA intends to run these events like a tournament, with the final in 2016 for a total prize of $2 million.
This would be fine and dandy, except there's one problem with fully automated systems: What if you need them to shut down, for whatever reason? By allowing a computer or software to fix itself, a situation appears that a fully-automated computer or software may think of legitimate reasons to keep itself running, but more importantly may not see bugs as anything other than features. By creating this, there is distinct possibility that it will start attacking humans in self-defense, sort of like Skynet did. It may not be like the Terminator movies, but it certainly will cause havoc for humans in some ways, such as taking over key pieces of infrastructure. As it is, a lot of Internet projects, both military and civilian, have started out with good intentions, only to end up wrecking people's lives.