The scope of operating system security is broad. This chapter focuses on some of the most important topics. The most prominent issue for OS security is countering thread from intruders and malicious software. Intruders attempt to gain unauthorized access to system resources, while malicious software is designed to penetrate system defenses and become executable on target systems. Countermeasures to both types of threat include intrusion detection systems, authentication protocols, access control mechanisms, and firewalls.
One of the most common techniques for compromising OS security is the buffer overflow attack. A condition at an interface under which more input can be placed into a buffer or data-holding area than the capacity allocated, overwriting other information. Attackers exploit such a condition to crash a system or to insert specially crafted code that allows them to gain control of the system. System designers use a variety of compile-time and runtime defenses to counter this type of attack.
Another important area of security defense is access control. Access control measures include those that secure access to file system and to the OS user interface. Traditional techniques for access control are referred to as discretionary access control. A more flexible approach that has gained considerable support is role-based access control, in which access depends not only on the identity of the user but on the specific role that user can assume for a specific task or set of tasks.