Memory Safety in Operating Systems ​

Data is the core component of computer systems, and without a way to reliably store and manage data a computer system would be extremely inefficient. As computers have evolved so has the strategies Operating Systems employ to manage memory. Fixed partitions were an early attempt at memory management in which main memory would be split into partitions and each partition could service a single job, this evolved into much more advanced virtual memory management strategies with paging and segmentation. Because memory is vital in computing, many exploits exist to compromise the data stored in memory and the chance of being compromised by such an exploit is very high.

“For example, Google’s Project Zero team analyzed vulnerabilities that were used in the wild by attackers before they were reported to software providers (also called “zero-day vulnerabilities”). Out of the 58 vulnerabilities for the year, 39, or 67% were memory corruption vulnerabilities.” (Lord, 2023).

This widespread issue has led to the creation of many strategies to defend against memory vulnerabilities. Virtual memory management, process isolation, memory safe languages, and more have been developed to increase memory security.

The United States Cybersecurity & Infrastructure Security Agency CISA states that memory safety is a key issue in the security of computing systems.

“Memory safety vulnerabilities are the most prevalent type of disclosed software vulnerability. They are a class of well-known and common coding errors that malicious actors routinely exploit. These vulnerabilities represent a major problem for the software industry as they cause manufacturers to continually release security updates and their customers to continually patch” (CISA, 2020)

Memory vulnerabilities are very widespread and often the result of poor programming choices, however, their impact doesn’t change. Buffer overflows occur when an attacker injects more data than is expected into a program, usually due to failing to check the bounds on user inputted data, causing the program to execute malicious code. These were more common before modern memory security practices were introduced in lower-level programming languages such as C, due to the nature of C in which the bounds checking is entirely up to the programmer. Out of bounds reads are another vulnerability in which a program returns data outside the bounds of what the program is supposed to return.  This has led to multiple high-profile vulnerabilities such as Heartbleed, or Meltdown and Specter, both of which allow an attacker to read data they should not have access to. Heartbleed allows attackers to read data from servers and Meltdown and Specter allow attackers to read data from individual users such as passwords or bank account information. Operating systems have developed strategies to decrease the chance a process can gain access to invalid memory, one of the main techniques is through virtual memory management.

Operating Systems or OS’s are the foundation every program builds upon to function. The OS abstracts away input hardware, storage devices, and memory to give developers a unified platform to build software for. One of the key jobs of operating systems is providing processes with memory which is achieved through virtual memory management.

“In a virtual memory system, the addresses a program may use to identify information are distinguished from the addresses the memory system uses to identify physical storage sites, and program-generated  addresses are translated automatically to the corresponding machine addresses.” (Denning, 1970)

Security through virtual memory is achieved through the use of the memory management unit, and by the OS having a complete map of processes and memory the OS can isolate processes from each other. Process isolation protects memory security by placing and enforcing boundaries on memory access. The operating system also places permissions on each page of memory indicating which processes are allowed to read, write, or execute the page of memory. Virtual memory management also allows for address space randomization in which the memory space given to each process is randomized.

“Address Space Layout Randomization (or ASLR) is the randomization of the place in memory where the program, shared libraries, the stack, and the heap are. This makes can make it harder for an attacker to exploit a service, as knowledge about where the stack, heap, or libc can't be re-used between program launches” (OSIRIS Lab, 2024)

ASLR is a key component of memory security due to the complex nature of software development and many programs relying on external libraries. Operating systems for instance include standard libraries to help software developers create programs,  these include common functions to interface with the OS such as writing input to a terminal or interacting with hardware devices. Lib-C is the standard Linux library providing functions such as printf(), malloc(), and free(), which are extremely common to use in software development. These functions can be used in a safe way, however due to their low-level nature it takes considerable effort on the programmers’ side to do so. Lib-C being shared by most all software developed for Linux creates a large target for attacks such as buffer overflow attacks. By knowing the memory address of your program an attacker could find where the Lib-C code is in memory and exploit poor handling of system functions to create an overflow and execute malicious code stored elsewhere in memory. This is where ASLR becomes very important. ASLR when a new process is launched automatically randomizes all the memory addresses for the process and related processes. This thwarts many common attacks which rely on overflowing or underflowing memory as the attacker has no way of knowing where to send the program to the malicious code residing in memory. This process of forcing a program to execute malicious code is known as control flow hijacking.

Modern operating systems have many protections in place to protect the integrity of control flow of a program to detect and stop the hijacking of control flow attempted by attackers.

“The CFI security policy dictates that software execution must follow a path of a Control-Flow Graph (CFG) determined ahead of time. The CFG in question can be defined by analysis—sourcecode analysis, binary analysis, or execution profiling.” (Abadi et al., 2009)

When developing software compilers can generate a map, or graph, of when and where every function can be called. The operating system then works with this data to detect when a program is hijacked because the current instruction pointer of the program will be laying outside of the predetermined control flow graph. This can either be implemented at the software level, or with more advanced computer systems this can be done at the hardware level. Intel CPU’s implement Intel Control-Flow Enforcement Technology CET which works in conjunction with the operating system to detect control flow hijacking. CET uses a shadow stack which is a copy of the current stack that stores the addresses of function calls. This shadow stack is implemented in hardware as a separate stack and this is separate from all the control and instruction logic on the CPU which makes it incredibly difficult for attackers to modify. CET also implements indirect branch tracking that leverages modern branch prediction algorithms that allow for dramatically increased processing speed by pre calculating the different branches a program can take. For example, a modern cpu can calculate both sides of an IF statement in a program and only continue down the branch that executes correctly. Intel CPU’s can take advantage of branch prediction by detecting when an attacker forces the code to appear in an unexpected branch.

“IBT is meant as a defense against jump-oriented programming; it works by trying to ensure that the target of every indirect branch is, in fact, intended to be reached that way. If enabled, he CPU will ensure that every indirect branch lands on a special instruction (endbr32 or endbr64), which executes as a no-op; if anything else is found, the processor will raise a control-protection (#CP) exception.“ (Corbet, 2022)

These advanced hardware features are leveraged by the operating system to create a more secure execution environment.

Operating systems can have systems in place to mitigate the risk of memory vulnerabilities however there are tools in software developers can use in the programming language level to also thwart these attacks. Lower level programming languages such as C or C++ are considered unsafe, not because they can’t be used to create safe software with but because creating memory safe software in these languages takes considerable effort on the programmer’s side. This has led to the development of so called memory safe programming languages.

“Memory safety consists of spatial safety and temporal safety. First, spatial safety is violated by out-of-bounds read or write, as pointers are dereferenced without any check in C/C++. Secondly, the representative example of the violation of temporal safety is use-after-free. In C/C++, allocation and revocation of memory are left entirely to a programmer. As there’s no binding between pointer and pointee, de-allocated region can still be dereferenced.” (Inyoung, 2024)

Several memory safe programming languages have been invented such as Java, C#, Python, all of which sacrifice speed for safety. Java for example has its own runtime, the Java Virtual Machine which handles all the allocation and deallocation of memory, and programs written in Java only have access to the memory inside the virtual machine. These higher level languages, due to being in their own isolated environment, can gracefully handle errors such as bounds checking or use after free and not compromise the rest of the system. Programming languages can also be used to thwart control flow hijacking attacks. Inline Reference Monitors or IRM’s are a form of control flow integrity at the software level. IRM’s work by rewriting programs, usually at compile time, and inserting code into the program to automatically keep track of a program’s state. The injected code into the program updates the IRM’s current state and if the state doesn’t match the expected state the IRM can notify the program that function execution has been hijacked.

“The IRM approach is facilitated by the trend towards using higher-level languages, especially type safe languages, for software development. Not only do those languages define application abstractions on which policies can be enforced, but they also provide strong guarantees that can be used to ensure a secured application cannot compromise its IRM” (Erlingsson, 2004)

The IRM system is very useful especially in applications that handle very sensitive data. IRM’s can also be implemented in memory safe languages to further safeguard the memory and function execution of the program. CISA the Cybersecurity and Infrastructure Security Agency in the United States has recognized this need to better secure memory in programs. One language they recommend is a newer language called Rust which looks to bridge the gap between lower level unsafe languages and higher level safe languages by explicitly controlling memory management through a tool called the borrow-checker. The borrow-checker and explicit mutability of program variables ensure that a software developer handle all edge cases in regards to memory allocation and deallocation allowing for the development of secure memory safe software.

Due to the exponential rise of computer use in our daily lives, the need for secure computing systems has also risen. Memory security is one of the key areas of concern for building secure systems. At the hardware level control flow enforcement technology allows the hardware to signal when control flow is hijacked, address space layout randomization allow the operating system to detect hijacking, and inline reference monitors injected into code can also detect the hijacking of control flow. This multi layered approach is key to ensure the safety of memory in a system, and each layer builds upon the last to deter cyber criminals. The rise in popularity of a programming language like Rust is promising for the future of memory safety in programming languages by decreasing the tradeoff of processing speed between memory safe and unsafe languages.

References ​

Abadi, M., Budiu, M., Erlingsson, U., & Ligatti, J. (2009). Control-flow integrity principles, implementations, and applications. ACM Transactions on Information and System Security (TISSEC), 13(1), 1–40.

CISA, U. (2023). The Case for Memory Safe Roadmaps: Why Both C-Suite Executives and Technical Experts Need to Take Memory Safe Coding Seriously.

Corbet, J. (2022, March 21). Indirect branch tracking for Intel CPUs. LWN.Net. https://lwn.net/Articles/889475/

CWE. (2024, July 19) CWE-125: Out-of-bounds read (4.17). https://cwe.mitre.org/data/definitions/125.html

Denning, P. J. (1970). Virtual memory. ACM Computing Surveys (CSUR), 2(3), 153–189.

Erlingsson, Ú. (2004) The inlined reference monitor approach to security policy enforcement. Cornell University.

Flynn, I. M., & McHoes, A. M. (2017) Understanding operating systems. Cengage Learning.

Inyoung, B. (2024) A Study of Memory Safety in Unsafe and Safe Languages.

Lord, B. (2023, December 6). The urgent need for memory safety in software products. Cybersecurity and Infrastructure Security Agency CISA. https://www.cisa.gov/news-events/news/urgent-need-memory-safety-software-products

OSIRIS Lab. (2024). Address space layout randomization (ASLR). CTF Handbook. https://ctf101.org/binary-exploitation/address-space-layout-randomization/