Memory Layout

Notes from this video

Notes on the Content

Introduction & Context

• Before understanding buffer overflows, one must first understand how programs are laid out in memory.
• Focus: Linux process model on x86 (32-bit or 64-bit) systems.
• Though details differ across OS and CPU, the core concepts are similar.

Process Memory Layout

• A program becomes a process when it starts, and is given memory by the OS.
• Processes see memory as if they own it entirely (virtual addresses), though OS maps to physical addresses.
• On a 32-bit system, memory ranges from 0 to 4GB.

🔢 Why 0 to 4GB on a 32-bit System?
1. Address Size & 32-bit CPUs

A 32-bit CPU means the processor’s registers (including the address registers) are 32 bits wide.

Each memory address is represented by a 32-bit binary number.

The maximum value a 32-bit unsigned integer can represent is about 4.29 billion unique addresses.

2. Byte Addressability

Modern systems are byte-addressable, meaning each memory address points to 1 byte (8 bits) of memory.

So, 4.29 billion addresses = 4.29 billion bytes ≈ 4 GB.

That’s why a 32-bit process can only see 4 GB of address space.

Memory Segments

1. Text segment (code)
   • Contains machine instructions.
2. Data segment
   • Initialized data area (e.g., int y = 10;).
   • Uninitialized data area (BSS) (e.g., int x;).
   • Global uninitialized variables default to 0, unlike local ones.
3. Heap
   • Managed by malloc / dynamic allocation.
   • Grows upward in memory.
4. Stack
   • Holds local variables, arguments, and function call metadata.
   • Grows downward in memory.

1. Command-line arguments & environment variables
   • Stored at the top of memory space when a process begins.

🔹 How Mapping Works: Virtual Memory & Paging

Virtual Address Space

When your program uses an address (e.g., 0x12345678), that’s a virtual address.

The CPU cannot directly access physical RAM with this — it must be translated.

Page Tables

The OS maintains a “page table” for each process.

The page table tells the CPU: “virtual page X corresponds to physical page Y.”

Example:

Virtual 0x12345000 → Physical 0x00AB3000


MMU (Memory Management Unit)

The CPU has a built-in component called the MMU.

On every memory access, the MMU looks up the virtual address in the page table to find the physical address.

This happens automatically in hardware and is very fast.

Pages & Frames

Memory is divided into fixed-size chunks called pages (commonly 4 KB).

A virtual page is mapped to a physical frame in RAM.

Some pages may be mapped to disk (swap) if RAM is full.

TLB (Translation Lookaside Buffer)

To speed things up, the CPU caches recent translations in the TLB.

Without this, looking up the page table every time would be too slow.

Stack Details

• The stack pointer tracks the top of the stack.
• When a function is called:
  • Arguments are pushed in reverse order (arg3, arg2, arg1).
  • The return address (instruction pointer) is pushed.
  • The caller’s frame pointer is saved.
  • A new frame pointer is set.
  • Local variables are pushed.
• When returning:
  • Old frame pointer is restored.
  • Instruction pointer is restored from the saved return address.

Frame Pointer & Accessing Variables

• Each function call creates a stack frame (arguments + locals + metadata).
• The frame pointer (EBP) is used to consistently access variables relative to a fixed location.
• Example: Local variable at EBP - 8.
• This ensures variables are addressable no matter where the function is called from.

mental model for EIP, EBP, ESP on 32-bit x86 and how they behave across a function call → execute → return.

---

What they are

EBP (Base/Frame Pointer): a stable reference to the current stack frame so the compiler can access locals/args via fixed offsets (e.g., locals at negative offsets, args at positive offsets). Typically saved/restored on entry/exit of a function.

EIP (Instruction Pointer): holds the virtual address of the next instruction to execute. call, ret, jmp, branches, and interrupts update EIP. The CPU also pushes the return address (the EIP of the next instruction) when you execute call.

ESP (Stack Pointer): points to the top of the stack (x86 stacks grow downward toward lower addresses). push decrements ESP and writes a value; pop reads and then increments ESP.