What to Look For and Why: Understanding Memory Analysis Objectives
Part 1: Program Initialization Analysis
What to Look For:
1. Entry point (`_start` or `main`)
- The very first instruction executed
- How the program sets up its environment
2. Stack initialization
- Initial stack pointer value
- How the stack frame is established
3. Global/static variable initialization
- Where and when global variables get their values
Why It's Important:
- Security: Malware often hijacks the entry point
- Debugging: Understanding initialization helps trace later issues
- Performance: Initialization patterns affect startup time
- Compatibility: Different systems initialize memory differently
Part 2: Stack Frame Analysis
What to Look For:
1. Function prologue
```assembly
push rbp ; Save old base pointer
mov rbp, rsp ; Set new base pointer
sub rsp, 0x20 ; Allocate space for locals
```
2. Local variable allocation
- Where local variables live relative to RBP
- How much stack space each function uses
3. Function epilogue
```assembly
mov rsp, rbp ; Clean up locals
pop rbp ; Restore old base pointer
ret ; Return to caller
```
Why It's Important:
- Buffer Overflow Detection: Understand where buffers are vulnerable
- Reverse Engineering: Helps reconstruct high-level code from assembly
- Optimization: Identify stack space waste
- Debugging: Locate corrupted stack frames
Part 3: Heap Memory Analysis
What to Look For:
1. malloc/calloc calls
- Allocation sizes and patterns
- Memory alignment considerations
2. Heap metadata
- How the heap manager tracks allocations
- Chunk headers and footers (in glibc)
3. Heap fragmentation
- Free chunks and their management
- Coalescing of adjacent free chunks
Why It's Important:
- Memory Leaks: Unfreed allocations waste resources
- Use-After-Free: Accessing freed memory causes crashes/exploits
- Heap Overflow: Overflowing one chunk corrupts adjacent ones
- Performance: Fragmentation slows allocation
Part 4: Buffer Overflow Analysis
What to Look For:
- Which way does the overflow go (up or down in memory)?
- What gets overwritten first?
3. Control flow hijacking
- Can you overwrite the return address?
- What value would redirect execution?
Why It's Important:
- Exploit Development: Understanding how attacks work
- Defense Implementation: Knowing what to protect
- Vulnerability Assessment: Identifying risky code patterns
- Forensics: Recognizing attack patterns after compromise
Part 5: Function Call Convention Analysis
What to Look For:
1. System V AMD64 ABI (Linux x86_64)
- Arguments: RDI, RSI, RDX, RCX, R8, R9, then stack
- Return value: RAX
- Callee-saved: RBX, RBP, R12-R15
- Caller-saved: Everything else
2. Stack alignment
- RSP must be 16-byte aligned before `call`
- Compiler ensures this with `sub rsp, X`
Why It's Important:
- Interoperability: Functions must agree on how to pass data
- Debugging: Understand where to find function arguments
- Optimization: Calling conventions affect performance
- Binary Analysis: Reconstruct function signatures
Part 6: Memory Layout Analysis
What to Look For:
Why It's Important:
- ASLR: Address Space Layout Randomization affects predictability
- Exploit Mitigation: Non-executable stacks/heaps
- Memory Protection: Different segments have different permissions
- Forensics: Memory layout reveals system and compiler details
Part 7: Control Flow Analysis
What to Look For:
1. Direct jumps/calls
- `jmp 0x401234`
- `call 0x401000`
2. Indirect jumps/calls
- `jmp rax`
- `call [rbx+0x8]`
3. Conditional branches
- `je`, `jne`, `jg`, etc.
- Based on flags register
Why It's Important:
- Program Understanding: Follow the execution path
- Malware Analysis: Detect obfuscated control flow
- Optimization: Identify hot paths and bottlenecks
- Vulnerability: Spot where control can be hijacked
Part 8: System Call Analysis
What to Look For:
1. Syscall invocation (x86_64)
```assembly
mov rax, 1 ; syscall number (write = 1)
mov rdi, 1 ; fd = stdout
mov rsi, buffer ; buffer address
mov rdx, len ; length
syscall
```
2. Common syscalls to monitor
- `execve` (process execution)
- `mmap` (memory mapping)
- `mprotect` (permission changes)
- `ptrace` (debugging/anti-debugging)
Why It's Important:
- Malware Detection: Unusual syscall patterns indicate malicious behavior
- Sandbox Evasion: Malware checks for debuggers via syscalls
- Resource Monitoring: Understand what resources the program uses
- Forensics: Syscalls leave traces in logs
Part 9: Data Structure Analysis
What to Look For:
1. Arrays
- Contiguous memory
- Index calculation: `base + index * size`
2. Structs
- Fields at fixed offsets
- Padding for alignment
3. Linked lists
- Nodes with data + next pointer
- Traversal via pointer chasing
Why It's Important:
- Memory Corruption: Understanding layout helps exploit/defend
- Performance: Data structure choices affect cache usage
- Reverse Engineering: Reconstruct high-level data types
- Debugging: Know what memory should contain
Part 10: Security Mechanism Analysis
What to Look For:
1. Stack canaries
- Random value before return address
- Checked before function return
2. NX/DEP (No-Execute)
- `mprotect` calls changing permissions
- Segments without execute permission
3. ASLR
- Randomized base addresses
- Different each run
4. RELRO
- Read-only GOT after relocation
- Prevents GOT overwrite attacks
Why It's Important:
- Attack Mitigation: Understand what protections are in place
- Exploit Development: Know what to bypass
- Security Assessment: Evaluate program hardening
- Forensics: Detect bypass attempts
Real-World Applications
1. Vulnerability Research
What to look for: Unsafe functions (`strcpy`, `gets`), lack of bounds checking, integer overflows
Why: These are the root causes of exploits in real software
2. Malware Analysis
What to look for: Unusual syscalls, code injection, anti-debugging tricks
Why: Malware uses memory manipulation to evade detection
3. Performance Optimization
What to look for: Memory access patterns, cache usage, allocation frequency
Why: Memory is often the bottleneck in high-performance software
4. Forensics
What to look for: Memory artifacts, execution traces, modified code
Why: Memory contains evidence even after files are deleted
5. Reverse Engineering
What to look for: Function boundaries, data structures, control flow
Why: Needed for interoperability, security audits, legacy systems
Assessment
Beginner:
1. What is the difference between stack and heap memory?
2. How does a function call affect the stack?
3. What register holds the return value?
Intermediate:
1. How would you detect a buffer overflow in this code?
2. What memory protections are enabled and why?
3. How could an attacker hijack control flow?
Advanced:
1. Write an exploit for the vulnerability
2. Implement a security mitigation
3. Optimize the memory usage
