A detour is a way of causing an API call to go through a different set of code to filter or modify the call in some way. Let's say for sake of example that you want to detour kernel32.dll!ReadProcessMemory to act like it failed every time.
Here's an example call to ReadProcessMemory:
; assume pID of target is in eax
push eax ; dwProcessId = eax
push 0 ; bInheritHandle = false
push 38h ; dwDesiredAccess = VM_READ | VM_WRITE | VM_OPERATION
call kernel32!OpenProcess
push 0h ; lpNumberOfBytesRead = NULL
push 100h ; nSize = 4096 bytes
push 2010A0h ; lpBuffer = 0x002010A0 [address of buffer where we store result]
push 12345h ; lpBaseAddress = 0x00012345 [base address we want to read]
push eax ; hProcess = eax
call kernel32!ReadProcessMemory
Notice that the prototypes specified on MSDN for the functions mentioned here have their parameters in the opposite order to the code above. This is because in the stdcall calling convention (the standard for Windows) parameters are pushed onto the stack in reverse order.
There are two ways to create the detour code. One is to utilize the empty padding at the end of the .code section (usually filled with zeros) in order to patch the binary executable on disk. Another method is to create a runtime patch by injecting a DLL or by remotely allocating a block of executable memory in the process, writing the code there, then calling CreateRemoteThread to perform the reroute. I'm going to show you the former method, in which we patch the executable to fake a failure.
To find the area of code we want to patch, simply load up the program in OllyDbg and scroll to the bottom of the .code section in the disassembler. You'll see a bunch of add instructions (they're actually just a representation of lots of zeros!) which can be overwritten. Scroll to where these instructions start, so you can maximize the amount of space you have to play with. These instructions are never executed normally, so we don't have to take any additional safety measures.
In the case of patching ReadProcessMemory, we look at the parameters list and notice that it has five parameters on the stack. The call will also have pushed the return address on top of that, so we need to pop that off safely. The only register we can safely use in this case is eax (unless we start using pushad/popad) because all other registers are expected to remain untouched. What we need to do is pop off the return address and store it, then remove the parameters, push the return address back onto the stack, set eax to a nonzero value (i.e. an error) and return. In this case we will also use SetLastError to fake an error code.
pop eax ; store return address in eax
add esp, 14h ; remove the next 5 DWORDs from the stack (5x4 = 20d = 14h)
push eax ; push return address back onto stack
push 5h ; dwErrCode = ERROR_ACCESS_DENIED
call kernel32.SetLastError
mov eax, -1 ; set eax to non-zero to fake the error
ret ; return to original code
In OllyDbg you can save this patch to the file, then modify the calls in the process to jump to your routine instead of the actual ReadProcessMemory API.
