There are several possible reasons.
One is that you are, in fact, not running the amd64 build. Another is that your motherboard and/or BIOS are broken and not reporting the correct amount of RAM.
Another is that you have video memory and other hardware resources taking up space in the 3-4 GB area of memory, and your motherboard/BIOS is incapable of hoisting the shadowed RAM to higher addresses so that it can be accessed.
Figuring out which requires looking at your dmesg output.