I love this question, and I would like to break this up into a few parts to discuss the different loan types/products.
The basic answer is mortgage backed securities. You have probably heard Fannie Mae (FNMA) and Freddie Mac, Ginnie MAE …
At the root there is a security bond sold (on the bond market on wall street etc.) at a set rate, an example would be the FNMA 5.5 30yr, or the FNMA 6.0 30yr. These are long term fixed income bonds.
The more money pouring into a particular bond drives the value up (and mortgage rates go down) the more money coming out of a security (and back into the stock market) drives the value of the bond down (rates go up)
Now…some loans are based on other indices plus a margin like the LIBOR (London inter bank offering rate) the COFI (cost of funds index. Etc.
The other factor is Risk. If a mortgage backed security is rated as risky then the value of the bond drops.
In a nutshell when the mortgage market crunch started, the biggest problem was that seemingly overnight investors could not tell the true value of the various securites that were backing subprime mortgages, and would not lend money to each other or continue to back the security. Which forced many companies that had funded subprime loans to “keep” them, and not be reimbursed.
That is the short answer! keep in mind that at the end each company has to earn a profit prior to selling it to wall street and as each company has different costs to cover they will have a variance in rate you can get on any given day.
Rates and fees should not be the determining factor in choosing a lender since a mortgage has a vast amount of influence on your overall financial and tax strategies, and in this market you are talking about a few hundred dollars saved (seemingly) that can cost you tens of thousands if your total financial picture is not considered.
Hope that helps!