It would be unusual, but not unprecedented, for a lender to modify your loan if you are paying as agreed. Typically, lenders aren’t motivated to do modifications unless they fear borrowers will default. If your lender’s refi department referred you to the mod department, one would hope they feel you would qualify for a modification…..or at the very least that you don’t qualify for a refinance for whatever reason. The 96% loan to value might or might not be part of the reason for the refi turn down, depending on your current loan’s eligibility for refi programs like a HARP, FHA streamline, or VA IRRRL. The modification department will have to tell you if you are eligible based on their specific guidelines and your particular circumstances. Do yourself a favor, and take detailed notes on every conversation with them, doing as much as possible via email so you have a record of what you’ve been told when, and by whom!