You may be OK here, assuming your son has adequate credit. Bigger question may be why you want to add him…..do you need his income to qualify? Does he live in the home? Are you solely trying to get him added to title, or to mortgage for interest income tax deduction? Are you doing a conforming, FHA, or VA loan? Best course of action is to discuss your situation (and motivation for adding your son) with your loan officer for more detailed answers.
If you and your husband qualify for the refinance without your adult child on the loan, why add him? Adding him will make him liable for the debt but will not give him an ownership interest in the property. If and when he wants to get credit for some other purpose (for example, to buy a home of his own, a car, etc), the debt on your home will be reflected on his credit report, which could make it harder for him to get additional credit. If you’re trying to convey your home to your son, this isn’t the way to do it.
As I suggested, you may want to include him on title, but not the mortgage unless you need his income to qualify for the loan!