Yes, it matters a great deal.
No one can tell you exactly how many “points” you will lose on your score (as this is determined by the rest of your credit history, with the lates in context, and is highly individualized).
Regardless, for most lenders and creditors, going 120 days late on your mortgage is the considered THE SAME AS A FORECLOSURE.
It does not matter whether or not your lender actually forecloses or takes legal action. Most future creditors will assume that they would have / should have – and you will be subject to the same future credit restrictions as someone who has actually experienced foreclosure proceedings. You do not want a 120 day late reporting on your credit report.
It may be possible to explain away a single 30 or 60 day late pay on your mortgage, or to rebuild your credit rating just 12-24 months after the late occurs. If you go “later” than 60 days, the incident will be viewed as much more severe, and you will likely find that you will be denied many credit opportunities in the 1-5+ years that follow.
If you are having trouble making your mortgage payments, contact your lender immediately. Let them know what is going on, what you are trying to do to fix it, and how much you can actually afford to pay. Many lenders will work with you to accept lower payments temporarily, or to modify the terms of your loan if you are experiencing a genuine hardship and cannot sell. Gather all the paperwork you have to document the difficulty, as well as paystubs and bank statements to document your income and savings, and call your lender right away. Be persistent, and document each conversation that you have and everything you do to help your own cause.
If at all possible, do not allow your mortgage to be “later” than 60 days.