This is more a question for your tax advisor, and the answer is generally yes. However, you and the tax advisor, by planning ahead can take advantage of legitimate ways to defer capital gains taxes. The keywords here are “ahead” and “defer”. With good advance planning you can legally defer capital gains taxes on your real estate investments forever under existing IRS rules.
Advance tax planning could include strategies such as the structured sale, charitable trust, installment sale, and a 1031 exchange.
The latter is a well known and often used strategy that basically defers capital gains taxes for purchasing a “like kind” property of equal or greater value than the one you are selling.
Your capital gains avoidance strategy would also have to take into account other aspects of your financial situation. For instance, you may have qualifying stock losses, that taken in the same year, will offset the capital gains tax you face from selling the vacation home.
Most importantly, note that IRS rules and tax laws are constantly changing or being refined, making it that much more important to consult with a good tax advisor or tax attorney.