You are basically asking two different questions. Let’s separate the two. Yes, gift money can be used towards a downpayment. FHA loans for example allow 100% of the downpayment to be gift funds. Most FannieMae lenders allow 100% of the downpayment to be gift funds as long as it is a 20% downpayment. Some lender/investors do have restrictions on this so it’s best to talk with your lender before making an assumption. If they don’t like your solution then shop around.
With less than 20% downpayment using conventional loans typically the Mortgage Insurance providers are going to require some percentage of the funds to come from your own account, typically that is 5%, but you should check with your lender to verify.
The final thing on gift funds is that typically they need to be from a “relative”, Parents, siblings, cousins, etc.
With regards to the IRS that’s really a tax question for a tax advisor to answer. In my experience there is no reporting between the gift letter/funds that are provided to the lender and the IRS.