For my son's 3rd birthday we asked for a donation to charity instead of giving toys, because my son has enough toys already. We also gave the option of books for those who really wanted to bring a gift, because one can't have too many books

Everyone ended up bringing a gift, one person gave a nice book and told us which charity they donated to, the majority gave nice books, and a few brought toys.
None of our friends told us they didn't like our request or were offended. Perhaps if we just said no gifts at all, only charity donation they might have been offended, as it seems most preferred bringing a gift.
We will likely only throw a large birthday party every few years and celebrate the rest of the birthdays in a fun way with just our family.