Birthday Dreams

So lets be serious for a minute, how old does a kid have to be before they have a birthday party that they will remember? Everyone says the first few are just for the parents since the kids don't know whats going on and won't remember anyway...2?...3? even 4?
Just in case you are wondering, I've probably been planning my baby's first birthday since I was 17. I love parties, I love themes and I LOVE PLANNING.
I think I am settled on age 2. For baby's first birthday (BTW we decided not to find out the gender) we can do something simple like shapes and colours. For their second, IT WILL BE AMAZING. I'm thinking garden party. This will work for a girl or a boy.
If we have a boy, we can dig in the dirt for bugs, make creepy crawlers and eat mud pie.
AND it fits for a girl too. We can make flower pots and plant fun flowers, play games with ladybugs and butterflies and still eat mud pie.
We can make bug and flower cut outs to decorate and serve pinwheel sandwiches and call them snails.
Baby is moving around a lot, I think they like the idea too!