That's a very good question.
There are a lot of in jokes, but the dialogue is witty and clever and the action is spectacular enough that I think the general public will like it. Whedon has tried to make the movie self-contained, and there are intros to the characters in the beginning, so I think most folks can jump right in and enjoy, regardless of background knowledge.
However, if you've never seen any of the movies, I would recommend checking out: http://marvel.com/avengers_movie/
and look at the "characters section". Browsing through that beforehand will get you up to speed quickly so you can get the most out of the film.