It's the longterm potential versus the short-term payout. Put the ads up early, people may not get super excited and share your site with all their friends. Wait too long, and you may not be able to pay your hosting bills. Either could result in the death of your site.
Personally, I think anyone who isn't familiarizing themselves from day one with the various options for display advertising, how their various back ends work, etc, is putting themselves behind in the game when they do get to reasonable traction.
The typical blog doesn't look remotely spammy if it has a single sidebar text-only adsense unit, especially if the overall site is well designed, and the ad is both clearly separated from the content and subtly pushed into the background. Especially when you're just starting out, you can be doing a tremendous service to your readers – sending them to another site with more/better content. Plus, you get paid $0.02 for it! Everyone wins!
The important bit is this: Once you have ads, there will be a tremendous opportunity, and accompanying pressure, to increase their performance and value.
If you're a small blog or site, these efforts all rob time that you could be using to make your own site better, or at least procrastinate by answering web dev questions on Quora. Hypothetically speaking, of course.