Have you ever wondered who is pulling out of the marina in those sleek yachts and buying those $5 million condos in Cabo San Lucas?

Apparently there are a good 47,000 people able to do that without breaking a sweat. A report out from the IRS earlier this week says the ranks of the ultrawealthy have grown 62 percent over the past decade.

The tax collection agency says 47,000 Americans had a net worth of $20 million or more in 2004, the latest year they have tabulated. (Really though, how bad is their computer system if it takes them four years to figure that out?!) Another 79,000 had a net worth of 10 to 20 million dollars, up from 51,000 in 1998.

Where are these people? The top two are states where the housing bubble has burst the most dramatically, so these figures may dip lower in years to come. Tops with a net worth of $1.5 million or more were California, Florida, New York, Texas, Illinois, Pennsylvania, and Massachusetts.

The super-rich definitely do not have all their money in real estate though. “Of the total income for the $20 million or more group, the biggest single asset category by far was publicly traded stock ($719.28 billion).”