Like the old tv show 'Taxi' with Danny Devito and Andy Kaufman, taxis are owned by companies with fleets of yellow cabs. Only large companies can afford to pay for the city-approved medallions, which each cab has. Some medallions go for as high as $400,000.
A limited number of medallions in New York keeps the population of yellow cabs fixed. Gypsy cabs are cars without medallions. They also don't have meters, so passengers need to agree on a price with their driver before taking a ride.
Riders are typically skeptical of gypsy cabs. (The name and the fact that they're usually black sedans doesn't help).
I've taken gypsy cabs before and can report that they are absolutely fine. There are many car services in Brooklyn rather than yellow cabs, and they're a reliable way to get around in a crunch.
Related posts: Fixing Flats in the Streets, Stuck on 8th Ave. and Taxi!.