When I initially crank my car (cold) it starts right up and charges properly from the alternator.

The Problem -
When the car reaches normal operating temperature, it will refuse to idle. Car will run if you press the gas pedal and will cruise on the highway fine.
But when you come to a stop, the idle will drop so low the car cuts off.
During this time, the alternator is charging still.

I've changed 2 alternators (one newly rebuilt and one my old standby stocker) and both do it.

I've traced the problem to the actual charging of the alternator.
If I unplug the car wiring harness from the alternator, the car will idle perfectly (cold, warm, or hot it does not matter).
Just as soon as I plug the harness in, the car will die.

With the serpentine belt off and car running, I've tried to turn the alternator by hand and it doesn't spin easy.
With the harness unplugged, the alternator spins perfectly fine.
I believe the alternator has too much of a load or something on it making it hard to spin and this causes the car to die.

So what do I need to look for to fix this? Is there a diode or something that could have gone bad causing this problem?
It just started about a week ago.

Thanks,
Ryan