Uber India — Driver denied duty

This has happened many times that I book a cab and drivers call me up asking me about the destination.
And when I tell them that I've to go to XYZ place, they say that they won't go. Neither do they cancel the ride.
Eventually I've had to cancel rides many times because driver denied duty.
And because I cancel the ride, uber charges me extra on the next ride.
How is this fair? What's even the point? Atleast we should be able to explain why we're cancelling the cab.