So what happens when there's large-scale political and/or civil unrest if they government holds the keys to your car? There's revolution so to slow down their enemies, the government deactivates every car with a self-driving chip? This is exactly the issue (well, one of them) with smart guns.
What happens if you're in the boonies and you get injured somehow and have to drive yourself to the hospital while bleeding profusely? Romp on the gas pedal all you like - the car doesn't care that you're dying. It will control your speed and calmly tell you "the speed limit is this area is 60 mph."
What happens if you take evasive action but the car doesn't sense the danger, so it cancels your steering wheel control because it thinks you fell asleep at the wheel? What happens if...etc, etc.
I know these are slightly outlandish examples, but they are no more improbable than the government coming for our guns (like we are all so defensive about). I for one will NEVER own a self driving car. And if the government mandates it, all the more reason to buy some old car and learn everything about it. I distrust the government far more than I distrust my own driving abilities.
Never bring a knife to a gun fight.
Carry gun: Springfield XD Tactical .45