But you also need to remember two things.
#1: Technology can falter. And when the technology fails, it is up to the AD team to visually ID the track. Many things can effect a visually ID. (ie range, speed, color) I will say the pilot does have to put a lot of faith in the training of the AD team.
#2: To err is human. Even a patriot battery, who would never have to visual ID, shot down a friendly during OIF. Not certain where the failure of system occurred but it happened. Once again, a lot of faith is put into the AD team.
I am personally not too worried about the aircrew misID of me. a good part of my job is to hide, which I do quite well.