Do you trust repairmen?

Would anyone else trust a repairman in their house alone? I recently hired one, and said he needed access to my house once in a while to check on equipment he was leaving there. I told him that I was leaving and I could drive back in a few days to let him in. Then he is like "well this is quite unusual, all our other customers leave us a key." LOL, what an insult does he think I am naive or something...on top of that I am 80% sure I saw his van one day outside in front of my mailbox, and if I found out he took my mail, I will be pissed. When I said I felt more comfortable being there to let him in, he looked at me and his coworker like I was some paranoid weirdo. Anyway, what do you guys think, would you let people in when you are not there? Thanks