Am i too worried about this?!?

I've owned cats for years and I also work with cats on a regular bases and I have never caught anything from them. As long as you keep your cat indoors, vaccinate your cat annually, then there shouldn't be a problem. Enjoy your kitty!