I have heard this said about CA's (superglues), but personally have not found it to be a problem, if it's even true. (Is it true?) I always use CA's for fillers, sometimes letting them sit overnight, and have had no big issues sanding them down and leaving the plastic intact.
For sanding, I don't have any paper coarser than 240 grit. I also use finger nail emery boards like you can buy in the cosmetics section, then follow with sandpaper. In most cases, I start out with 320 grit, then move to 400 and then to 600. Usually this is all that's necessary
Lately, I only wait about an hour for superglue to dry, then sand. It seems easier to work. If it hasn't quite set in an hour, I wait a few minutes then try again.