It is possible that plasticizer has migrated from the plastic through the primer. If that is the case, washing should fix the problem.
It is also possible that the surface of the primer is dusty, at least on a microscopic level. This will cause the same problem. A light sanding, followed by washing in detergent and water, should solve this problem.