I modelled many years without using any primer at all, so I'm fairly confident that you can still do it without too many problems. I only started using primer once I branched out into metal figures (for my dioramas) and realised that without a primer coat, the paint was fragile and did not resist long to any touching/handling...
Nowadays, most of my kits are 'upgraded' with metal, resin bits (wire, foil, photo-etched parts, accessories,..) and the primer give me an even surface to work on. Without it, I'd have paint sticking well to plastic, not so well to metal and possibly not at all on resin bits...
Also, many years ago, I did not do any weathering, while nowadays I do, and this treatment can be harsh (drybrushing in particular) to the kit. Unprimed resin and metal would suffer more than plastic...
So, all in all, I'd advise you to prime your models.