Probably. I used to do that, and I, too clean my brushes thoroughly. I did notice an accelerated deterioraton of the brushes, though, and once I didn't clean quite well enough…. As I moved more and more to acrylics, the problem became moot. But I recently started using artists' oil washes again, and I'm planning on moving into figures, where I will be using both oils and acrylics. So I just invested in a separate set of brushes for oils and the occasionaly enamel, only.
This is one of those things that you can get away with from time to time—until the time you don't. The worst case scenario would be to use the same brushes with both enamels and acrylics in roughly equal proportions. Much higher risk of something going wrong. For someone who rarely uses enamels (or acrylics), provided they are maniac brush cleaners (always good practice, IMHO) the risk may be acceptably low.
Most of the professional painters (art, architectural, and interior) I have known who used both mediums had two sets of brushes, one for enamels (or oils) and one for aqueous systems. At least one artist I knew had three sets: oil, acrylic, and enamel.
One tip that can greatly extend the life of any brush is to treat it with hair conditioner before storing, or periodically if you use the brush frequently. For those who "cross-paint," I strongly recommend it.