Indeed, most British frigates during the War of 1812 were smaller than Constitution, President and United States, more lightly built, had fewer, lighter long guns (18-pounders versus the American 24-pounders).
As others have noted, southern white oak was distinctly stronger than other materials commonly used in ship building during the War of 1812 era.
It should be noted that Constellation, Congress and Chesapeake were also smaller and had fewer guns than the 3 ships named above. They were rated as 38-gun frigates, similar to Guerriere and Java (both lost to Constitution) and Macedonian (captured by United States) which were all also rated as 38-gun frigates.
But the British were no pushovers nor were they timid. Chesapeake and President were lost to the British, a third of the US frigate force. And when Constitution fought Java, Java stern-raked Constitution - twice, before ultimately losing to her larger opponent.
Constitution's sistership USS President, while under the command of the very capable Stephen Decatur, was crippled by a smaller but faster frigate, HMS Endymion, leading directly to her capture. Endymion was able to stern-rake President three times during the engagement, causing significant damage and holing President's hull below the waterline. The damage to President was so serious Decatur was forced to strike his colors.* Endymion was also seriously damaged, so much so that she could not affect the capture. Two other 38-gun British frigates, HMS Pomone (French-built) and HMS Tenedos came up and their boarding parties took possession of President.
Despite the damage suffered by President, the three large American frigates were very strongly built for frigates, having the structure of much heavier line-of-battle ships. In "Chronicles of the Frigate Macedonian 1809-1922" by James Tertius de Kay, there is a very detailed passage describing the reaction of British 38-gun frigate Macedonian's captain, John Carden, when he is brought aboard United States after his surrender. Carden was so astonished at the structural superiority of the American frigate that he remarked in his letter notifying the admiralty of his surrender that the American frigate was "built with the scantline [structural framing] of a seventy four gun Ship". Carden then went on to describe United States' very heavy armament, even heavier than that of sistership Constitution (42-pounder carronades compared to Constitution's 32-pounders).
We can wonder whether or not Carden would have engaged the heavier American frigate in single combat had he known what he was really facing.
When the British admiralty realized that the three larger American frigates had significant advantages in structure and weaponry likely to result in an American victory, the admiralty decided those advantages could best be neutralized by a change in tactics. The admiralty forbade British frigate captains from engaging in single combat with an American frigate.
But even then, that was no guarantee of success. Constitution's very hard-fought simultaneous victories over sixth-rates HMS Levant (22 guns) and HMS Cyane (30 guns) is really remarkable, even though both British ships were significantly inferior in armament by themselves.
It should also be noted that some British frigates, such as Guerriere and Java, were originally French-built, captured from France then placed in Royal Navy service. Their structure was a bit different than typical British design.
Any American victory over a British ship did not come easy. Just ask Stephen Decatur.
* Endymion had successfully disabled six guns on President's starboard side by shooting them off their carriages or damaging the gun barrels. A total 10 of the 15 starboard upper deck gunports on President were hit and the gun crew reduced. It was recorded that shot from Endymion had pierced President below the waterline and six feet (1.8 m) of water was found in President's hold when Pomone boarded her. Shot from Endymion was even found inside President's magazine. - "The Challenge - Britain Against America in the War of 1812" by Andrew Lambert, (Faber and Faber) ISBN 978-0-571-27319-5