Any person or business that reports transactions to either the IRS or the SSA must use this number. An example, if a bank pays you interest on your account they, by law must report that amount to the IRS for income tax purposes, and you as the account holder are informed by the bank of that amount and must, by law report it to the IRS. There are also other laws covering money laundering and terrorist activities that the bank, by law must report to the proper Federal Agencies again your SSN is required. Banks can refuse you service if you do not provide them with this number and in some cases they must report that to the proper Federal Agency. Failure by the bank(and it's employees) can lead them to civil and criminal charges including lose of license to do business fines and prison time(both the bank and/or the employee).
Any other type of action except those dealing with the IRS or the SSA or those covered by money laundering or terrorist activities are covered by the Privacy act.
It is also a good idea to limit use of your SSA except where required by Federal laws and Requalations to aviod ID theft.
If you have any questions about use of your number to contact either the IRS or the SSA for clarifation