Isn't unsolicited mail free speech? How can you ban it?
"Freedom of speech" makes sure that the governments of many countries don't suppress the opinions of their citizens in public places. Of course, it doesn't apply in the same way to private businesses. For example, people can't stand up and share political opinions with other movie theater patrons during a screening of "Casablanca", then claim it's "free speech" when the ushers throw them out.
The Internet is not a government agency, nor is it a public common area like a town square or the broadcast spectrum. Moreover, the ISPs it comprises are not "common carriers" like the postal service or telephone company. Rather, the Internet is a network of private organizations who have agreed to exchange certain messages with each other (although they are under no obligation to handle any particular message). These organizations are self-supporting and do not receive payment when they accept email from other network members.
When you send a message, the person you send it to is, on average, paying for half the delivery cost. The "spam is free speech" argument would force other people to pay to receive your opinion, which is clearly unreasonable even if you're sending noncommercial content such as political or charitable messages. "Postage due" mail or collect phone calls from political or charitable organizations would be unwelcome in the offline world, too.
What makes a message "spam" is not the message's content. What matters is whether the recipient requested it and consented to receive it (and whether you can prove that they did if they complain that they didn't).