First, you would have to clearly define which characters you want to keep. Letters, numbers, punctuation, brackets, etc.
Once you have that defined, the easiest approach is to add code to /inc/datahandlers/post.php. I think adding to the verify_subject function would do the trick. Here's an example that strips out anything that is not letters, numbers, dashes or spaces.
function verify_subject()
{
global $db;
$post = &$this->data;
$subject = &$post['subject'];
$subject = trim_blank_chrs($subject);
//BEGIN MOD
$subject= preg_replace('/[^A-Za-z0-9-\s]/', '', $subject); //change regex to acceptable characters!
//END MOD
No warning is given and the characters are just stripped out. You could display an error if bad characters were included instead, but it sounds like you just want to strip them.
Make sure you test this first.