Is it possible to change the way mybb encryption?
I notice it's $stored_pass = md5(md5($salt).md5($plain_pass)); and I would like to change it to something (such as sha512), is it possible?
Yes but to do it for existing users you'll need to have them login, confirm, and rehash the password. It's more complex than it's worth.
Unsure what you believe you will achieve with a sha512 over the current method. I've never seen a password broken by anything other than dictionary method which would work against any encryption.
You'll need to tear apart the code starting with the inc/functions.php file.
My guess is that if you have to ask that you can't do with your current skillset. I advise you just leave well enough alone. Don't be convinced that the way MyBB secures passwords isn't secure enough for 99.9% of sites. It's just fine how it is. It's probably more important to add a config key than it is to alter to sha512.
If you want a little bit more security... increase the minimum password length requirement to 16 and max length to somewhere over 22... and enable the must be strong password option... that may give your users more protection... although they may have difficulty remembering their passwords... but you can put on the registration page a tip for making strong phrase passwords that are easy to remember. eg Save$Money10timesAjourney
You can check the strength of passwords here:
http://www.passwordmeter.com/