2015-03-23, 05:25 PM
I've been working on a very large database import for a source forum that has a lot of junk. The source forum is IPB 3.something. IPB doesn't have as nice of data integrity standards as MyBB does.
For example, I'm getting a ton of errors while merging the users because people (in particular, spammers) have URLs and other junk in pfields_content.field_4 (stores the person's ICQ number). Now I don't know anything about ICQ, but MyBB defines an ICQ username as varchar(10). Perhaps instead of having the merge script stop and barf out a nasty SQL error, just have it assume that if the ICQ is > 10 characters, it's garbage and insert null?
Another example would be topic subject length. In MyBB, the title is limited to 120 characters, but IPB allows much longer. Any time it hits a thread with more than that, it spits out another SQL error and stops. Perhaps the best course of action with these would be to truncate the subject at 120 characters if it's to long? When importing a forum that has hundreds of thousands of threads, it's tedious to fix them one by one. I ended up just running an update statement to trim them in the source database.
Just an idea... Does anyone prefer to see the SQL errors rather than "auto correcting" the problems?
For example, I'm getting a ton of errors while merging the users because people (in particular, spammers) have URLs and other junk in pfields_content.field_4 (stores the person's ICQ number). Now I don't know anything about ICQ, but MyBB defines an ICQ username as varchar(10). Perhaps instead of having the merge script stop and barf out a nasty SQL error, just have it assume that if the ICQ is > 10 characters, it's garbage and insert null?
Another example would be topic subject length. In MyBB, the title is limited to 120 characters, but IPB allows much longer. Any time it hits a thread with more than that, it spits out another SQL error and stops. Perhaps the best course of action with these would be to truncate the subject at 120 characters if it's to long? When importing a forum that has hundreds of thousands of threads, it's tedious to fix them one by one. I ended up just running an update statement to trim them in the source database.
Just an idea... Does anyone prefer to see the SQL errors rather than "auto correcting" the problems?