MyBB Community Forums

I'm currently looking at moving my forums to a different platform due to the way that they are running due to issues with my current sites which are not being addressed by the developers of that platform.

The most import site that I am looking at converting to another software within the next 7 days has the following stats:

Threads: 350,979, Posts: 3,027,715, Members: 41,836, Attachments: 0

The forum database is approximately 8-10GB.

I have not yet installed MyBB or attempted the conversion. Before I do so, I want to know if anyone has attempted conversions of this size before. If not, but conversions of over 1 million posts have been done successfully, then that is something that I can work with.

Do you have any advice at all for performing large merges. I've been over the documentation that is available, and searched the forums, but haven't found anything relevant although maybe I searched with the wrong terms as I'm not used to the terminology used here yet.

I have full root access to the server, so if command line instructions are required as the conversion script will simply not work through the browser interface than that is not an issue.

Also, how well does MyBB stand up with these sort of stats?

I am looking at converting multiple sites if I can find a platform that both gives timely answers to questions and is robust enough to stand up to the test.

Total stats of the sites that I am looking at converting are (possibly into 1 site, or keeping the existing multiple site structure - undecided yet but very tempted to put it all onto one site now):

Posts: approx 15,000,000
Threads: approx 3,000,000
Members: approx 400,000
Attachments: approx 5,000 (attachments got deleted after a bad server move a couple of years ago and my users don't use many attachments anyway)
Database size: approx 50GB

All advice is welcome. If MyBB has not been tested with these type of stats, then I won't be offended to hear that. The same goes for if large conversions have not been tested.

The Merge System was used to convert ncaabbs.com from phpBB 2 to MyBB - It was around 2 million posts and using a really old code base from the Merge System. I believe the new code bases should be a lot better.

I would say converting a forum the size of that could bring up issues that were never noticed or reproducible before now with the Merge System. If you would like I can assist in the conversion so I can fix any issues that pop-up and make the transition as smooth as possible.

MyBB itself should run fine no problem with those stats, although there are a few setting tweaks that should be made and some performance optimizations on the server end if they aren't already done and I would recommend applying these once you've installed MyBB: http://community.mybboard.net/thread-43165.html

Ryan

That's great to hear Ryan. Thanks for replying so quickly.

I'll get organized for giving it a whirl and report back if I have any issues or to let you know that it went according to plan.

Out of curiosity, what is the largest site currently running MyBB?

(2009-05-21, 02:20 AM)MHryano Wrote: [ -> ]That's great to hear Ryan. Thanks for replying so quickly.

I'll get organized for giving it a whirl and report back if I have any issues or to let you know that it went according to plan.

Out of curiosity, what is the largest site currently running MyBB?

Sounds good - If you give it a whirl, would you mind using the Beta 4? In addition, I'm 90% through with a codebase rewrite of the Merge System (Beta 5/RC/final depending on which route we decide to go) which offers a good size of improvements, complexity, simplicity, and stability. It's up to you which one you want to use.

And http://ncaabbs.com/ is currently the largest site running MyBB.

Sure, I'll use the beta 4 and see what happens. I can always hold off on the others for a bit if beta 5 is nearly finished and then test that out as well. The 3m posts site is the smallest one that I will be converting. There are two more at about 5.5m - 6m posts each as well (hence the 15m posts total for converting).

It's good to know that it handles 4m posts happily enough. I've looked at a few different systems and not had a lot of luck when visiting the bigger sites running them and ncaabbs is running nicely.

---

Seems like I've hit my first problem. I'm importing the users at the moment but it seems a bit backwards to import the users then the usergroups, especially if the importing the usergroups then alters all the users to set them to the correct usergroup. If it doesn't, then importing the usergroups first would solve the issue and also ensure that things such as banned users are still banned.

I'm also going to need to re-run some modules but that is something that you have prevented in the import system. Can you possibly advice of any changes to the files to allow the modules to be re-ran with the existing imported data cleared out.

Just something I've thought of as well while doing a bit more reading - does the importer support multiple consecutive imports?

---

Right, the users imported fine, as did the usergroups, but the banned users data has not been carried over. Running the threads importer now. Only the banned users data is a problem so far, so not too bad.

Source system is vBulletin 3.7 BTW

---

Argh, I need to be able to re-run the threads import, but as that is prevented, I have to reinstall and start over. It rather annoyingly decided that it was going to die with 700 threads left to import.

The threads import died again with 700 left to be imported. I was hoping I could get the posts importing before going out today.

Any suggestions?

I really, really, really, need a way to be able to clear out one modules imported data and then rerun that one without having to start over. Like this is annoying enough, but when if I get the threads imported and it dies in the posts module, or even worse, I get the threads and the posts imported and it dies with one of the last couple of modules! I really don't want to have to start over from the beginning again if it does that!

Also, this auto merge of posts thing is really annoying. Is there some way that we can try and do this without having the posts merged. I feel that the thread has been ignored because the original reply was read and then ones since then have not been.

Note: This reply is submitted at 10:02 GMT 22nd May 2009

---

Update @ 12:52 GMT 22nd May 2009

Managed to get through the threads. I narrowed it down to a problematic thread which I deleted and it continued after refreshing the page.

Polls and poll votes imported fine.

So... now onto the posts module

Quote:MyBB Merge System
Import vBulletin 3 Posts
There are 3121572 posts left to import and 3122 pages left at a rate of 1000 per page.

Inserting post #1...

That is as far as it has got. It will not import the first post!

I've checked the data and there is nothing wrong with the source post data, so I'm not sure why it won't import it. I'm hoping that some advice can be given soon as I would like to get it running the posts import as soon as possible as it will take a while.

Also, it doesn't give the choice of how many posts to import at a time, nor did it for threads. My server is beefy enough to handle 10-15k at a time, which would make it significantly quicker. Can this be changed from the default 1k? If so, how?

(2009-05-21, 02:43 AM)MHryano Wrote: [ -> ]Seems like I've hit my first problem. I'm importing the users at the moment but it seems a bit backwards to import the users then the usergroups, especially if the importing the usergroups then alters all the users to set them to the correct usergroup. If it doesn't, then importing the usergroups first would solve the issue and also ensure that things such as banned users are still banned.

I'm also going to need to re-run some modules but that is something that you have prevented in the import system. Can you possibly advice of any changes to the files to allow the modules to be re-ran with the existing imported data cleared out.

The modules are put in the order in which they should be executed.

Quote:Just something I've thought of as well while doing a bit more reading - does the importer support multiple consecutive imports?

As in merging? Yes

Quote:Right, the users imported fine, as did the usergroups, but the banned users data has not been carried over. Running the threads importer now. Only the banned users data is a problem so far, so not too bad.

Source system is vBulletin 3.7 BTW

---

Argh, I need to be able to re-run the threads import, but as that is prevented, I have to reinstall and start over. It rather annoyingly decided that it was going to die with 700 threads left to import.

Bans are only carried over if the usergroup is set to 'banned'. There isn't another way

(2009-05-22, 09:01 AM)MHryano Wrote: [ -> ]The threads import died again with 700 left to be imported. I was hoping I could get the posts importing before going out today.

Any suggestions?

I really, really, really, need a way to be able to clear out one modules imported data and then rerun that one without having to start over. Like this is annoying enough, but when if I get the threads imported and it dies in the posts module, or even worse, I get the threads and the posts imported and it dies with one of the last couple of modules! I really don't want to have to start over from the beginning again if it does that!

Please be patient. This is a beta product.

(2009-05-22, 09:01 AM)MHryano Wrote: [ -> ]Also, this auto merge of posts thing is really annoying. Is there some way that we can try and do this without having the posts merged. I feel that the thread has been ignored because the original reply was read and then ones since then have not been.

Note: This reply is submitted at 10:02 GMT 22nd May 2009

I have not read this thread until this morning. I've been really busy, please respect that this is free support and we are not available 24/7.

Quote:Managed to get through the threads. I narrowed it down to a problematic thread which I deleted and it continued after refreshing the page.

If you give me details about the thread (like what is different about it then others), then I can fix the issue.

Quote:So... now onto the posts module

Quote:MyBB Merge System
Import vBulletin 3 Posts
There are 3121572 posts left to import and 3122 pages left at a rate of 1000 per page.

Inserting post #1...

That is as far as it has got. It will not import the first post!

I've checked the data and there is nothing wrong with the source post data, so I'm not sure why it won't import it. I'm hoping that some advice can be given soon as I would like to get it running the posts import as soon as possible as it will take a while.

Do you have error reporting turned off in PHP or something? I can't really diagnose an issue with a blank page.

(2009-05-22, 09:01 AM)MHryano Wrote: [ -> ]Also, it doesn't give the choice of how many posts to import at a time, nor did it for threads. My server is beefy enough to handle 10-15k at a time, which would make it significantly quicker. Can this be changed from the default 1k? If so, how?

Yes, it does give you the option on each module. You have to click past that page to begin the conversion. You can change the values there.

Hi Ryan,

Thanks for getting back to me

Quote:I have not read this thread until this morning. I've been really busy, please respect that this is free support and we are not available 24/7.

I appreciate the support is free and people need to eat, sleep, have breaks, have a life, etc I was just worried in case my updates had been missed when the updates got merged into a previous post.

Quote:If you give me details about the thread (like what is different about it then others), then I can fix the issue.

I ended up having problems with two threads during the thread import. One had a blank first post (not sure how that happened, but it was an old thread so I just deleted it) and the other was a thread re-direct.

Quote:Do you have error reporting turned off in PHP or something? I can't really diagnose an issue with a blank page.

phpinfo: http://www.peterska2.com/info.php

I forgot I have display_errors off by default and just override it when I'm troubleshooting something. My bad there, sorry.

I forced display_errors to true in config.php and got

Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 16 bytes) in /var/www/vhosts/peterska2.com/httpdocs/mybb/inc/db_mysqli.php on line 425

I have my memory_limit set to 16M which should be enough for anything but I've added the following to config.php and got it running:

ini_set('display_errors', true);
ini_set('memory_limit', -1);

Quote:Yes, it does give you the option on each module. You have to click past that page to begin the conversion. You can change the values there.

I was expecting it to do that, but I never got the option. I can see that I am supposed to get it from the convert/boards/vbulletin3.php file, but for some reason it just didn't give me the form to submit and just started right away when I clicked go from the main module list.

Quote:The modules are put in the order in which they should be executed.

Yeah, it's just a suggestion as the users are imported before the usergroups in the vB3 importer and so the users all end up in the default registered users usergroup and do not retain their existing usergroups.

Quote:As in merging? Yes :p

excellent Just not MyBB to MyBB then

Quote:Bans are only carried over if the usergroup is set to 'banned'. There isn't another way

My banned usergroups in vBulletin have 'this usergroup is a banned group' set to Yes meaning that the usergroup is a banned usergroup. The users in them are also in the banned users list in vBulletin. Neither of these flags have been carried over.

So... the good news is that I've got the posts importing now. The bad news is that there are still 3075 pages of 1000 posts to go, so it's going to take a while.

I'll be back if I run into any more snags.

[edit @ 16:40]

I've checked my banned groups and they are def set as banned groups in vBulletin. One of the groups is also the default banned users usergroup in vBulletin too (contains just over 3k users).

I'll look into the thread redirects/blank first post issues stopping the importing the threads. I'll also look into the memory issue, but the merge system cache's a lot of data (i.e. it cache's the pid and import_pid of the posts your importing which could eventually easily add up to more then 16MB's if there are millions of them) to make it go fast so it could easily end up more then 16MB on a huge forum like yours.

I'll look into the usergroup issues and per-page options page as well