My board has grown to over 400,000 threads and now, despite no change in traffic, server is slammed with excessive MySQL requests. I've noticed a while ago that attempts to access forum sections with 100,000+ threads in them take forever. Browsing through pages of such popular sections is a major time consuming effort and resource hog.
Have I stepped over safe size MyBB is able to handle? What can I do to restore the board to normal speed and stop the scripts from needlessly slaughtering server?
Site is NSFW so I'm not posting a link here to avoid offending some members.
Not that I want to advertise but here: hackforums.net
My forum is the largest MyBB forum. DB size is about 12GB.
You can join the MyBB Big Board group and you might gain some tips on running large forums.
I checked that group out but it looks like too much hassle for nothing. You must do this, then do that, then fart in the water till two bubbles come out simultaneously and when it happens, then you can join but we will delete your account if you don't log in for 3 months. Seriously...
For some reason your site barely loads for me. I can hardly get the fron page but nothing from there. Must be some temporary server issue.
BTW, accessing sections that only have a few thousands threads in them goes without problems and they load fast, but trying to access one with 250,000 threads takes forever and slaps the $hite out of the server. I'm talking about the same MyBB installation. Same database, same everything. There definitely is a major weakness that won't handle larger boards and if you haven't experienced it yet, you are probably going to at some point. I don't know what the threshold is. Everything worked fine for me until a few days ago, though I noticed that the slowness became significant when the number of threads starting rising well above 6 figures.
(2011-07-23, 03:46 AM)Tranny Wrote: [ -> ]My board has grown to over 400,000 threads and now
How many posts there and what is your DB size?
(2011-07-23, 05:11 AM)labrocca Wrote: [ -> ]You can join the MyBB Big Board group and you might gain some tips on running large forums.
I have 403 Forbidden error when trying to visit your forum.
Just shrunk the DB to 3GB. I deactivated two of the busiest sections where 90% of all threads are and deleted everything (or at least I think it was everything) pre 2009. About 1/4 of all threads went. 500,000 posts at the moment.
Still, each of the two sections had over 150,000 threads each making them extremely slow and killing the server when used so I keep them deactivated. The sections with fewer threads work fine. Since it's the same database, it became obvious that it's not the DB size issue, it's the issue with number of threads in a particular section. Something is not handled the right way there so when this section is called, the script goes through way too many records in the database completely paralyzing the processor (my site is on a dedicated server).
Some interesting stuff there, Maj. Thanks a lot
On a second note - most of his advice, such as putting the CSS before javascript - broke my site and resulted in broken theme.
Thought about trying the 1604 tag for the SVN? It's worked wonders for my largest board.
There were quite a few optimizations regarding forumdisplay and showthread.
Since I'm a webmaster, not a programmer or server tech (though I have some basic knowledge), I'm not sure I know what exactly you mean.
As for the second part of your post, I'm pretty sure optimizing files will not be of much help. The issue is not with displaying threads or search results, the issue is with dislplaying large number of threads or search results. Where there are many threads or search results (I think it starts in the 180,000 neighborhood), then it gets extremely sluggish and annihilates the server. But you won't see the same thing happening, on the same site with the same databases when number of threads or search results is less than 150,000 (or there about - have not pinned that one down yet).
So in other words, if the script is programmed to run through all the records of this particular type in the database, it will run through those records and it will slap your server no matter how much time you spend optimizing it. If each click sends the script to run down 200,000 rows of records in the DB, it's gonna paralyze the server one way or another.
I saw it first hand. Simply disabling (de-activating) the section with 200,000+ threads made immediate difference. You can browse other sections of the same forum all you want and the server won't sweat. Click a thousand times on any of those sections but the load stays handled easily. But try just one single click on a sectionwith 200,000+ threads and you'll bring it to its knees. I had the domain locked out with only my IP enabled. All other traffic got 403 error message (root CHMOD to 0). I browsed the whole site without problem but I did one click on the section with 200,000 threads and woop the server went.
There may not have been this type of issue encountered by any MyBB user yet, hence it may never have been seen as anything that could cause an issue to the developers. I'm just saying what I have experienced and believe that if the developers have an intention for the script to join the big boys club with their script, they may need to address this issue. I've run this forum since 2006. But even if we merely take version 1.6 which has been out for a while itself – if there was to be anything that needs to be tweaked on my board, it would have shown up long time ago. But I never touched the files since last upgrade and all I did was upgrade as per upgrade instructions. Since then, all that was happening on a perfectly functioning site was – new threads and posts. And as new threads and posts kept coming – by the hundreds each day – the number of threads grew and grew and grew and then bam! Server down. Traffic to the site the same, but server went from 20% of its capacity to 180% and it was all because of the number of threads. I disabled the sections with many threads – the server load goes back down to 20%. Same traffic, same number of users. Same amount of activity, just no click on sections which have high number of threads. I enable the section, the load immediately spikes to 180% and paralyzes the server. All it would take is one single click on the section with many threads and it instantly kills the server.
(2011-07-23, 07:50 AM)Tranny Wrote: [ -> ]Since I'm a webmaster, not a programmer or server tech (though I have some basic knowledge), I'm not sure I know what exactly you mean.
The SVN is where the latest revisions of MyBB are stored.
(2011-07-23, 07:50 AM)Tranny Wrote: [ -> ]As for the second part of your post, I'm pretty sure optimizing files will not be of much help. The issue is not with displaying threads or search results, the issue is with dislplaying large number of threads or search results. Where there are many threads or search results (I think it starts in the 180,000 neighborhood), then it gets extremely sluggish and annihilates the server. But you won't see the same thing happening, on the same site with the same databases when number of threads or search results is less than 150,000 (or there about - have not pinned that one down yet).
I can assure you that it will help a lot. It's not as much the files as the queries they run. On larger boards, one unoptimized query can slow the whole thing down.
I had a forum with 600K+ posts which took around 25 seconds to load. That was with 1.6.3. As soon as I upgraded to 1.6.4, it took no more than 2 seconds to load.
The optimizations made in 1.6.4 aren't small ones. They have been made through out MyBB.
For search results, you should consider using the Sphinx Search plugin by Ryan Gordon. It's a must have for large boards.
(2011-07-23, 07:50 AM)Tranny Wrote: [ -> ]So in other words, if the script is programmed to run through all the records of this particular type in the database, it will run through those records and it will slap your server no matter how much time you spend optimizing it. If each click sends the script to run down 200,000 rows of records in the DB, it's gonna paralyze the server one way or another.
MyBB doesn't just pull all the threads from that forum. It uses the datecut and from what I can tell, will only query for that page of threads.
(2011-07-23, 07:50 AM)Tranny Wrote: [ -> ]I saw it first hand. Simply disabling (de-activating) the section with 200,000+ threads made immediate difference. You can browse other sections of the same forum all you want and the server won't sweat. Click a thousand times on any of those sections but the load stays handled easily. But try just one single click on a sectionwith 200,000+ threads and you'll bring it to its knees. I had the domain locked out with only my IP enabled. All other traffic got 403 error message (root CHMOD to 0). I browsed the whole site without problem but I did one click on the section with 200,000 threads and woop the server went.
As I've stated before, 1.6.4 contains many optimizations that will help towards speeding up larger boards. Of course, it also depends on how well optimized your server is.
(2011-07-23, 07:50 AM)Tranny Wrote: [ -> ]There may not have been this type of issue encountered by any MyBB user yet, hence it may never have been seen as anything that could cause an issue to the developers.
Issues regarding larger boards have been found and fixed, whether it be using custom plugins the developers have created, or fixing it in unreleased versions.
(2011-07-23, 07:50 AM)Tranny Wrote: [ -> ]I'm just saying what I have experienced and believe that if the developers have an intention for the script to join the big boys club with their script, they may need to address this issue. I've run this forum since 2006. But even if we merely take version 1.6 which has been out for a while itself – if there was to be anything that needs to be tweaked on my board, it would have shown up long time ago.
MyBB 1.6 is still in development. There are many updates and optimizations to come in 1.6.4 and 1.6.5.
(2011-07-23, 07:50 AM)Tranny Wrote: [ -> ]But I never touched the files since last upgrade and all I did was upgrade as per upgrade instructions. Since then, all that was happening on a perfectly functioning site was – new threads and posts. And as new threads and posts kept coming – by the hundreds each day – the number of threads grew and grew and grew and then bam! Server down. Traffic to the site the same, but server went from 20% of its capacity to 180% and it was all because of the number of threads. I disabled the sections with many threads – the server load goes back down to 20%. Same traffic, same number of users. Same amount of activity, just no click on sections which have high number of threads. I enable the section, the load immediately spikes to 180% and paralyzes the server. All it would take is one single click on the section with many threads and it instantly kills the server.
It's really up to you what you from here. You stated that optimizing the code won't help, so I'm not really sure what you expect MyBB to do about this.