MyBB Community Forums

Great

It's ticking along quite nicely now. It's done about 200k posts, so a good while to go for it yet, but as long as it going that is the main thing. Anything that we can come up with to make it tick along a bit quicker will be good. I'll probably have a look at it tonight and see if I can come up with anything too.

---

BTW - the forum titles didn't import. While not a problem on this one (only a mere 24 forums/catagories) it is going to be a problem on the others (one has about 200 forums and the other around 325 forums) so editing them and entering the names isn't exactly ideal.

From a guy that's used the merge system pretty often and for clients let me give you some advice.

You should do this locally imho via xampp. You have more control imho and it's faster normally. It won't be bogged down my server processes from your existing site.

Set the 1000 per page higher which is easier to do locally.

Delete a great deal of your forum data to start with...instead of 3 million posts delete them down to 100,000 and so forth. If possible get a smaller site database to work from so you can test and see how things work without going through hours of each module.


To Ryan: Yes I would love to see a "rerun" for some of the modules. Starting over when things go wrong is very depleting. Many are savvy enough to go and empty a table to try and rerun a module that failed. As it stands..any failure forces a restart no matter where you are. That's time consuming.

(2009-05-22, 06:25 PM)labrocca Wrote: [ -> ]To Ryan: Yes I would love to see a "rerun" for some of the modules. Starting over when things go wrong is very depleting. Many are savvy enough to go and empty a table to try and rerun a module that failed. As it stands..any failure forces a restart no matter where you are. That's time consuming.

How would you propose I do that then? Technically, it would be pretty hard to do.

(2009-05-22, 05:09 PM)MHryano Wrote: [ -> ]BTW - the forum titles didn't import. While not a problem on this one (only a mere 24 forums/catagories) it is going to be a problem on the others (one has about 200 forums and the other around 325 forums) so editing them and entering the names isn't exactly ideal.

Yeah, that's been reported and fixed for Beta 5

Quote:How would you propose I do that then? Technically, it would be pretty hard to do.

Technically you're the smart guy here.

hehe...

As it is you lock the button...why not just add a button that runs it again. Yes I know it's more involved than that but as I said...it's a problem now since the merge is buggy and often requires people to start over and over and over until it works. When I use it I do report any bugs I find but still it's a cumbersome task.

I'd love to get my hands on your new beta btw.

EDIT: I'll think about your question some more on a possible method to do this but what about an undo button on the last action? How feasible is that?

(2009-05-22, 06:42 PM)labrocca Wrote: [ -> ]As it is you lock the button...why not just add a button that runs it again. Yes I know it's more involved than that but as I said...

So they just end up with twice the data?

Someone else has pointed it out but it's something I have known for a long time. You build the merge system but you don't really use it. If you did you would see this a problem and fix it. As it stands you just fix bugs in the existing system instead of improving it.

I can think of a number of ways to solve this but ultimately you're the coder and you expressed that you're rebuilding parts of merge system.

Maybe you missed my edit...

Quote:EDIT: I'll think about your question some more on a possible method to do this but what about an undo button on the last action? How feasible is that?

(2009-05-22, 06:25 PM)labrocca Wrote: [ -> ]From a guy that's used the merge system pretty often and for clients let me give you some advice.

You should do this locally imho via xampp. You have more control imho and it's faster normally. It won't be bogged down my server processes from your existing site.

Set the 1000 per page higher which is easier to do locally.

Delete a great deal of your forum data to start with...instead of 3 million posts delete them down to 100,000 and so forth. If possible get a smaller site database to work from so you can test and see how things work without going through hours of each module.

I did a quick run of a small database on localhost first The amount of time it takes to backup, gzip, then download, restore to localhost, and then try for the amount of difference in available power is just not worth it. My webserver is built to withstand a lot of abuse, my computer is not as beefy, and I really don't want to go and beef up the ram to match my webserver just to test an import.

Pruning down to 100k would mean 32 imports to be ran. Not fun, and would mean 32 database backups and restores, plus 32 lots of stress on the server instead of just 1. When you are talking a small database (say even up to 1GB db size) then that's not too bad, but when you start talking serious databases it's a lot easier to change server configuration on a temporary basis than to start messing with repeating things and causing unnecessary strain on the server. I have full control of the dedicated server anyway, so I'm not limited by server limits other than the actual physical limits of what I have in the box itself.

Oh, and even with the import running, server load is under 5.00 so nothing to worry about

(2009-05-22, 06:36 PM)Ryan Gordon Wrote: [ -> ]

(2009-05-22, 05:09 PM)MHryano Wrote: [ -> ]BTW - the forum titles didn't import. While not a problem on this one (only a mere 24 forums/catagories) it is going to be a problem on the others (one has about 200 forums and the other around 325 forums) so editing them and entering the names isn't exactly ideal.

Yeah, that's been reported and fixed for Beta 5

Glad to hear it

(2009-05-22, 06:42 PM)labrocca Wrote: [ -> ]

Quote:How would you propose I do that then? Technically, it would be pretty hard to do.

Technically you're the smart guy here.

hehe...

As it is you lock the button...why not just add a button that runs it again. Yes I know it's more involved than that but as I said...it's a problem now since the merge is buggy and often requires people to start over and over and over until it works. When I use it I do report any bugs I find but still it's a cumbersome task.

I'd love to get my hands on your new beta btw.

EDIT: I'll think about your question some more on a possible method to do this but what about an undo button on the last action? How feasible is that?

Looking at the import code, it should be possible to have an option where 'Re-run' does a

DELETE FROM {table} WHERE import_* != '0';

for the applicable tables for the data imported there.

I think I have a solution for the banned users issue, but I'm not 100% sure yet, so I'll look at that on localhost next week (busy weekend) with a smaller database I use for testing stuff on and see if I can put something together for dealing with it. It might be quick and nasty, but as long as it works, especially with it being a one time run script, it won't be a big issue.

As it stands, it's slow progress with just doing 1k posts per page (I'll force that to more in the code before doing the next site), but it is still ticking along really nicely and hasn't had any further issues.

(2009-05-22, 08:51 PM)labrocca Wrote: [ -> ]Someone else has pointed it out but it's something I have known for a long time. You build the merge system but you don't really use it.

And how would you know that? I've run the merge system more then anyone has. ever. I test the Merge System as much as I can and I fix any and all problems I run across. I also have been working on an almost complete rewrite of the Merge System for the past several months, so don't say I don't try and improve it. Between school, clubs, MyBB Development, Merge System development, and other activities I don't have time to sit down for 8 hours a day, 5 days a week and program in things that I've read from people's minds so instead I compromise. I do what I can, and in return, I'd hope the user provides proper feedback and cooperates with me if they find problems instead of turning around reporting a problem without any information and never following up.

(2009-05-22, 08:51 PM)labrocca Wrote: [ -> ]

Quote:EDIT: I'll think about your question some more on a possible method to do this but what about an undo button on the last action? How feasible is that?

It might be possible to do but say for instance if you imported usergroups, users, threads and posts, you wouldn't be able to rerun threads without removing the posts first. In addition, it couldn't be done after the merge is already finished running since the import_* trackers are removed after the merge is completed. I might also mention that merged data doesn't show up on the forum until the merge is completed (so in the case of it being a live forum, guests/members don't see the import running) so there's no way of checking if everything is correct and then going back and rerunning it. And in addition, even if you rerun the module it would still cause the same problem unless you reported the problem [with proper detail], and I was able to come up with a fix, test it, post it, then you apply it, then rerun it.

So, as I said most solutions like that don't seem feasible or practical to me.

MHryano Wrote:As it stands, it's slow progress with just doing 1k posts per page (I'll force that to more in the code before doing the next site), but it is still ticking along really nicely and hasn't had any further issues.

That's because of the cache's I'd assume. It's probably overflowing into the disk paging file. Unfortunately I don't think there's anything I could do about it, but you could tweak with the server settings.

(2009-05-22, 11:48 PM)Ryan Gordon Wrote: [ -> ]That's because of the cache's I'd assume. It's probably overflowing into the disk paging file. Unfortunately I don't think there's anything I could do about it, but you could tweak with the server settings.

I'm not worried. I'm just leaving it to it while it is working and I'll find a way to do something about it before I do the bigger ones. I'm just using diskswapping after using all available memory. It's tweakable with a couple of minor edits to the import script, so its not a biggie. I'll probably stick an extra couple of gigs ram in the server before doing the others too which will help.

BTW - ignore the database size I said originally. I looked at the wrong database! The one that I am importing is 34.9 GB. The stats of the forum itself were right, just not the db size.