MyBB Community Forums

I'm sure we've all seen the 1.10 RFC thread by now.

I want to have a discussion about the MyBB project, the current state of play and our direction for the future.

As it stands at the minute, we have three ongoing projects:

• MyBB 1.8 - the current codebase, mostly stable. Full of bad practices and outdated code conventions, but it works (for the most part)
  
• MyBB 2.0 - a full rewrite started over two years ago, almost three (January 2015 according to the commit log). Far from completion, no sign of an alpha release, never mind a final release. Only 24 commits this year, over a period of 9 months. We're probably looking at at least another year before we have any kind of alpha version anywhere near usable, assuming contribution steps up by a long way.
  
• MyBB Merge System - untouched since April 2017, 6 months ago. Not up to date with current versions of external software, no active maintainer since Jones left the team.
  

We also have to look at the community activity, which as far as I can see is vastly reduced. In days gone by I'd check the community forums once a day at least using "View New Posts" and find at least a page of results, often several. Now, there's rarely ever even a full page of results when I check.

So, what can we do about it? Tom M and I have been talking on Slack, and obviously the current momentum and direction isn't going anywhere fast. By the time MyBB 2.0 is ever ready for an alpha release (if ever), will we have any users left using MyBB? Just this weekend I mentioned in Slack that the DreamHost support forum had moved from MyBB to Discourse, making another large forum leaving us. With the current pace of development and feature set that we offer in comparison to other free competitors (NodeBB, Discourse, Flarum, even phpBB), how are we competing?

The 1.10 RFC has a valid argument in my eyes. I think there might be real merit to pushing 2.0 (as it currently stands - a full rewrite in any case) further out into the future. Instead, we could concentrate our efforts on incrementally improving the 1.8 codebase, and naming it 2.0:

• Replace our current template system with Twig, using Composer.
  
• Replace our current database methods and queries with Eloquent or a similar DBAL.
  
• Introduce a new theme, making use of current technologies and approaches - if we're replacing the template engine, there's no worry about plugin support anyway. Whilst we're at it, we can remove tables and such to our heart's content.
  
• Bump our minimum PHP version requirement to PHP 7 and remove backwards compatible hacks.
  

This way we have a fresh release, without requiring quite so much work. We could then start focusing upon a 2.1 release, with further incremental improvements such as:

• Introducing the Laravel internalisation (language) system.
  
• Introducing a core API system.
  
• etc.
  

phpBB found themselves in a very similar position to we ourselves are in now just a few years back, and ended up going down the route of incremental improvement with 3.1. They introduced a responsive version of their theme, added alerts to their core and started introducing Symphony components into their codebase and embracing composer and third party libraries. I see no reason why we can't achieve the same thing.

2.0 as a full rewrite is a nice idea, but is it really attainable by the current team in a timeframe that makes sense or are we all just dreaming?

No matter what we decide to do, we must make sure that we make our plans completely crystal clear to the community, with a solid roadmap and task assignments to reach our stated goals. It's important to decide, in advance, whether this is going to be a staged release (2.0 being theme, 2.1 being database, 2.2 being language etc.) or if it's all going to be bang - all in 2.0. Whatever we decide, it must be made absolutely clear. In my eyes, getting the theme and templates sorted out first makes the most sense.

Sounds like a good idea to me.

I suggest going with the 1.x major number until all code of dubious quality is removed and to prevent confusion.

What we're missing is a technical assessment of 1.8 refactoring with a framework - it wouldn't make sense to go over the same places multiple times only to have it outsourced to Laravel / 3rd party module.
If that would happen, instead of performing package-wide search & replaces - such as with database queries - the focus should be placed on specific areas, rewriting the code using modern sources and practices.

The first stage should include some meta changes that would speed things up:

• introduce all planned organizational changes (how the package is built and published),
  
• introduce Gitflow to keep the repositories clean,
  
• reshape the file structure to roughly match 2.0 (introducing a simple routing system that wouldn't require many changes to entry point files),
  
• make it easy to distinguish old and new code,
  
• bump all requirements to remove concerns of compatibility with old software (keep in mind PHP 7.1 was released on 1 Dec 2016),
  

Some changes missing from the public discussion, yet of high priority:

• HTTPS-friendly resources (DVZ Secure Content functionality implementation),
  
• strong password algorithms (DVZ Hash functionality implementation),
  
• per-session authentication keys + maintenance tools (keys, passwords reset),
  
• safe self-check (DVZ Integrity Tools functionality + strong checksums + verified HTTPS connection)
  

Core functions like the cache system should also take priority.

Some modules in 1.x are separated enough to make it possible to replace them with adapters (if needed) and this should be one of the first targets - replacing old code with stable, external packages.

The 2.0's process could be either kept as it is or maintained as a proof-of-concept to make sure all new modules follow new practices when all 1.x code is stripped.

---

#110-discussion Wrote:tomm 1:54

An important note to consider is if we're going to do this all at once, or piece by piece.

1:55

The key will be a roadmap for the community (that's believable)

devilshakerz 1:57 AM

it would have to be gradual to be able to release in a reasonable time (still, deadline-less) since that's the major point

1:57

ironically, the first stage I've described lays the groundwork but doesn't include the fireworks that satisfy that point

1:58

the more fireworks the longer it takes, and the basics have to be done [...]

2.0 should follow some basic principles. Building on what devilshakerz mentions:

• A clear roadmap for the community and contributors to follow
  
• Modern contribution practises should be followed
  
• A stepped introduction of changes, just like Invision/phpBB, replacing core sections of the code with each step (2.0, 2.1, 2.2 etc.)
  
• A restructure of directory and codebase to closely match our vision of "2.0" (so that the jump between major versions is less revolutionary and more evolutionary) should be the first step -- increased password security, sessions and modern frontend/backend/codebase methods (things are done to forums from where there's no going back to 1.x)
  
• 1.10/2.0 should incorporate the features and methods we've laid out in 2.0 already (notifications, etc.)
  

This is a much more realistic vision than rebuilding MyBB from scratch. There's so many forum systems out there. We shouldn't be disheartened by how long things have taken, as other systems are taking just as long to catch up on what users want their forum system to do, in an ever more connected world.

This project should share Chris' vision of what MyBB should be from 2.0, with something that we can deliver in a realistic timescale.

I think one simple area that we could tackle pretty easily as a "proof of concept" idea of replacing components one at a time would be the ACP. It already has it's own front controller type of approach to routing requests, so we could fairly easily swap that out for a routing library. It also doesn't currently use templates, so refactoring it to use Twig would possibly be easier than refactoring the front-end.

The directory structure would slowly change over time. The first two changes I see happening are:

• Addition of a "vendor" folder to "inc/", where Composer dependencies are installed. This can be specified in the "composer.json" file.
  
• Addition of a "src" folder to "inc/" acting as a MyBB namespace route where classes are autoloaded by Composer.
  

The current directory structure of the "mybb2" repository isn't ideal at all - it simply follows the directory structure used by the Laravel framework. We could certainly improve that a lot.

The other question is whether we would fork the "mybb/mybb" repository to a new repository to work on these enhancements, or should they happen in the current tree on the "feature" branch (or on a new branch)?

If this is the route we decide to take, we need to select a clear set of changes to take in the first incremental version. In my eyes, the following are some good starting points - we can choose some more tasks, or remove some:

• Updated password hashing, using "password_hash".
  
• Improved session handling.
  
• Better HTTPS support.
  
• Introduce Composer to manage 3rd party dependencies rather than manually managing them.
  
• Require PHP 7 at a minimum, possibly PHP 7.1.
  
• Swap out our custom caching classes for 3rd party libraries, possibly with a wrapper to aid the transition (with an aim to remove said wrapper).
  
• Twig for templates, at least in the ACP to begin with.
  

We don't want to go wild and say that we want to hit hundreds of targets with the first version, otherwise we'll end up in the same position as we are now. We should select a set of attainable goals and start working towards them, with team members dedicated to individual tasks.

I see several issues with replacing major parts of the core over several steps:

• There is a huge gap between MyBB 1.8 and a modern PHP application. It's extremely difficult to do the needed transition over several steps without rewriting the same code multiple times.
  
• There would be a lot of breaking changes. Even though updating plugins was usually pretty simple so far it often took months until plugins were updated. Some never have been updated.
  
• We could end up with a suboptimal code base full of compatibility layers and compromises.
  

If we stuff too many changes into one step its development still takes a long time. If we have smaller steps there is a change of breaking plugins and themes again and again driving our community away.

I'm not sure what you mean about having to rewrite the same code multiple times, ca you elaborate on that?

Updating plugins will be an annoyance, true. However if updates are broken into chunks with a clear schedule, I think it's a manageable task. Also, many big popular plugins (SEO, alerts, likes, mentions, etc.) could be rolled into the core during the process. We could also take the chance as we're doing the work to improve our devloper documentation and make the transition and future work even easier for developers.

We could end up with a suboptimal codebase, yes - if we don't tackle things correctly. The same could be said of anything though, really.

phpBB and IPB went down this road in the past and managed to pull it off, so it's not like it's an impossible task.

(2017-10-02, 08:26 AM)Euan T Wrote: [ -> ]The current directory structure of the "mybb2" repository isn't ideal at all - it simply follows the directory structure used by the Laravel framework. We could certainly improve that a lot.

Is there an Issue opened for that? It could be at least planned for 2.0 to establish what we'll be aiming at.

Quote:The other question is whether we would fork the "mybb/mybb" repository to a new repository to work on these enhancements, or should they happen in the current tree on the "feature" branch (or on a new branch)?

By definition 1.8+ would be the continuation of 1.8 with most of the old code left, just the strategic chunks rebuilt; Gitflow would mean that master always contains the latest stable version (i.e. released) and the development would happen on feature.
The refactoring road would possibly take us to having to maintain 3 different branches so 1.8 would have to be EOL'd as early as possible, which might be a valid reason to have 1.8+ development in the current repository with 1.8 branched off to master/1.8 and feature/1.8.

(2017-10-02, 10:08 AM)StefanT Wrote: [ -> ]I see several issues with replacing major parts of the core over several steps:

• There is a huge gap between MyBB 1.8 and a modern PHP application. It's extremely difficult to do the needed transition over several steps without rewriting the same code multiple times.
  
• There would be a lot of breaking changes. Even though updating plugins was usually pretty simple so far it often took months until plugins were updated. Some never have been updated.
  
• We could end up with a suboptimal code base full of compatibility layers and compromises.
  

If we stuff too many changes into one step its development still takes a long time. If we have smaller steps there is a change of breaking plugins and themes again and again driving our community away.

IMO continuing to work on 2.0 is a safer bet and I agree with https://community.mybb.com/thread-213285.html (which is our current approach, actually).

We can have that while discussing what role 1.x would play in the transition. It can go from only reusing 2.0 modules in 1.8 (which can be the logical conclusion in some 1.8 Issues) to making it structurally compatible with 2.x (these would still be different scripts, but the development gap would be decreased).

(2017-10-02, 11:37 AM)Euan T Wrote: [ -> ]I'm not sure what you mean about having to rewrite the same code multiple times, ca you elaborate on that?

Without a technical plan some 1.x files could be modified multiple times because the changes would be applied in layers (corrections to automated changes in queries, caching system, etc.); and the focus on specific sections (files) would come later; files that would have received changes of 2+ types in 10+ places could be swapped entirely for a better, external package.

Quote:phpBB and IPB went down this road in the past and managed to pull it off, so it's not like it's an impossible task.

Refactoring is the recommended approach to old codebase with large communities and reduced number of developers, but every project is different and requires separate analysis of overhead. 1.x is not only structurally outdated (dependencies, code quality) but has numerous defects that make most of the old code dangerous to run.

(2017-10-02, 11:37 AM)Euan T Wrote: [ -> ]I'm not sure what you mean about having to rewrite the same code multiple times, ca you elaborate on that?

Unless we are implementing a new template system, database abstraction or plugin system (just some examples) in the same step we are rewriting large parts of the code again and again.

(2017-10-02, 11:37 AM)Euan T Wrote: [ -> ]However if updates are broken into chunks with a clear schedule, I think it's a manageable task.

Unfortunately most of the plugins get abandoned after a while. MyBB 1.8 is out for 3 years and users still ask to use older plugins (and they often work after changing the compatibility flag). It's tricky to find the right way to satisfy the users here. Some users still stay on 1.6 or even older.

(2017-10-02, 11:37 AM)Euan T Wrote: [ -> ]phpBB and IPB went down this road in the past and managed to pull it off, so it's not like it's an impossible task.

phpBB has improved their code but they are certainly not finished yet. There is still a lot of legacy and spaghetti code.

(2017-10-02, 12:05 PM)Devilshakerz Wrote: [ -> ]

(2017-10-02, 08:26 AM)Euan T Wrote: [ -> ]The current directory structure of the "mybb2" repository isn't ideal at all - it simply follows the directory structure used by the Laravel framework. We could certainly improve that a lot.

Is there an Issue opened for that? It could be at least planned for 2.0 to establish what we'll be aiming at.

Not currently, but with the current approach we're tied to Laravel, and any time they introduce directory layout changes in a framework update (as they did in 5.4, and other versions), it's quite a big job to replicate the changes just to follow the structure form the framework.

(2017-10-02, 12:05 PM)Devilshakerz Wrote: [ -> ]

Quote:The other question is whether we would fork the "mybb/mybb" repository to a new repository to work on these enhancements, or should they happen in the current tree on the "feature" branch (or on a new branch)?

By definition 1.8+ would be the continuation of 1.8 with most of the old code left, just the strategic chunks rebuilt; Gitflow would mean that master always contains the latest stable version (i.e. released) and the development would happen on feature.
The refactoring road would possibly take us to having to maintain 3 different branches so 1.8 would have to be EOL'd as early as possible, which might be a valid reason to have 1.8+ development in the current repository with 1.8 branched off to master/1.8 and feature/1.8.

The idea that Tom and I were discussing is that "1.8+" would actually be 2.0, with the current 2.0 rewrite then becoming 3.0 or "MyBB Next" or something. In that case,a  separate repository would make sense.

(2017-10-02, 12:05 PM)Devilshakerz Wrote: [ -> ]

(2017-10-02, 10:08 AM)StefanT Wrote: [ -> ]I see several issues with replacing major parts of the core over several steps:

• There is a huge gap between MyBB 1.8 and a modern PHP application. It's extremely difficult to do the needed transition over several steps without rewriting the same code multiple times.
  
• There would be a lot of breaking changes. Even though updating plugins was usually pretty simple so far it often took months until plugins were updated. Some never have been updated.
  
• We could end up with a suboptimal code base full of compatibility layers and compromises.
  

If we stuff too many changes into one step its development still takes a long time. If we have smaller steps there is a change of breaking plugins and themes again and again driving our community away.

IMO continuing to work on 2.0 is a safer bet and I agree with https://community.mybb.com/thread-213285.html (which is our current approach, actually).

We can have that while discussing what role 1.x would play in the transition. It can go from only reusing 2.0 modules in 1.8 (which can be the logical conclusion in some 1.8 Issues) to making it structurally compatible with 2.x (these would still be different scripts, but the development gap would be decreased).

Re-using 2.0 modules and making it structurally compatible over time is pretty much what Tom and I are proposing - rather than a full rewrite, adapt the current software into a more modern piece of software over time. Had we done this from the start, we'd be a lot further ahead than we are now.

(2017-10-02, 12:05 PM)Devilshakerz Wrote: [ -> ]

(2017-10-02, 11:37 AM)Euan T Wrote: [ -> ]I'm not sure what you mean about having to rewrite the same code multiple times, ca you elaborate on that?

Without a technical plan some 1.x files could be modified multiple times because the changes would be applied in layers (corrections to automated changes in queries, caching system, etc.); and the focus on specific sections (files) would come later; files that would have received changes of 2+ types in 10+ places could be swapped entirely for a better, external package.

Yes, true. Some updates wouldn't require too many changes and not for every plugin, others obviously would. With merging in some features currently provided by plugins, that effort should be reduced to a degree as well.

(2017-10-02, 12:05 PM)Devilshakerz Wrote: [ -> ]

Quote:phpBB and IPB went down this road in the past and managed to pull it off, so it's not like it's an impossible task.

Refactoring is the recommended approach to old codebase with large communities and reduced number of developers, but every project is different and requires separate analysis of overhead. 1.x is not only structurally outdated (dependencies, code quality) but has numerous defects that make most of the old code dangerous to run.

All I'm asking for is an analysis of overhead. Clearly the current approach isn't working and unless things change drastically, isn't going to suddenly start working.

(2017-10-02, 12:17 PM)StefanT Wrote: [ -> ]

(2017-10-02, 11:37 AM)Euan T Wrote: [ -> ]However if updates are broken into chunks with a clear schedule, I think it's a manageable task.

Unfortunately most of the plugins get abandoned after a while. MyBB 1.8 is out for 3 years and users still ask to use older plugins (and they often work after changing the compatibility flag). It's tricky to find the right way to satisfy the users here. Some users still stay on 1.6 or even older.

Yes, that is unfortunate, though I do think that our unclear policies in the past haven't helped us with that. During the 1.8 development cycle, we weren't great at letting our users know where we were headed or how we were progressing, which drove quite a lot of people away. We

(2017-10-02, 12:17 PM)StefanT Wrote: [ -> ]

(2017-10-02, 11:37 AM)Euan T Wrote: [ -> ]phpBB and IPB went down this road in the past and managed to pull it off, so it's not like it's an impossible task.

phpBB has improved their code but they are certainly not finished yet. There is still a lot of legacy and spaghetti code.

Yep, I'm not saying it's perfect, but it shows that an incremental approach is possible.