Suggestions for prioritising the release of 1.9

[Laird]

[Originally posted in a team-only private forum; moved to this public forum at Devilshakerz's suggestion]

Hi all,

As I wrote in my application to join the team, my highest priority, which I believe mirrors that of our users, is a default responsive theme, which means prioritising the release of 1.9. My analysis and recommendations in this respect after some thought follow.

Development discussions are, of course, better had publicly and transparently, however, given that the release of 1.9 has become a sensitive issue, with some in the community angry about its delay, I am posting this thread privately to start with. I will repost it publicly (edited if necessary) if the consensus here is that that's appropriate.

As a new and relatively uninformed team member, this all might come across as unjustifiably assertive, perhaps even presumptuous or arrogant. If so, please forgive me: my intention is simply to forge the best (immediate) path forward for our project. Where I get it wrong, especially due to ignorance or error, I hope that you will correct me. Where I overstep the bounds of my role, I hope that you will let me know.

That said, here is what I've come up with.

In summary, my recommendations for the initial 1.9 release are for:

1. 1.8.27 to be the last release prior to 1.9, and for the list of PRs planned to be merged prior to 1.8.27 to be ASAP (a) minimised and (b) frozen, and, thereafter, for all remaining PRs to be deferred to the 1.9 branch (with conflicts resolved as necessary) - but for even those PRs deferred to the 1.9 branch to not necessarily be applied prior to its initial release, the timeliness of which should be prioritised above the merging of (non-critical) PRs.
   
2. All non-critical unimplemented changes for 1.9 other than to the template system to be deferred to a later release.
   
3. Changes to the template system for 1.9 anyway to be minimised, except for backwards-incompatible changes, which should be kept to the bare minimum necessary to avoid future backwards-incompatible changes given the functionality we ultimately plan to, or will potentially, implement.
   

I have thought about issue #3689, 1.9 Theme System, to which Devilshakerz pointed me in my application thread. I do not currently have answers to all of the questions raised at the start, but here are some responses to, and considerations arising from, the discussion:

• I love Euan's idea of a JS manager similar to the stylesheet manager, also with support for the stipulation of dependencies. Although it is perhaps debatable as to whether or not this should be considered a non-backwards-compatible change that ought to be included in the initial 1.9 release, my view is that it is not, since we wouldn't be able to force plugin developers to use it anyway; the approach of inserting JS files into templates would always be available to them regardless. Thus, I do not think that we should aim to include it in the initial 1.9 release.
  
• I also love Devilshakerz's idea of enhancing Find Updated Templates so as to provide 'a comparison of what's changed in [a] specific upgrade in the master set; these "diffs" would be easier to recognize than master-custom comparison (as unrelated modifications are caught up in the report), along with a suitable UI (perhaps the diff - "suggested changes"/"copy these" - and an editor side-by-side).'
  

Also, I love the idea of providing the same sort of updating functionality - the sort that Devilshakerz suggests there that we extend with respect to templates - for stylesheets and Javascript files too. This is (with the caveats noted below) a backwards-compatible change though, so I recommend that we defer it beyond the initial 1.9 release.

Looking a little more closely into the current Find Updated Templates diff functionality, as Devilshakerz's comments suggest it is fruitful to do, I have an observation which might turn out to be false (and if you know it to be false, then please say so), but which right now seems to me to be true:

Because the diff is only two-way, and doesn't have access to nor take into account the prior Master template upon which the admin based their custom template, the explanation at the top of the diff report is not always accurate, that explanation being (colours not identical):

Changes that have been made between your previous version and this one are highlighted like this.
Any customizations you've made to your templates (the old ones) are highlighted like this.

It seems to me that if the new Master template deletes content present in the prior Master template, this will be shown in light red, for the admin to interpret as a "customization" that they had made to that prior template, which it is not, and which they had not.

It seems to me that this confusion can best be resolved by running a three-way (or at least a dual) diff, which does take into account the contents of the prior Master template from which the admin had generated their custom template. This, in turn, means storing that prior Master template and its relationship with the customised template. This would presumably be done automatically when the admin first saves an edited version of the (then-current) Master template.

This would allow us to extend Devilshakerz's idea a little: we could auto-generate a "patch" between the version of the Master template which the admin had customised, and the new version of the Master template, and then attempt to auto-apply that "patch". If there were no conflicts with the customisations made by the admin, then we could simply auto-apply the patch and avoid the need for the template to even show up in "Find Updated Templates" at all (perhaps, for safety, this would be configurable, as in a yes/no setting somewhere in the ACP something like "Attempt to auto-apply core updates to customised templates where possible?"). Again, this depends on the prior Master template being stored and related to the customised template.

I emphasise that final point because I think that it is of concern given the current goal of user-editable filesystem-based templates. In general, the problem is this: certain features such as the one I've just canvassed require that metadata is automatically generated and then associated with a customised template when that template is saved, however, it is impossible to automatically generate that metadata, let alone automatically store it against that template, when edits to templates are permitted via the filesystem. Too, in a filesystem-based approach, the best place to store that metadata might well be in the same file as the (user-editable) template, however, this would mix user-editable data with auto-generated data, which in my opinion is bad practice with meaningful potential for error.

Regarding inheritance, I agree with Wildcard in that issue's comments that the current system works well and seems to be optimal. Again, though, I have concerns about the implementation of inheritance given user-editable filesystem-based templates. In particular: how is inheritance to be implemented in files? If a user-editable filesystem template inherits from the Master template, then what does this look like in the filesystem? If it is implemented via an empty template file, then this confounds the user's editing experience, since the user then has to locate the Master template and manually copy its contents in on first edit. On the other hand, if it is implemented by a template file with contents identical to the Master template, then an inheriting template cannot be distinguished from one which does not inherit but happens to have the same contents as the Master template anyway. Neither of these options seem to me to be viable, and I can't see an alternative, viable option. If anybody else can, then please say so.

For these two reasons (and there might be others that I've overlooked), I think that filesystem-based templates, especially when user-editable independently of the MyBB UI, are not a good idea, and should be abandoned. If ease of editing is our goal, then incorporating into core Shade's FASTyle plugin, as suggested in the comments of issue #3686, Allow editing of Twig templates from the ACP, is a better idea. However, as this is a backwards-compatible change, my recommendation is that it be deferred beyond the initial 1.9 release.

I end this missive then with a fourth and final recommendation:

• That minimal changes be made to the templating system for the initial 1.9 release: Twig templates can and should be distributed, stored, and edited in the same way as templates for 1.8 - including by plugins via the find_replace_templatesets() function - except where this conflicts with a planned future feature which would otherwise be backwards-incompatible.
  

The only such feature which I am currently suggesting is the auto-patching of customised templates, which requires from the start that the version of the Master template upon which a custom template was based be stored against that custom template. Since we would ultimately want this same auto-patching to be possible for stylesheets though, we would also want 1.9 from the start to version stylesheets (just as for templates) and to store against each customised stylesheet the version of the Master stylesheet on which it is based.

I encourage others to suggest their own such features ASAP so that we can plan for them, and implement the necessary infrastructure for them in, and then roll out ASAP, MyBB 1.9 with a default responsive theme.

In the hope of the speediest possible release of 1.9,
Cheers, guys.

[Devilshakerz]

1.8.27 to be the last release prior to 1.9, and for the list of PRs planned to be merged prior to 1.8.27 to be ASAP (a) minimised and (b) frozen, and, thereafter, for all remaining PRs to be deferred to the 1.9 branch (with conflicts resolved as necessary) - but for even those PRs deferred to the 1.9 branch to not necessarily be applied prior to its initial release, the timeliness of which should be prioritised above the merging of (non-critical) PRs.

Pushing the updated 1.9 code a planned should make it more comfortable in a few areas:

• up-to-date comparison of how they diverge,
  
• merging into 1.9 by anyone without post-rebase conflicts,
  
• similarly, ability to require 1.8 Pull Requests to have a 1.9 equivalent,
  
• getting everyone used to actually running, and working with, 1.9 (which currently may have some basic errors that prevent free navigation)
  

We have a silent check that shows git conflicts for PRs as if the 1.9 branch was their base: https://github.com/mybb/mybb/runs/207434...#step:5:18.

Note .27 is unlikely to be the last one for practical purposes, but selective changes for high-priority Issues shouldn't cause much trouble for 1.9. Theoretically it could be the last maintenance release, leaving only security releases.

Once the 1.9 branch is active again, I agree with strict filtering of 1.8 changes, and if we continue with bringing 1.9 up to date by applying all changes at once, the developer(s) tasked with this should have such discretion.

It seems to me that if the new Master template deletes content present in the prior Master template, this will be shown in light red, for the admin to interpret as a "customization" that they had made to that prior template, which it is not, and which they had not.

Yes, no history is saved. Comparing master theme changes between versions shouldn't be too difficult to implement since the theme XML files are available remotely on GitHub. The same will apply to Twig template files and stylesheets.

I emphasise that final point because I think that it is of concern given the current goal of user-editable filesystem-based templates. In general, the problem is this: certain features such as the one I've just canvassed require that metadata is automatically generated and then associated with a customised template when that template is saved, however, it is impossible to automatically generate that metadata, let alone automatically store it against that template, when edits to templates are permitted via the filesystem. Too, in a filesystem-based approach, the best place to store that metadata might well be in the same file as the (user-editable) template, however, this would mix user-editable data with auto-generated data, which in my opinion is bad practice with meaningful potential for error.

In a tool (core, plugin, or external) to compare core theme changes, and to potentially create a patch (that may or may not be applicable to customized templates), which would take old and new MyBB versions as input, MyBB could use the theme's MyBB compatibility in its manifest file or use template file modification dates to make a best guess on the input values, but it should be satisfiable even when users have to enter these manually.

Note that third-party file comparison tools, as well as GitHub's own file comparison (between version tags), will be much more usable with the new theme system - non-technical users won't have to be familiar with XML syntax, have to strip CDATA, etc. to compare changes in individual templates and stylesheets.

Regarding inheritance, I agree with Wildcard in that issue's comments that the current system works well and seems to be optimal. Again, though, I have concerns about the implementation of inheritance given user-editable filesystem-based templates. In particular: how is inheritance to be implemented in files? If a user-editable filesystem template inherits from the Master template, then what does this look like in the filesystem?

Active themes (and any additional template sources), including the default base set (which I expect to be treated as static core files) and their inheritance path would result in a list where most specific sources have priority, and the template is fetched from the first source that can provide it.
In the case of the filesystem, files of templates inherited from a parent theme would simply not exist, which would be handled appropriately by the system.

In practice, the resolved solution for a particular list of template sources (e.g. final paths, functions, or template values) would be cached.

That minimal changes be made to the templating system for the initial 1.9 release: Twig templates can and should be distributed, stored, and edited in the same way as templates for 1.8 - including by plugins via the find_replace_templatesets() function - except where this conflicts with a planned future feature which would otherwise be backwards-incompatible.

Not sure if we can keep these functions working, given divergent syntax (doesn't mean we have to remove them), but a related idea is to leave the eval() database template system for extensions.
While plugins will have to be adapted for compatibility with core templates (which is often limited to adding new variables), they could add and use their own old templates to Global Templates and, if possible, attach old templates to 1.9 themes.

[Laird]

Thanks, Devilshakerz, for your informative reply.

Pushing the updated 1.9 code a planned should make it more comfortable in a few areas

Could you elaborate a little on the current plan? I ask because I am keen for us to short-circuit what seems to be an endless game of "rebase catch-up" that Euan has been forced to play for a very long time (years?), and I am curious to know how the current plan achieves that. It seems to me that the best way to achieve it is to halt further 1.8 PRs, deferring any existing 1.8 PRs to 1.9, even if they have to be rewritten, allow Euan to finally finish the rebase for good, and then - as you suggest in your bullet points - to (at least) require further 1.8 PRs to have a 1.9 equivalent. I write "at least" because better, IMO, would be to require all future PRs to be against 1.9, without even necessarily having a 1.8 equivalent except for critical - security-related - PRs.

We have a silent check that shows git conflicts for PRs as if the 1.9 branch was their base: https://github.com/mybb/mybb/runs/207434...#step:5:18.

The "cherry-pick test"? I had wondered what that referred to...

Note .27 is unlikely to be the last one for practical purposes, but selective changes for high-priority Issues shouldn't cause much trouble for 1.9. Theoretically it could be the last maintenance release, leaving only security releases.

Let's make that theory our practice.

It seems to me that if the new Master template deletes content present in the prior Master template, this will be shown in light red, for the admin to interpret as a "customization" that they had made to that prior template, which it is not, and which they had not.

Yes, no history is saved. Comparing master theme changes between versions shouldn't be too difficult to implement since the theme XML files are available remotely on GitHub. The same will apply to Twig template files and stylesheets.

Surely, though, saving history, and then comparing local files, is faster and more efficient than remote fetches? Moreover, it's less prone to errors - see below.

I emphasise that final point because I think that it is of concern given the current goal of user-editable filesystem-based templates. In general, the problem is this: certain features such as the one I've just canvassed require that metadata is automatically generated and then associated with a customised template when that template is saved, however, it is impossible to automatically generate that metadata, let alone automatically store it against that template, when edits to templates are permitted via the filesystem. Too, in a filesystem-based approach, the best place to store that metadata might well be in the same file as the (user-editable) template, however, this would mix user-editable data with auto-generated data, which in my opinion is bad practice with meaningful potential for error.

In a tool (core, plugin, or external) to compare core theme changes, and to potentially create a patch (that may or may not be applicable to customized templates), which would take old and new MyBB versions as input, MyBB could use the theme's MyBB compatibility in its manifest file

Can we guarantee though that a theme's compatibility string will be specific enough to identify a particular version, rather than something generic like 1.9.*? Even if we can, how can we know - if we don't store it when the custom template is first saved - which base theme with a specific version it was customised from? You suggest...

or use template file modification dates to make a best guess on the input values, but it should be satisfiable even when users have to enter these manually.

...however, these are fallible approaches. We don't have to settle for a fallible approach if we drop the idea of filesystem-based templates. We don't lose anything meaningful either - see below.

Note that third-party file comparison tools, as well as GitHub's own file comparison (between version tags), will be much more usable with the new theme system - non-technical users won't have to be familiar with XML syntax, have to strip CDATA, etc. to compare changes in individual templates and stylesheets.

But nor will they need to if we implement the sort of in-app diffing tool you suggested in your 1.9 Theme System issue - which, as I'm arguing, to avoid fallibility would need to rely on templates being stored with metadata in the database. We could go so far as to - where automated patching fails or is forbidden by an ACP setting - display individual merge conflicts in the UI and allow the user to choose a resolution to each one (or all combined) with the simple click of a button.

[Regarding inheritance:]
In the case of the filesystem, files of templates inherited from a parent theme would simply not exist, which would be handled appropriately by the system.

This was one of the horns of the dilemma that I posed though: one justification for filesystem-based templates is that they are more user-friendly because the user can load and edit them in his or her favourite editor, but having to create a non-existent template and copy in its content from another file, which first has to be correctly identified and then opened, is the opposite of user-friendly. In this respect, the current MyBB UI is far superior: whether or not the template is currently being inherited, if you try to edit it, its contents are pulled into the editor up-front with no copy-paste effort required.

In practice, the resolved solution for a particular list of template sources (e.g. final paths, functions, or template values) would be cached.

I agree that caching is the way to go, but isn't that a separate question from where the pre-cache sources are stored?

That minimal changes be made to the templating system for the initial 1.9 release: Twig templates can and should be distributed, stored, and edited in the same way as templates for 1.8 - including by plugins via the find_replace_templatesets() function - except where this conflicts with a planned future feature which would otherwise be backwards-incompatible.

Not sure if we can keep these functions working, given divergent syntax (doesn't mean we have to remove them)

Oh, but find_replace_templatesets() is syntax-agnostic. Likewise, the template column of the mybb_templates table is syntax-agnostic, as is the content of each <template> tag in install/resources/mybb_theme.xml. That's what I meant by minimal changes being necessary. All that needs to be changed is the - syntax-sensitive - renderer: from eval() to Twig.

but a related idea is to leave the eval() database template system for extensions.
While plugins will have to be adapted for compatibility with core templates (which is often limited to adding new variables), they could add and use their own old templates to Global Templates and, if possible, attach old templates to 1.9 themes.

Perhaps that could be a temporary but deprecated feature for backwards compatibility alone? I don't think we'd want to support the use of eval() in our codebase indefinitely, would we?

[Devilshakerz]

It may be best to hear from Euan about plans and own preferences (and how PHP 8 and similar updates should be handled, which may be contained in .27), but hopefully we can switch to clean and more controlled merges for 1.9.

Currently theme XML files contain a specific x.y.z version (as opposed to 18* used in plugins).

We want to offload most of the theme metadata to (eventually) Composer-compatible package.json manifest files (some data may be still left in the database, especially if we want to leave a skeleton for compatibility with extensions) - this file, as well as template files, can be generated and updated automatically.
The inheritance and version details would be stored there.

While leaving administrators to editing files manually may be a MVP, we do expect to keep (and extend) existing functionality of creating and customizing themes in the ACP (and the final difference would be no XML files to download/upload - just files to copy over).

Theme updates (especially non-core ones) is one of the missing extension management features. Some ideas here or elsewhere may only refer to dealing with core changes (in the base theme).

Assuming any derivative template is structured correctly (base theme ← theme from Extend ← board-specific customizations), we can adjust the structure and storage to allow multiple versions - old versions would be stored indefinitely with manual purging, which would allow to simply review changes, or to use them to update dependent themes.

In practice, the resolved solution for a particular list of template sources (e.g. final paths, functions, or template values) would be cached.

I agree that caching is the way to go, but isn't that a separate question from where the pre-cache sources are stored?

That's a minor point relating to eventually limiting the number of disk reads (multiple directories would need to be checked for each template).

Oh, but find_replace_templatesets() is syntax-agnostic. Likewise, the template column of the mybb_templates table is syntax-agnostic, as is the content of each <template> tag in install/resources/mybb_theme.xml. That's what I meant by minimal changes being necessary. All that needs to be changed is the - syntax-sensitive - renderer: from eval() to Twig.

Yes, it's mainly related to the values used with the function in plugin files, rather than its implementation. It may be better to fail explicitly, rather than have eval() syntax injected into new templates with the same name.

but a related idea is to leave the eval() database template system for extensions.
While plugins will have to be adapted for compatibility with core templates (which is often limited to adding new variables), they could add and use their own old templates to Global Templates and, if possible, attach old templates to 1.9 themes.

Perhaps that could be a temporary but deprecated feature for backwards compatibility alone? I don't think we'd want to support the use of eval() in our codebase indefinitely, would we?

Yes, I'd expect this would be removed in 1.10. We'd prefer to limit compatibility problems with plugins in 1.9, since themes will be left out completely.

[Laird]

It may be best to hear from Euan about plans and own preferences (and how PHP 8 and similar updates should be handled, which may be contained in .27), but hopefully we can switch to clean and more controlled merges for 1.9.

OK, thanks - hopefully Euan isn't too time-strapped to offer a response.

Currently theme XML files contain a specific x.y.z version (as opposed to 18* used in plugins).

We want to offload most of the theme metadata to (eventually) Composer-compatible package.json manifest files (some data may be still left in the database, especially if we want to leave a skeleton for compatibility with extensions) - this file, as well as template files, can be generated and updated automatically.
The inheritance and version details would be stored there.

While leaving administrators to editing files manually may be a MVP, we do expect to keep (and extend) existing functionality of creating and customizing themes in the ACP (and the final difference would be no XML files to download/upload - just files to copy over).

Thanks again. I understand the appeal of this approach, although my objections - which, to avoid becoming argumentative, I won't reiterate - remain. Let's again see what Euan has to say.

Theme updates (especially non-core ones) is one of the missing extension management features. Some ideas here or elsewhere may only refer to dealing with core changes (in the base theme).

A great point which until now had not been raised. I had wondered whether I was missing something in that, after installing a custom theme, there seemed to be no mechanism by which to later update it. I understand now that I wasn't missing something; rather, update functionality for custom themes is missing.

This seems to me to be somewhat complicated, in that a custom theme might initially be based on a particular version of the base theme, but that then its update might be based on a later version of the base theme - whilst containing updates of its own. Hmm. Not so straightforward to handle.

Assuming any derivative template is structured correctly (base theme ← theme from Extend ← board-specific customizations), we can adjust the structure and storage to allow multiple versions - old versions would be stored indefinitely with manual purging, which would allow to simply review changes, or to use them to update dependent themes.

Thanks. This adds some detail to my understanding of the proposed new functionality, although I am not sure that it addresses my objections, which, again, to avoid becoming argumentative, I won't reiterate.

but a related idea is to leave the eval() database template system for extensions.
While plugins will have to be adapted for compatibility with core templates (which is often limited to adding new variables), they could add and use their own old templates to Global Templates and, if possible, attach old templates to 1.9 themes.

Perhaps that could be a temporary but deprecated feature for backwards compatibility alone? I don't think we'd want to support the use of eval() in our codebase indefinitely, would we?

Yes, I'd expect this would be removed in 1.10. We'd prefer to limit compatibility problems with plugins in 1.9, since themes will be left out completely.

Great. That makes a lot of sense.

[Laird]

Copying in here a Discord exchange with @Devilshakerz from #staff-1x-organization:

@Laird RE https://community.mybb.com/thread-231966...pid1365473 are you concerned about individual template-level metadata?

@Devilshakerz yes, my concern is that some functionality that I would like to see supported - which is, as I wrote in the thread, the auto-patching of custom (individual) templates which were based on an earlier version of an updated Master template - would not be possible (or would only be fallibly possible) if we did not know what that earlier version of the template was, and it seems to me that the only way we can reliably know is if the version is stored as metadata alongside the custom (individual) template when that custom (individual) template is first saved. I use this only as an example of the general case in which we might imagine that some other scenario might exist in which we would similarly need to store metadata against an individual (customised) template, even though I can't bring to mind any other examples right now.
(I need to keep better track of new Discord messages because I would have liked to have responded to your message sooner).
(And I hope I haven't missed the point of your question).

we can save the whole tree in the manifest file - automatically when edited via the ACP:

{
  "theme": {
      "name": "My theme",
      "parent": "base",
      "templates": {
        "showthread": {
            "version": "3",
            "notes": "Added disclaimer to thread pages"
        }
      }
  }
}

Sure! I think that that would work. Whether it's preferable to a DB approach is not a question I'd ask at this point, although it would need to be asked at some point. Instead, I'll simply clarify: my objection to the proposed filesystem-based templates is not that they are filesystem-based as such, it is that they are proposed as being user-editable VIA the filesystem: in other words, that changes can be (and are encouraged to be) made to them independently of the MyBB UI; changes that avoid the automated addition of this sort of metadata. If we make it clear to users that templates should never be edited directly in the filesystem, and only be edited via the ACP, then in principle (with the caveat above), my concerns are addressed.

a development mode for template editing (where MyBB would rebuild the metadata as necessary) may do the trick

Agreed.

[martec]

I agree with almost everything (anything that might delay the launch should be postponed. ) except the part of using filesystem for template for me it should continue to store in database as 1.8 and keep the find_replace_templatesets() function and the hierarchy system is sure to be much easier to maintain if it is stored in a database.
But the issue of filesystem I have already commented on another topic.
anyway i am happy that with sharing the thoughts of the mybb team in relation to mybb 1.9.

*Keep in mind that this is just my personal opinion.

[Laird]

I agree with almost everything (anything that might delay the launch should be postponed. )

Yes, avoiding delay is my main concern too. Glad to see you agree.

except the part of using filesystem for template for me it should continue to store in database as 1.8 and keep the find_replace_templatesets() function and the hierarchy system is sure to be much easier to maintain if it is stored in a database. 
But the issue of filesystem I have already commented on another topic.

I agree, but am open to opposing views and arguments. As I mentioned already in the other thread, I understand that a blog post is planned to lay out the advantages and disadvantages of both approaches.

anyway i am happy that with sharing the thoughts of the mybb team in relation to mybb 1.9.

And it's good to get your thoughts. Thanks for sharing.

[martec]

I agree, but am open to opposing views and arguments. As I mentioned already in the other thread, I understand that a blog post is planned to lay out the advantages and disadvantages of both approaches.

Thanks for the reply.
So I will be waiting for the blog post.

[Alhimika]

Shoud I say AMEN to the most of existing plugins?