I've put together a summary and proposal text document which at Devilshakerz's invitation I reproduce in full below. I'm a little concerned that indentation will be lost, but we'll see how that works out.
Title : MyBB 1.9 Theme System Development Notes
Author : Laird
Version: 2
Date : 2021-10-23
Introduction:
This plain-text document compiles and summarises ideas suggested for the MyBB 1.9 theme system, answers some questions that have been posed of that system, and then proposes a minimalist set of development goals for that theme system for the very first release of MyBB 1.9.
Sources:
The following three sections of this document were compiled from several sources:
* The Community Forums thread "The 1.9 theme and template system":
https://community.mybb.com/thread-232217.html
* GitHub issue #3689 "1.9 Theme System", created by @Devilshakerz:
https://github.com/mybb/mybb/issues/3689
* A Discord discussion in the MyBB server's public 1x-development channel:
https://discord.com/channels/21587684763...9839220736
The subsequent section consists of my own proposed way forward in response to it all.
Definitions:
"Resource": A template, stylesheet, or, ultimately (beyond the initial release of 1.9), Javascript file or pre-processed CSS file. Other resource types might later be added.
"Theme property": Any relevant and/or necessary data associated with a theme, such as, for a theme's stylesheet, the page(s) to which it applies.
"Core theme": The primary theme shipped with MyBB, including all of its resources. Intended to be immutable other than when updating MyBB.
"Original theme": A custom theme, such as downloaded from Extend. Intended to be immutable other than when updating MyBB or the theme itself.
"Board theme": A mutable (by extensions and by admins via the ACP/filesystem) theme initially copied in full from (potentially via copy-on-write aka COW), and irrevocably associated with, either the core theme or an original theme.
Existing ideas:
1. Resource management for extensions:
1.1. Permanent additions:
This "extension layer" of new resources and their properties conceptually extends the core theme and is inherited by original and board themes.
1.2. Permanent modifications:
Resource modification functions directly modify the existing resources of board themes, similar to how MyBB 1.8's find_replace_templatesets() function works.
1.3. Dynamic modifications:
"Template hooks" provide for dynamic, compile-time modifications of resources, potentially using functions corresponding to those of 1.2 above (and see 3 below).
2. Resource tracking:
Resources added/modified by 1.1 and 1.2 above are tracked, so that additions/modifications can be undone even when the plugins that made them are removed.
3. Fine-grained modifications:
Modification functions for 1.2 above provide for detailed stipulation of changes, to support the resource tracking and undo functionality of 2 above.
For example (lightly edited extract of code provided by Devilshakerz):
// modify existing template
\MyBB\ExtendTheme\Template('index.twig')
->insertBefore(
'<a href="custom_page.php">Custom Page</a>',
'<!-- end: navigation -->',
);
4. Conditional applicability:
Any addition or modification of a resource per 1.1 to 1.3 above can be stipulated as applying to certain custom themes only, at least via the UI, and potentially programmatically too.
5. Retention of prior theme versions:
Whenever the core theme or an original theme is updated, its previous version is stored, so as to support enhanced updating and conflict resolution (see next).
6. Enhanced updating and conflict resolution:
During both the upgrade of MyBB core and the upload of a new version of an original theme, MyBB attempts - conceptually via generating and then applying a patch for each change - to automatically update all resources for all board themes, i.e., handling (passing through) prior modifications to board themes either by an extension or by an admin via the ACP. If automatic application of a conceptual patch fails, then the UI guides the admin through the resolution of the conflict.
7. The encouragement of good inheritance practices:
The resources of core and original themes are considered immutable (read-only); only board theme resources may be modified. This is enforced via the UI, but can only be encouraged for direct filesystem edits of templates.
8. Continued, but deprecated, eval()-based template support:
To make plugin migration easier, but only temporarily: to be removed in a later release.
9. Compatibility flagging:
The UI supports the marking of a board theme as compatible with the current version of a plugin. This suppresses the UI from otherwise displaying a need for conflict resolution.
10. Resource management for Javascript (JS) files:
Similar to the existing stylesheet management, with these features:
10.1. Dependency support: one JS resource can be flagged as depending on another, in which case the latter is loaded automatically if necessary.
10.2. Automatic combining, minifying, and caching of multiple JS resources, with cache busting.
11. The merge of the FAStyle plugin into core.
12. The packaging of themes as Composer packages:
An optional extra method of installation for theme developers to utilise, in addition to the standard installation procedure (uploading a file to the ACP).
13. Pre-processed CSS support:
In addition to stylesheet resources, pre-processed CSS files are also supported.
14. Colour scheme support:
Made possible via 13 above; includes a colour scheme editor.
Questions (and my answers):
The first question is mine; the rest are from issue #3689.
1. Should templates/stylesheets be stored in the filesystem or DB?
A: Most folk seem to prefer the filesystem, and I don't think my objections to that approach are strong enough, so let's stick with filesystem templates.
2. What theme properties should the manifest file include?
3. What metadata should the manifest file include?
A: I'm not entirely sure as to the difference between theme properties and metadata, so I'll consider them together. Let's just compile this list as we go: start from scratch, and, whenever, during development, we find a need (or high desirability) for a property / item of metadata, we add it. There should at least, though, be entries for version, author, type (core/original/board), parent (if none, the core theme is assumed to be parent), and the pages associated with any stylesheet(s). Additionally, for a board theme: the last version of the core/original theme with which it is associated (even if via a chain of inheritance) with which it has been synchronised.
4. How should theme installation & modification work (Composer, CSS pre-processors)?
A: Let's defer support for Composer and CSS pre-processors until after the initial 1.9 release. Until then, theme installation then should work similarly to how it already does: the theme's file, a ZIP archive with structure described below, is uploaded in the ACP, and its resources are unzipped and installed (with a similar approach as described for plugins below to be used as a fallback when PHP's Zip extension is missing).
5. How should MyBB compatibility updates work?
A: I'm not really sure what a compatibility update means in this context, so I can't answer this question.
6. How should stylesheet information be stored?
A: Each stylesheet file itself containing raw CSS should be stored in the filesystem (see proposed structure below). Its properties/metadata (e.g., the pages with which it is associated), should be stored as entries in its theme's manifest file.
7. How should the new system handle inheritance? How should that information be stored?
A: See my ideas below re filesystem structure.
8. What system could be implemented for applying theme updates?
A: The same sort of system as for applying core theme updates: the conceptual generation of a patch which MyBB attempts to auto-apply to associated board themes, and to board themes which inherit from those board themes (see below re inheritance and association), etc, with UI-assisted conflict resolution as necessary.
9. What level of support should be left for 1.8 templates & stylesheet to maintain 1.8 extension compatibility? How?
A: Continued, although deprecated, support for both, simply by not making or enforcing any changes to the existing support.
My proposals/suggestions for the initial 1.9 release:
1. That we defer until a later release all items from the "Existing ideas" list above except 1.1, 1.2, 5, 6, 7, and 8. This is so as to release MyBB 1.9 as soon as possible. Plugin/theme developers can then begin to migrate over to the new Twig templating system while we enhance theme functionality further by implementing other items from the list.
2. That re idea 1.2 in that list (permanent modifications of resources by plugins), we implement a simple function corresponding to find_replace_templatesets(), perhaps named something like find_replace_twig_tpls(), which does effectively the same job, perhaps with better indication of errors such as changes failing to apply. If/when we later implement idea 3 (fine-grained modification functions), we deprecate this function. I say "if" because I have concerns over the viability of this approach, in particular that changes cannot necessarily be reverted deterministically when *subsequent* changes have been made, although my concerns are somewhat alleviated by the observation that the existing find_replace_templatesets() approach is not immune from this criticism either.
3. That in implementing especially ideas 1.1 and 6, we fully support stylesheets, in particular for automated patching and assisted conflict resolution on core upgrades and on original theme updates, reusing the associated functionality for templates.
4. That re inheritance and association, we stipulate that inheritance applies only to board themes: only a board them can inherit, and only from another board theme. A board theme may alternatively (one or the other must be the case), though, be irrevocably *associated* with the (immutable) original or core theme of which it is a (mutable) copy, meaning that updates to that original/core theme are auto-applied to all associated board themes (and to those board themes which inherit from those board themes, etc). The metadata for an *original* theme may, though, additionally, indicate (for purely informative purposes) that it is "based on" or "inspired by" some other original theme - an original theme which need not be installed, and changes to which do not affect the inspired original theme in the local installation, whereas original themes which are inherited from or associated with board themes *do* need to be installed, and attempting to uninstall such a theme results in an error.
5. That re a filesystem structure, with support for both inheritance and plugin-generated resources per existing idea 1.1 (the extension layer) above, zipped themes (and the "extension layer" theme component of plugins) contain a coherent structure that can be simply unzipped and used as-is. The structure I suggest is as follows (the characters in all directory and file names are, I suggest, restricted to lowercase alphabetical characters (since on Windows, filenames are case-insensitive), digits, dashes, and underscores):
-> theme-name.json [the theme's manifest file; could be XML rather than JSON, but, in my view, JSON is simpler and neater; the theme name may not begin with 'ext-' for reasons made clear further down]
-> tpls [the root directory containing Twig templates]
-> <tpl-group> [each <tpl-group> entry is a subdirectory named for a template group; these subdirectories can be nested]
-> <tpl>.twig [each <tpl>.twig is a raw Twig template file where <tpl> is the name of the template, potentially within any level of nested <tpl-group> parent directories]
-> css [the root directory containing stylesheets]
-> <filename>.css [a raw stylesheet file containing CSS, where the characters in <filename> have the same restrictions as above]
-> jscripts [the root directory containing Javascript files; inapplicable to the initial release of 1.9]
-> <path-to>/<filename>.js [a raw Javascript file, where the characters in <filename> have the same restrictions as above; and <path-to> is either empty or represents any set of subdirectories]
Later, we can also add support for precompiled CSS files.
Each of these theme directories is placed within either:
-> inc/themes/immutable [for the core theme and original themes]
or
-> inc/themes/mutable [for board themes]
The structure for each theme under the respective mutable/immutable directory is:
-> theme-name [the root directory for the theme, named after the theme; must match the name of the theme in its manifest file]
-> current [the current version of the theme, with directory structure as above]
-> <version> [an historical version of the theme, again with directory structure as above, with version stipulated by <version>; this is a string compatible with PHP's version_compare() function]
In addition, plugins supply their resources per 1.1 in the "Existing ideas" list above in a structure identical to the theme structure described above, the only difference being that they are placed under:
-> inc/themes/immutable/ext-<plugin-codename>
but still contain a "current" subdirectory as well as any applicable historical "<version>" subdirectories.
Note: if "mutable" and "immutable" are considered to be too unfamiliar and non-user-friendly, we could instead use terms something like, respectively, "editable" and "read-only".
6. That inheritance be implemented as follows: a board theme's manifest file indicates from which board theme it inherits, or (mutually exclusively) with which original/core theme it is irrevocably associated. In the event that a file is not found in the board theme's directory structure, the inheritance chain is searched until a parent theme is found which contains the given file (the idea is to implement copy-on-write (COW), such that, unless modified, files will not be present in inheriting themes). Similarly, in editing the board theme in the ACP, the list of its files (templates or stylesheets) is generated by traversing its inheritance chain up to and including the core theme, gathering all files at all levels, and then listing them. When saving an edited file, COW again applies: if an inherited resource is modified but otherwise not present for the inheriting theme in the filesystem, then a new file is inserted into the filesystem for that inheriting theme.
7. Given that plugins will be supplying theme directories (in inc/themes/immutable/ext-<plugin-codename>) which we do not want to be simply copied over (since we do not want to overwrite the "current" subdirectory without first archiving it in a "<version>" subdirectory), the initial 1.9 release is a good occasion on which to introduce semi-automated plugin installation: rather than FTPing an unzipped plugin directory structure directly to the board's root directory and then installing it via the ACP's Plugins page, the ZIP archive itself would be uploaded via the ACP's Plugins page. In addition, although consisting in an ordinary ZIP file, the extension for MyBB plugin archives would be changed from .zip (to something perhaps like .mbp?), so as to avoid tempting members into unzipping them and uploading them as-is, thus mistakenly overwriting the plugin's "current" theme directory.
A fall-back for admins using servers without PHP's Zip extension would be to rename the file so that it has a .zip extension, then upload the unzipped contents to some sort of staging directory checked by the ACP's Plugins page - perhaps simply a subdirectory under the board's root named "plugin_staging".