The 1.9 theme and template system
#21
(2021-07-08, 05:09 PM)Laird Wrote:
(2021-07-08, 12:41 PM)Laird Wrote: Does that seem like a fair summary of your diagram/proposal?

In the interests of increasing bandwidth, I'm going to at this point assume that it is

Bad assumption. Thanks, @Devilshakerz, for helping to clear up my confusion, both in this thread and on Discord. As we seem to agree, some of the arrows in your diagram mean different things than other arrows. I had mistakenly understood them to all mean the same thing - something like "The source box is modified into the target box by some means" (with means including modifications by human theme designers - hence the "Human design" annotation in my alternative diagram).

Instead, the left column represents:

(2021-07-08, 09:56 PM)Devilshakerz Wrote: the inheritance chain.

As opposed to the "modifying" arrows, the arrows in that chain represent something like "The source box (ancestor) provides resources to the target box (descendant) where the descendant has no provision for the resource in itself" (and where resources can potentially be pulled in - inherited - from several ancestors back: from the closest one which provides the resource in question).

The modifications made to the core theme by a theme developer to arrive at a custom theme (an "original theme" in your terms, which I mistakenly suggested above that your diagram didn't actually include) are not covered by your diagram; they are simply understood to have been undertaken already.

I was a little obtuse, but hopefully we're on the same page now. My alternative diagram above can be ignored.

Dynamic modifications as the "third bite" aside, I like the example you provided on Discord which causes me to rethink my suggestion that extensions having "two bites of the cherry" is redundant given that only one "bite" seems necessary. In your example, a resource inserted by an extension's "first bite" at the top of the inheritance chain can then be overridden by a custom theme which is "aware" of that extension, which wouldn't be possible if that extension had instead been forced to use a "second bite" to change that resource in the custom theme, thus preventing the custom theme from overriding it. This seems like a valid reason to provide extensions with "two bites" at permanent modifications.

So, I agree after all with what @martec wrote:

(2021-05-13, 04:19 AM)martec Wrote: @Devilshakerz

What you are proposing seems in fact ideal, that is to say I am not disagreeing.

How much of this should be implemented in the initial 1.9 release is (from my perspective) now the pressing question.
Reply
#22
A more precise diagram from Discord discussions earlier:

[Image: 19-theme-system-v5.png]
devilshakerz.com/pgp (DF3A 34D9 A627 42E5 BC6A 6750 1F2F B8AA 28FF E1BC) ▪ keybase.io/devilshakerz
Reply
#23
I've put together a summary and proposal text document which at Devilshakerz's invitation I reproduce in full below. I'm a little concerned that indentation will be lost, but we'll see how that works out.



Title  : MyBB 1.9 Theme System Development Notes
Author : Laird
Version: 2
Date  : 2021-10-23


Introduction:

This plain-text document compiles and summarises ideas suggested for the MyBB 1.9 theme system, answers some questions that have been posed of that system, and then proposes a minimalist set of development goals for that theme system for the very first release of MyBB 1.9.


Sources:

The following three sections of this document were compiled from several sources:

  * The Community Forums thread "The 1.9 theme and template system": https://community.mybb.com/thread-232217.html

  * GitHub issue #3689 "1.9 Theme System", created by @Devilshakerz: https://github.com/mybb/mybb/issues/3689

  * A Discord discussion in the MyBB server's public 1x-development channel: https://discord.com/channels/21587684763...9839220736

The subsequent section consists of my own proposed way forward in response to it all.


Definitions:

  "Resource": A template, stylesheet, or, ultimately (beyond the initial release of 1.9), Javascript file or pre-processed CSS file. Other resource types might later be added.

  "Theme property": Any relevant and/or necessary data associated with a theme, such as, for a theme's stylesheet, the page(s) to which it applies.

  "Core theme": The primary theme shipped with MyBB, including all of its resources. Intended to be immutable other than when updating MyBB.

  "Original theme": A custom theme, such as downloaded from Extend. Intended to be immutable other than when updating MyBB or the theme itself.

  "Board theme": A mutable (by extensions and by admins via the ACP/filesystem) theme initially copied in full from (potentially via copy-on-write aka COW), and irrevocably associated with, either the core theme or an original theme.


Existing ideas:

  1. Resource management for extensions:

    1.1. Permanent additions:

          This "extension layer" of new resources and their properties conceptually extends the core theme and is inherited by original and board themes.

    1.2. Permanent modifications:

          Resource modification functions directly modify the existing resources of board themes, similar to how MyBB 1.8's find_replace_templatesets() function works.

    1.3. Dynamic modifications:

          "Template hooks" provide for dynamic, compile-time modifications of resources, potentially using functions corresponding to those of 1.2 above (and see 3 below).

  2. Resource tracking:

    Resources added/modified by 1.1 and 1.2 above are tracked, so that additions/modifications can be undone even when the plugins that made them are removed.

  3. Fine-grained modifications:

    Modification functions for 1.2 above provide for detailed stipulation of changes, to support the resource tracking and undo functionality of 2 above.

    For example (lightly edited extract of code provided by Devilshakerz):

        // modify existing template
        \MyBB\ExtendTheme\Template('index.twig')
            ->insertBefore(
                '<a href="custom_page.php">Custom Page</a>',
                '<!-- end: navigation -->',
            );

  4. Conditional applicability:

    Any addition or modification of a resource per 1.1 to 1.3 above can be stipulated as applying to certain custom themes only, at least via the UI, and potentially programmatically too.

  5. Retention of prior theme versions:

    Whenever the core theme or an original theme is updated, its previous version is stored, so as to support enhanced updating and conflict resolution (see next).

  6. Enhanced updating and conflict resolution:

    During both the upgrade of MyBB core and the upload of a new version of an original theme, MyBB attempts - conceptually via generating and then applying a patch for each change - to automatically update all resources for all board themes, i.e., handling (passing through) prior modifications to board themes either by an extension or by an admin via the ACP. If automatic application of a conceptual patch fails, then the UI guides the admin through the resolution of the conflict.

  7. The encouragement of good inheritance practices:

    The resources of core and original themes are considered immutable (read-only); only board theme resources may be modified. This is enforced via the UI, but can only be encouraged for direct filesystem edits of templates.

  8. Continued, but deprecated, eval()-based template support:

    To make plugin migration easier, but only temporarily: to be removed in a later release.

  9. Compatibility flagging:

    The UI supports the marking of a board theme as compatible with the current version of a plugin. This suppresses the UI from otherwise displaying a need for conflict resolution.

  10. Resource management for Javascript (JS) files:

      Similar to the existing stylesheet management, with these features:

      10.1. Dependency support: one JS resource can be flagged as depending on another, in which case the latter is loaded automatically if necessary.

      10.2. Automatic combining, minifying, and caching of multiple JS resources, with cache busting.

  11. The merge of the FAStyle plugin into core.

  12. The packaging of themes as Composer packages:

      An optional extra method of installation for theme developers to utilise, in addition to the standard installation procedure (uploading a file to the ACP).

  13. Pre-processed CSS support:

      In addition to stylesheet resources, pre-processed CSS files are also supported.

  14. Colour scheme support:

      Made possible via 13 above; includes a colour scheme editor.


Questions (and my answers):

The first question is mine; the rest are from issue #3689.

  1. Should templates/stylesheets be stored in the filesystem or DB?

    A: Most folk seem to prefer the filesystem, and I don't think my objections to that approach are strong enough, so let's stick with filesystem templates.

  2. What theme properties should the manifest file include?
  3. What metadata should the manifest file include?

    A: I'm not entirely sure as to the difference between theme properties and metadata, so I'll consider them together. Let's just compile this list as we go: start from scratch, and, whenever, during development, we find a need (or high desirability) for a property / item of metadata, we add it. There should at least, though, be entries for version, author, type (core/original/board), parent (if none, the core theme is assumed to be parent), and the pages associated with any stylesheet(s). Additionally, for a board theme: the last version of the core/original theme with which it is associated (even if via a chain of inheritance) with which it has been synchronised.

  4. How should theme installation & modification work (Composer, CSS pre-processors)?

    A: Let's defer support for Composer and CSS pre-processors until after the initial 1.9 release. Until then, theme installation then should work similarly to how it already does: the theme's file, a ZIP archive with structure described below, is uploaded in the ACP, and its resources are unzipped and installed (with a similar approach as described for plugins below to be used as a fallback when PHP's Zip extension is missing).

  5. How should MyBB compatibility updates work?

    A: I'm not really sure what a compatibility update means in this context, so I can't answer this question.

  6. How should stylesheet information be stored?

    A: Each stylesheet file itself containing raw CSS should be stored in the filesystem (see proposed structure below). Its properties/metadata (e.g., the pages with which it is associated), should be stored as entries in its theme's manifest file.

  7. How should the new system handle inheritance? How should that information be stored?

    A: See my ideas below re filesystem structure.

  8. What system could be implemented for applying theme updates?

    A: The same sort of system as for applying core theme updates: the conceptual generation of a patch which MyBB attempts to auto-apply to associated board themes, and to board themes which inherit from those board themes (see below re inheritance and association), etc, with UI-assisted conflict resolution as necessary.

  9. What level of support should be left for 1.8 templates & stylesheet to maintain 1.8 extension compatibility? How?

    A: Continued, although deprecated, support for both, simply by not making or enforcing any changes to the existing support.


My proposals/suggestions for the initial 1.9 release:

  1. That we defer until a later release all items from the "Existing ideas" list above except 1.1, 1.2, 5, 6, 7, and 8. This is so as to release MyBB 1.9 as soon as possible. Plugin/theme developers can then begin to migrate over to the new Twig templating system while we enhance theme functionality further by implementing other items from the list.

  2. That re idea 1.2 in that list (permanent modifications of resources by plugins), we implement a simple function corresponding to find_replace_templatesets(), perhaps named something like find_replace_twig_tpls(), which does effectively the same job, perhaps with better indication of errors such as changes failing to apply. If/when we later implement idea 3 (fine-grained modification functions), we deprecate this function. I say "if" because I have concerns over the viability of this approach, in particular that changes cannot necessarily be reverted deterministically when *subsequent* changes have been made, although my concerns are somewhat alleviated by the observation that the existing find_replace_templatesets() approach is not immune from this criticism either.

  3. That in implementing especially ideas 1.1 and 6, we fully support stylesheets, in particular for automated patching and assisted conflict resolution on core upgrades and on original theme updates, reusing the associated functionality for templates.

  4. That re inheritance and association, we stipulate that inheritance applies only to board themes: only a board them can inherit, and only from another board theme. A board theme may alternatively (one or the other must be the case), though, be irrevocably *associated* with the (immutable) original or core theme of which it is a (mutable) copy, meaning that updates to that original/core theme are auto-applied to all associated board themes (and to those board themes which inherit from those board themes, etc). The metadata for an *original* theme may, though, additionally, indicate (for purely informative purposes) that it is "based on" or "inspired by" some other original theme - an original theme which need not be installed, and changes to which do not affect the inspired original theme in the local installation, whereas original themes which are inherited from or associated with board themes *do* need to be installed, and attempting to uninstall such a theme results in an error.

  5. That re a filesystem structure, with support for both inheritance and plugin-generated resources per existing idea 1.1 (the extension layer) above, zipped themes (and the "extension layer" theme component of plugins) contain a coherent structure that can be simply unzipped and used as-is. The structure I suggest is as follows (the characters in all directory and file names are, I suggest, restricted to lowercase alphabetical characters (since on Windows, filenames are case-insensitive), digits, dashes, and underscores):

      -> theme-name.json [the theme's manifest file; could be XML rather than JSON, but, in my view, JSON is simpler and neater; the theme name may not begin with 'ext-' for reasons made clear further down]
      -> tpls [the root directory containing Twig templates]
          -> <tpl-group> [each <tpl-group> entry is a subdirectory named for a template group; these subdirectories can be nested]
            -> <tpl>.twig [each <tpl>.twig is a raw Twig template file where <tpl> is the name of the template, potentially within any level of nested <tpl-group> parent directories]
      -> css [the root directory containing stylesheets]
          -> <filename>.css [a raw stylesheet file containing CSS, where the characters in <filename> have the same restrictions as above]
      -> jscripts [the root directory containing Javascript files; inapplicable to the initial release of 1.9]
          -> <path-to>/<filename>.js [a raw Javascript file, where the characters in <filename> have the same restrictions as above; and <path-to> is either empty or represents any set of subdirectories]

    Later, we can also add support for precompiled CSS files.

    Each of these theme directories is placed within either:

      -> inc/themes/immutable [for the core theme and original themes]

      or

      -> inc/themes/mutable [for board themes]

    The structure for each theme under the respective mutable/immutable directory is:
      -> theme-name [the root directory for the theme, named after the theme; must match the name of the theme in its manifest file]
          -> current [the current version of the theme, with directory structure as above]
          -> <version> [an historical version of the theme, again with directory structure as above, with version stipulated by <version>; this is a string compatible with PHP's version_compare() function]

    In addition, plugins supply their resources per 1.1 in the "Existing ideas" list above in a structure identical to the theme structure described above, the only difference being that they are placed under:
      -> inc/themes/immutable/ext-<plugin-codename>
    but still contain a "current" subdirectory as well as any applicable historical "<version>" subdirectories.

    Note: if "mutable" and "immutable" are considered to be too unfamiliar and non-user-friendly, we could instead use terms something like, respectively, "editable" and "read-only".

  6. That inheritance be implemented as follows: a board theme's manifest file indicates from which board theme it inherits, or (mutually exclusively) with which original/core theme it is irrevocably associated. In the event that a file is not found in the board theme's directory structure, the inheritance chain is searched until a parent theme is found which contains the given file (the idea is to implement copy-on-write (COW), such that, unless modified, files will not be present in inheriting themes). Similarly, in editing the board theme in the ACP, the list of its files (templates or stylesheets) is generated by traversing its inheritance chain up to and including the core theme, gathering all files at all levels, and then listing them. When saving an edited file, COW again applies: if an inherited resource is modified but otherwise not present for the inheriting theme in the filesystem, then a new file is inserted into the filesystem for that inheriting theme.

  7. Given that plugins will be supplying theme directories (in inc/themes/immutable/ext-<plugin-codename>) which we do not want to be simply copied over (since we do not want to overwrite the "current" subdirectory without first archiving it in a "<version>" subdirectory), the initial 1.9 release is a good occasion on which to introduce semi-automated plugin installation: rather than FTPing an unzipped plugin directory structure directly to the board's root directory and then installing it via the ACP's Plugins page, the ZIP archive itself would be uploaded via the ACP's Plugins page. In addition, although consisting in an ordinary ZIP file, the extension for MyBB plugin archives would be changed from .zip (to something perhaps like .mbp?), so as to avoid tempting members into unzipping them and uploading them as-is, thus mistakenly overwriting the plugin's "current" theme directory.

    A fall-back for admins using servers without PHP's Zip extension would be to rename the file so that it has a .zip extension, then upload the unzipped contents to some sort of staging directory checked by the ACP's Plugins page - perhaps simply a subdirectory under the board's root named "plugin_staging".
Reply
#24
This user has been denied support. This user has been denied support.
(2021-10-23, 01:44 PM)Laird Wrote:   8. Continued, but deprecated, eval()-based template support:

    To make plugin migration easier, but only temporarily: to be removed in a later release.

eval() is evil. 1.9 is a good time as any to finally let it go, no matter how much it hurts. How much effort does it even involve to continue supporting it? I'm afraid if you actually implement it and it works well, it has to stay around forever because people would continue using it, regardless of the security risks.
Reply
#25
(2021-10-23, 01:44 PM)Laird Wrote:   13. Pre-processed CSS support:

      In addition to stylesheet resources, pre-processed CSS files are also supported.

  14. Colour scheme support:

      Made possible via 13 above; includes a colour scheme editor.

Of interest here is a possibility I thought up shortly after the source Discord conversation but forgot to bring up:

Core already supports color schemes as CSS (albeit seeing rather limited use by theme developers), so this new editor could be extended to support saving individual custom palettes as core color styles, should the theme developer or administrator want to.
Reply
#26
(2021-10-23, 05:39 PM)Nebulon Ranger Wrote:
(2021-10-23, 01:44 PM)Laird Wrote:   13. Pre-processed CSS support:

      In addition to stylesheet resources, pre-processed CSS files are also supported.

  14. Colour scheme support:

      Made possible via 13 above; includes a colour scheme editor.

Of interest here is a possibility I thought up shortly after the source Discord conversation but forgot to bring up:

Core already supports color schemes as CSS (albeit seeing rather limited use by theme developers), so this new editor could be extended to support saving individual custom palettes as core color styles, should the theme developer or administrator want to.

While these may not be implemented in 1.9.0, CSS variables would be an easy way to allow administrators to customize themes (declared variables would be editable in the ACP, and it should be easy enough to add a front-end widget for adjusting colors with live preview).

It's worth keeping an eye on supported types of the attr() CSS function (that helps making variable data available in CSS), and, with regard to pre-processors, native support for some SCSS features like CSS nesting.
devilshakerz.com/pgp (DF3A 34D9 A627 42E5 BC6A 6750 1F2F B8AA 28FF E1BC) ▪ keybase.io/devilshakerz
Reply
#27
(2021-10-23, 06:38 PM)Devilshakerz Wrote:
(2021-10-23, 05:39 PM)Nebulon Ranger Wrote:
(2021-10-23, 01:44 PM)Laird Wrote:   13. Pre-processed CSS support:

      In addition to stylesheet resources, pre-processed CSS files are also supported.

  14. Colour scheme support:

      Made possible via 13 above; includes a colour scheme editor.

Of interest here is a possibility I thought up shortly after the source Discord conversation but forgot to bring up:

Core already supports color schemes as CSS (albeit seeing rather limited use by theme developers), so this new editor could be extended to support saving individual custom palettes as core color styles, should the theme developer or administrator want to.

While these may not be implemented in 1.9.0, CSS variables would be an easy way to allow administrators to customize themes (declared variables would be editable in the ACP, and it should be easy enough to add a front-end widget for adjusting colors with live preview).

This is effectively what WoltLab Suite, the example I used in the conversation, does, in addition to allowing the administrator to override SCSS variables. I have a local install of the core, so here are a few screenshots (scaled to 640px wide so as to not murder the post container):

[Image: rH0HQtp.png]
[Image: LNwz8rG.png]
[Image: sNLPB7f.png]
Reply
#28
(2021-10-23, 01:44 PM)Laird Wrote: 2. Resource tracking:

Resources added/modified by 1.1 and 1.2 above are tracked, so that additions/modifications can be undone even when the plugins that made them are removed.

Technically, this can be done by saving { extension, extension version, serialized instructions, diff of changes for redundancy }.
This information can be understood as theme "macros", and the feature may later be extended to allow saving and (re)applying sets of pre-defined custom changes.

(2021-10-23, 01:44 PM)Laird Wrote: 9. Compatibility flagging:

The UI supports the marking of a board theme as compatible with the current version of a plugin. This suppresses the UI from otherwise displaying a need for conflict resolution.

Depending on how the plugin structure is organized, administrators with some technical experience could be able to create a custom layer of changes (modification instructions) for a given theme (but unless these are stored in a separate location, it may break best-practice concepts of not interfering with third party packages).

We could also use the "macro" approach from above, where a set of changes is associated with a plugin and its version as an alternative to provided modification instructions. This wouldn't be limited to resolving conflicts, but also making arbitrary customizations.

E.g. to change the location of a custom widget:
  1. Install the plugin
  2. Revert the plugin's changes to selected theme using advanced options (resulting in an "attention needed" status for the theme)
  3. Create a custom "macro" with desired modifications (programmatically / using the ACP / "recording" it and reviewing the end result)
  4. Mark the "macro" as resolving a particular change request (the theme system may suggest possible options, since it knows which plugin's changes are not currently applied).

    It's worth considering allowing a macro to resolve multiple change requests, which would help with manual resolution of conflicts between change requests of different plugins.

  5. Apply the new "macro" to the theme (resulting in the "attention needed" status removed)

In this case, we should consider how the "macros" would be stored (it may make sense to store serialized data in theme directories so they are available for export, and displayed on installations after being imported, even if the plugins are not available).

Quote: 1. Should templates/stylesheets be stored in the filesystem or DB?

A: Most folk seem to prefer the filesystem, and I don't think my objections to that approach are strong enough, so let's stick with filesystem templates.

This should result in third-party/exportable content (Resources + Resource properties) stored in the filesystem, and dynamic data stored in the database.
New uploaded themes would be detected in the ACP, and database records would be created to maintain relations within the database.

Note the new system would increase reliance on identifiers in the filesystem and manifests, so it would be more important to keep them unique (the Extend platform already issues unique codenames).

Quote: 2. What theme properties should the manifest file include?
3. What metadata should the manifest file include?

A: I'm not entirely sure as to the difference between theme properties and metadata, so I'll consider them together. Let's just compile this list as we go: start from scratch, and, whenever, during development, we find a need (or high desirability) for a property / item of metadata, we add it. There should at least, though, be entries for version, author, type (core/original/board), parent (if none, the core theme is assumed to be parent), and the pages associated with any stylesheet(s).

We can leave the possibility of having separate files to store theme metadata - i.e. non-extra fields of the composer.json schema, and Resource properties, so that:
  • Resource properties are not nested in the extra field
  • the Composer file - mostly related to packaging & distribution matters - is not modified each time a Resource property is changed

Whether the new theme and plugin systems each are completely compatible with Composer or not, the manifest files should use the established names and structure if there's no added cost.

Quote:A: Let's defer support for Composer and CSS pre-processors until after the initial 1.9 release.

Note the current 1.9 CSS is generated using SCSS: https://github.com/mybb/1.9-ui-framework

If MyBB handles pre-processing (which may not have been discussed specifically, but "development mode" and comparing file modification times was considered), it may make things easier for theme developers, who won't have to set up external tools that watch source files.

Quote: 8. What system could be implemented for applying theme updates?

A: The same sort of system as for applying core theme updates: the conceptual generation of a patch which MyBB attempts to auto-apply to associated board themes, and to board themes which inherit from those board themes (see below re inheritance and association), etc, with UI-assisted conflict resolution as necessary.

The database information will likely include a reference to a version of the theme (pointing to a specific inc/themes/theme-name/* version directory).
Should the system expose an option to change this version?
Should the parent theme information include a version number that, unless bumped, will cause the child theme to continue to inherit from the old version (but issue "attention needed" warnings)?
Should changes in parent themes, and in plugins' modification instructions, be propagated silently or manually (following "attention needed" warnings, with possible edit suggestions)?

Should changes to Board themes, which are not versioned, be logged (e.g. in diff format), especially given the automation involved?

Quote: 2. That re idea 1.2 in that list (permanent modifications of resources by plugins), we implement a simple function corresponding to find_replace_templatesets(), perhaps named something like find_replace_twig_tpls(), which does effectively the same job, perhaps with better indication of errors such as changes failing to apply. If/when we later implement idea 3 (fine-grained modification functions), we deprecate this function. I say "if" because I have concerns over the viability of this approach, in particular that changes cannot necessarily be reverted deterministically when *subsequent* changes have been made, although my concerns are somewhat alleviated by the observation that the existing find_replace_templatesets() approach is not immune from this criticism either.

Yes, it may take some time to arrive at a good API. The MVP, however, would be one where theme interactions are no worse than in 1.8, so we'd need functions for:
  • adding custom Resources in 1.8.x format, but attached to 1.9.x themes (PluginLibrary-like, may need modifications)
  • adding custom Resources in 1.9.x format
  • adding stylesheets and modifying properties in 1.9.x format


Quote: 4. That re inheritance and association, we stipulate that inheritance applies only to board themes: only a board them can inherit, and only from another board theme. A board theme may alternatively (one or the other must be the case), though, be irrevocably *associated* with the (immutable) original or core theme of which it is a (mutable) copy, meaning that updates to that original/core theme are auto-applied to all associated board themes (and to those board themes which inherit from those board themes, etc). The metadata for an *original* theme may, though, additionally, indicate (for purely informative purposes) that it is "based on" or "inspired by" some other original theme - an original theme which need not be installed, and changes to which do not affect the inspired original theme in the local installation, whereas original themes which are inherited from or associated with board themes *do* need to be installed, and attempting to uninstall such a theme results in an error.

Does that limitation come from expected practical problems with exporting and importing themes (which would necessitate moving packages of multiple themes at once, when they form an enforced inheritance chain)?

It may be sometimes reasonable to require parent themes to be present, even when using a downloaded theme (e.g. adjustments for right-to-left languages, or for high contrast - otherwise all three would have to be distributed with duplicated Resources).

Note 1.8 supports collecting all inherited Resources when exporting themes - a similar tool can be considered for the 1.9 system to address the problem (and when creating an archive, all parents can be simply added to it instead of merging everything).

In this scenario, do all themes continue to inherit from the Core theme (so Core changes result in "attention needed" warnings, even if an Original theme is the closest parent)?

In the latest diagram, an Original theme would inherit either from another Original theme or the Core theme, and the inheritance chain would be composed of optional, but uninterrupted series of Board and Original themes, up to the Core theme.


Quote: -> <tpl-group> [each <tpl-group> entry is a subdirectory named for a template group; these subdirectories can be nested]
-> <tpl>.twig [each <tpl>.twig is a raw Twig template file where <tpl> is the name of the template, potentially within any level of nested <tpl-group> parent directories]
-> css [the root directory containing stylesheets]
-> <filename>.css [a raw stylesheet file containing CSS, where the characters in <filename> have the same restrictions as above]

Are parent directories going to be supported for CSS files? If SCSS is supported, should these source files be placed in e.g. css/_scss (and all subdirectories, or those beginning with _, would be ignored when copying resources to publicly available locations)?


Quote: 3. What metadata should the manifest file include?

[...] type (core/original/board), [...]
Quote: Each of these theme directories is placed within either:

-> inc/themes/immutable [for the core theme and original themes]

or

-> inc/themes/mutable [for board themes]

Are separate directories only intended to hint to users which files should (not) be modified? It may be insufficient as long as we can't include a more detailed warning ("visit the ACP to modify your themes").

If merged, should the core theme be stored separately (currently inc/views), or also merged for internal clarity? What about Core theme's resources other than templates (currently not present)?

Quote: In addition, plugins supply their resources per 1.1 in the "Existing ideas" list above in a structure identical to the theme structure described above, the only difference being that they are placed under:
-> inc/themes/immutable/ext-<plugin-codename>
but still contain a "current" subdirectory as well as any applicable historical "<version>" subdirectories.

In the diagram's terms, is each active plugin's ext-* directory a set of New resources, making the collection of all of them an Inheritance base when combined with the Core theme, forming the root of the inheritance chain?

Should New resources be saved as physical files, rather than be assigned dynamically from arbitrary files shipped with the plugin (as in the PHP code examples)?
Should they be saved in the directory for themes, rather than plugins?

If they're packaged and located in the plugin directory, we can consider a structure similar to:
plugin-name/
    theme/
        current/
            templates/
            stylesheets/
            theme-changes/
                core/
                    1.9.1/
                        extend.php
                    1.9.0/
                        extend.php
            properties.json
        v1/
    manifest.json
The theme-related resources would be placed in a dedicated directory that applies the concept of version-stamping of directory names, so that New resources and Modification instructions are versioned.

If we follow the concept of separating metadata and Resource properties, the properties.json file would contain Resource properties for custom Resources (like templates/ and stylesheets/), and modifications of existing Resources' properties would be coded in extend.php files, which in a MVP can simply contain function/PluginLibrary calls known from 1.8, to eventually take advantage of the new API:
\MyBB\ExtendTheme\Stylesheet('global.css')
    ->setAttachedTo('custom_page.php');

This structure would be processed by the extensions system. Note the modification instructions are always tied to a specific theme version (and the most recent version would be used to install the modifications), and plugins can be distributed with just the most recent versions of these instructions (so overwriting plugin files would preserve instructions for previous versions).

Other arbitrary files and directories (PHP code, etc.) would be ignored.

The manifest.json could be used as an optional alternative to inc/plugins/*.php files and *_info() array manifest functions, but major plugin system changes unrelated to the theme system are probably out of scope for 1.9.


Quote: 7. Given that plugins will be supplying theme directories (in inc/themes/immutable/ext-<plugin-codename>) which we do not want to be simply copied over (since we do not want to overwrite the "current" subdirectory without first archiving it in a <version> subdirectory), the initial 1.9 release is a good occasion on which to introduce semi-automated plugin installation: rather than FTPing an unzipped plugin directory structure directly to the board's root directory and then installing it via the ACP's Plugins page, the ZIP archive itself would be uploaded via the ACP's Plugins page. In addition, although consisting in an ordinary ZIP file, the extension for MyBB plugin archives would be changed from .zip (to something perhaps like .mbt?), so as to avoid tempting members into unzipping them and uploading them as-is, thus mistakenly overwriting the plugin's "current" theme directory.

A fall-back for admins using servers without PHP's Zip extension would be to rename the file so that it has a .zip extension, then upload the unzipped contents to some sort of staging directory checked by the ACP's Plugins page - perhaps simply a subdirectory under the board's root named "plugin_staging".

Does that also apply to theme updates?

To follow the version-stamping approach, it looks like we have 3 choices:
  • Version-stamp during distribution

    Require developers to name inc/plugins/plugin-name/theme/*/ directories according to the shipped version.

    To properly preserve version history in code repositories, the names could remain static and be changed automatically by a building tool, but that would require all extension developers to set up a build process (e.g. using GitHub Actions), and end users who download extensions from repositories directly could face issues.

    The directory structure and names may be validated in an automated inspection of Extend builds.

  • Version-stamp on detection

    When visiting ACP's extension management pages, or upon extension activation, the system would scan extension directories for placeholder names, and rename them using versions in manifest files.

    Attempting this, and dealing with possible errors (related to filesystem permissions) on activation may be the most intuitive approach (the administrator is already focused on a specific extension, and solving the problem is relevant to their activity).

    This may cause unexpected side-effects in certain cases (e.g. some editors may close "lost" open files and discard edit history when an extension is "stamped" by the system).

  • Version-stamp on archiving

    When processing extension updates, rename directories with placeholder names using versions in manifest files, effectively marking them as archived versions.

    Requires UX-related architecture changes to avoid overwriting files when uploading updated extensions.



Generally, regarding uploading archives, IMO it may be best to only use HTTP uploads in the ACP for convenience purposes, and default to direct uploads. This approach would be more automation-friendly - multiple packages (MyBB, themes, and plugins) could be combined by merging the file structure (including, but not limited to, Composer use - without additional processing of extension packages).

Also worth noting that in 1.8, new themes had to be uploaded in 2 places (.xml files with metadata, some Resources, and their properties in the ACP; remaining files through FTP), while in 1.9, according to where we're headed, it would be sufficient to simply upload all files.



Meta:
  • Note that any names (like current/ and theme-changes/) are only suggestions.

  • The chosen schemes and automated name changes should be reviewed to make sure they're intuitive and not unexpected.

  • We can create lists with high-level descriptions of how the system operates when installing/uninstalling/upgrading extensions under various conditions (state, versions, and position in the inheritance chain).

  • Quote:eventually we can maintain a Markdown Blog draft as a main architecture outline - may help whether it gets published or not
devilshakerz.com/pgp (DF3A 34D9 A627 42E5 BC6A 6750 1F2F B8AA 28FF E1BC) ▪ keybase.io/devilshakerz
Reply
#29
(2021-10-23, 03:25 PM)frostschutz Wrote: eval() is evil. 1.9 is a good time as any to finally let it go, no matter how much it hurts. How much effort does it even involve to continue supporting it? I'm afraid if you actually implement it and it works well, it has to stay around forever because people would continue using it, regardless of the security risks.

I get where you're coming from.

There's a pro and a con to temporarily maintaining support for 1.8's eval()-based templates. You've expressed the con well: eval() is evil and has security risks. The pro is that the initial release of 1.9 doesn't immediately and catastrophically break all (most?) existing MyBB plugins - it gives plugin developers time to update them for 1.9 and allows admins to continue using (most?) 1.8 plugins while they do that.

There's no effort involved in continuing to support it as it was never removed from 1.9, and there are even a few remaining usages of it in the current 1.9 code (which of course need to be removed before release).

I'm not saying you've made the wrong call though. Maybe we should just suck up an initial catastrophic plugin breakage to force plugin developers to update their plugins.
Reply
#30
I think it might be worth considering removing usage of eval(), if not by the security concerns alone, because keeping it for backward compatibility reasons will ultimately result in its continue use. We might end up only delaying the inevitable.

The theme and tempalte systems are already suffering massive changes, it is likely there won't be a better chance in a long time.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)