The 1.9 theme and template system

[Laird]

@Devilshakerz,

I like your macro idea. As you say, the main challenge is developing a reliable instruction and API set. I wonder whether there's an existing PHP library for this that has been well-tested. I'd kind of expect that there is. A little googling didn't turn one up, but it's hard to know what the right keywords are.

The macro-based functionality you describe seems pretty useful, although - I think we agree - beyond the scope of the initial release.

Re storage of theme-related data in the filesystem versus the DB, you seem to envisage a hybrid approach, which I hadn't expected. I had been under the impression that everything theme-related would be stored in the filesystem alone, and I was comfortable with that. Is it really necessary/preferable to store anything theme-related in the DB?

Re storage of macros, perhaps they could simply be stored under inc/themes/macros.

Agreed that increased reliance on identifiers in the filesystem and manifests makes it more important to keep them unique.

Re composer.json: oh, I hadn't realised that that's what you meant by a theme's manifest file. In that case, yes, I do think it's worth having a separate theme JSON file with its own custom schema including theme properties/metadata. This is where I had understood all theme-related data would be stored, eliminating the need for storage in the DB.

Re the current 1.9 CSS being generated using SCSS: thanks, I hadn't been aware of that or had forgotten. I imagine that the answer is "No, it can be deferred", but just to confirm: do you think the MVP for our initial 1.9 release requires MyBB to handle pre-processing, or can we defer that to later?

8. What system could be implemented for applying theme updates?

A: The same sort of system as for applying core theme updates: the conceptual generation of a patch which MyBB attempts to auto-apply to associated board themes, and to board themes which inherit from those board themes (see below re inheritance and association), etc, with UI-assisted conflict resolution as necessary.

The database information will likely include a reference to a version of the theme (pointing to a specific inc/themes/theme-name/* version directory).
Should the system expose an option to change this version?

Probably, but it should probably be accompanied by a strong "Only if you know what you are doing" warning.

Should the parent theme information include a version number that, unless bumped, will cause the child theme to continue to inherit from the old version (but issue "attention needed" warnings)?

Probably, yes.

Should changes in parent themes, and in plugins' modification instructions, be propagated silently or manually (following "attention needed" warnings, with possible edit suggestions)?

Upon upload of a new version of a theme (including the core them during an upgrade) or resource-generating plugin, the admin should probably be given, during the upgrade/install procedure, the choice of automatic (except where manual conflict resolution is required) or manual update of board themes per modification instructions. Presumably, admins would generally choose the automatic option.

Should changes to Board themes, which are not versioned, be logged (e.g. in diff format), especially given the automation involved?

Do you mean so that they can be rolled back if necessary? This might be a good idea, but maybe deferred to a later release?

The MVP, however, would be one where theme interactions are no worse than in 1.8, so we'd need functions for:

• adding custom Resources in 1.8.x format, but attached to 1.9.x themes (PluginLibrary-like, may need modifications)
  
• adding custom Resources in 1.9.x format
  
• adding stylesheets and modifying properties in 1.9.x format
  

I'm not quite sure what you mean by Resources in 1.8.x format, but maybe, per frostschultz and Omar's suggestions, we should drop 1.8 support from the very start? Make a clean break?

4. That re inheritance and association, we stipulate that inheritance applies only to board themes: only a board them can inherit, and only from another board theme. A board theme may alternatively (one or the other must be the case), though, be irrevocably *associated* with the (immutable) original or core theme of which it is a (mutable) copy, meaning that updates to that original/core theme are auto-applied to all associated board themes (and to those board themes which inherit from those board themes, etc). The metadata for an *original* theme may, though, additionally, indicate (for purely informative purposes) that it is "based on" or "inspired by" some other original theme - an original theme which need not be installed, and changes to which do not affect the inspired original theme in the local installation, whereas original themes which are inherited from or associated with board themes *do* need to be installed, and attempting to uninstall such a theme results in an error.

Does that limitation come from expected practical problems with exporting and importing themes (which would necessitate moving packages of multiple themes at once, when they form an enforced inheritance chain)?

That might be a consideration too, yes, but I was thinking more of practical problems given the new concept of a board theme which is mutable whereas original themes are immutable, except when being updated with a developer-supplied new version. When such an update is applied, and multiple original themes form an inheritance chain, tricky questions like this arise: what if the original theme is higher up on the chain, and a change to it made by the developer conflicts with a child original theme (which isn't simultaneously being updated) - does that change propagate through to (a) the child original theme (here, the answer seems to be: "No, because original themes are immutable"), and (b) the (mutable) board theme(s) at the bottom of the inheritance chain (here, the answer seems to be: "Only if, after the system flags a conflict that needs to be resolved by the admin, the admin resolves that conflict")?

And what if the admin resolves the conflict in favour of the new version of the parent original theme, such that the change gets incorporated into the board theme at the bottom of the chain, but, upon later updating the child theme, a change is made to the same area of the same resource - does *that* change get propagated to the board theme, or does another conflict get raised with the admin?

It just seems to me that allowing for these sort of inheritance chains for original themes leads to unnecessary complications in respect of these sort of questions. It seems to me that more sensible is for original theme developers to manually incorporate any changes from other original themes on which their theme is based, and then to release the complete new version of the theme to end-users, which avoids the need for admins to resolve such potential conflicts.

Board admins can then tinker with spawning child board themes of board themes to any degree of inheritance without any of these sort of conflicts arising.

In this scenario, do all themes continue to inherit from the Core theme (so Core changes result in "attention needed" warnings, even if an Original theme is the closest parent)?

My idea is: yes. Changes to the core theme always propagate to board themes, bypassing original themes.

In the latest diagram, an Original theme would inherit either from another Original theme or the Core theme, and the inheritance chain would be composed of optional, but uninterrupted series of Board and Original themes, up to the Core theme.

Yes, I realised in writing up my summary that I don't think that this part of the diagram makes total sense: original themes are immutable, so core changes *can't* propagate to them (except via a new version supplied by the theme developer) - but core changes *should* propagate to board themes, which *are* mutable: without core changes propagating to board themes, board functionality dependent on those changes might fail.

I need to clarify though that when I talk about changes propagating, I'm referring to the case in which a copy-on-write resource *has* been written, i.e., the resource exists in the child, and is not being pulled in from the parent.

This clarification also applies to my above considerations re original themes inheriting from other original themes.

Are parent directories going to be supported for CSS files?

Sure, that was an omission on my part.

If SCSS is supported, should these source files be placed in e.g. css/_scss (and all subdirectories, or those beginning with _, would be ignored when copying resources to publicly available locations)?

I'm not familiar with CSS pre-compilers, so I defer to you on the first part of that question. On the parenthetical part, yes, that seems sensible.

Each of these theme directories is placed within either:

-> inc/themes/immutable [for the core theme and original themes]

or

-> inc/themes/mutable [for board themes]

Are separate directories only intended to hint to users which files should (not) be modified?

Yes, exactly.

It may be insufficient as long as we can't include a more detailed warning ("visit the ACP to modify your themes").

What would you suggest instead/additionally then? Setting file/directory permissions to disallow writing?

If merged, should the core theme be stored separately (currently inc/views), or also merged for internal clarity?

My idea was to store the core theme just like an original theme, under inc/themes/immutable/core.

What about Core theme's resources other than templates (currently not present)?

Likewise: stored in the same subdirectories under inc/themes/immutable/core as for any original theme.

In addition, plugins supply their resources per 1.1 in the "Existing ideas" list above in a structure identical to the theme structure described above, the only difference being that they are placed under:
-> inc/themes/immutable/ext-<plugin-codename>
but still contain a "current" subdirectory as well as any applicable historical "<version>" subdirectories.

In the diagram's terms, is each active plugin's ext-* directory a set of New resources, making the collection of all of them an Inheritance base when combined with the Core theme, forming the root of the inheritance chain?

Yes, exactly.

Should New resources be saved as physical files

Yes. Just as for themes.

Should they be saved in the directory for themes, rather than plugins?

I'm comfortable with acceding to your apparent preference for the plugins directory rather than the themes directory.

Am also comfortable with your idea of extend.php files.

This structure would be processed by the extensions system. Note the modification instructions are always tied to a specific theme version (and the most recent version would be used to install the modifications), and plugins can be distributed with just the most recent versions of these instructions (so overwriting plugin files would preserve instructions for previous versions).

I'm not quite sure what you mean by this (especially the being tied to a specific version part) and how you see it working - can you please elaborate?

The manifest.json could be used as an optional alternative to inc/plugins/*.php files and *_info() array manifest functions, but major plugin system changes unrelated to the theme system are probably out of scope for 1.9.

Yes, there are lots of exciting possibilities for plugin manifest files, but, I do think they're out of scope for at least the first release of 1.9.

7. Given that plugins will be supplying theme directories (in inc/themes/immutable/ext-<plugin-codename>) which we do not want to be simply copied over (since we do not want to overwrite the "current" subdirectory without first archiving it in a <version> subdirectory), the initial 1.9 release is a good occasion on which to introduce semi-automated plugin installation: rather than FTPing an unzipped plugin directory structure directly to the board's root directory and then installing it via the ACP's Plugins page, the ZIP archive itself would be uploaded via the ACP's Plugins page. In addition, although consisting in an ordinary ZIP file, the extension for MyBB plugin archives would be changed from .zip (to something perhaps like .mbt?), so as to avoid tempting members into unzipping them and uploading them as-is, thus mistakenly overwriting the plugin's "current" theme directory.

A fall-back for admins using servers without PHP's Zip extension would be to rename the file so that it has a .zip extension, then upload the unzipped contents to some sort of staging directory checked by the ACP's Plugins page - perhaps simply a subdirectory under the board's root named "plugin_staging".

Does that also apply to theme updates?

Yes.

Re your preference for uploading unzipped and not zipped files, that's OK with me - that idea of mine was only to help with versioning. My understanding was that the current version of a theme or plugin's theme-related files would always reside in the "current" directory. Its directory name would not be explicitly versioned, though its version could be pulled out of its manifest/properties file. Updating the theme or plugin's theme-related files then would entail moving the "current" directory to the "<version>" directory where <version> is pulled out of the manifest/properties file, and replacing it with the new "current" directory.

The sort of possibility I suggested above might work then: under the MyBB root is a "staging" directory. An unzipped plugin/theme is uploaded to that directory, and the ACP picks up on the new/updated plugin/theme staging directory, and, if it is an update, automatically moves the "current" directory to the "<version>" directory for the existing install location for that plugin/theme, and then moves the plugin/theme files to their appropriate locations, overwriting any existing files.

eventually we can maintain a Markdown Blog draft as a main architecture outline - may help whether it gets published or not

Good idea.

[Devilshakerz]

The macro-based functionality you describe seems pretty useful, although - I think we agree - beyond the scope of the initial release.

Yes, which would mean the initial system wouldn't be capable of tracking plugins' changes in themes (as is the case in 1.8), if that's the way the related information would be stored.

Re storage of theme-related data in the filesystem versus the DB, you seem to envisage a hybrid approach, which I hadn't expected. I had been under the impression that everything theme-related would be stored in the filesystem alone, and I was comfortable with that. Is it really necessary/preferable to store anything theme-related in the DB?]

While it may be possible to have all theme references (e.g. mybb_users.style) correlated with a text identifier (pointing to a theme directory), it would be practical to keep and extend relationships within the database, and to keep instance-specific references (like group IDs for theme permissions) separated from data included in the theme directory.
Currently, MyBB's database table definitions don't explicitly indicate foreign keys and constraints, but they may in the future (related: https://github.com/mybb/mybb/issues/4388).

If we reuse the themes table, the potential 1.9 changes may be described as:

CREATE TABLE mybb_themes (
  tid smallint unsigned NOT NULL auto_increment, -- activated theme ID - keep
  name varchar(100) NOT NULL default '',         -- display name - potentially keep to allow customizing
  pid smallint unsigned NOT NULL default '0',    -- parent ID: may be discarded if parent themes are referenced by their filesystem identifier in the used theme's manifest file, and if parent themes don't need to be activated (nor to have a record in this database table)
  def tinyint(1) NOT NULL default '0',           -- global default status - keep (unless moved elsewhere)
  properties text NOT NULL,                      -- saved in the properties file - discard/deprecate
  stylesheets text NOT NULL,                     -- saved in the properties file - discard/deprecate
  allowedgroups text NOT NULL,                   -- instance-specific references to user groups - keep (may later be normalized as a pivot table)
  PRIMARY KEY (tid)
)

Notably, in 1.8, themes (which contain stylesheets and Resource properties) can point to template sets (containing templates), and are considered separate entities - in the 1.9 concept, these would be merged. Instead, we would have these entities:

• a package in the filesystem that includes all Resources, their properties, and metadata
  
• theme instances - runtime information about an activated theme, most of which may be stored in the database
  

While 1.9's equivalent of 1.8's Colors (presents with pre-defined values) should be included in the package, the choice of a specific present and data configured through a potential variable system would likely belong to runtime information.

We could allow creating multiple theme instances from a single package, which would enable administrators to keep multiple variations created using presents and variables (in 1.8, you would create child themes with different Color presents).

BTW: The 1.9 "Default" theme would probably be a Board theme that inherits directly from the Core theme.

Re the current 1.9 CSS being generated using SCSS: thanks, I hadn't been aware of that or had forgotten. I imagine that the answer is "No, it can be deferred", but just to confirm: do you think the MVP for our initial 1.9 release requires MyBB to handle pre-processing, or can we defer that to later?

It may be possible to require core and theme developers to use external tools, but I wouldn't expect significant overhead, especially if static files are already expected to be handled in some way (like copying to public locations).

Should changes to Board themes, which are not versioned, be logged (e.g. in diff format), especially given the automation involved?

Do you mean so that they can be rolled back if necessary? This might be a good idea, but maybe deferred to a later release?

Yes; agreed.

This would provide redundancy (at least in some cases) to macros.

The MVP, however, would be one where theme interactions are no worse than in 1.8, so we'd need functions for:

• adding custom Resources in 1.8.x format, but attached to 1.9.x themes (PluginLibrary-like, may need modifications)
  
• adding custom Resources in 1.9.x format
  
• adding stylesheets and modifying properties in 1.9.x format
  

I'm not quite sure what you mean by Resources in 1.8.x format, [...]

Templates intended for eval() and stylesheets (which may not require special attention, but plugins will attempt to insert them into database tables with 1.8 structure).

[...] but maybe, per frostschultz and Omar's suggestions, we should drop 1.8 support from the very start? Make a clean break?

It's possible (and was the 1.9 starting point), but IMO considering the health of the extensions ecosystem, it's worth supporting the old way if it doesn't overly complicate things. In some sense, it would follow the idea of refactoring in that we don't break too many things at once.

In this scenario, do all themes continue to inherit from the Core theme (so Core changes result in "attention needed" warnings, even if an Original theme is the closest parent)?

My idea is: yes. Changes to the core theme always propagate to board themes, bypassing original themes.

Wouldn't that effectively result in scenarios intended to be avoided by limiting possible inheritance (e.g. what when an Original theme is updated to account for Core theme changes, and these changes need to be propagated to Board themes, but that was already done when applying them directly from Core theme, bypassing Original theme as suggested)?

And what if the admin resolves the conflict in favour of the new version of the parent original theme, such that the change gets incorporated into the board theme at the bottom of the chain, but, upon later updating the child theme, a change is made to the same area of the same resource - does *that* change get propagated to the board theme, or does another conflict get raised with the admin?

In the latest diagram, an Original theme would inherit either from another Original theme or the Core theme, and the inheritance chain would be composed of optional, but uninterrupted series of Board and Original themes, up to the Core theme.

Yes, I realised in writing up my summary that I don't think that this part of the diagram makes total sense: original themes are immutable, so core changes *can't* propagate to them (except via a new version supplied by the theme developer) - but core changes *should* propagate to board themes, which *are* mutable: without core changes propagating to board themes, board functionality dependent on those changes might fail.

This is an accepted cost in the described scheme (at the benefit of maintaining correct information about packages, which would be the foundation of the theme system). Administrators who take on applying upstream changes manually may understand the need of reviewing potential conflicts that appear when the more immediate parents are updated.

Themes could contain information about all parent themes in the chain (in 1.8, the MyBB version is used, making it possible to apply changes that come from the core template set, but not intermediate parents). The system wouldn't need to particularly "understand" the relationships of versioning across the Core/Original/Board boundaries to issue "attention needed" warnings whenever a newer version of one of the parents is available. When a "bypass" change is applied, the source parent's version would be bumped in the target theme, bypassing the version specified in the more immediate parent.

We can also make it easier to replace a parent Original theme with an editable Board theme, which would belong to "advanced" options dedicated to theme developers (conceptually, it might simply involve copying the theme directory and changing theme type).

It may be insufficient as long as we can't include a more detailed warning ("visit the ACP to modify your themes").

What would you suggest instead/additionally then? Setting file/directory permissions to disallow writing?

I'd probably keep them in the same place at the cost of potential user errors (which shouldn't be catastrophic, but result in avoidable maintenance overhead when updating).

This structure would be processed by the extensions system. Note the modification instructions are always tied to a specific theme version (and the most recent version would be used to install the modifications), and plugins can be distributed with just the most recent versions of these instructions (so overwriting plugin files would preserve instructions for previous versions).

I'm not quite sure what you mean by this (especially the being tied to a specific version part) and how you see it working - can you please elaborate?

This structure would be equivalent to checking the version of the theme a plugin intends to modify in order to use the correct values with find_replace_templatesets() in 1.8. Consider:

1. Original theme version v1 exists, and a Board theme inherits from it, with a plugin installed
   
2. New theme version v2 is uploaded
   
3. New plugin version is uploaded, which makes it compatible with theme version v2
   
4. The Board theme's parent is switched to new version v2 (possibly resulting in warnings/conflicts as expected)
   
5. The system reverts/discards the plugin's v1 edits applied in the Board theme
   
6. The system applies the plugin's changes appropriate for the v2 theme version in the Board theme
   

The sort of possibility I suggested above might work then: under the MyBB root is a "staging" directory. An unzipped plugin/theme is uploaded to that directory, and the ACP picks up on the new/updated plugin/theme staging directory, and, if it is an update, automatically moves the "current" directory to the "<version>" directory for the existing install location for that plugin/theme, and then moves the plugin/theme files to their appropriate locations, overwriting any existing files.

Is that approach better than Version-stamp during distribution and Version-stamp on detection? Note that we could use these two simultaneously (if supporting stamping on detection, stamping on distribution should come at no cost).

[Laird]

The macro-based functionality you describe seems pretty useful, although - I think we agree - beyond the scope of the initial release.

Yes, which would mean the initial system wouldn't be capable of tracking plugins' changes in themes (as is the case in 1.8), if that's the way the related information would be stored.

Maybe, if we can involve multiple developers, we could develop the macro system in parallel with the rest of the system, and include it in the initial release - it would be a "nice to have if we can develop it in time but not necessary" component.

  name varchar(100) NOT NULL default '',        -- display name - potentially keep to allow customizing

We'd also want to add something like codename too, so as to be able to identify the theme's directory in the filesystem, right?

  pid smallint unsigned NOT NULL default '0',    -- parent ID: may be discarded if parent themes are referenced by their filesystem identifier in the used theme's manifest file, and if parent themes don't need to be activated (nor to have a record in this database table)

Agreed: can discard.

  def tinyint(1) NOT NULL default '0',          -- global default status - keep (unless moved elsewhere)

My preference is to move it elsewhere, because of redundancy and the hypothetical possibility of data conflicts as-is: multiple themes having a def value of one.

Notably, in 1.8, themes (which contain stylesheets and Resource properties) can point to template sets (containing templates), and are considered separate entities - in the 1.9 concept, these would be merged. Instead, we would have these entities:

• a package in the filesystem that includes all Resources, their properties, and metadata
  
• theme instances - runtime information about an activated theme, most of which may be stored in the database
  

While 1.9's equivalent of 1.8's Colors (presents with pre-defined values) should be included in the package, the choice of a specific present and data configured through a potential variable system would likely belong to runtime information.

We could allow creating multiple theme instances from a single package, which would enable administrators to keep multiple variations created using presents and variables (in 1.8, you would create child themes with different Color presents).

BTW: The 1.9 "Default" theme would probably be a Board theme that inherits directly from the Core theme.

That all sounds good to me.

Re the current 1.9 CSS being generated using SCSS: thanks, I hadn't been aware of that or had forgotten. I imagine that the answer is "No, it can be deferred", but just to confirm: do you think the MVP for our initial 1.9 release requires MyBB to handle pre-processing, or can we defer that to later?

It may be possible to require core and theme developers to use external tools, but I wouldn't expect significant overhead, especially if static files are already expected to be handled in some way (like copying to public locations).

OK, so you'd like automated pre-processing by MyBB to be in the initial release of 1.9.

[...] but maybe, per frostschultz and Omar's suggestions, we should drop 1.8 support from the very start? Make a clean break?

It's possible (and was the 1.9 starting point), but IMO considering the health of the extensions ecosystem, it's worth supporting the old way if it doesn't overly complicate things. In some sense, it would follow the idea of refactoring in that we don't break too many things at once.

There seem to be different views on this then - maybe it needs more discussion from more stakeholders.

In this scenario, do all themes continue to inherit from the Core theme (so Core changes result in "attention needed" warnings, even if an Original theme is the closest parent)?

My idea is: yes. Changes to the core theme always propagate to board themes, bypassing original themes.

Wouldn't that effectively result in scenarios intended to be avoided by limiting possible inheritance (e.g. what when an Original theme is updated to account for Core theme changes, and these changes need to be propagated to Board themes, but that was already done when applying them directly from Core theme, bypassing Original theme as suggested)?

A fair question. It seems then that my criticism of multiple inheritance of original themes doesn't hold up given that the same criticism can be made of core theme "bypasses".

In the latest diagram, an Original theme would inherit either from another Original theme or the Core theme, and the inheritance chain would be composed of optional, but uninterrupted series of Board and Original themes, up to the Core theme.

Yes, I realised in writing up my summary that I don't think that this part of the diagram makes total sense: original themes are immutable, so core changes *can't* propagate to them (except via a new version supplied by the theme developer) - but core changes *should* propagate to board themes, which *are* mutable: without core changes propagating to board themes, board functionality dependent on those changes might fail.

This is an accepted cost in the described scheme (at the benefit of maintaining correct information about packages, which would be the foundation of the theme system). Administrators who take on applying upstream changes manually may understand the need of reviewing potential conflicts that appear when the more immediate parents are updated.

Themes could contain information about all parent themes in the chain (in 1.8, the MyBB version is used, making it possible to apply changes that come from the core template set, but not intermediate parents). The system wouldn't need to particularly "understand" the relationships of versioning across the Core/Original/Board boundaries to issue "attention needed" warnings whenever a newer version of one of the parents is available. When a "bypass" change is applied, the source parent's version would be bumped in the target theme, bypassing the version specified in the more immediate parent.

We can also make it easier to replace a parent Original theme with an editable Board theme, which would belong to "advanced" options dedicated to theme developers (conceptually, it might simply involve copying the theme directory and changing theme type).

OK. That all seems clear and workable.

I'd probably keep them in the same place at the cost of potential user errors (which shouldn't be catastrophic, but result in avoidable maintenance overhead when updating).

OK.

This structure would be processed by the extensions system. Note the modification instructions are always tied to a specific theme version (and the most recent version would be used to install the modifications), and plugins can be distributed with just the most recent versions of these instructions (so overwriting plugin files would preserve instructions for previous versions).

I'm not quite sure what you mean by this (especially the being tied to a specific version part) and how you see it working - can you please elaborate?

This structure would be equivalent to checking the version of the theme a plugin intends to modify in order to use the correct values with find_replace_templatesets() in 1.8. Consider:

1. Original theme version v1 exists, and a Board theme inherits from it, with a plugin installed
   
2. New theme version v2 is uploaded
   
3. New plugin version is uploaded, which makes it compatible with theme version v2
   
4. The Board theme's parent is switched to new version v2 (possibly resulting in warnings/conflicts as expected)
   
5. The system reverts/discards the plugin's v1 edits applied in the Board theme
   
6. The system applies the plugin's changes appropriate for the v2 theme version in the Board theme
   

That makes sense to me. A question I have though is what happens if an original theme is installed of which the plugin is unaware (i.e., fails to stipulate version compatibility with)? The answer, I assume, is something like: a plugin must/should at least stipulate its compatibility with versions of the core theme, so that "bypass" changes can be made to board themes of any original themes of which it is unaware. Is that in the ballpark?

The sort of possibility I suggested above might work then: under the MyBB root is a "staging" directory. An unzipped plugin/theme is uploaded to that directory, and the ACP picks up on the new/updated plugin/theme staging directory, and, if it is an update, automatically moves the "current" directory to the "<version>" directory for the existing install location for that plugin/theme, and then moves the plugin/theme files to their appropriate locations, overwriting any existing files.

Is that approach better than Version-stamp during distribution and Version-stamp on detection? Note that we could use these two simultaneously (if supporting stamping on detection, stamping on distribution should come at no cost).

I think it's better than Version-stamp during distribution in the sense that it doesn't require developers to use build tools or otherwise remember to manually update directory names prior to shipping.

I also think it's better than Version-stamp on detection in the sense that placeholder names also require developers to make changes prior to shipping: they couldn't develop the plugin while using the placeholder name, so, again, they would have to use a build tool to change their "current" directory to whichever placeholder name is chosen, or to otherwise manually make this change prior to shipping.

The main drawback that I can see to it is that admins - especially those migrating from 1.8 - might not realise that they need to upload plugins to the new "staging" directory rather than to the board root as is typically the case for 1.8 plugins.

Here's another possibility - a variation on the "on detection" approach: the "current" directory is archived to its version directory (as pulled out of its manifest file) immediately upon installation (during detection). This means it is fine if admins overwrite that directory on upgrade, because it is already archived. The main drawback to this approach is redundancy in the filesystem.

[Devilshakerz]

  name varchar(100) NOT NULL default '',        -- display name - potentially keep to allow customizing

We'd also want to add something like codename too, so as to be able to identify the theme's directory in the filesystem, right?

Yes.

A question I have though is what happens if an original theme is installed of which the plugin is unaware (i.e., fails to stipulate version compatibility with)? The answer, I assume, is something like: a plugin must/should at least stipulate its compatibility with versions of the core theme, so that "bypass" changes can be made to board themes of any original themes of which it is unaware. Is that in the ballpark?

Yes.

Regarding versions, the first implementation can select them in a simple manner (e.g. exact match, otherwise most recent).
A better one would pick best candidates according to semantic versioning (so changes for v1.2 would be used instead of v2 when theme v1.2.3 is installed), and administrators could be presented a choice with a preview of changes & conflict status (less likely candidates can have less conflicts, but that may be coincidental).
When done automatically, non-exact version matches could be reported post-installation (and this information would be attached to the macro).

The sort of possibility I suggested above might work then: under the MyBB root is a "staging" directory. An unzipped plugin/theme is uploaded to that directory, and the ACP picks up on the new/updated plugin/theme staging directory, and, if it is an update, automatically moves the "current" directory to the "<version>" directory for the existing install location for that plugin/theme, and then moves the plugin/theme files to their appropriate locations, overwriting any existing files.

Is that approach better than Version-stamp during distribution and Version-stamp on detection? Note that we could use these two simultaneously (if supporting stamping on detection, stamping on distribution should come at no cost).

I think it's better than Version-stamp during distribution in the sense that it doesn't require developers to use build tools or otherwise remember to manually update directory names prior to shipping.

This may be addressed by allowing - but not requiring - stamping on distribution (so in packages on Extend). Then it would be a question of whether this deviation from conformance and standardization of packages is acceptable.

As long as the version in the manifest is correct, it may be acceptable to have themes (with inc/themes/[stampable]/ structure) and plugins (with inc/plugins/theme/[stampable]/ structure) potentially distributed with directories already renamed to specific versions - then, the purpose of the system's behavior would be to normalize placeholders to versions, if not done already.

The potential validation by the Extend platform would be to make sure the manifest & directory versions match (unless the directories use placeholder names), and the ACP could display a warning if the latest stamped version and the manifest version are different.

I also think it's better than Version-stamp on detection in the sense that placeholder names also require developers to make changes prior to shipping: they couldn't develop the plugin while using the placeholder name, so, again, they would have to use a build tool to change their "current" directory to whichever placeholder name is chosen, or to otherwise manually make this change prior to shipping.

This downside also applies to copying development files to code repositories, where static names may be preferred.

The presented choices only accounted for how static names are treated forward, but not the other way around - from live development copies to repositories (and packages, if placeholder names are required) - which seems to be the remaining issue.

Here's another possibility - a variation on the "on detection" approach: the "current" directory is archived to its version directory (as pulled out of its manifest file) immediately upon installation (during detection). This means it is fine if admins overwrite that directory on upgrade, because it is already archived. The main drawback to this approach is redundancy in the filesystem.

This may be acceptable, since just the theme-related directories would be copied (and we expect numerous archives like this to be created during upgrades over time).

Would the current/ or the "archived" directory be used by the system for fetching Resources and including files?

[Laird]

Regarding versions, the first implementation can select them in a simple manner (e.g. exact match, otherwise most recent).
A better one would pick best candidates according to semantic versioning (so changes for v1.2 would be used instead of v2 when theme v1.2.3 is installed), and administrators could be presented a choice with a preview of changes & conflict status (less likely candidates can have less conflicts, but that may be coincidental).
When done automatically, non-exact version matches could be reported post-installation (and this information would be attached to the macro).

Sounds good.

The sort of possibility I suggested above might work then: under the MyBB root is a "staging" directory. An unzipped plugin/theme is uploaded to that directory, and the ACP picks up on the new/updated plugin/theme staging directory, and, if it is an update, automatically moves the "current" directory to the "<version>" directory for the existing install location for that plugin/theme, and then moves the plugin/theme files to their appropriate locations, overwriting any existing files.

Is that approach better than Version-stamp during distribution and Version-stamp on detection? Note that we could use these two simultaneously (if supporting stamping on detection, stamping on distribution should come at no cost).

I think it's better than Version-stamp during distribution in the sense that it doesn't require developers to use build tools or otherwise remember to manually update directory names prior to shipping.

This may be addressed by allowing - but not requiring - stamping on distribution (so in packages on Extend). Then it would be a question of whether this deviation from conformance and standardization of packages is acceptable.

As long as the version in the manifest is correct, it may be acceptable to have themes (with inc/themes/[stampable]/ structure) and plugins (with inc/plugins/theme/[stampable]/ structure) potentially distributed with directories already renamed to specific versions - then, the purpose of the system's behavior would be to normalize placeholders to versions, if not done already.

The potential validation by the Extend platform would be to make sure the manifest & directory versions match (unless the directories use placeholder names), and the ACP could display a warning if the latest stamped version and the manifest version are different.

I'm probably missing something, but I don't see how that addresses the problem. The problem as I see it is that plugin/theme developers are going to be working with, and committing to their repositories with, a "current" directory, and that the Version-stamp during distribution approach requires them to rename that directory on distribution. I don't understand how your suggestions eliminate the need for the "current" directory to be in some way renamed on distribution.

I also think it's better than Version-stamp on detection in the sense that placeholder names also require developers to make changes prior to shipping: they couldn't develop the plugin while using the placeholder name, so, again, they would have to use a build tool to change their "current" directory to whichever placeholder name is chosen, or to otherwise manually make this change prior to shipping.

This downside also applies to copying development files to code repositories, where static names may be preferred.

True.

Here's another possibility - a variation on the "on detection" approach: the "current" directory is archived to its version directory (as pulled out of its manifest file) immediately upon installation (during detection). This means it is fine if admins overwrite that directory on upgrade, because it is already archived. The main drawback to this approach is redundancy in the filesystem.

This may be acceptable, since just the theme-related directories would be copied (and we expect numerous archives like this to be created during upgrades over time).

Would the current/ or the "archived" directory be used by the system for fetching Resources and including files?

The current/ directory. It's the simplest and most direct approach, avoiding the need for various checks/determinations.

[HLFadmin]

Gentlemen,

Thank you for this public discussion, it is very interesting to me for two reasons.

It is an opportunity to see what goes on when considering a major feature change in software. It is also a tiny glimpse into what 1.9 and beyond holds in store. While I'm certainly not developer-grade skilled, I know enough to tinker, perhaps to break, maybe to fix again, and in the process, learn more about this evolving system.

Please continue. Again, thanks.

[Laird]

I sure hope you do tinker, break, and fix as 1.9 starts to shape up, HLFadmin. We'll need as many folk to test and refine the system as we can get.

[Devilshakerz]

I'm probably missing something, but I don't see how that addresses the problem. The problem as I see it is that plugin/theme developers are going to be working with, and committing to their repositories with, a "current" directory, and that the Version-stamp during distribution approach requires them to rename that directory on distribution. I don't understand how your suggestions eliminate the need for the "current" directory to be in some way renamed on distribution.

We can be more lenient and accept packages with both placeholders (current/) and stamped versions, so that if developers:

• copy files from a live forum, where directories are already stamped
  
• make a package from a repository where a placeholder (current/) is used, without making any changes
  

the resulting packages could still work: the system would only need to look for leftover placeholder names (current/) and rename/"archive" them before use. If version numbers were already applied, it would do nothing.

This assumes no inherent need for current/ directories for production use.

Would the current/ or the "archived" directory be used by the system for fetching Resources and including files?

The current/ directory. It's the simplest and most direct approach, avoiding the need for various checks/determinations.

It may lead to - potentially unexpected - implementation difficulties when specific versions have to resolve to "live" directories (and if the deviation described above also applies to this approach, the system would need to accept the latest version directory in the absence of current/).

Generally, with the described redundancy, all data that may be needed during production use could be resolved using standardized paths/"archives" (with stamped versions, where current/ is considered equivalent to a specific v*/ directory).
When developing extensions, though, only the current/ content would be modified - so e.g. when a parent theme's Resource is edited there, and its "archived" version becomes stale, it would only propagate down when queries for the parent's Resources are properly resolved to inc/themes/[parent]/current.

This could be simple enough (e.g. the current/ directory would have priority in arrays of template directories for Twig), but unless the system is well-aware of the filesystem state (i.e. maintains an accurate cache of which extensions have a current/ directory, and which versions they override), it may be necessary to additionally read manifest files to make sure the override directories are intended for the queried versions to avoid unintended behavior.
Resources and metadata for any version of any extension may be queried at any time (especially when the system is expected to track and analyze extension interactions - vertical [inheritance] and horizontal [themes-plugins]), so this resolution would need be involved always, and globally.

An example resolution function could look like this:

function getThemePackageStylesheetsPath(string $name, string $version): string
{
    if (
        // `current/` doesn't apply to old "archives"
        $version === getThemePackageLatestVersion(themeName: $name) &&

        // `current/` may not exist (if our approach allows that)
        themePackageHasCurrentDirectory(themeName: $name) &&

        // `current/` may declare a version different than the one queried
        $version === getThemePackageVersionFromManifest(themeName: $name, versionDirectory: 'current')
    ) {
        $liveVersionDirectory = 'current';
    } else {
        $liveVersionDirectory = 'v' . $version;
    }

    return THEMES_DIRECTORY . '/' . $name . '/' . $liveVersionDirectory . '/stylesheets';
}

(the 3 conditions may also be "hidden" in a current/-aware getThemePackageLatestVersion() function)

Some side effects could appear during development - current/ directories with a manifest file declaring an unknown version would be "archived", but further changes would not be copied.
The unnecessary "archives" would first be ignored by the system (since the current/ directories are expected to override them), but may be confusing when looking at the contents of inc/themes/[name] (so some developers may copy the automatically created directory - indicating a version they just authored - that may be stale, instead of current/).
Additionally, when further version bumps are made, such potentially invalid "archives" may then be actually used as sources of information for previous versions on the development installation.

Alternatively to such permanent routing, we can consider:

• current/ as a temporary override
  Content in current/ directories would override the version directory specified in the manifest (and prevent the system from renaming current/, if that's being done).
  
  This would still require checking for potential override directories when attempting to fetch data from "archives", but the behavior would be limited to a "development mode".
  
  
• current/ as source
  Detect current/ changes and copy them to the relevant version directory (compatible with the redundancy approach).
  
  In this case, a more accurate name for the current/ directories would be source/ or origin/:
  • the system watches its content for changes, which are copied to version directories indicated by versions in manifest files (we can add some protection against overwriting anything else than the latest version directory for each extension)
    
  • they're read-only for the whole application
    
  • they're never "live" (other than the mirroring operation, no component is allowed to use/include/read its content)
    
  
  This could take the current/-related routing out of consideration for the extension system, and all data could be fetched from locations with standardized paths, without additional conditions or exceptions.
  
  It may be acceptable to require less restrictive filesystem permissions for development, and contents of a stamped directory could still be modified instead.
  
  This may also necessitate adding/extending a "development mode" to reduce performance impact of the additional filesystem activity.
  

[Ben]

Commenting specifically on the use of eval() in 1.9, it does worry me somewhat on the amount of plugins which would be ported over to 1.9. There is a significant amount of plugins created by users who are perhaps no longer active here, and we may be shooting ourselves in the foot by removing that compatibility.

[Wazzyl]

Maybe the team of MyBB should have plugin testers, so you guys can focus on the version of 1.9 and the plugin testers can focus on the plugins, it will help you both with the community and the extends.