The 1.9 theme and template system
#41
@Devilshakerz,

Please pardon the delay - I've been low on energy and then catching up on tasks that built up during my low-energy period. Also, for some reason I've found it difficult to parse your ideas. I don't know why, because you write clearly, but it means that I've had to spend a lot of time trying to understand and then develop a response.

(2021-10-26, 01:53 PM)Devilshakerz Wrote: We can be more lenient and accept packages with both placeholders and stamped versions, so that if developers:
  • copy files from a live forum, where directories are already stamped
  • make a package from a repository where a placeholder is used, without making any changes
the resulting packages could still work: the system would only need to look for leftover placeholder names and rename/"archive" them before use. If version numbers were already applied, it would do nothing.

This assumes no inherent need for current/ directories for production use.

I understand you to be saying that in the absence of a current/ directory, the system selects in its place the directory with the most recent version (and this seems to be affirmed by that which you write in parentheses below). Yes?

If so, that's what I meant when I wrote that if we can always rely on the current/ directory being present, then we avoid the need for various checks/determinations: in that case, we don't have to perform (even if inexpensive) checks to determine which versioned directory is most recent; we just always use current/.

Also, re placeholder directories: can you give me an example of when a placeholder directory would ever be used/useful given the possibility of using a current/ directory instead? Why do/would we need both, or, at least, why is/would it be useful to support both ("current" as well as placeholder directories)?

(2021-10-26, 01:53 PM)Devilshakerz Wrote:
(2021-10-25, 07:41 PM)Devilshakerz Wrote:
Quote:Would the current/ or the "archived" directory be used by the system for fetching Resources and including files?

The current/ directory. It's the simplest and most direct approach, avoiding the need for various checks/determinations.

It may lead to - potentially unexpected - implementation difficulties when specific versions have to resolve to "live" directories (and if the deviation described above also applies to this approach, the system would need to accept the latest version directory in the absence of current/).

I think you might be suggesting here that automatically keeping current/ and its archived version in sync is prone to errors, or am I misunderstanding?

(2021-10-26, 01:53 PM)Devilshakerz Wrote: Generally, with the described redundancy, all data that may be needed during production use could be resolved using standardized paths/"archives" (with stamped versions, where latest/ is considered equivalent to a specific v*/ directory).

Can you elaborate on the nature of this new directory, latest/? What would its contents be, how does it fit in, and why is it useful/necessary?

(2021-10-26, 01:53 PM)Devilshakerz Wrote: When developing extensions, though, only the current/ content would be modified - so e.g. when a parent theme's Resource is edited there, and its "archived" version becomes stale, it would only propagate down when queries for the parent's Resources are properly resolved to inc/themes/[parent]/current.

That's something I hadn't considered when suggesting this "redundant" approach: during development, the "archived" version of current/ would go stale when changes were made to current/. That seems to me to be a serious problem.

I think that we could avoid problems like that by adopting the "Upload zipped plugins/themes via the ACP, or upload them unzipped to a 'staging' directory and then visit the ACP" approach that I suggested earlier. I am not sure whether you have objections to that approach, and, if so, what they are, but it seems to me to be the least problematic approach during both development and production.

(2021-10-26, 01:53 PM)Devilshakerz Wrote: This could be simple enough (e.g. the current/ directory would have priority in arrays of template directories for Twig), but unless the system is well-aware of the filesystem state (i.e. maintains an accurate cache of which extensions have a current/ directory, and which versions they override), it may be necessary to additionally read manifest files to make sure the override directories are intended for the queried versions to avoid unintended behavior.
Resources and metadata for any version of any extension may be queried at any time (especially when the system is expected to track and analyze extension interactions - vertical [inheritance] and horizontal [themes-plugins]), so this resolution would need be involved always, and globally.

Again, if we ensured via the approach I reraised above that the current/ directory is always available and - well, current - then no awareness nor caching is necessary. It seems to me to be the simplest approach.

(2021-10-26, 01:53 PM)Devilshakerz Wrote: An example resolution function could look like this:
function getThemePackageStylesheetsPath(string $name, string $version): string
{
    if (
        // `current/` doesn't apply to old "archives"
        $version === getThemePackageLatestVersion(themeName: $name) &&

        // `current/` may not exist (if our approach allows that)
        themePackageHasCurrentDirectory(themeName: $name) &&

        // `current/` may declare a version different than the one queried
        $version === getThemePackageVersionFromManifest(themeName: $name, versionDirectory: 'current')
    ) {
        $liveVersionDirectory = 'current';
    } else {
        $liveVersionDirectory = 'v' . $version;
    }

    return THEMES_DIRECTORY . '/' . $name . '/' . $liveVersionDirectory . '/stylesheets';
}
(the 3 conditions may also be "hidden" in a current/-aware getThemePackageLatestVersion() function)

Similarly to my immediately previous comment: the approach I've suggested seems to avert the need for code like this in that the current/ directory would never be duplicated via an archived version directory - that corresponding version directory would only be created under controlled circumstances in which its contents are being replaced by the upload of a new current directory with a new version in its manifest file.

Do you see any problems with this approach?

(2021-10-26, 01:53 PM)Devilshakerz Wrote: Some side effects could appear during development - current/ directories with a manifest file declaring an unknown version would be "archived", but further changes would not be copied.

Wait, when would a manifest file ever declare an "unknown" version? Surely that would be/invoke an error?

(2021-10-26, 01:53 PM)Devilshakerz Wrote: The unnecessary "archives" would first be ignored by the system (since the current/ directories are expected to override them), but may be confusing when looking at the contents of inc/themes/[name] (so some developers may copy the automatically created directory - indicating a version they just authored - that may be stale, instead of current/).

Again, this problem does not occur on the approach I've suggested.

(2021-10-26, 01:53 PM)Devilshakerz Wrote: Additionally, when further version bumps are made, such potentially invalid "archives" may then be actually used as sources of information for previous versions on the development installation.

On the approach I've suggested, when a developer released a new version, s/he would simply make a copy on his/her development machine of the newly released current/ directory to its versioned directory, and then update the version number in the manifest file for the current/ directory as suitable for the next planned release.

(2021-10-26, 01:53 PM)Devilshakerz Wrote: Alternatively to such permanent routing, we can consider:
  • current/ as a temporary override
    Content in current/ directories would override the version directory specified in the manifest (and prevent the system from renaming current/, if that's being done).

    This would still require checking for potential override directories when attempting to fetch data from "archives", but the behavior would be limited to a "development mode".

A temporary override of whatever the latest versioned directory is?

(2021-10-26, 01:53 PM)Devilshakerz Wrote:
  • current/ as source
    Detect current/ changes and copy them to the relevant version directory (compatible with the redundancy approach).

    In this case, a more accurate name for the current/ directories would be source/ or origin/:
    • the system watches its content for changes, which are copied to version directories indicated by versions in manifest files (we can add some protection against overwriting anything else than the latest version directory for each extension)
    • they're read-only for the whole application
    • they're never "live" (other than the mirroring operation, no component is allowed to use/include/read its content)

    This could take the current/-related routing out of consideration for the extension system, and all data could be fetched from locations with standardized paths, without additional conditions or exceptions.

    It may be acceptable to require less restrictive filesystem permissions for development, and contents of a stamped directory could still be modified instead.

    This may also necessitate adding/extending a "development mode" to reduce performance impact of the additional filesystem activity.

That all sounds very error-prone, especially watching content for changes.
Reply
#42
In addition to current/, I also used latest/, [stampable], and directories with placeholder names. These refer to the same concept: a directory that generally contains the most recent code, and may be renamed to indicate a specific version (i.e. be stamped in a v* format compatible with version_compare(), becoming an archive, or versioned directory). The name of that original directory would indicate how it's used, and/or what happens to it.

We expect these directories to contain theme-related information, so they would be found directly under:
  • inc/themes/[theme codename]/ (all data of a theme, possibly excluding the manifest file)
  • inc/plugins/[plugin codename]/theme/ (all theme resources and modification instructions of a plugin)

References to other extensions are not expected to use this concept, i.e. if inc/plugin/[plugin codename]/theme/[plugin version]/theme-changes/[theme codename]/[theme version] directories contain resources & modification instructions relevant to a given theme, [theme version] always indicate a specific version of that theme.



(2021-10-30, 04:43 AM)Laird Wrote:
(2021-10-26, 01:53 PM)Devilshakerz Wrote: We can be more lenient and accept packages with both placeholders and stamped versions, so that if developers:
  • copy files from a live forum, where directories are already stamped
  • make a package from a repository where a placeholder is used, without making any changes
the resulting packages could still work: the system would only need to look for leftover placeholder names and rename/"archive" them before use. If version numbers were already applied, it would do nothing.

This assumes no inherent need for current/ directories for production use.

I understand you to be saying that in the absence of a current/ directory, the system selects in its place the directory with the most recent version (and this seems to be affirmed by that which you write in parentheses below). Yes?

Yes.

Quote:I think you might be suggesting here that automatically keeping current/ and its archived version in sync is prone to errors, or am I misunderstanding?

Yes, in that the system may query the two directories - initially expected to be equivalent - interchangeably, and when their contents differ, it may be problematic.

Quote:
(2021-10-26, 01:53 PM)Devilshakerz Wrote: An example resolution function could look like this:
function getThemePackageStylesheetsPath(string $name, string $version): string
{
    if (
        // `current/` doesn't apply to old "archives"
        $version === getThemePackageLatestVersion(themeName: $name) &&

        // `current/` may not exist (if our approach allows that)
        themePackageHasCurrentDirectory(themeName: $name) &&

        // `current/` may declare a version different than the one queried
        $version === getThemePackageVersionFromManifest(themeName: $name, versionDirectory: 'current')
    ) {
        $liveVersionDirectory = 'current';
    } else {
        $liveVersionDirectory = 'v' . $version;
    }

    return THEMES_DIRECTORY . '/' . $name . '/' . $liveVersionDirectory . '/stylesheets';
}
(the 3 conditions may also be "hidden" in a current/-aware getThemePackageLatestVersion() function)

Similarly to my immediately previous comment: the approach I've suggested seems to avert the need for code like this in that the current/ directory would never be duplicated via an archived version directory - that corresponding version directory would only be created under controlled circumstances in which its contents are being replaced by the upload of a new current directory with a new version in its manifest file.

Do you see any problems with this approach?

We'd still need some logic to resolve explicit versions to current/, if they happen to be most recent (otherwise - the archives).

Quote:
(2021-10-26, 01:53 PM)Devilshakerz Wrote: Some side effects could appear during development - current/ directories with a manifest file declaring an unknown version would be "archived", but further changes would not be copied.

Wait, when would a manifest file ever declare an "unknown" version? Surely that would be/invoke an error?
Quote:
(2021-10-26, 01:53 PM)Devilshakerz Wrote: Additionally, when further version bumps are made, such potentially invalid "archives" may then be actually used as sources of information for previous versions on the development installation.

On the approach I've suggested, when a developer released a new version, s/he would simply make a copy on his/her development machine of the newly released current/ directory to its versioned directory, and then update the version number in the manifest file for the current/ directory as suitable for the next planned release.

During development of a new version of an extension, the version number can be changed (incremented, e.g. from 1.0.0 to 1.0.1) by the developer when the extension is already active.

If the system creates an archive automatically using current/, so that the information about all versions can be queried from archives: a new 1.0.1 archive is made so the mechanism can continue to work, but subsequent changes to current/ (where further 1.0.1 work is saved) would not be copied to the 1.0.1 archive.
In this scenario, the developer would have to continue copying changes to the archive manually so the data can be queried properly (or the other way around, intending to keep the git-friendly directory up to date).

If the system skips archiving for the latest version: the version information may no longer be continuous on the developer's installation, since there was no 1.0.0 archive (because current/ was used instead, and by then it's used as the source for 1.0.1).
In this scenario, the developer would have to create an archive manually before starting work on the next release (which seems like the solution you describe).

Quote:
(2021-10-26, 01:53 PM)Devilshakerz Wrote: Alternatively to such permanent routing, we can consider:
  • current/ as a temporary override
    Content in current/ directories would override the version directory specified in the manifest (and prevent the system from renaming current/, if that's being done).

    This would still require checking for potential override directories when attempting to fetch data from "archives", but the behavior would be limited to a "development mode".

A temporary override of whatever the latest versioned directory is?

Yes, but we may not need to limit it to the latest version: the system can read the manifest file in the override directory (current/ would then have that role), and queries related to that version would use that directory, rather than the archive.

Quote:
(2021-10-26, 01:53 PM)Devilshakerz Wrote:
  • current/ as source
    Detect current/ changes and copy them to the relevant version directory (compatible with the redundancy approach).

    In this case, a more accurate name for the current/ directories would be source/ or origin/:
    • the system watches its content for changes, which are copied to version directories indicated by versions in manifest files (we can add some protection against overwriting anything else than the latest version directory for each extension)
    • they're read-only for the whole application
    • they're never "live" (other than the mirroring operation, no component is allowed to use/include/read its content)

    This could take the current/-related routing out of consideration for the extension system, and all data could be fetched from locations with standardized paths, without additional conditions or exceptions.

    It may be acceptable to require less restrictive filesystem permissions for development, and contents of a stamped directory could still be modified instead.

    This may also necessitate adding/extending a "development mode" to reduce performance impact of the additional filesystem activity.

That all sounds very error-prone, especially watching content for changes.

A less I/O-intensive approach could use symlinks, where current/ points to the version archive according to the manifest file.
devilshakerz.com/pgp (DF3A 34D9 A627 42E5 BC6A 6750 1F2F B8AA 28FF E1BC) ▪ keybase.io/devilshakerz
Reply
#43
(2021-10-30, 02:55 PM)Devilshakerz Wrote: In addition to current/, I also used latest/, [stampable], and directories with placeholder names. These refer to the same concept: a directory that generally contains the most recent code, and may be renamed to indicate a specific version (i.e. be stamped in a v* format compatible with version_compare(), becoming an archive, or versioned directory). The  name of that original directory would indicate how it's used, and/or what happens to it.

Why would we allow a bunch of different names for the same concept? Isn't that unnecessarily confusing matters, especially since it means that more than one of these differently-named but conceptually identical directories might exist? Why not simply accept only the one name, current/, for the one concept? In any case: what different uses/happenings do the various different names indicate?

(2021-10-30, 02:55 PM)Devilshakerz Wrote: We expect these directories to contain theme-related information, so they would be found directly under:
  • inc/themes/[theme codename]/ (all data of a theme, possibly excluding the manifest file)
  • inc/plugins/[plugin codename]/theme/ (all theme resources and modification instructions of a plugin)

References to other extensions are not expected to use this concept, i.e. if inc/plugin/[plugin codename]/theme/[plugin version]/theme-changes/[theme codename]/[theme version] directories contain resources & modification instructions relevant to a given theme, [theme version] always indicate a specific version of that theme.

Fair enough.

(2021-10-30, 02:55 PM)Devilshakerz Wrote:
Quote:I think you might be suggesting here that automatically keeping current/ and its archived version in sync is prone to errors, or am I misunderstanding?

Yes, in that the system may query the two directories - initially expected to be equivalent - interchangeably, and when their contents differ, it may be problematic.

I agree. This is problematic. We should avoid redundancy where possible. I threw it out as a possible alternative solution, but I don't think that it's ultimately the best solution (I've indicated what I think the best solution is).

(2021-10-30, 02:55 PM)Devilshakerz Wrote:
Quote:Similarly to my immediately previous comment: the approach I've suggested seems to avert the need for code like this in that the current/ directory would never be duplicated via an archived version directory - that corresponding version directory would only be created under controlled circumstances in which its contents are being replaced by the upload of a new current directory with a new version in its manifest file.

Do you see any problems with this approach?

We'd still need some logic to resolve explicit versions to current/, if they happen to be most recent (otherwise - the archives).

You're right. However, this seems to me to be a very minor problem, especially since most of the time the system will want to pull out the latest theme resource, and can then simply pull it out of the current/ directory.

Are there any other problems that you envisage with this approach?

(2021-10-30, 02:55 PM)Devilshakerz Wrote: During development of a new version of an extension, the version number can be changed (incremented, e.g. from 1.0.0 to 1.0.1) by the developer when the extension is already active.

If the system creates an archive automatically using current/, so that the information about all versions can be queried from archives: a new 1.0.1 archive is made so the mechanism can continue to work, but subsequent changes to current/ (where further 1.0.1 work is saved) would not be copied to the 1.0.1 archive.
In this scenario, the developer would have to continue copying changes to the archive manually so the data can be queried properly (or the other way around, intending to keep the git-friendly directory up to date).

Yes: this, too, I think, makes the "current plus redundant archived version" approach sub-optimal, given my preferred alternative.

(2021-10-30, 02:55 PM)Devilshakerz Wrote: If the system skips archiving for the latest version: the version information may no longer be continuous on the developer's installation, since there was no 1.0.0 archive (because current/ was used instead, and by then it's used as the source for 1.0.1).
In this scenario, the developer would have to create an archive manually before starting work on the next release (which seems like the solution you describe).

Yes, but that's only assuming that the explicitly-versioned directory would ever even be used. As I wrote above, in the vast majority of cases, the system is going to simply be interested in the current (latest) version of the theme.

It seems to me to be a minor price for developers to pay for the otherwise ease-of-use of the "upload a zip to the ACP (or unzipped to the staging directory)" approach.

(2021-10-30, 02:55 PM)Devilshakerz Wrote:
Quote:
(2021-10-26, 01:53 PM)Devilshakerz Wrote: Alternatively to such permanent routing, we can consider:
  • current/ as a temporary override
    Content in current/ directories would override the version directory specified in the manifest (and prevent the system from renaming current/, if that's being done).

    This would still require checking for potential override directories when attempting to fetch data from "archives", but the behavior would be limited to a "development mode".

A temporary override of whatever the latest versioned directory is?

Yes, but we may not need to limit it to the latest version: the system can read the manifest file in the override directory (current/ would then have that role), and queries related to that version would use that directory, rather than the archive.

This seems to me again to be over-complicated and confusing. When would it be useful? I think it is much more useful to be able to rely on current/ being - well, current - rather than standing in for some non-current directory.

(2021-10-30, 02:55 PM)Devilshakerz Wrote: A less I/O-intensive approach could use symlinks, where current/ points to the version archive according to the manifest file.

I considered that approach too, but it seems to me to be fraught with peril. For one, symlinks on Windows (as "shortcuts") don't seem to work so well with web servers. For another, symlinks are difficult (impossible?) to include in zip archives. [Edit: Doing some googling, I see that recent versions of Windows have full UNIX-like support for symlinks over and above "shortcuts", so maybe my criticism is no longer valid for recent Windows versions. I don't use Windows much these days and haven't kept up.]
Reply
#44
(2021-10-31, 04:04 AM)Laird Wrote: Why would we allow a bunch of different names for the same concept? Isn't that unnecessarily confusing matters, especially since it means that more than one of these differently-named but conceptually identical directories might exist? Why not simply accept only the one name, current/, for the one concept? In any case: what different uses/happenings do the various different names indicate?

Yes, I only want to use a single name. Various names were used in this thread since it's not set in stone yet. I edited this post to remove the accidental latest/ reference (it should be current/) and note that placeholder name also refers to current/.
devilshakerz.com/pgp (DF3A 34D9 A627 42E5 BC6A 6750 1F2F B8AA 28FF E1BC) ▪ keybase.io/devilshakerz
Reply
#45
Please keep the discussion on topic and avoid spamming.
Reply
#46
(2021-10-31, 04:04 AM)Laird Wrote:
(2021-10-30, 02:55 PM)Devilshakerz Wrote: A less I/O-intensive approach could use symlinks, where current/ points to the version archive according to the manifest file.

I considered that approach too, but it seems to me to be fraught with peril. For one, symlinks on Windows (as "shortcuts") don't seem to work so well with web servers. For another, symlinks are difficult (impossible?) to include in zip archives. [Edit: Doing some googling, I see that recent versions of Windows have full UNIX-like support for symlinks over and above "shortcuts", so maybe my criticism is no longer valid for recent Windows versions. I don't use Windows much these days and haven't kept up.]

The shortcuts themselves wouldn't need to exist outside the MyBB instance, but it looks like it would only work as expected on Windows (using junctions, which require administrator access to set up).
devilshakerz.com/pgp (DF3A 34D9 A627 42E5 BC6A 6750 1F2F B8AA 28FF E1BC) ▪ keybase.io/devilshakerz
Reply
#47
The majority of web servers, and indeed users, use a *NIX server such as GNU/Linux or the BSDs or whatever. I understand a minority of users may use Windows server, or there may be an over-representation of WAMP/XAMPP solutions among the dev team, but before such solutions are even bikeshedded would it not perish the thought to get one of those free tlds for yourself and a $5/mo shared hosting plan if you yourself lack a GNU/Linux machine?

Don't take this the wrong way -- but Windows should be the last priority just based on marketshare. The majority of Windows server deployments, overwhelmingly, use ASP, ASP.NET and .NET Core type apps (I included classic ASP only because you'd be surprised at how often people ask about it) and many windows hosts have no htaccess support (IIS uses webconfig files) or other rewrite support. So uh, yeah. If you guys actually genuinely need a *NIX testbed, I would suggest a cheap method like that.
Reply
#48
I agree that *NIX is the most common web server OS, and it's the OS I work with in both dev and production environments, however, unless I misunderstand, MyBB does support Windows deployments, so, unless we are to drop that support, then we have to make sure that the design of the new theme system is Windows-compatible.
Reply
#49
Hey Laird,

I was in no way saying that. I was however saying that proposing solutions that only work in WAMP or WIndows/IIS env is... not good optics, especially when it's probably the case outside of the dev team you could name the number of active, updated and running mybb boards on an IIS deployment on two hands.

I don't have a direct role in MyBB dev at all, I'm just an admin/operator and owner/web dev for my forum. But I'd encourage future publicly discussed solutions to the problems of template editing here to... perhaps use a bit more forethought. If the situation were inverted (a templating solution that excluded Windows or macOS support) you could make the case that such a thing makes perfect sense (Windows being at best an ancillary target to your userbase).
Reply
#50
While most live instances don't run on Windows, this statistic may not apply to devices that extension developers use, which is more relevant in this case, as the "shortcuts" are supposed to aid their local workflow.

The described limitation (and others related to git and Docker) would make this approach infeasible.

(2021-11-23, 06:31 PM)Raion Wrote: But I'd encourage future publicly discussed solutions to the problems of template editing here to... perhaps use a bit more forethought.

It's natural for a FLOSS project to prefer open development, including open brainstorming, and prioritize quality over optics.

MyBB's development discussions weren't always public, so it's possible to misinterpret the status of these discussions.

Note that the system is also being discussed in #1x-development on Discord, and we're working on a document that will compile the outstanding issues and eventually outline the desired design, and everyone is welcome to pitch in.
devilshakerz.com/pgp (DF3A 34D9 A627 42E5 BC6A 6750 1F2F B8AA 28FF E1BC) ▪ keybase.io/devilshakerz
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)