Problems with the Help system and ideas on how to fix them

[Laird]

This post is based on a little document I put together at Ben's request on Discord. He asked me for my ideas on the problems with help docs, referencing several GitHub issues:

Help documents search results does not match document #2100
Default Help Documents #3193
Help Documents Enhacements #1450

In turn I referenced the related thread I'd started How a plugin can best use the built-in help document functionality and told him I'd put something together for him.

My having done that, Omar then suggested I quote it publicly so we don't lose track of the conversation, so here it is.

Setting the scene: How MyBB's Help system is broken

1. It is redundant in the worst sense - that of inconsistent redundancy - in that it uses both database entries and language files in different contexts, with no guarantee that the one (a database entry) will match the other (a language file entry). Issues #2100 and #3193 (as linked to by @Ben above) relate to this problematic redundancy. More specifically:
   
   • For search/display:
     
     • The language file is used by default when displaying help entries at misc.php?action=help and misc.php?action=help&hid=[hid]. What do I mean by 'by default'? I mean that even though the database is queried in this scenario, when the usetranslation column of the relevant table is set to 1, which it currently is for every core help document/section, the results of the query are discarded, and language file entries (if they exist, which by default they do) are used instead (although, if the language file entries *don't* exist - presumably because the admin has deleted them - then the results of the DB query are *not* discarded, and are used, after all).
       
     • The database is used when searching via the "Search" form at the bottom of misc.php?action=help. Due to the above, when a search result is clicked on, the resulting help document is pulled from the language file, and thus may differ from the apparent search result (which, again, instead uses the database).
       
   • For editing:
     
     • The database help entries can be edited via the admin/index.php?module=config-help_documents&action=edit&hid=[hid] form.
       
     • The equivalent language file help docs can be edited via admin/index.php?module=config-languages&action=edit&lang=english&editwith=&file=helpdocs.lang.php. Nothing prevents the database entries from desynchronising with those of the language file (and, indeed, they are desynchronised from the start - see the next point).
       
   • For default data: even prior to editing conflicts between database and language files, the original, default, installed data is inconsistent. For example:
     
     • The database entry for the helpdocs table with hid of 7 in install/resources/mysql_db_inserts.php begins as follows: "You can use MyCode, a simplified version of HTML, in your posts", whereas:
       
     • The equivalent language file entry in inc/languages/english/helpdocs.lang.php begins (differently) as follows: "You can use MyCode, also known as BB Codes to add effects or formatting to your posts."
       
2. Translations of help documents/sections are not at all supported in the database; they are supported only in language files. This relates to issue #1450 as linked to by @Ben above.
   
   
3. MyBB's help system provides no reliable means for plugins to offer translatable help sections/documents. More specifically: keys for language file help documents include hard-coded integers (correlating to the sid column of both the helpsections and helpdocs tables, and the hid column of the helpdocs table ), but no means is provided for a plugin to generate new, unique integers which are guaranteed not to be shared with any other plugin such that translators can rely upon them when generating translations (in language files). My community thread linked to above affirms this problem.
   

Recommendations

Recommendation #1:

Dispense with help text in language files and store help sections/documents solely in the database.

Rationale:

1. Customisations to language files might be overwritten on upgrade. We can ensure that database help entries are not (although an interface like "Find Updated Templates" for Help documents would be ideal).
   
2. Database help section/document searchability is already implemented. Going with language files would require us to implement a new search function.
   

Recommendation #2:

Convert the sid and hid fields of the helpsections and helpdocs tables from integers to short strings, and rename them something like section_code and document_code, respectively. The values for these string columns would be unique (relying on community support to ensure this). Add a language column to both of those tables to support translations.

Rationale: This provides for reliable translatability of plugin help sections/documents (although the uniqueness of the converted fields would require the same collective coordination as is required for plugin codenames - and, indeed, a plugin's codename could form the basis of the string). Perhaps, for a start, core code would, by convention, prefix each of them with c_, and plugins would prefix each of them with p_.

Recommendation #3:

Provide, separately, for ordering of help sections/documents by the values of the above-suggested section_code and document_code columns. This could be via a simple list stored, for example, in the persistent cache.

Rationale: This is simplest to implement compared to storing order numbers in the rows of the section/document database help tables themselves, and making sure that those numbers are consistent.

Recommendation #4:

Dispense with the usetranslation column. All help sections/documents should be translatable, even if a translation does not currently exist.

Rationale: I can't see any reason why any given help section/document shouldn't be translatable, and where there is no good cause to restrict functionality, then we ought not to restrict it.

Recommendation #5:

Provide a default and fallback help section/document where there is not a translation available in the current language. If, e.g., the user's language is set to Polish, and a plugin provides its help only in Swedish, and no other translation is available, then show the Swedish translation rather than an empty section/document.

Rationale: Something is better than nothing, even if it requires Google Translate to make sense of.

--- End document ---

Note that Omar quoted recommendation #1 from this document in my How a plugin can best use the built-in help document functionality thread linked to above a little while ago prior to my making this document public. He seemed unsure as to whether or not that first recommendation was a good idea, but among other comments, did agree that:

the language help doc system is broken, so much IMO that a revamp is justified if necessary. My personal vision of the Help Docs module would be something in the direction (if not as complex but extendable) as my OUGC Pages plugin.

I would suggest to work on it for 1.9 instead of 1.8 too.

Please feel free to share your own thoughts on how the Help system is broken (or not) and how to fix (or extend) it...

[Omar G.]

* I did correct myself in here, in that it is the help doc system that is broken, not the language system.

I think we need more feedback on how we want to handle the document data store process (if either DB or files).

I think it is important because there are other places where language support is (kind of) expected or planned and this (fixing the help doc module) could define the approach we want to handle language support in other places.

Some examples:
* Template groups offer language support, but it breaks if the plugin that loads the language variable is inactive..
* Suggestions to add language support for prefixes has been made.

[Laird]

* I did correct myself in here, in that it is the help doc system that is broken, not the language system.

Updated in my quote of you above. I knew what you meant and figured everybody else did too, so didn't mention that minor typo.

I think we need more feedback on how we want to handle the document data store process (if either DB or files).

Fair call. I prefer DB because it comes with built in searchability via SQL, as well as all of the other benefits of SQL queries over the filesystem.

I think it is important because there are other places where language support is (kind of) expected or planned and this (fixing the help doc module) could define the approach we want to handle language support in other places.

Again: fair call.

Some examples:
* Template groups offer language support, but it breaks if the plugin that loads the language variable is inactive..

Can you elaborate on this a little? I'm probably being a little dense but I'm not quite sure what you mean.

* Suggestions to add language support for prefixes has been made.

Yes, I saw that briefly when it came up. Again, I think that the DB (optionally backed by a filesystem cache) is the preferable approach here.

[Omar G.]

Can you elaborate on this a little? I'm probably being a little dense but I'm not quite sure what you mean.

You can use <lang:lang_key> (or something of this sort) inside template group names, these are replaced by language strings if these are present ($lang->lang_key), but if it is not present then it fails (i.e: the plugin that loads the language string was disabled). This is how the core template groups are translated but it isn't functional for plugins.

I think that language support should be consistent across all features, but we can't rely on the current file system for all cases.

Could you elaborate into the DB approach? You think a new table will be needed to store translations? I suppose INT identifies will be used in that case (match translations to sections or pages)?

[Laird]

You can use <lang:lang_key> (or something of this sort) inside template group names, these are replaced by language strings if these are present ($lang->lang_key), but if it is not present then it fails (i.e: the plugin that loads the language string was disabled). This is how the core template groups are translated but it isn't functional for plugins.

OK. Thanks. I kinda sorta think I get it.

I think that language support should be consistent across all features, but we can't rely on the current file system for all cases.

Consistency is a good aim/principle.

Could you elaborate into the DB approach? You think a new table will be needed to store translations?

I don't think we need a new table, just mods to the existing tables, to (1) replace the hid and sid integer column IDs with string IDs, and (2) add a language column, which supports translations. Rows with the same values for the (string) IDs but different values for the language column would be language variants (translations) of the same help sections/documents.

I suppose INT identifies will be used in that case (match translations to sections or pages)?

I think we should steer clear of int identifiers because it is too difficult to reliably associate each one with a unique help context (e.g., the help document for a specific plugin), especially when they are auto-incrementing. String identifiers can be deliberately chosen to be unique; not so for (especially auto-incrementing) ints.