[ATI.eu] Replacement, regex issues (Content styling)

[Johann]

search:

/\s*<h5 id=["']([^"']+?)['"]>​([^<>]+?)<\/h5>\s*/

replace:

\n\n=== //$2// ===\n<span anchor #$1></span>\n\n

Sehr simple.

[Johann]

As it infected all non-standard characters: maybe batchedit has any process which deals with char-sets and which was possible interrupted by connectivity. Sometimes, during no reaction, Atma would send orders also twice which might "disturb" ongoing prozesses.

h3 and h4 has been made with similar regex before, but didn't touch that sample page (maybe 20at all infected).

Something un-usual is that this change is printed in gray in the list, possible pointing on something.

[Johann]

Interesting is also that the replacments, althought looking similar, are uniqu on each page. So the additional added invisible characters are different while the string appears to be equal.

[Moritz]

Still no clear idea.

But it seems I have also not yet seen the full picture of what all went wrong.


So, apparently, something must have gone wrong when saving the page http://accesstoinsight.eu/en/lib/authors/thanissaro/bmc/section0059

between the subsequent revisions
2019/07/30 06:36 -- div headings
and
2019/08/01 16:10 -- <p id

where the UTF8 encoding of special characters somehow was garbled, which can be seen in the comparison between these two directly subsequent revisions.

That is the obvious error that I have seen. Everything else mentioned is not clear to me:

Plain text replacement &amp; -> & seems to have "destroyed" certain characters, mostly Pali.

More seemingly a simple header replacement of h5-tag. The action caused invisible characters in &.. html values.

I have not found where invisible characters appear in relation to h5-tag replacements or elsewhere.

Interesting is also that the replacments, althought looking similar, are uniqu on each page. So the additional added invisible characters are different while the string appears to be equal.

I have not seen any other page than BMC section 59 where the mentioned UTF8 encoding error happened, exactly between these two revisions .

Can Bhante point to other pages where this happened?


In any case this looks like an encoding error, which happened only one or maybe more times (which I have not seen) when saving some page(s) with BatchEdit.

My first idea would have been that maybe the file(s) was/were edited in an external program in between which could not deal properly with the UTF8 encoding and saved it wrongly.
DokuWiki has some mechanism for recognizing external edits and including them as such in the revision history. But it might be that it does not always work, does not always become "aware" when something was changed from outside (I think the check only happens when saving), so that the change would appear included in another change (in this case a replacement which was mostly correct, but based on a file that had already been modified and wrongly encoded in between).
Not sure about that, if that would be possible that especially BatchEdit might skip the mechanism of "being aware" if something has changed from outside.

Apart from that, I have seen that BatchEdit has become a lot more complex since the last time I looked into the code. Many things happen which I don't understand so quickly. For example it looks like the results of a BatchEdit search (with matches and replacements, even before they are applied) are stored in a certain structure in some temporary files, and most likely they are read back again from those files when finally applying the replacements. Maybe it could happen somehow that the data gets wrongly encoded there sometimes and reloaded from there afterwards for some reason with the wrong encoding. But that is just vague speculation now since I don't really yet see how it all works.


As long as not possible to replicate the error in some clear test case I think it will be difficult to figure it out. 

* Moritz now probably not having much time the next days or week to find the reason.

[Johann]

The idea of invisible characters came because a search of the defect strings would only match the page where copied from. As for which pages are effected, just the next, next, ... page of BMC, seemingly only in the BMC2 part.... ohh, ... No. Because the next page links are wrong, always the same page.

So it's just one page. My person then guesses it's because of a connection problem caused certain action not to be fullfilled. Atma also thinks that it would not be that worthy to investigate the plugin fully and redevelop it. Maybe, how ever, good if informing the developer that such happens.

Sadhu for care and much joy with good undertakings, Nyom Moritz .