I first posted this question :
Regex matching nested beginning and ending tags
It was answered perfectly by Wiktor Stribiżew. Now, I wanted to upgrade my Regex expression so that my parameters supports a JSON object (or almost, because lonely '{' and '[' aren't supported).
I have two expressions: one for paired tags, one for lonely tags. I first use the paired one, when all replacements done, I execute the lonely one. The modified lonely one works fine on regex101.com (https://www.regex101.com/r/HIEQZk/9), but the paired one tells me "castatrophic backtracking" (https://www.regex101.com/r/HIEQZk/8) even though in PHP in doesn't crash.
So could anyone help me optimize/fix this fairly huge regex.
Even though there seems to be useless escaping, it is because begin/end markers and the splitter can be customized and thus have to be escaped. (The paired one is not as escaped because it is not the one generated by PHP, but the one made by Wiktor Stribiżew with the modifications I did.)
The only part that I think that shall be optimized/fixed is the "parameters" group which I just modified to support JSON objects. (Tests of these can be seen in the earlier versions of the same regex101 url. The ones here are with a real HTML to parse.)
Lonely expression
~
\{\{ #Instruction start
([^\^\{\}]+) # (Group 1) Instruction name OR variable to reach if nothing else after then
(?:
\^
(?:([^\\^\{\}]*)\^)? #(Group 2) Specific delimiter
([^\{\}]*{(?:[^{}\[\]]+|(?3))+}[^\{\}]*|[^\{\}]*\[(?:[^{}\[\]]+|(?3))+\][^\{\}]*|[^\{\}]+) # (Group 3) Parameters
)?
\}\} #Instruction end
~xg
Paired expression
~{{ # Opening tag start
(\w+) # (Group 1) Tag name
(?: # Not captured group for optional parameters
(?: # Not captured group for optional delimiter
\^ # Aux delimiter
([^^\{\}]?) # (Group 2) Specific delimiter
)?
\^ # Aux delimiter
([^\{\}]*{(?:[^{}\[\]]+|(?3))+}[^\{\}]*|[^\{\}]*\[(?:[^{}\[\]]+|(?3))+\][^\{\}]*|[^\{\}]+) # (Group 3) Parameters
)?
}} # Opening tag end
( # (Group 4)
(?>
(?R) # Repeat the whole pattern
| # or match all that is not the opening/closing tag
[^{]*(?:\{(?!{/?\1[^\{\}]*}})[^{]*)*
)* # Zero or more times
)
{{/\1}} # Closing tag
~ix
Try to replace your (?: non-capturing groups with (?> atomic groups to prevent/reduce backtracking wherever possible. Those are non capturing as well. And/or experiment with possessive quantifiers while watching the stepscounter/debugger in regex101.
Wherever you don't want the engine to go back and try different other ways.
This is your updated demo where I just changed the first (?: to (?>
Related
I am using a regex to parse some BBCode, so the regex has to work recursively to also match tags inside others. Most of the BBCode has an argument, and sometimes it's quoted, though not always.
A simplified equivalent of the regex I'm using (with html style tags to reduce the escaping needed) is this:
'~<(\")?a(?(1)\1)> #Match the tag, and require a closing quote if an opening one provided
([^<]+ | (?R))* #Match the contents of the tag, including recursively
</a>~x'
However, if I have a test string that looks like this:
<"a">Content<a>Also Content</a></a>
it only matches the <a>Also Content</a> because when it tries to match from the first tag, the first matching group, \1, is set to ", and this is not overwritten when the regex is run recursively to match the inner tag, which means that because it isn't quoted, it doesn't match and that regex fails.
If instead I consistently either use or don't use quotes, it works fine, but I can't be sure that that will be the case with the content that I have to parse. Is there any way to work around this?
The full regex that I'm using, to match [spoiler]content[/spoiler], [spoiler=option]content[/spoiler] and [spoiler="option"]content[/spoiler], is
"~\[spoiler\s*+ #Match the opening tag
(?:=\s*+(\"|\')?((?(1)(?!\\1).|[^\]]){0,100})(?(1)\\1))?+\s*\] #If an option exists, match that
(?:\ *(?:\n|<br />))?+ #Get rid of an extra new line before the start of the content if necessary
((?:[^\[\n]++ #Capture all characters until the closing tag
|\n(?!\[spoiler]) Capture new line separately so backtracking doesn't run away due to above
|\[(?!/?spoiler(?:\s*=[^\]*])?) #Also match all tags that aren't spoilers
|(?R))*+) #Allow the pattern to recurse - we also want to match spoilers inside spoilers,
# without messing up nesting
\n? #Get rid of an extra new line before the closing tag if necessary
\[/spoiler] #match the closing tag
~xi"
There are a couple of other bugs with it as well though.
The simplest solution is to use alternatives instead:
<(?:a|"a")>
([^<]++ | (?R))*
</a>
But if you really don't want to repeat that a part, you can do the following:
<("?)a\1>
([^<]++ | (?R))*
</a>
Demo
I've just put the conditional ? inside the group. This time, the capturing group always matches, but the match can be empty, and the conditional isn't necessary anymore.
Side note: I've applied a possessive quantifier to [^<] to avoid catastrophic backtracking.
In your case I believe it's better to match a generic tag than a specific one. Match all tags, and then decide in your code what to do with the match.
Here's a full regex:
\[
(?<tag>\w+) \s*
(?:=\s*
(?:
(?<quote>["']) (?<arg>.{0,100}?) \k<quote>
| (?<arg>[^\]]+)
)
)?
\]
(?<content>
(?:[^[]++ | (?R) )*+
)
\[/\k<tag>\]
Demo
Note that I added the J option (PCRE_DUPNAMES) to be able to use (?<arg>...) twice.
(?(1)...) only checks if the group 1 has been defined, so the condition is true once the group is defined the first time. That is why you obtain this result (it is not related with the recursion level or whatever).
So when <a> is reached in the recursion, the regex engine try to match <a"> and fails.
If you want to use a conditional statement, you can write <("?)a(?(1)\1)> instead. In this way the group 1 is redefined each times.
Obviously you can write your pattern in a more efficient way like this:
~<(?:a|"a")>[^<]*+(?:(?R)[^<]*)*+</a>~
For your particular problem, I will use this kind of pattern to match any tags:
$pattern = <<<'EOD'
~
\[ (?<tag>\w+) \s*
(?:
= \s*
(?| " (?<option>[^"]*) " | ' ([^']*) ' | ([^]\s]*) ) # branch reset feature
)?
\s* ]
(?<content> [^[]*+ (?: (?R) [^[]*)*+ )
\[/\g{tag}]
~xi
EOD;
If you want to impose a specific tag at the ground level, you can add (?(R)|(?=spoiler\b)) before the tag name.
I am in the process of making a templating engine that is quite complex as it will feature typical constructs in programming languages such as if statements and loops.
Currently, I am working on the lexer, which I believe, deals with the job of converting a stream of characters into tokens. What I want to do is capture certain structures within the HTML document, which later can be worked on by the parser.
This is an example of the syntax:
<head>
<title>Template</title>
<meta charset="utf-8">
</head>
<body>
<h1>{{title}}</h1>
<p>This is also being matched.</p>
{{#myName}}
<p>My name is {{myName}}</p>
{{/}}
<p>This content too.</p>
{{^myName}}
<p>I have on name.</p>
{{/}}
<p>No matching here...</p>
</body>
I am trying to scan only for everything between the starting '{{' characters and ending '}}' characters. So, {{title}} would be one match, along with {{#myName}}, the text and content leading up to {{/}}, this should then be the second match.
I am not particularly the best at regular expressions, and I am pretty sure it is an issue with the pattern I have devised, which is this:
({{([#\^]?)([a-zA-Z0-9\._]+)}}([\w\W]+){{\/?}})
I read this as match two { characters, then either # or ^ any words containing uppercase or lowercase letters, along with any digits, dots, or underscores. Match anything that comes after the closing }} characters, until either the {{/}} characters are met, but the /}} part is optional.
The problem is visible in the link below. It is matching text that is not within the {{ and }} blocks. I am wondering it is linked to the use of the \w and \W, because if I specify specifically what characters I want to match against in the set, it seems to then work.
The regular expression test is here. I did look at the regular expression is the shared list for capturing all text that isn't HTML, and I noticed it is using lookaheads which I just cannot grasp, or understand why they would help me.
Can someone help me by pointing out the problem with the regular expression, or whether or not I am going the wrong way about it in terms of creating the lexer?
I hope I've provided enough information, and thank you for any help!
Your pattern doesn't work because [\w\W]+ take all possible characters until the last {{/}} of your string. Quantifiers (i.e. +, *, {1,3}, ?) are greedy by default. To obtain a lazy quantifier you must add a ? after it: [\w\W]+?
A pattern to deal with nested structures:
$pattern = <<<'LOD'
~
{{
(?| # branch reset group: the interest of this feature is that
# capturing group numbers are the same in all alternatives
([\w.]++)}} # self-closing tag: capturing group 1: tag name
| # OR
([#^][\w.]++)}} # opening tag: capturing group 1: tag name
( # capturing group 2: content
(?> # atomic group: three possible content type
[^{]++ # all characters except {
| # OR
{(?!{) # { not followed by another {
| # OR
(?R) # an other tag is met, attempt the whole pattern again
)* # repeat the atomic group 0 or more times
) # close the second capturing group
{{/}} # closing tag
) # close the branch reset group
~x
LOD;
preg_match_all($pattern, $html, $matches);
var_dump($matches);
To obtain all nested levels you can use this pattern:
$pattern = <<<'LOD'
~
(?=( # open a lookahead and the 1st capturing group
{{
(?|
([\w.]++)}}
|
([#^][\w.]++)}}
( # ?R was changed to ?1 because I don't want to
(?>[^{]++|{(?!{)|(?1))* # repeat the whole pattern but only the
) # subpattern in the first capturing group
{{/}}
)
) # close the 1st capturing group
) # and the lookahead
~x
LOD;
preg_match_all($pattern, $html, $matches);
var_dump($matches);
This pattern is only the first pattern enclosed in a lookahead and a capturing group. This construct allows to capture overlapping substrings.
More informations about regex features used in these two patterns:
possessive quantifiers ++atomic groups (?>..)lookahead (?=..), (?!..)branch reset group (?|..|..)recursion (?R), (?1)
I'm trying to write a PHP template engine.
Consider the following string:
#foreach($people as $person)
<p></p>
$end
I am able to use the following regex to find it:
#[\w]*\(.*?\).*?#end
But if I have this string:
#cake()
#cake()
#fish()
#end
#end
#end
The regex fails, this is what it finds:
#cake()
#cake()
#fish()
#end
Thanks in advance.
You can match nested functions, example:
$pattern = '~(#(?<func>\w++)\((?<param>[^)]*+)\)(?<content>(?>[^#]++|(?-4))*)#end)~';
or without named captures:
$pattern = '~(#(\w++)\(([^)]*+)\)((?>[^#]++|(?-4))*)#end)~';
Note that you can have all the content of all nested functions, if you put the whole pattern in a lookahead (?=...)
pattern details:
~ # pattern delimiter
( # open the first capturing group
#(\w++) # function name in the second capturing group
\( # literal (
([^)]*+) # param in the third capturing group
\) # literal )
( # open the fourth capturing group
(?> # open an atomic group
[^#]++ # all characters but # one or more times
| # OR
(?-4) # the first capturing group (the fourth on the left, from the current position)
)* # close the atomic group, repeat zero or more times
) # close the fourth capturing group
#end
)~ # close the first capturing group, end delimiter
You have nesting, which takes you out of the realm of a regular grammar, which means that you can't use regular expressions. Some regular expression engines (PHP's included, probably) have features that let you recognize some nested expressions, but that'll only take you so far. Look into traditional parsing tools, which should be able to handle your work load. This question goes into some of them.
I have a string that looks like this:
[if-abc] 12345 [if-def] 67890 [/if][/if]
I have the following regex:
/\[if-([a-z0-9-]*)\]([^\[if]*?)\[\/if\]/s
This matches the inner brackets just like I want it to. However, when I replace the 67890 with text (ie. abcdef), it doesn't match it.
[if-abc] 12345 [if-def] abcdef [/if][/if]
I want to be able to match ANY characters, including line breaks, except for another opening bracket [if-.
This part doesn't work like you think it does:
[^\[if]
This will match a single character that is neither of [, i or f. Regardless of the combination. You can mimic the desired behavior using a negative lookahead though:
~\[if-([a-z0-9-]*)\]((?:(?!\[/?if).)*)\[/if\]~s
I've also included closing tags in the lookahead, as this avoid the ungreedy repetition (which is usually worse performance-wise). Plus, I've changed the delimiters, so that you don't have to escape the slash in the pattern.
So this is the interesting part ((?:(?!\[/?if).)*) explained:
( # capture the contents of the tag-pair
(?: # start a non-capturing group (the ?: are just a performance
# optimization). this group represents a single "allowed" character
(?! # negative lookahead - makes sure that the next character does not mark
# the start of either [if or [/if (the negative lookahead will cause
# the entire pattern to fail if its contents match)
\[/?if
# match [if or [/if
) # end of lookahead
. # consume/match any single character
)* # end of group - repeat 0 or more times
) # end of capturing group
Modifying a little results in:
/\[if-([a-z0-9-]+)\](.+?)(?=\[if)/s
Running it on [if-abc] 12345 [if-def] abcdef [/if][/if]
Results in a first match as: [if-abc] 12345
Your groups are: abc and 12345
And modifying even further:
/\[if-([a-z0-9-]+)\](.+?)(?=(?:\[\/?if))/s
matches both groups. Although the delimiter [/if] is not captured by either of these.
NOTE: Instead of matching the delimeters I used a lookahead ((?=)) in the regex to stop when the text ahead matches the lookahead.
Use a period to match any character.
How should this, /(?>[^<>]+)/, be interpreted please? (PHP RegExp Engine)
Thank you.
(?> # I had to look this up, but apparently this syntax prevents the regex
# parser from backtracking into whatever is matched in this group if
# the rest of the pattern fails
[^<>]+ # match ANY character except '<' or '>', 1 or more times.
) # close non-backtrackable group.
For anyone interested in the once-only pattern, check out the section Once-only subpatterns in http://www.regextester.com/pregsyntax.html