349 lines
14 KiB
Plaintext
349 lines
14 KiB
Plaintext
Technical Notes about PCRE
|
|
--------------------------
|
|
|
|
These are very rough technical notes that record potentially useful information
|
|
about PCRE internals.
|
|
|
|
Historical note 1
|
|
-----------------
|
|
|
|
Many years ago I implemented some regular expression functions to an algorithm
|
|
suggested by Martin Richards. These were not Unix-like in form, and were quite
|
|
restricted in what they could do by comparison with Perl. The interesting part
|
|
about the algorithm was that the amount of space required to hold the compiled
|
|
form of an expression was known in advance. The code to apply an expression did
|
|
not operate by backtracking, as the original Henry Spencer code and current
|
|
Perl code does, but instead checked all possibilities simultaneously by keeping
|
|
a list of current states and checking all of them as it advanced through the
|
|
subject string. In the terminology of Jeffrey Friedl's book, it was a "DFA
|
|
algorithm". When the pattern was all used up, all remaining states were
|
|
possible matches, and the one matching the longest subset of the subject string
|
|
was chosen. This did not necessarily maximize the individual wild portions of
|
|
the pattern, as is expected in Unix and Perl-style regular expressions.
|
|
|
|
Historical note 2
|
|
-----------------
|
|
|
|
By contrast, the code originally written by Henry Spencer (which was
|
|
subsequently heavily modified for Perl) compiles the expression twice: once in
|
|
a dummy mode in order to find out how much store will be needed, and then for
|
|
real. (The Perl version probably doesn't do this any more; I'm talking about
|
|
the original library.) The execution function operates by backtracking and
|
|
maximizing (or, optionally, minimizing in Perl) the amount of the subject that
|
|
matches individual wild portions of the pattern. This is an "NFA algorithm" in
|
|
Friedl's terminology.
|
|
|
|
OK, here's the real stuff
|
|
-------------------------
|
|
|
|
For the set of functions that form the "basic" PCRE library (which are
|
|
unrelated to those mentioned above), I tried at first to invent an algorithm
|
|
that used an amount of store bounded by a multiple of the number of characters
|
|
in the pattern, to save on compiling time. However, because of the greater
|
|
complexity in Perl regular expressions, I couldn't do this. In any case, a
|
|
first pass through the pattern is needed, for a number of reasons. PCRE works
|
|
by running a very degenerate first pass to calculate a maximum store size, and
|
|
then a second pass to do the real compile - which may use a bit less than the
|
|
predicted amount of store. The idea is that this is going to turn out faster
|
|
because the first pass is degenerate and the second pass can just store stuff
|
|
straight into the vector, which it knows is big enough. It does make the
|
|
compiling functions bigger, of course, but they have become quite big anyway to
|
|
handle all the Perl stuff.
|
|
|
|
Traditional matching function
|
|
-----------------------------
|
|
|
|
The "traditional", and original, matching function is called pcre_exec(), and
|
|
it implements an NFA algorithm, similar to the original Henry Spencer algorithm
|
|
and the way that Perl works. Not surprising, since it is intended to be as
|
|
compatible with Perl as possible. This is the function most users of PCRE will
|
|
use most of the time.
|
|
|
|
Supplementary matching function
|
|
-------------------------------
|
|
|
|
From PCRE 6.0, there is also a supplementary matching function called
|
|
pcre_dfa_exec(). This implements a DFA matching algorithm that searches
|
|
simultaneously for all possible matches that start at one point in the subject
|
|
string. (Going back to my roots: see Historical Note 1 above.) This function
|
|
intreprets the same compiled pattern data as pcre_exec(); however, not all the
|
|
facilities are available, and those that are do not always work in quite the
|
|
same way. See the user documentation for details.
|
|
|
|
Format of compiled patterns
|
|
---------------------------
|
|
|
|
The compiled form of a pattern is a vector of bytes, containing items of
|
|
variable length. The first byte in an item is an opcode, and the length of the
|
|
item is either implicit in the opcode or contained in the data bytes that
|
|
follow it.
|
|
|
|
In many cases below "two-byte" data values are specified. This is in fact just
|
|
a default. PCRE can be compiled to use 3-byte or 4-byte values (impairing the
|
|
performance). This is necessary only when patterns whose compiled length is
|
|
greater than 64K are going to be processed. In this description, we assume the
|
|
"normal" compilation options.
|
|
|
|
A list of all the opcodes follows:
|
|
|
|
Opcodes with no following data
|
|
------------------------------
|
|
|
|
These items are all just one byte long
|
|
|
|
OP_END end of pattern
|
|
OP_ANY match any character
|
|
OP_ANYBYTE match any single byte, even in UTF-8 mode
|
|
OP_SOD match start of data: \A
|
|
OP_SOM, start of match (subject + offset): \G
|
|
OP_CIRC ^ (start of data, or after \n in multiline)
|
|
OP_NOT_WORD_BOUNDARY \W
|
|
OP_WORD_BOUNDARY \w
|
|
OP_NOT_DIGIT \D
|
|
OP_DIGIT \d
|
|
OP_NOT_WHITESPACE \S
|
|
OP_WHITESPACE \s
|
|
OP_NOT_WORDCHAR \W
|
|
OP_WORDCHAR \w
|
|
OP_EODN match end of data or \n at end: \Z
|
|
OP_EOD match end of data: \z
|
|
OP_DOLL $ (end of data, or before \n in multiline)
|
|
OP_EXTUNI match an extended Unicode character
|
|
|
|
|
|
Repeating single characters
|
|
---------------------------
|
|
|
|
The common repeats (*, +, ?) when applied to a single character use the
|
|
following opcodes:
|
|
|
|
OP_STAR
|
|
OP_MINSTAR
|
|
OP_PLUS
|
|
OP_MINPLUS
|
|
OP_QUERY
|
|
OP_MINQUERY
|
|
|
|
In ASCII mode, these are two-byte items; in UTF-8 mode, the length is variable.
|
|
Those with "MIN" in their name are the minimizing versions. Each is followed by
|
|
the character that is to be repeated. Other repeats make use of
|
|
|
|
OP_UPTO
|
|
OP_MINUPTO
|
|
OP_EXACT
|
|
|
|
which are followed by a two-byte count (most significant first) and the
|
|
repeated character. OP_UPTO matches from 0 to the given number. A repeat with a
|
|
non-zero minimum and a fixed maximum is coded as an OP_EXACT followed by an
|
|
OP_UPTO (or OP_MINUPTO).
|
|
|
|
|
|
Repeating character types
|
|
-------------------------
|
|
|
|
Repeats of things like \d are done exactly as for single characters, except
|
|
that instead of a character, the opcode for the type is stored in the data
|
|
byte. The opcodes are:
|
|
|
|
OP_TYPESTAR
|
|
OP_TYPEMINSTAR
|
|
OP_TYPEPLUS
|
|
OP_TYPEMINPLUS
|
|
OP_TYPEQUERY
|
|
OP_TYPEMINQUERY
|
|
OP_TYPEUPTO
|
|
OP_TYPEMINUPTO
|
|
OP_TYPEEXACT
|
|
|
|
|
|
Match by Unicode property
|
|
-------------------------
|
|
|
|
OP_PROP and OP_NOTPROP are used for positive and negative matches of a
|
|
character by testing its Unicode property (the \p and \P escape sequences).
|
|
Each is followed by two bytes that encode the desired property as a type and a
|
|
value.
|
|
|
|
Repeats of these items use the OP_TYPESTAR etc. set of opcodes, followed by
|
|
three bytes: OP_PROP or OP_NOTPROP and then the desired property type and
|
|
value.
|
|
|
|
|
|
Matching literal characters
|
|
---------------------------
|
|
|
|
The OP_CHAR opcode is followed by a single character that is to be matched
|
|
casefully. For caseless matching, OP_CHARNC is used. In UTF-8 mode, the
|
|
character may be more than one byte long. (Earlier versions of PCRE used
|
|
multi-character strings, but this was changed to allow some new features to be
|
|
added.)
|
|
|
|
|
|
Character classes
|
|
-----------------
|
|
|
|
If there is only one character, OP_CHAR or OP_CHARNC is used for a positive
|
|
class, and OP_NOT for a negative one (that is, for something like [^a]).
|
|
However, in UTF-8 mode, the use of OP_NOT applies only to characters with
|
|
values < 128, because OP_NOT is confined to single bytes.
|
|
|
|
Another set of repeating opcodes (OP_NOTSTAR etc.) are used for a repeated,
|
|
negated, single-character class. The normal ones (OP_STAR etc.) are used for a
|
|
repeated positive single-character class.
|
|
|
|
When there's more than one character in a class and all the characters are less
|
|
than 256, OP_CLASS is used for a positive class, and OP_NCLASS for a negative
|
|
one. In either case, the opcode is followed by a 32-byte bit map containing a 1
|
|
bit for every character that is acceptable. The bits are counted from the least
|
|
significant end of each byte.
|
|
|
|
The reason for having both OP_CLASS and OP_NCLASS is so that, in UTF-8 mode,
|
|
subject characters with values greater than 256 can be handled correctly. For
|
|
OP_CLASS they don't match, whereas for OP_NCLASS they do.
|
|
|
|
For classes containing characters with values > 255, OP_XCLASS is used. It
|
|
optionally uses a bit map (if any characters lie within it), followed by a list
|
|
of pairs and single characters. There is a flag character than indicates
|
|
whether it's a positive or a negative class.
|
|
|
|
|
|
Back references
|
|
---------------
|
|
|
|
OP_REF is followed by two bytes containing the reference number.
|
|
|
|
|
|
Repeating character classes and back references
|
|
-----------------------------------------------
|
|
|
|
Single-character classes are handled specially (see above). This applies to
|
|
OP_CLASS and OP_REF. In both cases, the repeat information follows the base
|
|
item. The matching code looks at the following opcode to see if it is one of
|
|
|
|
OP_CRSTAR
|
|
OP_CRMINSTAR
|
|
OP_CRPLUS
|
|
OP_CRMINPLUS
|
|
OP_CRQUERY
|
|
OP_CRMINQUERY
|
|
OP_CRRANGE
|
|
OP_CRMINRANGE
|
|
|
|
All but the last two are just single-byte items. The others are followed by
|
|
four bytes of data, comprising the minimum and maximum repeat counts.
|
|
|
|
|
|
Brackets and alternation
|
|
------------------------
|
|
|
|
A pair of non-capturing (round) brackets is wrapped round each expression at
|
|
compile time, so alternation always happens in the context of brackets.
|
|
|
|
Non-capturing brackets use the opcode OP_BRA, while capturing brackets use
|
|
OP_BRA+1, OP_BRA+2, etc. [Note for North Americans: "bracket" to some English
|
|
speakers, including myself, can be round, square, curly, or pointy. Hence this
|
|
usage.]
|
|
|
|
Originally PCRE was limited to 99 capturing brackets (so as not to use up all
|
|
the opcodes). From release 3.5, there is no limit. What happens is that the
|
|
first ones, up to EXTRACT_BASIC_MAX are handled with separate opcodes, as
|
|
above. If there are more, the opcode is set to EXTRACT_BASIC_MAX+1, and the
|
|
first operation in the bracket is OP_BRANUMBER, followed by a 2-byte bracket
|
|
number. This opcode is ignored while matching, but is fished out when handling
|
|
the bracket itself. (They could have all been done like this, but I was making
|
|
minimal changes.)
|
|
|
|
A bracket opcode is followed by LINK_SIZE bytes which give the offset to the
|
|
next alternative OP_ALT or, if there aren't any branches, to the matching
|
|
OP_KET opcode. Each OP_ALT is followed by LINK_SIZE bytes giving the offset to
|
|
the next one, or to the OP_KET opcode.
|
|
|
|
OP_KET is used for subpatterns that do not repeat indefinitely, while
|
|
OP_KETRMIN and OP_KETRMAX are used for indefinite repetitions, minimally or
|
|
maximally respectively. All three are followed by LINK_SIZE bytes giving (as a
|
|
positive number) the offset back to the matching OP_BRA opcode.
|
|
|
|
If a subpattern is quantified such that it is permitted to match zero times, it
|
|
is preceded by one of OP_BRAZERO or OP_BRAMINZERO. These are single-byte
|
|
opcodes which tell the matcher that skipping this subpattern entirely is a
|
|
valid branch.
|
|
|
|
A subpattern with an indefinite maximum repetition is replicated in the
|
|
compiled data its minimum number of times (or once with OP_BRAZERO if the
|
|
minimum is zero), with the final copy terminating with OP_KETRMIN or OP_KETRMAX
|
|
as appropriate.
|
|
|
|
A subpattern with a bounded maximum repetition is replicated in a nested
|
|
fashion up to the maximum number of times, with OP_BRAZERO or OP_BRAMINZERO
|
|
before each replication after the minimum, so that, for example, (abc){2,5} is
|
|
compiled as (abc)(abc)((abc)((abc)(abc)?)?)?.
|
|
|
|
|
|
Assertions
|
|
----------
|
|
|
|
Forward assertions are just like other subpatterns, but starting with one of
|
|
the opcodes OP_ASSERT or OP_ASSERT_NOT. Backward assertions use the opcodes
|
|
OP_ASSERTBACK and OP_ASSERTBACK_NOT, and the first opcode inside the assertion
|
|
is OP_REVERSE, followed by a two byte count of the number of characters to move
|
|
back the pointer in the subject string. When operating in UTF-8 mode, the count
|
|
is a character count rather than a byte count. A separate count is present in
|
|
each alternative of a lookbehind assertion, allowing them to have different
|
|
fixed lengths.
|
|
|
|
|
|
Once-only subpatterns
|
|
---------------------
|
|
|
|
These are also just like other subpatterns, but they start with the opcode
|
|
OP_ONCE.
|
|
|
|
|
|
Conditional subpatterns
|
|
-----------------------
|
|
|
|
These are like other subpatterns, but they start with the opcode OP_COND. If
|
|
the condition is a back reference, this is stored at the start of the
|
|
subpattern using the opcode OP_CREF followed by two bytes containing the
|
|
reference number. If the condition is "in recursion" (coded as "(?(R)"), the
|
|
same scheme is used, with a "reference number" of 0xffff. Otherwise, a
|
|
conditional subpattern always starts with one of the assertions.
|
|
|
|
|
|
Recursion
|
|
---------
|
|
|
|
Recursion either matches the current regex, or some subexpression. The opcode
|
|
OP_RECURSE is followed by an value which is the offset to the starting bracket
|
|
from the start of the whole pattern. From release 6.5, OP_RECURSE is
|
|
automatically wrapped inside OP_ONCE brackets (because otherwise some patterns
|
|
broke it). OP_RECURSE is also used for "subroutine" calls, even though they
|
|
are not strictly a recursion.
|
|
|
|
|
|
Callout
|
|
-------
|
|
|
|
OP_CALLOUT is followed by one byte of data that holds a callout number in the
|
|
range 0 to 254 for manual callouts, or 255 for an automatic callout. In both
|
|
cases there follows a two-byte value giving the offset in the pattern to the
|
|
start of the following item, and another two-byte item giving the length of the
|
|
next item.
|
|
|
|
|
|
Changing options
|
|
----------------
|
|
|
|
If any of the /i, /m, or /s options are changed within a pattern, an OP_OPT
|
|
opcode is compiled, followed by one byte containing the new settings of these
|
|
flags. If there are several alternatives, there is an occurrence of OP_OPT at
|
|
the start of all those following the first options change, to set appropriate
|
|
options for the start of the alternative. Immediately after the end of the
|
|
group there is another such item to reset the flags to their previous values. A
|
|
change of flag right at the very start of the pattern can be handled entirely
|
|
at compile time, and so does not cause anything to be put into the compiled
|
|
data.
|
|
|
|
Philip Hazel
|
|
June 2006
|