From 0fd3f5c0ddf830162b66067e536bdbf0975523e5 Mon Sep 17 00:00:00 2001 From: Aaron Gyes Date: Thu, 14 Apr 2016 22:21:36 -0700 Subject: Update to pcre2 10.21 Point build tools at 10.21 --- pcre2-10.21/README | 843 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 843 insertions(+) create mode 100644 pcre2-10.21/README (limited to 'pcre2-10.21/README') diff --git a/pcre2-10.21/README b/pcre2-10.21/README new file mode 100644 index 00000000..48d2ffdd --- /dev/null +++ b/pcre2-10.21/README @@ -0,0 +1,843 @@ +README file for PCRE2 (Perl-compatible regular expression library) +------------------------------------------------------------------ + +PCRE2 is a re-working of the original PCRE library to provide an entirely new +API. The latest release of PCRE2 is always available in three alternative +formats from: + + ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre2-xxx.tar.gz + ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre2-xxx.tar.bz2 + ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre2-xxx.zip + +There is a mailing list for discussion about the development of PCRE (both the +original and new APIs) at pcre-dev@exim.org. You can access the archives and +subscribe or manage your subscription here: + + https://lists.exim.org/mailman/listinfo/pcre-dev + +Please read the NEWS file if you are upgrading from a previous release. +The contents of this README file are: + + The PCRE2 APIs + Documentation for PCRE2 + Contributions by users of PCRE2 + Building PCRE2 on non-Unix-like systems + Building PCRE2 without using autotools + Building PCRE2 using autotools + Retrieving configuration information + Shared libraries + Cross-compiling using autotools + Making new tarballs + Testing PCRE2 + Character tables + File manifest + + +The PCRE2 APIs +-------------- + +PCRE2 is written in C, and it has its own API. There are three sets of +functions, one for the 8-bit library, which processes strings of bytes, one for +the 16-bit library, which processes strings of 16-bit values, and one for the +32-bit library, which processes strings of 32-bit values. There are no C++ +wrappers. + +The distribution does contain a set of C wrapper functions for the 8-bit +library that are based on the POSIX regular expression API (see the pcre2posix +man page). These can be found in a library called libpcre2posix. Note that this +just provides a POSIX calling interface to PCRE2; the regular expressions +themselves still follow Perl syntax and semantics. The POSIX API is restricted, +and does not give full access to all of PCRE2's facilities. + +The header file for the POSIX-style functions is called pcre2posix.h. The +official POSIX name is regex.h, but I did not want to risk possible problems +with existing files of that name by distributing it that way. To use PCRE2 with +an existing program that uses the POSIX API, pcre2posix.h will have to be +renamed or pointed at by a link. + +If you are using the POSIX interface to PCRE2 and there is already a POSIX +regex library installed on your system, as well as worrying about the regex.h +header file (as mentioned above), you must also take care when linking programs +to ensure that they link with PCRE2's libpcre2posix library. Otherwise they may +pick up the POSIX functions of the same name from the other library. + +One way of avoiding this confusion is to compile PCRE2 with the addition of +-Dregcomp=PCRE2regcomp (and similarly for the other POSIX functions) to the +compiler flags (CFLAGS if you are using "configure" -- see below). This has the +effect of renaming the functions so that the names no longer clash. Of course, +you have to do the same thing for your applications, or write them using the +new names. + + +Documentation for PCRE2 +----------------------- + +If you install PCRE2 in the normal way on a Unix-like system, you will end up +with a set of man pages whose names all start with "pcre2". The one that is +just called "pcre2" lists all the others. In addition to these man pages, the +PCRE2 documentation is supplied in two other forms: + + 1. There are files called doc/pcre2.txt, doc/pcre2grep.txt, and + doc/pcre2test.txt in the source distribution. The first of these is a + concatenation of the text forms of all the section 3 man pages except the + listing of pcre2demo.c and those that summarize individual functions. The + other two are the text forms of the section 1 man pages for the pcre2grep + and pcre2test commands. These text forms are provided for ease of scanning + with text editors or similar tools. They are installed in + /share/doc/pcre2, where is the installation prefix + (defaulting to /usr/local). + + 2. A set of files containing all the documentation in HTML form, hyperlinked + in various ways, and rooted in a file called index.html, is distributed in + doc/html and installed in /share/doc/pcre2/html. + + +Building PCRE2 on non-Unix-like systems +--------------------------------------- + +For a non-Unix-like system, please read the comments in the file +NON-AUTOTOOLS-BUILD, though if your system supports the use of "configure" and +"make" you may be able to build PCRE2 using autotools in the same way as for +many Unix-like systems. + +PCRE2 can also be configured using CMake, which can be run in various ways +(command line, GUI, etc). This creates Makefiles, solution files, etc. The file +NON-AUTOTOOLS-BUILD has information about CMake. + +PCRE2 has been compiled on many different operating systems. It should be +straightforward to build PCRE2 on any system that has a Standard C compiler and +library, because it uses only Standard C functions. + + +Building PCRE2 without using autotools +-------------------------------------- + +The use of autotools (in particular, libtool) is problematic in some +environments, even some that are Unix or Unix-like. See the NON-AUTOTOOLS-BUILD +file for ways of building PCRE2 without using autotools. + + +Building PCRE2 using autotools +------------------------------ + +The following instructions assume the use of the widely used "configure; make; +make install" (autotools) process. + +To build PCRE2 on system that supports autotools, first run the "configure" +command from the PCRE2 distribution directory, with your current directory set +to the directory where you want the files to be created. This command is a +standard GNU "autoconf" configuration script, for which generic instructions +are supplied in the file INSTALL. + +Most commonly, people build PCRE2 within its own distribution directory, and in +this case, on many systems, just running "./configure" is sufficient. However, +the usual methods of changing standard defaults are available. For example: + +CFLAGS='-O2 -Wall' ./configure --prefix=/opt/local + +This command specifies that the C compiler should be run with the flags '-O2 +-Wall' instead of the default, and that "make install" should install PCRE2 +under /opt/local instead of the default /usr/local. + +If you want to build in a different directory, just run "configure" with that +directory as current. For example, suppose you have unpacked the PCRE2 source +into /source/pcre2/pcre2-xxx, but you want to build it in +/build/pcre2/pcre2-xxx: + +cd /build/pcre2/pcre2-xxx +/source/pcre2/pcre2-xxx/configure + +PCRE2 is written in C and is normally compiled as a C library. However, it is +possible to build it as a C++ library, though the provided building apparatus +does not have any features to support this. + +There are some optional features that can be included or omitted from the PCRE2 +library. They are also documented in the pcre2build man page. + +. By default, both shared and static libraries are built. You can change this + by adding one of these options to the "configure" command: + + --disable-shared + --disable-static + + (See also "Shared libraries on Unix-like systems" below.) + +. By default, only the 8-bit library is built. If you add --enable-pcre2-16 to + the "configure" command, the 16-bit library is also built. If you add + --enable-pcre2-32 to the "configure" command, the 32-bit library is also + built. If you want only the 16-bit or 32-bit library, use --disable-pcre2-8 + to disable building the 8-bit library. + +. If you want to include support for just-in-time compiling, which can give + large performance improvements on certain platforms, add --enable-jit to the + "configure" command. This support is available only for certain hardware + architectures. If you try to enable it on an unsupported architecture, there + will be a compile time error. + +. When JIT support is enabled, pcre2grep automatically makes use of it, unless + you add --disable-pcre2grep-jit to the "configure" command. + +. If you do not want to make use of the support for UTF-8 Unicode character + strings in the 8-bit library, UTF-16 Unicode character strings in the 16-bit + library, or UTF-32 Unicode character strings in the 32-bit library, you can + add --disable-unicode to the "configure" command. This reduces the size of + the libraries. It is not possible to configure one library with Unicode + support, and another without, in the same configuration. + + When Unicode support is available, the use of a UTF encoding still has to be + enabled by setting the PCRE2_UTF option at run time or starting a pattern + with (*UTF). When PCRE2 is compiled with Unicode support, its input can only + either be ASCII or UTF-8/16/32, even when running on EBCDIC platforms. It is + not possible to use both --enable-unicode and --enable-ebcdic at the same + time. + + As well as supporting UTF strings, Unicode support includes support for the + \P, \p, and \X sequences that recognize Unicode character properties. + However, only the basic two-letter properties such as Lu are supported. + Escape sequences such as \d and \w in patterns do not by default make use of + Unicode properties, but can be made to do so by setting the PCRE2_UCP option + or starting a pattern with (*UCP). + +. You can build PCRE2 to recognize either CR or LF or the sequence CRLF, or any + of the preceding, or any of the Unicode newline sequences, as indicating the + end of a line. Whatever you specify at build time is the default; the caller + of PCRE2 can change the selection at run time. The default newline indicator + is a single LF character (the Unix standard). You can specify the default + newline indicator by adding --enable-newline-is-cr, --enable-newline-is-lf, + --enable-newline-is-crlf, --enable-newline-is-anycrlf, or + --enable-newline-is-any to the "configure" command, respectively. + + If you specify --enable-newline-is-cr or --enable-newline-is-crlf, some of + the standard tests will fail, because the lines in the test files end with + LF. Even if the files are edited to change the line endings, there are likely + to be some failures. With --enable-newline-is-anycrlf or + --enable-newline-is-any, many tests should succeed, but there may be some + failures. + +. By default, the sequence \R in a pattern matches any Unicode line ending + sequence. This is independent of the option specifying what PCRE2 considers + to be the end of a line (see above). However, the caller of PCRE2 can + restrict \R to match only CR, LF, or CRLF. You can make this the default by + adding --enable-bsr-anycrlf to the "configure" command (bsr = "backslash R"). + +. In a pattern, the escape sequence \C matches a single code unit, even in a + UTF mode. This can be dangerous because it breaks up multi-code-unit + characters. You can build PCRE2 with the use of \C permanently locked out by + adding --enable-never-backslash-C (note the upper case C) to the "configure" + command. When \C is allowed by the library, individual applications can lock + it out by calling pcre2_compile() with the PCRE2_NEVER_BACKSLASH_C option. + +. PCRE2 has a counter that limits the depth of nesting of parentheses in a + pattern. This limits the amount of system stack that a pattern uses when it + is compiled. The default is 250, but you can change it by setting, for + example, + + --with-parens-nest-limit=500 + +. PCRE2 has a counter that can be set to limit the amount of resources it uses + when matching a pattern. If the limit is exceeded during a match, the match + fails. The default is ten million. You can change the default by setting, for + example, + + --with-match-limit=500000 + + on the "configure" command. This is just the default; individual calls to + pcre2_match() can supply their own value. There is more discussion on the + pcre2api man page. + +. There is a separate counter that limits the depth of recursive function calls + during a matching process. This also has a default of ten million, which is + essentially "unlimited". You can change the default by setting, for example, + + --with-match-limit-recursion=500000 + + Recursive function calls use up the runtime stack; running out of stack can + cause programs to crash in strange ways. There is a discussion about stack + sizes in the pcre2stack man page. + +. In the 8-bit library, the default maximum compiled pattern size is around + 64K. You can increase this by adding --with-link-size=3 to the "configure" + command. PCRE2 then uses three bytes instead of two for offsets to different + parts of the compiled pattern. In the 16-bit library, --with-link-size=3 is + the same as --with-link-size=4, which (in both libraries) uses four-byte + offsets. Increasing the internal link size reduces performance in the 8-bit + and 16-bit libraries. In the 32-bit library, the link size setting is + ignored, as 4-byte offsets are always used. + +. You can build PCRE2 so that its internal match() function that is called from + pcre2_match() does not call itself recursively. Instead, it uses memory + blocks obtained from the heap to save data that would otherwise be saved on + the stack. To build PCRE2 like this, use + + --disable-stack-for-recursion + + on the "configure" command. PCRE2 runs more slowly in this mode, but it may + be necessary in environments with limited stack sizes. This applies only to + the normal execution of the pcre2_match() function; if JIT support is being + successfully used, it is not relevant. Equally, it does not apply to + pcre2_dfa_match(), which does not use deeply nested recursion. There is a + discussion about stack sizes in the pcre2stack man page. + +. For speed, PCRE2 uses four tables for manipulating and identifying characters + whose code point values are less than 256. By default, it uses a set of + tables for ASCII encoding that is part of the distribution. If you specify + + --enable-rebuild-chartables + + a program called dftables is compiled and run in the default C locale when + you obey "make". It builds a source file called pcre2_chartables.c. If you do + not specify this option, pcre2_chartables.c is created as a copy of + pcre2_chartables.c.dist. See "Character tables" below for further + information. + +. It is possible to compile PCRE2 for use on systems that use EBCDIC as their + character code (as opposed to ASCII/Unicode) by specifying + + --enable-ebcdic --disable-unicode + + This automatically implies --enable-rebuild-chartables (see above). However, + when PCRE2 is built this way, it always operates in EBCDIC. It cannot support + both EBCDIC and UTF-8/16/32. There is a second option, --enable-ebcdic-nl25, + which specifies that the code value for the EBCDIC NL character is 0x25 + instead of the default 0x15. + +. If you specify --enable-debug, additional debugging code is included in the + build. This option is intended for use by the PCRE2 maintainers. + +. In environments where valgrind is installed, if you specify + + --enable-valgrind + + PCRE2 will use valgrind annotations to mark certain memory regions as + unaddressable. This allows it to detect invalid memory accesses, and is + mostly useful for debugging PCRE2 itself. + +. In environments where the gcc compiler is used and lcov version 1.6 or above + is installed, if you specify + + --enable-coverage + + the build process implements a code coverage report for the test suite. The + report is generated by running "make coverage". If ccache is installed on + your system, it must be disabled when building PCRE2 for coverage reporting. + You can do this by setting the environment variable CCACHE_DISABLE=1 before + running "make" to build PCRE2. There is more information about coverage + reporting in the "pcre2build" documentation. + +. The pcre2grep program currently supports only 8-bit data files, and so + requires the 8-bit PCRE2 library. It is possible to compile pcre2grep to use + libz and/or libbz2, in order to read .gz and .bz2 files (respectively), by + specifying one or both of + + --enable-pcre2grep-libz + --enable-pcre2grep-libbz2 + + Of course, the relevant libraries must be installed on your system. + +. The default size (in bytes) of the internal buffer used by pcre2grep can be + set by, for example: + + --with-pcre2grep-bufsize=51200 + + The value must be a plain integer. The default is 20480. + +. It is possible to compile pcre2test so that it links with the libreadline + or libedit libraries, by specifying, respectively, + + --enable-pcre2test-libreadline or --enable-pcre2test-libedit + + If this is done, when pcre2test's input is from a terminal, it reads it using + the readline() function. This provides line-editing and history facilities. + Note that libreadline is GPL-licenced, so if you distribute a binary of + pcre2test linked in this way, there may be licensing issues. These can be + avoided by linking with libedit (which has a BSD licence) instead. + + Enabling libreadline causes the -lreadline option to be added to the + pcre2test build. In many operating environments with a sytem-installed + readline library this is sufficient. However, in some environments (e.g. if + an unmodified distribution version of readline is in use), it may be + necessary to specify something like LIBS="-lncurses" as well. This is + because, to quote the readline INSTALL, "Readline uses the termcap functions, + but does not link with the termcap or curses library itself, allowing + applications which link with readline the to choose an appropriate library." + If you get error messages about missing functions tgetstr, tgetent, tputs, + tgetflag, or tgoto, this is the problem, and linking with the ncurses library + should fix it. + +The "configure" script builds the following files for the basic C library: + +. Makefile the makefile that builds the library +. src/config.h build-time configuration options for the library +. src/pcre2.h the public PCRE2 header file +. pcre2-config script that shows the building settings such as CFLAGS + that were set for "configure" +. libpcre2-8.pc ) +. libpcre2-16.pc ) data for the pkg-config command +. libpcre2-32.pc ) +. libpcre2-posix.pc ) +. libtool script that builds shared and/or static libraries + +Versions of config.h and pcre2.h are distributed in the src directory of PCRE2 +tarballs under the names config.h.generic and pcre2.h.generic. These are +provided for those who have to build PCRE2 without using "configure" or CMake. +If you use "configure" or CMake, the .generic versions are not used. + +The "configure" script also creates config.status, which is an executable +script that can be run to recreate the configuration, and config.log, which +contains compiler output from tests that "configure" runs. + +Once "configure" has run, you can run "make". This builds whichever of the +libraries libpcre2-8, libpcre2-16 and libpcre2-32 are configured, and a test +program called pcre2test. If you enabled JIT support with --enable-jit, another +test program called pcre2_jit_test is built as well. If the 8-bit library is +built, libpcre2-posix and the pcre2grep command are also built. Running +"make" with the -j option may speed up compilation on multiprocessor systems. + +The command "make check" runs all the appropriate tests. Details of the PCRE2 +tests are given below in a separate section of this document. The -j option of +"make" can also be used when running the tests. + +You can use "make install" to install PCRE2 into live directories on your +system. The following are installed (file names are all relative to the + that is set when "configure" is run): + + Commands (bin): + pcre2test + pcre2grep (if 8-bit support is enabled) + pcre2-config + + Libraries (lib): + libpcre2-8 (if 8-bit support is enabled) + libpcre2-16 (if 16-bit support is enabled) + libpcre2-32 (if 32-bit support is enabled) + libpcre2-posix (if 8-bit support is enabled) + + Configuration information (lib/pkgconfig): + libpcre2-8.pc + libpcre2-16.pc + libpcre2-32.pc + libpcre2-posix.pc + + Header files (include): + pcre2.h + pcre2posix.h + + Man pages (share/man/man{1,3}): + pcre2grep.1 + pcre2test.1 + pcre2-config.1 + pcre2.3 + pcre2*.3 (lots more pages, all starting "pcre2") + + HTML documentation (share/doc/pcre2/html): + index.html + *.html (lots more pages, hyperlinked from index.html) + + Text file documentation (share/doc/pcre2): + AUTHORS + COPYING + ChangeLog + LICENCE + NEWS + README + pcre2.txt (a concatenation of the man(3) pages) + pcre2test.txt the pcre2test man page + pcre2grep.txt the pcre2grep man page + pcre2-config.txt the pcre2-config man page + +If you want to remove PCRE2 from your system, you can run "make uninstall". +This removes all the files that "make install" installed. However, it does not +remove any directories, because these are often shared with other programs. + + +Retrieving configuration information +------------------------------------ + +Running "make install" installs the command pcre2-config, which can be used to +recall information about the PCRE2 configuration and installation. For example: + + pcre2-config --version + +prints the version number, and + + pcre2-config --libs8 + +outputs information about where the 8-bit library is installed. This command +can be included in makefiles for programs that use PCRE2, saving the programmer +from having to remember too many details. Run pcre2-config with no arguments to +obtain a list of possible arguments. + +The pkg-config command is another system for saving and retrieving information +about installed libraries. Instead of separate commands for each library, a +single command is used. For example: + + pkg-config --libs libpcre2-16 + +The data is held in *.pc files that are installed in a directory called +/lib/pkgconfig. + + +Shared libraries +---------------- + +The default distribution builds PCRE2 as shared libraries and static libraries, +as long as the operating system supports shared libraries. Shared library +support relies on the "libtool" script which is built as part of the +"configure" process. + +The libtool script is used to compile and link both shared and static +libraries. They are placed in a subdirectory called .libs when they are newly +built. The programs pcre2test and pcre2grep are built to use these uninstalled +libraries (by means of wrapper scripts in the case of shared libraries). When +you use "make install" to install shared libraries, pcre2grep and pcre2test are +automatically re-built to use the newly installed shared libraries before being +installed themselves. However, the versions left in the build directory still +use the uninstalled libraries. + +To build PCRE2 using static libraries only you must use --disable-shared when +configuring it. For example: + +./configure --prefix=/usr/gnu --disable-shared + +Then run "make" in the usual way. Similarly, you can use --disable-static to +build only shared libraries. + + +Cross-compiling using autotools +------------------------------- + +You can specify CC and CFLAGS in the normal way to the "configure" command, in +order to cross-compile PCRE2 for some other host. However, you should NOT +specify --enable-rebuild-chartables, because if you do, the dftables.c source +file is compiled and run on the local host, in order to generate the inbuilt +character tables (the pcre2_chartables.c file). This will probably not work, +because dftables.c needs to be compiled with the local compiler, not the cross +compiler. + +When --enable-rebuild-chartables is not specified, pcre2_chartables.c is +created by making a copy of pcre2_chartables.c.dist, which is a default set of +tables that assumes ASCII code. Cross-compiling with the default tables should +not be a problem. + +If you need to modify the character tables when cross-compiling, you should +move pcre2_chartables.c.dist out of the way, then compile dftables.c by hand +and run it on the local host to make a new version of pcre2_chartables.c.dist. +Then when you cross-compile PCRE2 this new version of the tables will be used. + + +Making new tarballs +------------------- + +The command "make dist" creates three PCRE2 tarballs, in tar.gz, tar.bz2, and +zip formats. The command "make distcheck" does the same, but then does a trial +build of the new distribution to ensure that it works. + +If you have modified any of the man page sources in the doc directory, you +should first run the PrepareRelease script before making a distribution. This +script creates the .txt and HTML forms of the documentation from the man pages. + + +Testing PCRE2 +------------ + +To test the basic PCRE2 library on a Unix-like system, run the RunTest script. +There is another script called RunGrepTest that tests the pcre2grep command. +When JIT support is enabled, a third test program called pcre2_jit_test is +built. Both the scripts and all the program tests are run if you obey "make +check". For other environments, see the instructions in NON-AUTOTOOLS-BUILD. + +The RunTest script runs the pcre2test test program (which is documented in its +own man page) on each of the relevant testinput files in the testdata +directory, and compares the output with the contents of the corresponding +testoutput files. RunTest uses a file called testtry to hold the main output +from pcre2test. Other files whose names begin with "test" are used as working +files in some tests. + +Some tests are relevant only when certain build-time options were selected. For +example, the tests for UTF-8/16/32 features are run only when Unicode support +is available. RunTest outputs a comment when it skips a test. + +Many (but not all) of the tests that are not skipped are run twice if JIT +support is available. On the second run, JIT compilation is forced. This +testing can be suppressed by putting "nojit" on the RunTest command line. + +The entire set of tests is run once for each of the 8-bit, 16-bit and 32-bit +libraries that are enabled. If you want to run just one set of tests, call +RunTest with either the -8, -16 or -32 option. + +If valgrind is installed, you can run the tests under it by putting "valgrind" +on the RunTest command line. To run pcre2test on just one or more specific test +files, give their numbers as arguments to RunTest, for example: + + RunTest 2 7 11 + +You can also specify ranges of tests such as 3-6 or 3- (meaning 3 to the +end), or a number preceded by ~ to exclude a test. For example: + + Runtest 3-15 ~10 + +This runs tests 3 to 15, excluding test 10, and just ~13 runs all the tests +except test 13. Whatever order the arguments are in, the tests are always run +in numerical order. + +You can also call RunTest with the single argument "list" to cause it to output +a list of tests. + +The test sequence starts with "test 0", which is a special test that has no +input file, and whose output is not checked. This is because it will be +different on different hardware and with different configurations. The test +exists in order to exercise some of pcre2test's code that would not otherwise +be run. + +Tests 1 and 2 can always be run, as they expect only plain text strings (not +UTF) and make no use of Unicode properties. The first test file can be fed +directly into the perltest.sh script to check that Perl gives the same results. +The only difference you should see is in the first few lines, where the Perl +version is given instead of the PCRE2 version. The second set of tests check +auxiliary functions, error detection, and run-time flags that are specific to +PCRE2. It also uses the debugging flags to check some of the internals of +pcre2_compile(). + +If you build PCRE2 with a locale setting that is not the standard C locale, the +character tables may be different (see next paragraph). In some cases, this may +cause failures in the second set of tests. For example, in a locale where the +isprint() function yields TRUE for characters in the range 128-255, the use of +[:isascii:] inside a character class defines a different set of characters, and +this shows up in this test as a difference in the compiled code, which is being +listed for checking. For example, where the comparison test output contains +[\x00-\x7f] the test might contain [\x00-\xff], and similarly in some other +cases. This is not a bug in PCRE2. + +Test 3 checks pcre2_maketables(), the facility for building a set of character +tables for a specific locale and using them instead of the default tables. The +script uses the "locale" command to check for the availability of the "fr_FR", +"french", or "fr" locale, and uses the first one that it finds. If the "locale" +command fails, or if its output doesn't include "fr_FR", "french", or "fr" in +the list of available locales, the third test cannot be run, and a comment is +output to say why. If running this test produces an error like this: + + ** Failed to set locale "fr_FR" + +it means that the given locale is not available on your system, despite being +listed by "locale". This does not mean that PCRE2 is broken. There are three +alternative output files for the third test, because three different versions +of the French locale have been encountered. The test passes if its output +matches any one of them. + +Tests 4 and 5 check UTF and Unicode property support, test 4 being compatible +with the perltest.sh script, and test 5 checking PCRE2-specific things. + +Tests 6 and 7 check the pcre2_dfa_match() alternative matching function, in +non-UTF mode and UTF-mode with Unicode property support, respectively. + +Test 8 checks some internal offsets and code size features; it is run only when +the default "link size" of 2 is set (in other cases the sizes change) and when +Unicode support is enabled. + +Tests 9 and 10 are run only in 8-bit mode, and tests 11 and 12 are run only in +16-bit and 32-bit modes. These are tests that generate different output in +8-bit mode. Each pair are for general cases and Unicode support, respectively. +Test 13 checks the handling of non-UTF characters greater than 255 by +pcre2_dfa_match() in 16-bit and 32-bit modes. + +Test 14 contains a number of tests that must not be run with JIT. They check, +among other non-JIT things, the match-limiting features of the intepretive +matcher. + +Test 15 is run only when JIT support is not available. It checks that an +attempt to use JIT has the expected behaviour. + +Test 16 is run only when JIT support is available. It checks JIT complete and +partial modes, match-limiting under JIT, and other JIT-specific features. + +Tests 17 and 18 are run only in 8-bit mode. They check the POSIX interface to +the 8-bit library, without and with Unicode support, respectively. + +Test 19 checks the serialization functions by writing a set of compiled +patterns to a file, and then reloading and checking them. + + +Character tables +---------------- + +For speed, PCRE2 uses four tables for manipulating and identifying characters +whose code point values are less than 256. By default, a set of tables that is +built into the library is used. The pcre2_maketables() function can be called +by an application to create a new set of tables in the current locale. This are +passed to PCRE2 by calling pcre2_set_character_tables() to put a pointer into a +compile context. + +The source file called pcre2_chartables.c contains the default set of tables. +By default, this is created as a copy of pcre2_chartables.c.dist, which +contains tables for ASCII coding. However, if --enable-rebuild-chartables is +specified for ./configure, a different version of pcre2_chartables.c is built +by the program dftables (compiled from dftables.c), which uses the ANSI C +character handling functions such as isalnum(), isalpha(), isupper(), +islower(), etc. to build the table sources. This means that the default C +locale which is set for your system will control the contents of these default +tables. You can change the default tables by editing pcre2_chartables.c and +then re-building PCRE2. If you do this, you should take care to ensure that the +file does not get automatically re-generated. The best way to do this is to +move pcre2_chartables.c.dist out of the way and replace it with your customized +tables. + +When the dftables program is run as a result of --enable-rebuild-chartables, +it uses the default C locale that is set on your system. It does not pay +attention to the LC_xxx environment variables. In other words, it uses the +system's default locale rather than whatever the compiling user happens to have +set. If you really do want to build a source set of character tables in a +locale that is specified by the LC_xxx variables, you can run the dftables +program by hand with the -L option. For example: + + ./dftables -L pcre2_chartables.c.special + +The first two 256-byte tables provide lower casing and case flipping functions, +respectively. The next table consists of three 32-byte bit maps which identify +digits, "word" characters, and white space, respectively. These are used when +building 32-byte bit maps that represent character classes for code points less +than 256. The final 256-byte table has bits indicating various character types, +as follows: + + 1 white space character + 2 letter + 4 decimal digit + 8 hexadecimal digit + 16 alphanumeric or '_' + 128 regular expression metacharacter or binary zero + +You should not alter the set of characters that contain the 128 bit, as that +will cause PCRE2 to malfunction. + + +File manifest +------------- + +The distribution should contain the files listed below. + +(A) Source files for the PCRE2 library functions and their headers are found in + the src directory: + + src/dftables.c auxiliary program for building pcre2_chartables.c + when --enable-rebuild-chartables is specified + + src/pcre2_chartables.c.dist a default set of character tables that assume + ASCII coding; unless --enable-rebuild-chartables is + specified, used by copying to pcre2_chartables.c + + src/pcre2posix.c ) + src/pcre2_auto_possess.c ) + src/pcre2_compile.c ) + src/pcre2_config.c ) + src/pcre2_context.c ) + src/pcre2_dfa_match.c ) + src/pcre2_error.c ) + src/pcre2_find_bracket.c ) + src/pcre2_jit_compile.c ) + src/pcre2_jit_match.c ) sources for the functions in the library, + src/pcre2_jit_misc.c ) and some internal functions that they use + src/pcre2_maketables.c ) + src/pcre2_match.c ) + src/pcre2_match_data.c ) + src/pcre2_newline.c ) + src/pcre2_ord2utf.c ) + src/pcre2_pattern_info.c ) + src/pcre2_serialize.c ) + src/pcre2_string_utils.c ) + src/pcre2_study.c ) + src/pcre2_substitute.c ) + src/pcre2_substring.c ) + src/pcre2_tables.c ) + src/pcre2_ucd.c ) + src/pcre2_valid_utf.c ) + src/pcre2_xclass.c ) + + src/pcre2_printint.c debugging function that is used by pcre2test, + + src/config.h.in template for config.h, when built by "configure" + src/pcre2.h.in template for pcre2.h when built by "configure" + src/pcre2posix.h header for the external POSIX wrapper API + src/pcre2_internal.h header for internal use + src/pcre2_intmodedep.h a mode-specific internal header + src/pcre2_ucp.h header for Unicode property handling + + sljit/* source files for the JIT compiler + +(B) Source files for programs that use PCRE2: + + src/pcre2demo.c simple demonstration of coding calls to PCRE2 + src/pcre2grep.c source of a grep utility that uses PCRE2 + src/pcre2test.c comprehensive test program + src/pcre2_printint.c part of pcre2test + src/pcre2_jit_test.c JIT test program + +(C) Auxiliary files: + + 132html script to turn "man" pages into HTML + AUTHORS information about the author of PCRE2 + ChangeLog log of changes to the code + CleanTxt script to clean nroff output for txt man pages + Detrail script to remove trailing spaces + HACKING some notes about the internals of PCRE2 + INSTALL generic installation instructions + LICENCE conditions for the use of PCRE2 + COPYING the same, using GNU's standard name + Makefile.in ) template for Unix Makefile, which is built by + ) "configure" + Makefile.am ) the automake input that was used to create + ) Makefile.in + NEWS important changes in this release + NON-AUTOTOOLS-BUILD notes on building PCRE2 without using autotools + PrepareRelease script to make preparations for "make dist" + README this file + RunTest a Unix shell script for running tests + RunGrepTest a Unix shell script for pcre2grep tests + aclocal.m4 m4 macros (generated by "aclocal") + config.guess ) files used by libtool, + config.sub ) used only when building a shared library + configure a configuring shell script (built by autoconf) + configure.ac ) the autoconf input that was used to build + ) "configure" and config.h + depcomp ) script to find program dependencies, generated by + ) automake + doc/*.3 man page sources for PCRE2 + doc/*.1 man page sources for pcre2grep and pcre2test + doc/index.html.src the base HTML page + doc/html/* HTML documentation + doc/pcre2.txt plain text version of the man pages + doc/pcre2test.txt plain text documentation of test program + install-sh a shell script for installing files + libpcre2-8.pc.in template for libpcre2-8.pc for pkg-config + libpcre2-16.pc.in template for libpcre2-16.pc for pkg-config + libpcre2-32.pc.in template for libpcre2-32.pc for pkg-config + libpcre2posix.pc.in template for libpcre2posix.pc for pkg-config + ltmain.sh file used to build a libtool script + missing ) common stub for a few missing GNU programs while + ) installing, generated by automake + mkinstalldirs script for making install directories + perltest.sh Script for running a Perl test program + pcre2-config.in source of script which retains PCRE2 information + testdata/testinput* test data for main library tests + testdata/testoutput* expected test results + testdata/grep* input and output for pcre2grep tests + testdata/* other supporting test files + +(D) Auxiliary files for cmake support + + cmake/COPYING-CMAKE-SCRIPTS + cmake/FindPackageHandleStandardArgs.cmake + cmake/FindEditline.cmake + cmake/FindReadline.cmake + CMakeLists.txt + config-cmake.h.in + +(E) Auxiliary files for building PCRE2 "by hand" + + pcre2.h.generic ) a version of the public PCRE2 header file + ) for use in non-"configure" environments + config.h.generic ) a version of config.h for use in non-"configure" + ) environments + +Philip Hazel +Email local part: ph10 +Email domain: cam.ac.uk +Last updated: 16 October 2015 -- cgit v1.2.3