The Apache FOP Project

The Apache™ FOP Project

Apache™ FOP: Hyphenation

Hyphenation Support

Introduction

Apache™ FOP uses Liang's hyphenation algorithm, well known from TeX. It needs language specific pattern and other data for operation.

Because of licensing issues (and for convenience), all hyphenation patterns for FOP are made available through the Objects For Formatting Objects project. If you have made improvements to an existing FOP hyphenation pattern, or if you have created one from scratch, please consider contributing these to OFFO so that they can benefit other FOP users as well. Please inquire on the FOP User mailing list.

License Issues

Many of the hyphenation files distributed with TeX and its offspring are licenced under the LaTeX Project Public License (LPPL), which prevents them from being distributed with Apache software. The LPPL puts restrictions on file names in redistributed derived works which we feel can't guarantee. Some hyphenation pattern files have other or additional restrictions, for example against use for commercial purposes.

Although Apache FOP cannot redistribute hyphenation pattern files that do not conform with its license scheme, that does not necessarily prevent users from using such hyphenation patterns with FOP. However, it does place on the user the responsibility for determining whether the user can rightly use such hyphenation patterns under the hyphenation pattern license. The user is responsible to settle license issues for hyphenation pattern files that are obtained from non-Apache sources.

Sources of Custom Hyphenation Pattern Files

The most important source of hyphenation pattern files is the CTAN TeX Archive.

Installing Custom Hyphenation Patterns

To install a custom hyphenation pattern for use with FOP:

  1. Convert the TeX hyphenation pattern file to the FOP format. The FOP format is an xml file conforming to the DTD found at {fop-dir}/hyph/hyphenation.dtd.

  2. Name this new file following this schema: languageCode_countryCode.xml. The country code is optional, and should be used only if needed. For example:

  3. en_US.xml would be the file name for American English hyphenation patterns.

  4. it.xml would be the file name for Italian hyphenation patterns. The language and country codes must match the XSL-FO input, which follows ISO 639 (languages) and ISO 3166 (countries). NOTE: The ISO 639/ISO 3166 convention is that language names are written in lower case, while country codes are written in upper case. FOP does not check whether the language and country specified in the FO source are actually from the current standard, but it relies on it being two letter strings in a few places. So you can make up your own codes for custom hyphenation patterns, but they should be two letter strings too (patches for proper handling extensions are welcome)

  5. There are basically three ways to make the FOP-compatible hyphenation pattern file(s) accessible to FOP:

  6. Download the precompiled JAR from OFFO and place it either in the {fop-dir}/lib directory, or in a directory of your choice (and append the full path to the JAR to the environment variable FOP_HYPHENATION_PATH).

  7. Download the desired FOP-compatible hyphenation pattern file(s) from OFFO, and/or take your self created hyphenation pattern file(s),

    • place them in the directory {fop-dir}/hyph,

    • or place them in a directory of your choice and set the Ant variable user.hyph.dir to point to that directory (in build-local.properties), and run Ant with build target jar-hyphenation. This will create a JAR containing the compiled patterns in {fop-dir}/build that will be added to the classpath on the next run. (When FOP is built from scratch, and there are pattern source file(s) present in the directory pointed to by the user.hyph.dir variable, this JAR will automatically be created from the supplied pattern(s)).

  8. Put the pattern source file(s) into a directory of your choice and configure FOP to look for custom patterns in this directory, by setting the configuration option.

Either of these three options will ensure hyphenation is working when using FOP from the command-line. If FOP is being embedded, remember to add the location(s) of the hyphenation JAR(s) to the CLASSPATH (option 1 and 2) or to set the configuration option programmatically (option 3).

Hyphenation Patterns

If you would like to build your own hyphenation pattern files, or modify existing ones, this section will help you understand how to do so. Even when creating a pattern file from scratch, it may be beneficial to start with an existing file and modify it. See OFFO's Hyphenation page for examples. Here is a brief explanation of the contents of FOP's hyphenation patterns: The remaining content of this section should be considered "draft" quality. It was drafted from theoretical literature, and has not been tested against actual FOP behavior. It may contain errors or omissions. Do not rely on these instructions without testing everything stated here. If you use these instructions, please provide feedback on the FOP User mailing list, either confirming their accuracy, or raising specific problems that we can address.

If you want to convert a TeX hyphenation pattern file, you have to undo the TeX encoding for non-ASCII text. FOP uses Unicode, and the patterns must be proper Unicode too. You should be aware of the XML encoding issues, preferably use a good Unicode editor.

Note that FOP does not do Unicode character normalization. If you use combining chars for accents and other character decorations, you must declare character classes for them, and use the same sequence of base character and combining marks in the XSLFO source, otherwise the pattern wouldn't match. Fortunately, Unicode provides precomposed characters for all important cases in common languages, until now nobody run seriously into this issue. Some dead languages and dialects, especially ancient ones, may pose a real problem though.

If you want to generate your own patterns, an open-source utility called patgen is available on many Unix/Linux distributions and every TeX distribution which can be used to assist in creating pattern files from dictionaries. Pattern creation for languages like english or german is an art. If you can, read Frank Liang's original paper "Word Hy-phen-a-tion by Com-pu-ter" (yes, with hyphens). It is not available online. The original patgen.web source, included in the TeX source distributions, contains valuable comments, unfortunately technical details obscure often the high level issues. Another important source is The TeX Book, appendix H (either read the TeX source, or run it through TeX to typeset it). Secondary articles, for example the works by Petr Sojka, may also give some much needed insight into problems arising in automated hyphenation.