I'm starting to use poedit for PHP text translations. Now I'm confused what to do with long multiline texts, like a page with Terms and Conditions. I'm having a discussion about this with a colleague.
I see two options:
Per paragraph or line one poedit field
This keeps the text inside a echo _('lorem ipsum'); small.
Disadvantage: if you change the text and need to add lines, you would have to add lines in the code, which is not desirable. You probably wouldn't do this, and then use one identifier to add two paragraphs, which goes against this method.
All text in one poedit field
Disadvantage: This would result in a very long identifier. I'm not going to give an example - you know how long they can be. The identifier text could span more than a screen.
Identifier to describe what should be displayed
The identifier could be replaced by a short line describing what it is about, like this:
<?php _('Terms and conditions: complete text'); ?>
Disadvantage: if a new language is added, and this text is not translated, the identifier will show.
Advantage: if paragraphs are added or removed, you can do so without having to change the source code.
Advantage: sometimes one English word is translated into different meanings depending on context, like "bold":
Font weight: bold - translates to "fett" in German.
How are you feeling today: bold - translates to "mutig" in German.
NB: you should read this as a form field where you can select what font-weight you want: (1) normal, (2) bold, or (3) italic, not as one text "Font weight: bold".
In these two cases an identifier like "bold-font" and "bold-feeling" would provide context.
Questions
How do you handle long texts in poedit? What is best practise?
Gettext is not suitable for long free form texts at all. The correct answer is a third one you didn’t mention: just don’t. Use e.g. localized text files instead.
To comment on some of your ideas:
Per paragraph or line one poedit field
Chopping the text into individual lines is an extremely bad idea. It would loose important context.
Identifier to describe what should be displayed
...
Advantage: if paragraphs are added or removed, you can do so without having to change the source code.
Very bad idea as well (generally, not just for long text). If the source (English) text changes, there’s nothing to alert you to do the need to update the translation. Inevitably, the translations will get out of sync, undetectably. This is not an “advantage” at all.
There’s a reason why gettext uses source text as the key: it works.
Advantage: sometimes one English word is translated into different meanings depending on context, like "bold":
I don’t see how are different meanings of same words/short texts related to the question at all. That’s handled naturally by gettext with message contexts (msgctxt) as discussed in the manual.
Related
I've a rich text editor for news messages.
The frontend shows one paragraph and the user can read the full message once the user clicks "read more".
However this recognition is now done by <div></div> tags, while the editor works with tags (two for a paragraph).
My current regex is:
"/<div>([^`]*?)<\/div>/is"
How can i extend this to also recognize two tags right after each other. (Notice, those br tags might contain attributes).
As discussed above, beware that using regex to parse HTML, especially for "complex" problems, is generally a bad idea. The following is not a perfect solution, but may be good enough to the simple requirements you've given above:
/(?<=<div>).*?(?=<\/div>)|(?<=<br>\s*<br>).*?(?=<div>|<br>\s*<br>)/is
The (?<=...) and (?=...) are look behinds/aheads, i.e. they assert that those sections of the pattern are present, but are not included in the match result.
I have also used \s* to help catch scenarios where the user types something like:
<br> <br>
Or:
<br>
<br>
...But as I say, this is still not a perfect solution. If you find the pattern gets too complex, then seriously consider using an XML parser instead. (Or, how about just letting the user enter new lines, and converting these into paragraphs for them? ... Or even, just use an existing WYSIHTML5 library, or a markdown library?)
I'm using po files to translate my application using the gettext function.
I have a lot of strings using formatting characters like spaces, colons, question marks, etc....
What's best practice here?
E.g.:
_('operating database: '). DB_NAME. _(' on ').DB_HOST;
_('Your name:');
or
_('operating database').': '. DB_NAME.' '._('on').' '.DB_HOST;
_('Your name').':';
Should I keep them in translation or is it better to let them hardcoded? What are the pros and cons?
Neither of your examples is good.
The best practice is to have one string per one self-contained displayed unit of text. If you're showing a message box, for example, then all of its content should be one translatable string, even if it has more than one sentence. A label: one string; a message: one string.
Never, unless you absolutely cannot avoid it, break a displayed piece of text into multiple strings concatenated in code, as the above examples do. Instead, use string formatting:
sprintf(_('operating database: %s on %s'), $DB_NAME, $DB_HOST);
The reason is that a) some translations may need to put the arguments in different order and b) it gives the translator some context to work with. For example, "on" alone can be translated quite differently in different sentences, even in different uses in your code, so letting the translator translate just that word would inevitably lead to poor, hard to understand, broken translations.
The GNU gettext manual has a chapter on this as well.
If you keep them in translation than all translations will duplicate them. This means all this spaces, colons and etc will be duplicated for each language. What for?
I'm standing for translating just the meaning parts of the strings (second variant).
I have been working on web developement for quite some time now and I have always struggled to find a clean solution for a problem I have encountered during i18n of HTML strings, mostly anchor tags.
First of let me show you a typical problematic example. This is a frequently encountered string in HTML templates:
Welcome to my site. Check out our cool products
you should not miss.
How do I translate this string while still having the following properties:
Dynamic generation of the URL (e.g. using a router)
A translatable string that is as readable as possible (so translators can do it w/o looking at the code)
Because the string contains HTML, I probably want to escape some parts I insert (e.g. the URL), so I don't make myself vulnerable to XSS if this URL contains user input
It should look as good as possible in the code as well
How do you translate your strings when they contain dynamic content and HTML?
When I now want to apply i18n to that string, I probably turn to gettext or a framework function. Since I come from the PHP/Joomla! world, I used JText::_ before, which acts very similar to gettext. In Python I now use Babel. Both share the same problem and probably more languages, too. All code I share here is my way of doing it in Python, more explicitly, in my Mako templates
Of course, the problem is: There is HTML in our string to be translated (and a URL, for that matter). Here are my options, which I will each explain afterwards:
Passing the raw string to gettext
Splitting the text into three bits
Surrounding linked word with variables
Using one variable that gets build seperately
Passing the raw string to gettext
This one seems the first approach one might take, if not aware of the implications.
Approach 1:
_('Welcome to my site. Check out our cool products \
you should not miss.')
For this msgid you could now translate it, keeping the HTML intact.
Advantages:
This looks very clean in the code and is easy to understand
If the translator is keeping the HTML intact this does not produce any problems
Disadvantages:
The translator has to know at least a little HTML
The string is completely unflexible, e.g. if the URL changes, all translations have to be adjusted
It does not allow for dynamic generation of the URL using something like a router
So as a conclusion, while I used this I quickly hit my limit. My next idea was:
Splitting the text into three bits
Approach 2:
_('Welcome to my site. Check out our cool ') + '<a href="/products">' +\
_('products') + '</a>' + _(' you should not miss.')
Advantages:
The URL is completely flexible now
Only actual text for the translators
Disadvantages:
Splits a sentence into three parts
Translator has to know which parts relate together or he might not be able to produce meaningful sentences
Not very pretty in code
The msgid may be a single word, which can cause problems (beware of contexts) but can be fixed.
I used this technique for some time because I did not know about printf style strings in PHP (which I used back then). Because this looked so ugly, I tried a different approach:
Surrounding linked word with variables
Approach 3:
_('Welcome to my site. Check out our cool %sproducts%s you should not miss.' % \
('', '')
Advantages:
Single string to translate, a complete sentence
Translator gets the context right from the string
Code is not that ugly
Disadvantages:
Translator has to take care that no %s goes missing (might be confusion as it reads like sproducts)
Introduces two format string variables for every URL, one being only </a>
Using one variable that gets build seperately
From here I had some different approaches, but I finally came of with the one I currently use (which might look like overkill, but I perfer it for now).
Approach 4:
_('Welcome to my site. Check out our cool %s \
you should not miss.') % ('%s' % ('/products', _('products')))
Let me take some time to reason this (seemingly lunatic) approach. First of all, the actual translation string looks like this:
_('Welcome to my site. Checkout our cool ${product_url} \
you should not miss.')
Which leaves a translator with the information what is inserted there (that's the translationstring version). Second, I want to ensure that I can manually escape all parts that are inserted into the HTML. While Mako provides automatic escaping, this does not make sense in a statement like this:
${'This is a url'}
It would destroy the url so I have to apply the |n filter to remove any escaping. However, if any argument of that is user supplied, it also opens up to XSS which I want to prevent. Not taking any risk, I can just escape any input (the same way good template engines do by defualt) and then remove Mako's escaping for this one string. So
'%s' % ('/products', _('products'))
actually looks like
'%s' % (escape('/products'), _('products'))
where escape is imported from markupsafe (see Markupsafe).
The final part now is dynamic URLs through a router: request.route_url('products_view')
To combine each of these possibilities, I have to produce something very ugly (note that this uses the mapping keyword argument of translationstring (translationstring.TranslationString) but that combines all the benefits I want/need from translation:
Final result:
_('Welcome to my site. Checkout our cool ${product_url} \
you should not miss.', mapping={'product_url': '%s' %\
(escape(request.route_url('products_view')), _('products'))})
Advantages:
Full HTML escpaing
Fully dynamic
Very good msgids for translation
Disadvantages:
An extremely ugly construct in the template (or the program anyway)
The lingua extractor doesn't catch _('products') so we have do extract that manually
So that is it, this concludes my approaches to this problem. Maybe I am doing something way to complicated and you have a lot better ideas or maybe this is a problem that depends on specific types of translatable text (and one has to choose the right approach).
Did I miss any solution or anything that would improve my approach?
The purpose of the <bdi> tag in HTML5 is to isolate bidirectional text from it's context. And that's precisely what I'm looking for.
A left-to-right username displays like this:
Welcome, Generic User. [Logout]
With a right-to-left username it would turn into this awful thing:
Welcome, [tougoL] .resU cireneG
or even worse depending on the context, displaying everything around (not just the users' name) in backwards.
The problem is that no browsers support the <bdi> tag yet, so I was wandering, is there a way to simulate it? What HTML tags could isolate it aswell? I know <span> and <div> do not.
I wouldn't like to remove all BIDI characters, but the way I see it, the importance of my site to display properly > the right for bidirectional-language users to participate.
With a right-to-left username it would turn into this awful thing
It shouldn't. The text in an (eg) Arabic username would be rendered right-to-left but it wouldn't affect the flow of Latin text around it.
The problem you may be thinking of is when a username includes a Unicode BDO (bi-directional override) control character. This affects all inline text following it, which is often a Bad Thing for web sites templating text into HTML.
Probably the simplest solution to this problem is input filtering to remove control characters, both the normal ASCII ones (0x00–0x1F) and the Unicode ones. There is a group of characters designated by Unicode and W3 as unsuitable for use in markup in this Note which web applications will generally want to remove from data. It includes the BDO characters and several others that can cause odd effects to leak outside of their own stretch of text.
The string in the question does not trigger wrong display order, unless there are control characters in the username string, but e.g. a message of the form
User (N badges) wrote:
would do that, if User were replaced by a name in Arabic letters, say أحمد, and N were replaced by a number, say 3. The rendering would then be
أحمد (3 badges) said:
Technically, this is not a bug; it follows from Unicode bidirectionality rules – the strong right-to-left (RTL) directionality of Arabic letters affects characters with weak directionality like parentheses. But it is all wrong in practical terms, of course. Any string that may contain RTL characters in a generally left-to-right context should be protected, isolated. In HTML documents, there are three ways to do that:
Character level: use the control characters U+202B (right-to-left embedding, RLE) before and U+202C (pop directional formatting, PDF) after the string. In HTML, you could use and for them. This is supported by IE 9 but not by most other browsers.
Markup level: use the <bdi> markup. As mentioned, it is not supported by browsers yet.
Stylesheet: use unicode-bidi: embed. This is generally supported by modern browsers.
You can combine the stylesheet approach with the markup approach. It’s logical to do so, and in future browsers, this double approach will work even with stylesheets disabled:
<script>
document.createElement('bdi');
</script>
<style>
bdi { unicode-bidi: bidi-override; }
</style>
...
<bdi>أحمد</bdi> (3 badges) wrote:
The script code is there to make older versions of IE recognize the <bdi> element, so that styles will take effect on it. This would of course fail when scripting is disabled, so it would be slightly safer to use <span> with class, and you could still wrap it inside <bdi>. So an alternative is
<style>
.bdi { unicode-bidi: bidi-override; }
</style>
...
<bdi><span class=bdi>أحمد</span></bdi> (3 badges) wrote:
I'm currently in the process of building a PHP Parser written in PHP, as no existing parser came up in my previous question. The parser itself works fairly well.
Now obviously a parser by itself does little good (apart from static analysis). I would like to apply transformations to the AST and then compile it back to source code. Applying the transformations isn't much of a problem, a normal Visitor pattern should do.
What my problem currently is, is how to compile the AST back to source. There are basically two possibilities I see:
Compile the code using some predefined scheme
Keep the formatting of the original code and apply 1. only on Nodes that were changed.
For now I would like to concentrate on 1. as 2. seems pretty hard to accomplish (but if you got tips concerning that, I would like to hear them).
But I'm not really sure which design pattern can be used to compile the code. The easiest way I see to implement this, is to add a ->compile method to all Nodes. The drawback I see here, is that it would be pretty hard to change the formatting of the generated output. One would need to change the Nodes itself in order to do that. Thus I'm looking for a different solution.
I have heard that the Visitor pattern can be used for this, too, but I can't really imagine how this is supposed to work. As I understand the visitor pattern you have some NodeTraverser that iterates recursively over all Nodes and calls a ->visit method of a Visitor. This sounds pretty promising for node manipulation, where the Visitor->visit method could simply change the Node it was passed, but I don't know how it can be used for compilation. An obvious idea would be to iterate the node tree from leaves to root and replace the visited nodes with source code. But this somehow doesn't seem a very clean solution?
The problem of converting an AST back into source code is generally called "prettyprinting". There are two subtle variations: regenerating the text matching the original as much as possible (I call this "fidelity printing"), and (nice) prettyprinting, which generates nicely formatted text. And how you print matters
depending on whether coders will be working on the regenerated code (they often want fidelity printing) or your only
intention is to compile it (at which point any legal prettyprinting is fine).
To do prettyprinting well requires usually more information than a classic parser collects, aggravated by the fact that most parser generators don't support this extra-information collection. I call parsers that collect enough information to do this well "re-engineering parsers". More details below.
The fundamental way prettyprinting is accomplished is by walking the AST ("Visitor pattern" as you put it), and generating text based on the AST node content. The basic trick is: call children nodes left-to-right (assuming that's the order of the original text) to generate the text they represent, interspersing additional text as appropriate for this AST node type. To prettyprint a block of statements you might have the following psuedocode:
PrettyPrintBlock:
Print("{"}; PrintNewline();
Call PrettyPrint(Node.children[1]); // prints out statements in block
Print("}"); PrintNewline();
return;
PrettyPrintStatements:
do i=1,number_of_children
Call PrettyPrint(Node.children[i]); Print(";"); PrintNewline(); // print one statement
endo
return;
Note that this spits out text on the fly as you visit the tree.
There's a number of details you need to manage:
For AST nodes representing literals, you have to regenerate the literal value. This is harder than it looks if you want the answer to be accurate. Printing floating point numbers without losing any precision is a lot harder than it looks (scientists hate it when you damage the value of Pi). For string literals, you have to regenerate the quotes and the string literal content; you have to be careful to regenerate escape sequences for characters that have to be escaped. PHP doubly-quoted string literals may be a bit more difficult, as they are not represented by single tokens in the AST. (Our PHP Front End (a reengineering parser/prettyprinter represents them essentially as an expression that concatenates the string fragments, enabling transformations to be applied inside the string "literal").
Spacing: some languages require whitespace in critical places. The tokens ABC17 42 better not be printed as ABC1742, but it is ok for the tokens ( ABC17 ) to be printed as (ABC17). One way to solve this problem is to put a space wherever it is legal, but people won't like the result: too much whitespace. Not an issue if you are only compiling the result.
Newlines: languages that allow arbitrary whitespace can technically be regenerated as a single line of text. People hate this, even if you are going to compile the result; sometimes you have to look at the generated code and this makes it impossible. So you need a way to introduce newlines for AST nodes representing major language elements (statements, blocks, methods, classes, etc.). This isn't usually hard; when visiting a node representing such a construct, print out the construct and append a newline.
You will discover, if you want users to accept your result, that you will have to preserve some properties of the source text that you wouldn't normally think to store
For literals, you may have to regenerate the radix of the literal; coders having entered a number as a hex literal are not happy when you regenerate the decimal equivalent even though it means exactly the same thing. Likewise strings have to have the "original" quotes; most languages allow either " or ' as string quote characters and people want what they used originally. For PHP, which quote you use matters, and determines which characters in the string literal has to be escaped.
Some languages allow upper or lower case keywords (or even abbreviations), and upper and lower case variable names meaning the same variable; again the original authors typically want their original casing back. PHP has funny characters in different type of identifiers (e.g., "$") but you'll discover that it isn't always there (see $ variables in literal strings). Often people want their original layout formatting; to do this you have to store at column-number information for concrete tokens, and have prettyprinting rules about when to use that column-number data to position prettyprinted text where in the same column when possible, and what to do if the so-far-prettyprinted line is filled past that column.
Comments: Most standard parsers (including the one you implemented using the Zend parser, I'm pretty sure) throw comments away completely. Again, people hate this, and will reject a prettyprinted answer in which comments are lost. This is the principal reason that some prettyprinters attempt to regenerate code by using the original text
(the other is to copy the original code layout for fidelity printing, if you didn't capture column-number information). IMHO, the right trick is to capture the comments in the AST, so that AST transformations can inspect/generate comments too, but everybody makes his own design choice.
All of this "extra" information is collected by a good reenginering parser. Conventional parsers usually don't collect any of it, which makes printing acceptable ASTs difficult.
A more principled approach distinguishes prettyprinting whose purpose is nice formatting, from fidelity printing whose purpose is to regenerate the text to match the original source to a maximal extent. It should be clear that at the level of the terminals, you pretty much want fidelity printing. Depending on your purpose, you can pretty print with nice formatting, or fidelity printing. A strategy we use is to default to fidelity printing when the AST hasn't been changed, and prettyprinting where it has (because often the change machinery doesn't have any information about column numbers or number radixes, etc.). The transformations stamp the AST nodes that are newly generated as "no fidelity data present".
An organized approach to prettyprinting nicely is to understand that virtually all text-based programming language are rendered nicely in terms of rectangular blocks of text. (Knuth's TeX document generator has this idea, too). If you have some set of text boxes representing pieces of the regenerated code (e.g., primitive boxes generated directly for the terminal tokens), you can then imagine operators for composing those boxes: Horizontal composition (stack one box to the right of another), Vertical (stack boxes on top of each other; this in effect replaces printing newlines), Indent (Horizontal composition with a box of blanks), etc. Then you can construct your prettyprinter by building and composing text boxes:
PrettyPrintBlock:
Box1=PrimitiveBox("{"); Box2=PrimitiveBox("}");
ChildBox=PrettyPrint(Node.children[1]); // gets box for statements in block
ResultBox=VerticalBox(Box1,Indent(3,ChildBox),Box2);
return ResultBox;
PrettyPrintStatements:
ResultBox=EmptyBox();
do i=1,number_of_children
ResultBox=VerticalBox(ResultBox,HorizontalBox(PrettyPrint(Node.children[i]); PrimitiveBox(";")
endo
return;
The real value in this is any node can compose the text boxes produced by its children in arbitrary order with arbitrary intervening text. You can rearrange huge blocks of text this way (imagine VBox'ing the methods of class in method-name order). No text is spit out as encountered; only when the root is reached, or some AST node where it is known that all the children boxes have been generated correctly.
Our DMS Software Reengineering Toolkit uses this approach to prettyprint all the languages it can parse (including PHP, Java, C#, etc.). Instead of attaching the box computations to AST nodes via visitors, we attach the box computations in a domain-specific text-box notation
H(...) for Horizontal boxes
V(....) for vertical boxes
I(...) for indented boxes)
directly to the grammar rules, allowing us to succinctly express the grammar (parser) and the prettyprinter ("anti-parser") in one place. The prettyprinter box rules are compiled automatically by DMS into a visitor. The prettyprinter machinery has to be smart enough to understand how comments play into this, and that's frankly a bit arcane but you only have to do it once. An DMS example:
block = '{' statements '}' ; -- grammar rule to recognize block of statements
<<PrettyPrinter>>: { V('{',I(statements),'}'); };
You can see a bigger example of how this is done for Wirth's Oberon programming language PrettyPrinter showing how grammar rules and prettyprinting rules are combined. The PHP Front End looks like this but its a lot bigger, obviously.
A more complex way to do prettyprinting is to build a syntax-directed translator (means,
walk the tree and build text or other data structures in tree-visted order) to produce text-boxes in a special text-box AST. The text-box AST is then prettyprinted by another tree walk, but the actions for it are basically trivial: print the text boxes.
See this technical paper: Pretty-printing for software reengineering
An additional point: you can of course go build all this machinery yourself. But the same reason that you choose to use a parser generator (its a lot of work to make one, and that work doesn't contribute to your goal in an interesting way) is the same reason you want to choose an off-the-shelf prettyprinter generator. There are lots of parser generators around. Not many prettyprinter generators. [DMS is one of the few that has both built in.]