TL;DR Summary: I need a single command-line application which I can use to get synonyms and other related words. It needs to be multi-lingual and works cross platform. Can anyone suggest a suitable program for me, or help me with the ones I've already found? Thanks.
Longer version:
I've been tasked with writing a system in PHP that can come up with alternative suggestions for words entered by the user. I need to find a thesaurus application / API or similar which I can use to generate these suggestions.
Importantly, it needs to be multilingual (English, Danish, French and German). This rules out most of the software that I managed to find using Google. It also needs to be cross-platform (it needs to work on Linux and Windows).
My research has let me to two promising candidates: WordNet and Stardict.
I've been focusing on WordNet so far, calling it from PHP using the shell_exec() function, and I've managed to use it to create a very promising prototype PHP page, but so far in English only. I'm struggling with how to use it multi-lingual.
The Wordnet site has external links to Wordnet projects in other language (eg DanNet for Danish), but although they're often called Wordnet, they seem to use a variety of database formats and software, which makes them unsuitable for me. I need a consistent interface that I can call from my PHP program.
Stardict looked more promising from that perspective: they provide dictionaries in many languages in a standard DB format for the one application.
But the down-side of Stardict is that its primarily a GUI app. Calling it from the command-line launches the GUI. There is apparently a command-line version (SDCV), but it seems quite out of date (last update 2006), and only for Linux.
Can anyone help me with my problems with either of these programs? Or else, can anyone suggest any other alternative software or API that I could use?
Many thanks.
You could try to leverage PostgreSQL's full text search functionality:
http://www.postgresql.org/docs/9.0/static/textsearch.html
You can configure it with any of the available languages and all sorts of collations to fit your needs. PostgreSQL 9.1 adds some extra collation functionality that you may want to look into if the approach seems reasonable.
The basic steps would be (for each language):
Create the needed table (collated appropriately). For our sake, a single column is enough, e.g.:
create table dict_en (
word text check (word = lower(word)) primary key
);
Fetch the needed dictionary/thesaurus files (those from aspell/Open-Office should work).
Configure text search (see link above, namely section 12.6) using the relevant files.
Insert the whole dictionary into the table. (Surely there's a csv file somewhere...)
And finally index the vector, e.g.:
create index on dict_en using gin (to_tsvector('english', word));
You can now run queries that use this index:
-- Find words related to `:word`
select word
from dict_en
where to_tsvector('english', word) ## plainto_tsquery('english', :word)
and word <> :word;
You might need to create a separate database or schema for each language, and add an additional field (tsvector) if Postgres refuses to index the expression because of the language parameter. (I read the full text docs a long time ago). The details on this would be in section 12.2, and I'm sure you'll know how to adjust the above if this is the case.
Whichever the implementation details, though, I believe the approach should work.
There is a PHP example for a thesaurus API usage here...
http://thesaurus.altervista.org/testphp
Available for Italian, English, French, Deutsch, Spanish and Portuguese.
This seems to be an option, though I'm not sure whether its multilingual:
http://developer.dictionary.com/products/synonyms
I also found the following site which does something similar to your end goal, maybe you could try contacting the owner and ask him how he did it:
http://www.synonymlab.com/
Related
I'm writing a webapp in german, so all buttons,text,tooltips etc. are in german for now. But I want to use some kind of template file for the webapp so I can quick change to another language if needed. I thought about textfile that I explode with "\n" and load into a sessionvariable to have always all text the user will need in his session. An other approach would be to parse such a file i.e. a XML document like this:
<?xml version="1.0"?>
<phrase>
<placeholder></placeholder>
<value></value>
</phrase>
where every field has it's own name/value that represents a textsnippet or button or whatever on the website, and then cast it into an object an cache it for everyone. I think the second approach is the best for working with multiple languages for a webapp. Anybody perhaps some pointers what I could do even better, or just post how you did this kind of languagetemplating for mutlinational webpages/webapps in the past.
Since you are looking for a translation solution, I understand you don't use a framework to develop your site, since most of them provide you with solutions to handle translations.
Most frameworks and apps I've seen in php use arrays, where the original sentence is the key and the translation is the value. So, to make easier to translate it to several languages, the key is in english.
In case you use gettext as suggested, or another aproach, it'll be useful also to parse your code to catch all strings to be translated automatically, since it can be a mess doing it manually when the base code grows and you want to keep up to date your translations.
Take a look at GNU Gettext, its very handy for multilanguage support.
The main idea is that you just wrap your words or phrases into a function, like
echo _g('Hello');
so you do not have any engine changes. You will have to add translation files for each language you are using.
You've come up with 2 solutions for storing the data, but I suggest you need to think further about the architecture and take a more complete view of the lifecycle of each request.
Regarding architecture: neither solution scales up to describe an extensive vocabulary very well - although for one or 2 pages it will suffice. The alternative approach, to manage a translation database (such as gettext) which might be overkill - and performs less optimally with small numbers of pages but importantly performance does not deteriorate significantly with large/multiple dictionaries. A compromise solution might be to have a dataset for each URL/language (which might be extracted from a consolidated database).
If it were me, I would not use either method you proposed for storing the data: parsing XML creates a sginficant overhead for each page request : using \n as a delimiter precludes the use of \n within a translation. Using a serialized PHP array seems to be the least expensive solution.
I'm relatively new to Magento and working on a site build for a client and they simply need a list of phrases used throughout the site to be sent to a translator. I'm a little surprised that there isn't something simple and built into Magento for easily pulling this stuff out, which is why I'm writing here now. Is there a relatively simple way to extract translation phrases from a Magento app? Something built in that might not be obvious (to me)? or some useful library out there? This includes everything used in the templates (or controllers) like so:
$this->__("Some phrase on my website...");
... as well as cases in the layout XML where the 'translate' attribute is set, etc.
And taking this one step further, I'm aware of the available translations available from Magento here: http://www.magentocommerce.com/translations -- is there something simple to make sure I'm not double-doing it for phrases that might already exist in these packs?
And further still, is there a something to pull all translatables out of the database?
And if the answer for all this turns out to be 'no', I need to be very thorough with this so any advice on pitfalls or particular spots I need to be aware of where I might not think of pulling translations from, how you might have achieved something like this before, etc. -- I would love to hear your tips. Thanks!
I know I'm late, but anyway, I’ve just uploaded an extension that does that: Language CSV Files Generator.
It only extracts strings from .php and .phtml files, I have no idea of how to get the .xml ones. Hope that someone out there could share some idea.
hope you like it
Take a look in /app/locale/(language_country)/*.csv files.
There are different solutions to get the strings from CSV files of Magento:
check the links The Ultimate Guide to Translating Magento (using Translation Memory software)
and How to translate Magento using OmegaT software
I am working on a large multilingual website and I am considering different approaches for making it multilingual. The possible alternatives I can think of are:
The Gettext functions with generation of .po files
One MySQL table with the translations and a unique string ID for each text
PHP-files with arrays containing the different translations with unique string IDs
As far as I have understood the Gettext functions should be most efficient, but my requirement is that it should be possible to change a text string in the original reference language (English) without the other translations of that string automatically reverting back to English just because a couple of words changed. Is this possible with Gettext?
What is the least resource demanding solution?
Is using the Gettext functions or PHP files with arrays more or less equally resource demanding?
Any other suggestions for more efficient solutions?
A few considerations:
1. Translations
Who will be doing the translations? People that are also connected to the site? A translation agency? When using Gettext you'll be working with 'pot' (.po) files. These files contain the message ID and the message string (the translation). Example:
msgid "A string to be translated would go here"
msgstr ""
Now, this looks just fine and understandable for anyone who needs to translate this. But what happens when you use keywords, like Mike suggests, instead of full sentences? If someone needs to translate a msgid called "address_home", he or she has no clue if this is should be a header "Home address" or that it's a full sentence. In this case, make sure to add comments to the file right before you call on the gettext function, like so:
/// This is a comment that will be included in the pot file for the translators
gettext("ready_for_lost_episode");
Using xgettext --add-comments=/// when creating the .po files will add these comments. However, I don't think Gettext is ment to be used this way. Also, if you need to add comments with every text you want to display you'll a) probably make an error at some point, b) you're whole script will be filled with the texts anyway, only in comment form, c) the comments needs to be placed directly above the Gettext function, which isn't always convient, depending on the position of the function in your code.
2. Maintenance
Once your site grows (even further) and your language files along with it, it might get pretty hard to maintain all the different translations this way. Every time you add a text, you need to create new files, send out the files to translators, receive the files back, make sure the structure is still intact (eager translators are always happy to translate the syntax as well, making the whole file unusable :)), and finish with importing the new translations. It's doable, sure, but be aware with possible problems on this end with large sites and many different languages.
Another option: combine your 2nd and 3rd alternative:
Personally, I find it more useful to manage the translation using a (simple) CMS, keeping the variables and translations in a database and export the relevent texts to language files yourself:
add variables to the database (e.g.: id, page, variable);
add translations to these variables (e.g.: id, varId, language, translation);
select relevant variables and translations, write them to a file;
include the relevant language file in your site;
create your own function to display a variables text:
text('var'); or maybe something like __('faq','register','lost_password_text');
Point 3 can be as simple as selecting all the relevant variables and translations from the database, putting them in an array and writing the serlialized array to a file.
Advantages:
Maintenance. Maintaining the texts can be a lot easier for big projects. You can group variables by page, sections or other parts within your site, by simply adding a column to your database that defines to which part of the site this variable belongs. That way you can quickly pull up a list of all the variables used in e.g. the FAQ page.
Translating. You can display the variable with all the translations of all the different languages on a single page. This might be useful for people who can translate texts into multiple languages at the same time. And it might be useful to see other translations to get a feel for the context so that the translation is as good as possible. You can also query the database to find out what has been translated and what hasn't. Maybe add timestamps to keep track of possible outdated translations.
Access. This depends on who will be translating. You can wrap the CMS with a simple login to grant access to people from a translation agency if need be, and only allow them to change certain languages or even certain parts of the site. If this isn't an option you can still output the data to a file that can be manually translated and import it later (although this might come with the same problems as mentioned before.). You can add one of the translations that's already there (English or another main language) as context for the translator.
All in all I think you'll find that you'll have a lot more control over the translations this way, especially in the long run. I can't tell you anything about speed or efficiency of this approach compared to the native gettext function. But, depending on the size of the language files, I don't think it'll be a big difference. If you group the variables by page or section, you can alway include only the required parts.
After some testing I finally decided to go more or less with the lines of Alecs' combination of the second and third alternative.
Gettext problem
I tried to set up the whole gettext-system first to try it out, but it turned out to be much more complicated then I thought. The problem is that Windows and Unix systems use different language shortnames for setlocale(). For the moment I'm running my dev-server on Windows with Wamp, while the final site will run on Linux. After I went through a couple of dozen guides, forums, questions etc. and restarting the server after each modification. I couldn't get it setup properly in any easy way it seemed. Additionally gettext is not threadsafe, to update the language file the server needs to be restarted or a hack needs to be used, there is no easy way of handling different versions of language files or handling the original English text without modifying the source or using Mikes suggestion, which as Alec pointed out isn't optimal.
Solution
So I ended up with what I think is the best solution based on Alecs response:
Save all the translations in a DB with the fields; language, page, var_key, version, revision and last_modified_time - where the version is corresponds to versions of the original translation (English), while revision allows the translator to modify/correct the finalized translations within a version.
Use a kind of CMS for translation, which is connected to the DB and handles different versions and allows for an easy overview of which languages are translated, in which version and how complete the translations are.
When a revision of a version is finalized a cache files are generated - each file contains an array with only the var_key and text-translation for one language and one page and are named with the ISO 639-1 names of the languages and the page name like: lang/en_index.php These language files are then simply included and wrapped in a function t($var_key) which allows for using the DB during the development, while then changed to only use the cache files.
Performance
I never got around to test gettext, but according to the link Mike posted the difference in performance between using an array and gettext is totally acceptable for me for the benefits which a custom system gives as described above. However, I compared using an array with 20 translated text-strings in an array compared to retrieving the same 20 text-strings from a MySQL DB. It turned out that using an array included from a file was aeound 6 times faster than retrieving all the 20 strings at the same time from the MySQL DB. It was no really scientific benchmark and the results may surely vary on different systems and setups, but it clearly shows exactly what I expected - that it would be much slower using a DB than using an array directly, which is why I choose to generate cache-files for the array instead of using the DB.
As a comparison I also tested how fast it was to only output simple echos with the same text. It turned out to be around 20 times faster than using arrays from an included file, but well - then it is not possible to translate without having different versions of the page for different languages, which defies the purpose of dynamic pages. Then it is better to also use a good cachesystem.
Performance test source files:
PHP: http://pastie.org/964082
MySQL table: http://pastie.org/964115
It is surely not perfect, but at least creates an idea about the performance differences.
Rather than having to use the English text as the keys you could arbitrarily do this but also provide english translations i.e.
gettext key is 'hello'
You then have your various language translations of this and an english translation of this that is also 'hello', then if you want to update the english version of the string you can leave the key alone and just update the english translation.
I'm developing a website that will be available in different languages. It is a LAMP (Linux, Apache, MySQL, PHP) setup, and it makes use of Smarty, mostly for the template engine.
The way we currently translate is by a self-written smarty plugin, which will recognize certain tags in the HTML files, and will find the corresponding tag in an earlier defined language file.
The HTML could look as follows:
<p>Hi, welcome to $#gamedesc;!</p>
And the language file could look like this:
gamedesc:Poing 2009$;
welcome:this is another tag$;
Which would then output
<p>Hi, welcome to Poing 2009!</p>
This system is very basic, but it is pretty hard to control, if I f.e. would like to keep track of what has been translated so far, or give certain users the rights to translate only certain tags.
I've been looking at some alternative ways to approach this, by either replacing the text-file with XML files which could store some extra meta-data, or by perhaps storing all the texts in the database, and retrieving it there.
My question is, what would be the best way to make this system both maintainable and perform well with high user-traffic? Are there perhaps any (lightweight) plugins I could take a look at?
You could give a shot at gettext. It is the way it is done in most C/C++ linux applications and it is an extension to PHP too. The idea is not very different from what you're already doing, but there are tools that ease the mantainance of translations (i.e. poedit).
For user rights to translations, gettext won't be of much help, I think you'll need to do it on your own or look at some frameworks if they have smarter solutions.
Maybe taking a look to gettext lib could help you get some hints http://php.net/manual/en/book.gettext.php hope it helps!
You will need to have a table in your database that you can use to store strings of text, each with an composite ID. the composite ID will be made up of language ID and text node ID.
You will need to give the user a chance to select a preferred language. You should make sure that you either have a default "this has not been translated" for every language you use, or a default language that your entire site can be vied in.
For every bit of text with in your web site, rather then store the text with in the page, you just assign it an ID.
When serving the page, look up the text node ID and preferred language ID and load that string of text, or the string for the default.
in our project, http://pkp.sfu.ca/ojs, we use XML files to store translation key-value pairs. Browse our code: http://github.com/pkp/pkp-lib/blob/master/classes/i18n/PKPLocale.inc.php
We use that class to read the XML files for each locale and in our code we use Locale::translate('locale.key.name');. Similar to gettext, but using an XML file for easier updating.
Looking around at web stuff today I came across this website: http://translateth.is/
It looks simple to use... copy paste in some javascript.
Im writing a php script to compare the similarity of 2 strings. This works pretty good at the moment, but what I would like to do is match words when one is a synonym of the first.
Any thoughts?
You might want to try looking for a thesaurus service that allows you to query the synonyms for a word and have it return an XML list of synonyms.
Here is something to look at: http://nbii-thesaurus.ornl.gov/thesaurus/
I don't know if this would be helpful for you but time ago I have been working on a PHP (CodeIgniter) library for Google Search that gets related terms by using the ~ on searches.
Maybe you can digg on the source code codeigniter-googlesearch-api
Formally aren't synonymous but depending on the application that you have in mind it could be useful (for example for SEO purposes).
As a side note, if you put ~term in Google, then it will bold you the terms that are related. Try it with ~investment for example.