I had some trouble while playing with the Google Language API regarding the number of characters that can be sent using a GET request: Although the specification of the HTTP protocol does not specify any maximum length, URLs over 2,000 characters will not work in the most popular web browser.

Sounds okay for me, but we had also to remember that query strings have to be URLEncoded: I have large texts that need to be split in several parts so that it could sent using a GET request. So I need to calculate where to cut the text to have less than 2000 characters.

FLEX/AS3

I wanted to use encodeURI() function to calculate the size of my URLEncoded string to split it wisely. In the documentation, you can read :

Why not…

Using HTTPService, unless you specify the contentType to be application/xml, the parameters are urlencoded which makes sense, of course.

Now the fun part:

source String – 59chars :
Phrase_choc – j’adore quand un plan se déroule sans accroc!

encodeURI() – 82chars :
Phrase_choc%20-%20j’adore%20quand%20un%20plan%20se%20d%C3%A9roule%20sans%20accroc!

HTTPService parameter – 90chars:
Phrase%5Fchoc%20%2D%20j%27adore%20quand%20un%20plan%20se%20d%C3%A9roule%20sans%20accroc%21

So there’s a difference between the encoding done by HTTPService and  using encodeURI function?! HTTPService is encoding also the characters from the last two rows in the above table… is that “normal” ?  More fun ?

If you’re sending the encodedURI String via the HTTPService you will get a nice 112 chars long string :

Phrase%5Fchoc%2520%2D%2520j%27adore%2520quand%2520un%2520plan
%2520se%2520d%25C3%25A9roule%2520sans%2520accroc%21

Let’s compare with other languages.

COLDFUSION

Using URLEncodedFormat() I got the following encoded string:

UntitledPhrase%5Fchoc%20%2D%20j%27adore%20quand%20un%20plan%20se%20d%C3%A9roule%20sans%20accroc%21

same as the one from HTTPService.

JAVASCRIPT

Using encodeURI() function I got the following encoded string:

Phrase_choc%20-%20j’adore%20quand%20un%20plan%20se%20d%C3%A9roule%20sans%20accroc!

same as the one from AS3 encodeURI().

Well,  I’m finding this a bit disturbing, and the W3C is not really helping.

The same encoding method may be used for encoding characters whose use, although technically allowed in a URL, would be unwise due to problems of corruption by imperfect gateways or misrepresentation due to the use of variant character sets, or which would simply be awkward in a given environment. Because a % sign always indicates an encoded character, a URL may be made safer simply by encoding any characters considered unsafe, while leaving already encoded characters still encoded. Similarly, in cases where a larger set of characters is acceptable, % signs can be selectively and reversibly expanded.

The reserved characters shall however never be arbitrarly encoded and decoded.

Any thoughts ?

 
The Google AJAX Language API lets you translate and detect the language of blocks of text. For non-Javascript environments, it exposes also a simple RESTful interface to use in combination with JSON to decode the result. Let’s try a simple Flex example of language detection :

Demo (right click for the source code)

It’s pretty easy to achieve. Here is how to call the service

var srv:HTTPService = new HTTPService(); 
srv.url = 'http://ajax.googleapis.com/ajax/services/language/detect'; 
srv.request.v = '1.0'; /*version - might change in the future*/ 
srv.request.q = /*the text to send*/; 
srv.addEventListener(ResultEvent.RESULT, /*a resultHandler function*/); 
srv.addEventListener(FaultEvent.FAULT, /*a faultHandler function*/); 
srv.send();

The JSON Response is the following :

{
  "responseData" : {
    "language" : the-detected-language,
    "isReliable" : the-reliability-of-the-detect,
    "confidence" : the-confidence-level-of-the-detect
  },
  "responseDetails" : null | string-on-error,
  "responseStatus" : 200 | error-code
}

Okay, nice but Why am I so enthusiast about that API ?

Those last years, many of my projects involved to deal with translations in the 22 (formally 23 with the Gaelic) official EU languages. When I was thinking about adding more automation, I was always facing the problem of being sure about the language used in the document - It’s something that I already discussed with some guys at 360|Flex Europe in April. Now I could achieve this quite easily :-)

 Demo (right click for the source code)

Maybe it deserves a better explanation To be able to detect the language in a document (a MS Word document only at the moment, but I will try to support more document types later), we need to send plain text to the Google API. To do this, I’m using the Apache POI project and their HWPF extractor wrapper to extract the text, and then I split it in several parts of 300 characters (500 is the maximum supported but it makes my code freezing) and send them to the Google detection API. After recovering all of them, I compile and display the language(s) found. 

My demo is not 100% perfect but it’s working pretty well and it’s really fast. I’m using ColdFusion to upload the texts and call the POI library, and I let Flex deal with the multiple calls to the Google API. For an easier implementation, I created (or at least tried) a component detectLanguage that manage all steps from extraction to language detection.

If you need some MS Word documents for testing, you can have a look here, most of the texts are available in 22 languages since 2007. I’m waiting for you feedback and suggestions. Enjoy !