forked from Ponysearch/Ponysearch
[mod] add locale.get_engine_locale to get predictable results
The match_language function sometimes returns incorrect results which is why a
new function get_engine_locale is required.
A bugfix of the match_language is not easily possible, because there is almost
no documentation for it and already the call parameters are undefined. E.g. the
function processes values like the ones from yahoo::
"yahoo": [
"ar",
...
"zh_chs",
"zh_cht"
]
The get_engine_locale has been documented in detail, there is a clear
description of the assumptions as well as the requirements and approximation
rules (read doc-string for more details)::
Argument ``engine_locales`` is a python dict that maps *SearXNG locales* to
corresponding *engine locales*:
<engine>: {
# SearXNG string : engine-string
'ca-ES' : 'ca_ES',
'fr-BE' : 'fr_BE',
'fr-CA' : 'fr_CA',
'fr-CH' : 'fr_CH',
'fr' : 'fr_FR',
...
'pl-PL' : 'pl_PL',
'pt-PT' : 'pt_PT'
}
.. hint::
The *SearXNG locale* string has to be known by babel!
In the following you will find a comparison:
>>> import babel.languages
>>> from searx.utils import match_language
>>> from searx.locales import get_engine_locale
Assume we have an engine that supports the follwoing locales:
>>> lang_list = {
... "zh-CN": "zh_CN",
... "zh-HK": "zh_HK",
... "nl-BE": "nl_BE",
... "fr-CA": "fr_CA",
... }
Assumption:
A. When a user selects a language the results should be optimized according to
the selected language.
B. When user selects a language and a territory the results should be
optimized with first priority on territory and second on language.
----
Example: (Assumption A.)
A user selects region 'zh-TW' which should end in zh_HK
hint:
CN is 'Hans' and HK ('Hant') fits better to TW ('Hant')
>>> get_engine_locale('zh-TW', lang_list)
'zh_HK'
>>> lang_list[match_language('zh-TW', lang_list)]
'zh_CN'
----
Example: (Assumption A.)
A user selects only the language 'zh' which should end in CN
>>> get_engine_locale('zh', lang_list)
'zh_CN'
>>> lang_list[match_language('zh', lang_list)]
'zh_CN'
----
Example: (Assumption B.)
A user selects region 'fr-BE' which should end in nl-BE
hint:
priority should be on the territory the user selected. If the user
prefers 'fr' he will select 'fr' without a region tag.
>>> get_engine_locale('fr-BE', lang_list, default='unknown')
'nl_BE'
>>> match_language('fr-BE', lang_list, fallback='unknown')
'fr-CA'
----
Example: (Assumption A.)
A user selects only the language 'fr' which should end in fr_CA
>>> get_engine_locale('fr', lang_list)
'fr_CA'
>>> lang_list[match_language('fr', lang_list)]
'fr_CA'
----
The difference in priority on the territory is best shown with a engine that
supports the following locales:
>>> lang_list = {
... "fr-FR": "fr_FR",
... "fr-CA": "fr_CA",
... "en-GB": "en_GB",
... "nl-BE": "nl_BE",
... }
----
Example: (Assumption A.)
A user selects only a language
>>> get_engine_locale('en', lang_list)
'en_GB'
>>> match_language('en', lang_list)
'en-GB'
hint: the engine supports fr_FR and fr_CA since no territory is given, fr_FR
takes priority ..
>>> get_engine_locale('fr', lang_list)
'fr_FR'
>>> lang_list[match_language('fr', lang_list)]
'fr_FR'
----
Example: (Assumption B.)
A user selects region 'fr-BE' which should end in nl-BE
>>> get_engine_locale('fr-BE', lang_list)
'nl_BE'
>>> lang_list[match_language('fr-BE', lang_list)]
'fr_FR'
----
If the user selects a language and there are two locales like the following:
>>> lang_list = {
... "fr-BE": "fr_BE",
... "fr-CH": "fr_CH",
... }
>>>
>>> get_engine_locale('fr', lang_list)
'fr_BE'
>>> lang_list[match_language('fr', lang_list)]
'fr_BE'
Looks like both functions return the same value, but match_language depends on the
order of the dictionary (which is not predictable):
>>> lang_list = {
... "fr-CH": "fr_CH",
... "fr-BE": "fr_BE",
... }
>>> get_engine_locale('fr', lang_list)
'fr_BE'
>>> lang_list[match_language('fr', lang_list)]
'fr_CH'
>>>
The get_engine_locale selects the locale by looking at the "population percent"
and this percentage has an higher amount in BE (68.%) compared to CH (21%)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser
parent
75bb8c45d0
commit
9ae409a05a
1 changed files with 137 additions and 0 deletions
137
searx/locales.py
137
searx/locales.py
|
|
@ -10,6 +10,8 @@ import pathlib
|
||||||
|
|
||||||
from babel import Locale
|
from babel import Locale
|
||||||
from babel.support import Translations
|
from babel.support import Translations
|
||||||
|
import babel.languages
|
||||||
|
import babel.core
|
||||||
import flask_babel
|
import flask_babel
|
||||||
import flask
|
import flask
|
||||||
from flask.ctx import has_request_context
|
from flask.ctx import has_request_context
|
||||||
|
|
@ -150,3 +152,138 @@ def locales_initialize(directory=None):
|
||||||
LOCALE_NAMES[tag] = get_locale_descr(locale, dirname)
|
LOCALE_NAMES[tag] = get_locale_descr(locale, dirname)
|
||||||
if locale.text_direction == 'rtl':
|
if locale.text_direction == 'rtl':
|
||||||
RTL_LOCALES.add(tag)
|
RTL_LOCALES.add(tag)
|
||||||
|
|
||||||
|
|
||||||
|
def get_engine_locale(searxng_locale, engine_locales, default=None):
|
||||||
|
"""Return engine's language (aka locale) string that best fits to argument
|
||||||
|
``searxng_locale``.
|
||||||
|
|
||||||
|
Argument ``engine_locales`` is a python dict that maps *SearXNG locales* to
|
||||||
|
corresponding *engine locales*:
|
||||||
|
|
||||||
|
<engine>: {
|
||||||
|
# SearXNG string : engine-string
|
||||||
|
'ca-ES' : 'ca_ES',
|
||||||
|
'fr-BE' : 'fr_BE',
|
||||||
|
'fr-CA' : 'fr_CA',
|
||||||
|
'fr-CH' : 'fr_CH',
|
||||||
|
'fr' : 'fr_FR',
|
||||||
|
...
|
||||||
|
'pl-PL' : 'pl_PL',
|
||||||
|
'pt-PT' : 'pt_PT'
|
||||||
|
}
|
||||||
|
|
||||||
|
.. hint::
|
||||||
|
|
||||||
|
The *SearXNG locale* string has to be known by babel!
|
||||||
|
|
||||||
|
If there is no direct 1:1 mapping, this functions tries to narrow down
|
||||||
|
engine's language (locale). If no value can be determined by these
|
||||||
|
approximation attempts the ``default`` value is returned.
|
||||||
|
|
||||||
|
Assumptions:
|
||||||
|
|
||||||
|
A. When user select a language the results should be optimized according to
|
||||||
|
the selected language.
|
||||||
|
|
||||||
|
B. When user select a language and a territory the results should be
|
||||||
|
optimized with first priority on terrirtory and second on language.
|
||||||
|
|
||||||
|
First approximation rule (*by territory*):
|
||||||
|
|
||||||
|
When the user selects a locale with terrirtory (and a language), the
|
||||||
|
territory has priority over the language. If any of the offical languages
|
||||||
|
in the terrirtory is supported by the engine (``engine_locales``) it will
|
||||||
|
be used.
|
||||||
|
|
||||||
|
Second approximation rule (*by language*):
|
||||||
|
|
||||||
|
If "First approximation rule" brings no result or the user selects only a
|
||||||
|
language without a terrirtory. Check in which territories the language
|
||||||
|
has an offical status and if one of these territories is supported by the
|
||||||
|
engine.
|
||||||
|
|
||||||
|
"""
|
||||||
|
# pylint: disable=too-many-branches
|
||||||
|
|
||||||
|
engine_locale = engine_locales.get(searxng_locale)
|
||||||
|
|
||||||
|
if engine_locale is not None:
|
||||||
|
# There was a 1:1 mapping (e.g. "fr-BE --> fr_BE" or "fr --> fr_FR"), no
|
||||||
|
# need to narrow language nor territory.
|
||||||
|
return engine_locale
|
||||||
|
|
||||||
|
locale = babel.Locale.parse(searxng_locale, sep='-')
|
||||||
|
|
||||||
|
# SearXNG's selected locale is not supported by the engine ..
|
||||||
|
|
||||||
|
if locale.territory:
|
||||||
|
# Try to narrow by *offical* languages in the territory (??-XX).
|
||||||
|
|
||||||
|
for official_language in babel.languages.get_official_languages(locale.territory, de_facto=True):
|
||||||
|
searxng_locale = official_language + '-' + locale.territory
|
||||||
|
engine_locale = engine_locales.get(searxng_locale)
|
||||||
|
if engine_locale is not None:
|
||||||
|
return engine_locale
|
||||||
|
|
||||||
|
# Engine does not support one of the offical languages in the territory or
|
||||||
|
# there is only a language selected without a territory.
|
||||||
|
|
||||||
|
# Now lets have a look if the searxng_lang (the language selected by the
|
||||||
|
# user) is a offical language in other territories. If so, check if
|
||||||
|
# engine does support the searxng_lang in this other territory.
|
||||||
|
|
||||||
|
if locale.language:
|
||||||
|
|
||||||
|
searxng_lang = locale.language
|
||||||
|
if locale.script:
|
||||||
|
searxng_lang += '_' + locale.script
|
||||||
|
|
||||||
|
terr_lang_dict = {}
|
||||||
|
for territory, langs in babel.core.get_global("territory_languages").items():
|
||||||
|
if not langs.get(searxng_lang, {}).get('official_status'):
|
||||||
|
continue
|
||||||
|
terr_lang_dict[territory] = langs.get(searxng_lang)
|
||||||
|
|
||||||
|
# first: check fr-FR, de-DE .. is supported by the engine
|
||||||
|
|
||||||
|
territory = locale.language.upper()
|
||||||
|
if terr_lang_dict.get(territory):
|
||||||
|
searxng_locale = locale.language + '-' + territory
|
||||||
|
engine_locale = engine_locales.get(searxng_locale)
|
||||||
|
if engine_locale is not None:
|
||||||
|
return engine_locale
|
||||||
|
|
||||||
|
# second: sort by population_percent and take first match
|
||||||
|
|
||||||
|
# drawback of "population percent": if there is a terrirtory with a
|
||||||
|
# small number of people (e.g 100) but the majority speaks the
|
||||||
|
# language, then the percentage migth be 100% (--> 100 people) but in
|
||||||
|
# a different terrirtory with more people (e.g. 10.000) where only 10%
|
||||||
|
# speak the language the total amount of speaker is higher (--> 200
|
||||||
|
# people).
|
||||||
|
#
|
||||||
|
# By example: The population of Saint-Martin is 33.000, of which 100%
|
||||||
|
# speak French, but this is less than the 30% of the approximately 2.5
|
||||||
|
# million Belgian citizens
|
||||||
|
#
|
||||||
|
# - 'fr-MF', 'population_percent': 100.0, 'official_status': 'official'
|
||||||
|
# - 'fr-BE', 'population_percent': 38.0, 'official_status': 'official'
|
||||||
|
|
||||||
|
terr_lang_list = []
|
||||||
|
for k, v in terr_lang_dict.items():
|
||||||
|
terr_lang_list.append((k, v))
|
||||||
|
|
||||||
|
for territory, _lang in sorted(terr_lang_list, key=lambda item: item[1]['population_percent'], reverse=True):
|
||||||
|
searxng_locale = locale.language + '-' + territory
|
||||||
|
engine_locale = engine_locales.get(searxng_locale)
|
||||||
|
if engine_locale is not None:
|
||||||
|
return engine_locale
|
||||||
|
|
||||||
|
# No luck: narrow by "language from territory" and "territory from language"
|
||||||
|
# does not fit to a locale supported by the engine.
|
||||||
|
|
||||||
|
if engine_locale is None:
|
||||||
|
engine_locale = default
|
||||||
|
|
||||||
|
return default
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue