Many things have been changed since last review of this engine. This patch fix
xpath selectors, implements suggestion and is a complete review / rewrite of the
engine.
Signed-off-by: Markus Heiser <markus@darmarit.de>
When initing engines a "SearxEngineResponseException" is logged very verbose,
including full traceback information:
ERROR:searx.engines:yggtorrent engine: Fail to initialize
Traceback (most recent call last):
File "share/searx/searx/engines/__init__.py", line 293, in engine_init
init_fn(get_engine_from_settings(engine_name))
File "share/searx/searx/engines/yggtorrent.py", line 42, in init
resp = http_get(url, allow_redirects=False)
File "share/searx/searx/poolrequests.py", line 197, in get
return request('get', url, **kwargs)
File "share/searx/searx/poolrequests.py", line 190, in request
raise_for_httperror(response)
File "share/searx/searx/raise_for_httperror.py", line 60, in raise_for_httperror
raise_for_captcha(resp)
File "share/searx/searx/raise_for_httperror.py", line 43, in raise_for_captcha
raise_for_cloudflare_captcha(resp)
File "share/searx/searx/raise_for_httperror.py", line 30, in raise_for_cloudflare_captcha
raise SearxEngineCaptchaException(message='Cloudflare CAPTCHA', suspended_time=3600 * 24 * 15)
searx.exceptions.SearxEngineCaptchaException: Cloudflare CAPTCHA, suspended_time=1296000
For SearxEngineResponseException this is not needed. Those types of exceptions
can be a normal use case. E.g. for CAPTCHA errors like shown in the example
above. It should be enough to log a warning for such issues:
WARNING:searx.engines:yggtorrent engine: Fail to initialize // Cloudflare CAPTCHA, suspended_time=1296000
closes: #2612
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Newer CentOS-7 images from https://images.linuxcontainers.org do no longer
include the which command.
Issue:
$ sudo -H ./utils/lxc.sh cmd searx-centos7 ./utils/filtron.sh install all
INFO: [searx-centos7] ./utils/filtron.sh install all
...
Install Go in user's HOME
-------------------------
download and install go binary ..
...
-bash: line 1: which: command not found
-->|ERROR - Go Installation not found in PATH!?!
-bash: line 2: which: command not found
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The wrong and unnecessary `pip install .` is executed in /usr/local/searx and is
responsible for the error message:
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /usr/local/searx
The correct pip-install comes right after changing to `cd ${SEARX_SRC}`.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
LTS distros like Ubuntu 18.04 do not ship a up-to-date version of git.::
$ sudo -H ./utils/lxc.sh cmd searx-ubu1804 git --version
...
git version 2.17.1
The option `--show-current` was added in git v2.22, the alternative to this
option is::
git rev-parse --abbrev-ref HEAD
Issue when using option `--show-current`::
[searx-ubu1804] Clone searx sources
[searx-ubu1804] -------------------
[searx-ubu1804]
[searx-ubu1804] error: unknown option `show-current'
[searx-ubu1804] usage: git branch [<options>] [-r | -a] [--merged | --no-merged]
....
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
EOL of Ubuntu 16.04 (Xenial Xerus) is in April 2021 but we do not support 16.04
since a longer time. Issues are comming from dependencies (e.g. pip)::
[searx-ubu1604] |searx| SyntaxError: invalid syntax
[searx-ubu1604] |searx| Traceback (most recent call last):
[searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/bin/pip", line 7, in <module>
[searx-ubu1604] |searx| from pip._internal.cli.main import main
[searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60
[searx-ubu1604] |searx| sys.stderr.write(f"ERROR: {exc}")
[searx-ubu1604] |searx| ^
[searx-ubu1604] |searx| SyntaxError: invalid syntax
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The old xpath configuration for google scholar did not work and is replaced by a
python implementation.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- unittest2 is a backport of the new features added to the unittest testing
framework in Python 2.7
- unittest2 was only needed in py2 and can be dropped now
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
