Commit graph

5580 commits

Author SHA1 Message Date
Alexandre Flament
a1a492baed
Merge pull request #2641 from dalf/disable_http_by_default
[mod] by default allow only HTTPS, not HTTP
2021-03-12 19:21:46 +01:00
Alexandre Flament
af3e969c5a
Merge pull request #2642 from return42/fix-apkmirror
[fix] APKMirror engine - update xpath selectors and fix img_src
2021-03-11 09:48:31 +01:00
Alexandre Flament
8b650e6a2d
Merge pull request #2643 from return42/fix-makefile
[fix] make targets engines.languages and useragents.update
2021-03-11 09:44:30 +01:00
Alexandre Flament
cb04d42806 [mod] oscar: update README.rst 2021-03-11 09:33:04 +01:00
Alexandre Flament
86912e2272 [mod] oscar: get bootstrap and typeahead from NPM 2021-03-11 09:33:01 +01:00
Alexandre Flament
44407353ef [mod] oscar: get leaflet and jquery from NPM
easy to upgrade (package.json)
2021-03-11 09:32:22 +01:00
Alexandre Flament
c7133efb12 [mod] oscar: move compiled files to the src directory 2021-03-10 19:28:51 +01:00
Alexandre Flament
eda3b513ac [mod] oscar: remove polyfills for Internet Explorer 2021-03-10 19:01:16 +01:00
Alexandre Flament
1268910274 [mod] oscar: remove unused images 2021-03-10 19:01:16 +01:00
Alexandre Flament
bdb41bea7b [mod] theme: remove require-2.1.15.min.js
See https://github.com/requirejs/requirejs/issues/1816

requirejs loads one file: leaflet.

This commit:
* removes requirejs
* load leaflet using <script src...> HTML tag in searx/templates/oscar/base.html
2021-03-10 19:01:15 +01:00
Alexandre Flament
2f3d5ec2af [mod] oscar: upgrade npm dependencies 2021-03-10 19:01:14 +01:00
Markus Heiser
96422e5c9f [fix] APKMirror engine - update xpath selectors and fix img_src
BTW: make the code slightly more readable

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-09 08:34:57 +01:00
Alexandre Flament
ccf5ac9801
Merge pull request #2640 from return42/fix-yahoo-news
[fix] rewrite Yahoo-News engine
2021-03-08 19:03:41 +01:00
Markus Heiser
c0d3183593 [fix] make targets engines.languages and useragents.update
Since [PR 2600] is merged the update scripts for languages and useragent has
been moved to folder:

    searx_extra/update/

[PR 2600] https://github.com/searx/searx/pull/2600

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-08 16:14:33 +01:00
Markus Heiser
d2faea423a [fix] rewrite Yahoo-News engine
Many things have been changed since last review of this engine.  This patch fix
xpath selectors, implements suggestion and is a complete review / rewrite of the
engine.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-03-08 11:43:34 +01:00
Alexandre Flament
99e0651cea [mod] by default allow only HTTPS, not HTTP
Related to https://github.com/searx/searx/pull/2373
2021-03-08 11:35:08 +01:00
Michael Ilsaas
5549d58de3 Add Solid Torrents engine 2021-03-07 18:14:30 +01:00
Noémi Ványi
0d8b369b5b
Merge pull request #2615 from searx/engine-data
Add ability to send engine data to subsequent requests
2021-03-06 12:17:10 +01:00
Adam Tauber
44f4a9d49a [enh] add ability to send engine data to subsequent requests 2021-03-06 12:12:35 +01:00
Alexandre Flament
87f4cc4a9e
Merge pull request #2631 from searx/update_data_update_languages.py
Update searx.data - update_languages.py
2021-03-06 10:03:00 +01:00
Alexandre Flament
63e696372b
Merge pull request #2634 from return42/fedora
[mod] LXC switch to Fedora 33 / Fedora 31 reached its EOL
2021-03-05 21:56:18 +01:00
Alexandre Flament
70c38a93d0
Merge pull request #2635 from return42/einit
[mod] don't dump traceback of SearxEngineResponseException on init
2021-03-05 21:55:00 +01:00
Markus Heiser
4845183128 [mod] don't dump traceback of SearxEngineResponseException on init
When initing engines a "SearxEngineResponseException" is logged very verbose,
including full traceback information:

    ERROR:searx.engines:yggtorrent engine: Fail to initialize
    Traceback (most recent call last):
      File "share/searx/searx/engines/__init__.py", line 293, in engine_init
        init_fn(get_engine_from_settings(engine_name))
      File "share/searx/searx/engines/yggtorrent.py", line 42, in init
        resp = http_get(url, allow_redirects=False)
      File "share/searx/searx/poolrequests.py", line 197, in get
        return request('get', url, **kwargs)
      File "share/searx/searx/poolrequests.py", line 190, in request
        raise_for_httperror(response)
      File "share/searx/searx/raise_for_httperror.py", line 60, in raise_for_httperror
        raise_for_captcha(resp)
      File "share/searx/searx/raise_for_httperror.py", line 43, in raise_for_captcha
        raise_for_cloudflare_captcha(resp)
      File "share/searx/searx/raise_for_httperror.py", line 30, in raise_for_cloudflare_captcha
        raise SearxEngineCaptchaException(message='Cloudflare CAPTCHA', suspended_time=3600 * 24 * 15)
    searx.exceptions.SearxEngineCaptchaException: Cloudflare CAPTCHA, suspended_time=1296000

For SearxEngineResponseException this is not needed.  Those types of exceptions
can be a normal use case.  E.g. for CAPTCHA errors like shown in the example
above. It should be enough to log a warning for such issues:

    WARNING:searx.engines:yggtorrent engine: Fail to initialize // Cloudflare CAPTCHA, suspended_time=1296000

closes: #2612

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-05 17:26:22 +01:00
Markus Heiser
0305775e29 [mod] LXC switch to Fedora 33 / Fedora 31 reached its EOL
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-05 13:17:47 +01:00
Alexandre Flament
0165e14a7f
Merge pull request #2632 from searx/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-03-05 11:59:44 +01:00
Alexandre Flament
152f6fc1da
Merge pull request #2630 from searx/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2021-03-05 11:59:20 +01:00
Alexandre Flament
d7fb55a774
Merge pull request #2633 from searx/update_data_update_currencies.py
Update searx.data - update_currencies.py
2021-03-05 11:59:05 +01:00
dalf
1e8b846954 Update searx.data - update_currencies.py 2021-03-05 10:56:57 +00:00
dalf
2f8a708481 Update searx.data - update_wikidata_units.py 2021-03-05 10:56:49 +00:00
dalf
d9dc3376d0 Update searx.data - update_languages.py 2021-03-05 10:56:46 +00:00
dalf
2857473553 Update searx.data - update_ahmia_blacklist.py 2021-03-05 10:56:33 +00:00
Alexandre Flament
10ecc303c9
Fix integration.yml
Don't run twice the workflow on PR opened from the same repository.
2021-03-05 11:54:45 +01:00
Alexandre Flament
245f8626dd
Fix data-update.yml, run once a month 2021-03-05 11:12:31 +01:00
Alexandre Flament
d9a35fc28e
Fix data-update.yml 2021-03-05 10:24:54 +01:00
Alexandre Flament
917dff3aab
Update data-update.yml
Allow to manually running the workflow

See:
* https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow
* https://docs.github.com/en/actions/reference/events-that-trigger-workflows#workflow_dispatch
2021-03-05 10:16:59 +01:00
Alexandre Flament
b97273df6b
Update data-update.yml
Remove cache
2021-03-05 09:49:39 +01:00
Alexandre Flament
aaae9a209e
Merge pull request #2600 from dalf/searx-extra
Add searx_extra package
2021-03-05 09:43:39 +01:00
Alexandre Flament
b8cd326464 Add searx_extra package
Split the utils directory into:
* searx_extra contains update scripts, standalone_searx.py
* utils contains the files to build and setup searx.
2021-03-04 11:59:14 +01:00
Alexandre Flament
1d10ae175c
Merge pull request #2618 from thezeroalpha/master
Fix security vulnerabilities in provided nginx configuration
2021-03-04 11:27:03 +01:00
Alexandre Flament
6ba37777f7
Merge pull request #2623 from return42/fix-centos7
[fix] add package `which` to CentOS-7 boilerplate
2021-03-04 11:19:05 +01:00
Alexandre Flament
d26261c5ab
Merge pull request #2619 from return42/drop-ubu1604
[mod] Drop Ubuntu 16.04 (Xenial Xerus) support
2021-03-04 11:18:21 +01:00
Alexandre Flament
4c2a8aea39
Merge pull request #2620 from return42/fix-git
[fix] support git versions <v2.22
2021-03-04 11:17:24 +01:00
Alex Balgavy
8736f5bd70 Use $host in nginx morty.conf template 2021-03-04 11:16:27 +01:00
Alexandre Flament
aac37f288f
Merge pull request #2593 from dalf/update-autocomplete
Update autocomplete
2021-03-04 10:51:09 +01:00
Noémi Ványi
111d38cd8f
Merge pull request #2621 from return42/fix-searx.sh
[fix] utils/serax.sh create_pyenv() - drop duplicate 'pip install .'
2021-03-03 20:08:37 +01:00
Markus Heiser
c355bc3481 [fix] add package 'which' to CentOS-7 boilerplate
Newer CentOS-7 images from https://images.linuxcontainers.org do no longer
include the which command.

Issue:

    $ sudo -H ./utils/lxc.sh cmd searx-centos7 ./utils/filtron.sh install all
    INFO:  [searx-centos7] ./utils/filtron.sh install all
    ...
    Install Go in user's HOME
    -------------------------

    download and install go binary ..
    ...
    -bash: line 1: which: command not found
    -->|ERROR - Go Installation not found in PATH!?!
    -bash: line 2: which: command not found

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 19:37:03 +01:00
Markus Heiser
2e58988191 [fix] utils/serax.sh create_pyenv() - drop duplicate 'pip install .'
The wrong and unnecessary `pip install .` is executed in /usr/local/searx and is
responsible for the error message:

    ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /usr/local/searx

The correct pip-install comes right after changing to `cd ${SEARX_SRC}`.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 18:57:20 +01:00
Markus Heiser
237b1cfdb3 [fix] support git versions <v2.22
LTS distros like Ubuntu 18.04 do not ship a up-to-date version of git.::

    $ sudo -H ./utils/lxc.sh cmd searx-ubu1804 git --version
    ...
    git version 2.17.1

The option `--show-current` was added in git v2.22, the alternative to this
option is::

    git rev-parse --abbrev-ref HEAD

Issue when using option `--show-current`::

    [searx-ubu1804] Clone searx sources
    [searx-ubu1804] -------------------
    [searx-ubu1804]
    [searx-ubu1804] error: unknown option `show-current'
    [searx-ubu1804] usage: git branch [<options>] [-r | -a] [--merged | --no-merged]
    ....

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 18:22:41 +01:00
Markus Heiser
daa4b676df [mod] Drop Ubuntu 16.04 (Xenial Xerus) support
EOL of Ubuntu 16.04 (Xenial Xerus) is in April 2021 but we do not support 16.04
since a longer time.  Issues are comming from dependencies (e.g. pip)::

    [searx-ubu1604]   |searx| SyntaxError: invalid syntax
    [searx-ubu1604]   |searx| Traceback (most recent call last):
    [searx-ubu1604]   |searx|   File "/usr/local/searx/searx-pyenv/bin/pip", line 7, in <module>
    [searx-ubu1604]   |searx|     from pip._internal.cli.main import main
    [searx-ubu1604]   |searx|   File "/usr/local/searx/searx-pyenv/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60
    [searx-ubu1604]   |searx|     sys.stderr.write(f"ERROR: {exc}")
    [searx-ubu1604]   |searx|                                    ^
    [searx-ubu1604]   |searx| SyntaxError: invalid syntax

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 17:52:50 +01:00
Alex Balgavy
6b59800dc6 Fix security vulnerabilities in suggested nginx configuration
The suggested configurations for nginx found in the documentation and
templates lead to vulnerabilities allowing host spoofing [1] and path
traversal [2], as reported by Gixy [3]. This commit fixes those issues.

[1] https://github.com/yandex/gixy/blob/master/docs/en/plugins/hostspoofing.md
[2] https://github.com/yandex/gixy/blob/master/docs/en/plugins/aliastraversal.md
[3] https://github.com/yandex/gixy
2021-03-03 12:34:22 +01:00