Commit graph

5607 commits

Author SHA1 Message Date
Markus Heiser
274edd523c [fix] get common used languages back (min_engines_per_lang = 13)
In commit 4d3f2f48d common used languages has been droped.  By reducing the
number of `min_engines_per_lang` from 15 to 13 we get theses languages back.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-01 13:04:24 +02:00
Markus Heiser
0fc1dcbfa5
Merge pull request #243 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-08-01 09:42:01 +00:00
Markus Heiser
33ba700c3b
Merge pull request #240 from searxng/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2021-08-01 09:22:38 +00:00
dalf
7fae5647d3 Update searx.data - update_wikidata_units.py 2021-08-01 06:24:23 +00:00
dalf
2963c83daa Update searx.data - update_currencies.py 2021-08-01 06:24:13 +00:00
dalf
4d3f2f48d2 Update searx.data - update_languages.py 2021-08-01 06:24:12 +00:00
dalf
f5389fd722 Update searx.data - update_firefox_version.py 2021-08-01 06:24:01 +00:00
Markus Heiser
d2d6234f92
Merge pull request #239 from return42/fix-doc-git
[fix] documentation - remove brand.git_url & brand.git_branch
2021-08-01 05:29:41 +00:00
Markus Heiser
5e3525a298
Merge pull request #236 from dalf/manage_without_brand
manage script: remove unused reference to utils/brand.env and .config.sh
2021-08-01 05:29:24 +00:00
Alexandre Flament
a7a181064f manage script: remove unused reference to utils/brand.env and .config.sh 2021-08-01 07:05:00 +02:00
Markus Heiser
aee613d256 [fix] documentation - remove brand.git_url & brand.git_branch
In commit 4b43775c9 the brand.git_url & brand.git_branch has been removed, with
this patch these settings are removed from the documentation.

docs/admin/engines/settings.rst
  - Remove brand.git_url & brand.git_branch

docs/dev/makefile.rst
  - Remove brand.git_url & brand.git_branch

modified   docs/dev/lxcdev.rst
  - Remove brand.git_url & brand.git_branch
  - fix searXNG to SearXNG

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-01 06:57:14 +02:00
Alexandre Flament
4ca318aad2
dependabot.yml: check npm packages 2021-07-31 09:56:57 +02:00
Alexandre Flament
38ee88b390
Merge pull request #229 from dalf/version-from-git
version based on git
2021-07-30 18:57:00 +02:00
Markus Heiser
3b97657f3b
Merge pull request #233 from searxng/dependabot/pip/master/sphinx-4.1.2
Bump sphinx from 4.1.1 to 4.1.2
2021-07-30 16:27:44 +00:00
Markus Heiser
a2fc031e7a
Merge pull request #232 from searxng/dependabot/pip/master/pylint-2.9.6
Bump pylint from 2.9.5 to 2.9.6
2021-07-30 16:22:01 +00:00
Markus Heiser
5b3eeb6e54 [mod] searx.sh - improve some messages prompted while installation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-30 14:53:09 +02:00
Markus Heiser
85e0908e4f [fix] searx.sh - get_installed_version_variables()
Fix the quotation marks in the command line to run

    python -m searx.version

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-30 14:53:09 +02:00
Markus Heiser
64d64535a2 [fix] buildenv: first unset environment / then import searx package
The first import of names from the searx package implies loading the
settings.yml.  Before this is done, the enviroment variables must be unset to
not overwrite the values from the settings.yml

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-30 14:53:09 +02:00
Alexandre Flament
4b43775c91 version based on the git repository
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH:
there are read from the git repository.

It is possible to call python -m searx.version freeze to freeze the current version.
Useful when the code is installed outside git (distro package, docker, etc...)
2021-07-30 14:40:09 +02:00
dependabot[bot]
d95720b0dc
Bump sphinx from 4.1.1 to 4.1.2
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.1.1 to 4.1.2.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.1.1...v4.1.2)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-30 07:04:19 +00:00
dependabot[bot]
146b4f4b42
Bump pylint from 2.9.5 to 2.9.6
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.5 to 2.9.6.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.5...v2.9.6)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-30 07:04:12 +00:00
Alexandre Flament
feef577f99
Merge pull request #230 from dalf/remove-zh-hant-tw
Remove zh_Hant_TW translation
2021-07-28 08:34:09 +02:00
Alexandre Flament
0b2dca088f Remove zh_Hant_TW translation
See https://github.com/searxng/searxng/issues/195#issuecomment-887518808
2021-07-27 18:54:25 +02:00
Alexandre Flament
ed0db4d61a
Merge pull request #226 from return42/hardening
[mod] hardening SearXNG instances by default (formats)
2021-07-27 11:22:18 +02:00
Alexandre Flament
24fcd7475a Revert "[brand] !!DO NOT MERGE!! this patch uses branch fix-searx.sh @return42"
This reverts commit e946c3462c.
2021-07-27 08:40:53 +02:00
Markus Heiser
beff764d74
Merge pull request #216 from return42/fix-searx.sh
normalize environment of installation tasks (shell) with YAML settings
2021-07-27 05:32:57 +00:00
Markus Heiser
80f5241387
Merge pull request #225 from return42/docs-nosql
[docs] document NoSLQ engines MongoDB and redis
2021-07-27 05:32:26 +00:00
Markus Heiser
c5cfc24467
Merge pull request #227 from return42/fix-seznam
[fix] Seznam engine - some XPath selectors has been changed
2021-07-27 05:20:39 +00:00
Samuel Dudik
7a7ef9cea6 [fix] Seznam engine - some XPath selectors has been changed
Merged from https://github.com/dudik/searx/commit/5a4207759

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-27 07:13:41 +02:00
Markus Heiser
7b40c736e6 [mod] hardening SearXNG instances by default (formats)
Deny formats has been implemented in 6ed4616d.

To harden SearXNG instances by default, other formats than HTML should be
denied.  Most of JSON, RSS and CSV requests are bots [1]::

    Bots are the only users of this feature on a public instance, and they abuse
    it too much that the engines rate limit pretty quickly the IP address of the
    instance.

[1] https://github.com/searxng/searxng/issues/95

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 12:59:09 +02:00
Markus Heiser
e02b5469f0 [mod] use tests/unit/settings/test_settings.yml in unit tests
In unit tests settings from

    searx/settings.yml

and the user settings from:

    unit/settings/test_settings.yml

are used.  In the latter, settings can be activated that are needed in the unit
test but should not activated by default in production.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 12:59:09 +02:00
Noémi Ványi
5e648e77ab [docs] document NoSLQ engines MongoDB and redis
Slightly modified merge of commit [3b192e638]

[3b192e638] https://github.com/searx/searx/commit/3b192e638

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 11:39:11 +02:00
Alexandre Flament
48fe83b901
Merge pull request #221 from dalf/fix-peertube_fetch_supported_languages
[fix] peertube: update _fetch_supported_languages
2021-07-25 10:30:53 +02:00
Alexandre Flament
cb4220657c
Merge pull request #223 from return42/fix-qwant
[fix] qwant engine - hardening the engine
2021-07-25 10:30:41 +02:00
Markus Heiser
e946c3462c [brand] !!DO NOT MERGE!! this patch uses branch fix-searx.sh @return42
Do not merge this patch in master branch of SearXNG!  This branch exists only
for testing the feature branch fix-searx.sh @return42.

This patch changes the buildenv to::

    GIT_URL='https://github.com/return42/searxng'
    GIT_BRANCH='fix-searx.sh'
    SEARX_PORT='7777'
    SEARX_BIND_ADDRESS='127.0.0.12'

To test installation procedure, clone feature branch (fix-searx.sh)::

    $ cd ~/Downloads
    $ git clone --branch fix-searx.sh https://github.com/return42/searxng searxng
    $ cd searxng

    $ ./utils/searx.sh install all
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
7196a9b5dd [fix] ./utils/filtron.sh - FILTRON_TARGET from YAML settings
The filtron target is the SearXNG installation and the default of FILTRON_TARGET
is taken from the YAML configuration ('server.port' & 'server.bind_address').

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
61d3914b63 [fix] test.robot - do not use environment from ./utils/brand.env
Some defaults in the settings.yml are taken from the environment.
By example;

The manage scripts sources the ./utils/brand.env and sets SEARX_PORT
environment.  This enviroment *wins over* any settings in a YAML file.
Whe we run a::

    make test.robot

The searx/settings_robot.yml is used, in this file the server settings are::

    server:
      port: 11111
      bind_address: 127.0.0.1

To get in use of the 'port: 11111' we have to unset the SEARX_PORT environment
which was sourced before.

The function buildenv.unset_env() can be called in all use cases where the
enviroment from ./utils/brand.env is not wanted.  ATM it unset the enviroment
variables::

    unset GIT_URL
    unset GIT_BRANCH
    unset SEARX_URL
    unset SEARX_PORT
    unset SEARX_BIND_ADDRESS

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
b6a55e223c [mod] reduce enviroment variables in shell scripts to what is needed
Not all settings from the 'brand:' section of the YAML files are needed in the
shell scripts.  This patch reduce the variables in ./utils/brand.env to what is
needed.  The following ('brand:' settings) can be removed from this file:

- ISSUE_URL
- DOCS_URL
- PUBLIC_INSTANCES
- WIKI_URL

Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:

- GIT_URL            <--> brand.git_url
- GIT_BRANCH         <--> brand.git_branch
- SEARX_URL          <--> server.base_url  (aka PUBLIC_URL)
- SEARX_PORT         <--> server.port
- SEARX_BIND_ADDRESS <--> server.bind_address

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
57b8f340a6 [mod] utils/searx.sh - add command 'inspect settings'
Inspect YAML setting <key> from SearXNG instance (${SEARX_SRC})::

  utils/searx.sh inspect settings server.base_url

utils/lib_install.sh
  should not log on stdout which is used for the output of function
  prompt_installation_setting().  Turned build_msg (stdout) into
  info_msg (stderr).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:24:22 +02:00
Markus Heiser
fe67f1478f [fix] qwant engine - prevent API locale exception on lang 'all'
Has been reported in [1], error message::

    Error
        Error: searx.exceptions.SearxEngineAPIException
        Percentage: 0
        Parameters: ('API error::locale must be a string,locale must be one of
        the following values: en_gb, en_ie, en_us, en_ca, en_in, en_my, en_au,
        en_nz, cy_gb, gd_gb, de_de, de_ch, de_at, fr_fr, br_fr, fr_be, fr_ch,
        fr_ca, fr_ad, fc_ca, ec_ca, co_fr, es_es, es_ar, es_cl, es_co, es_mx,
        es_pe, es_ad, ca_es, ca_ad, ca_fr, eu_es, eu_fr, it_it, it_ch, pt_br,
        pt_pt, pt_ad, nl_be, nl_nl, pl_pl, zh_hk, zh_cn, fi_fi, bg_bg, et_ee,
        hu_hu, da_dk, nb_no, sv_se, ko_kr, th_th, cs_cz, ro_ro, el_gr',)
        File name: searx/engines/qwant.py:114
        Function: response
        Code: raise SearxEngineAPIException('API error::' + msg)

[1] https://github.com/searxng/searxng/issues/222

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 14:48:27 +02:00
Markus Heiser
ca57c7421b [fix] qwant engine - prevent exception on date/time value is None
Has been reported in [1], error messages::

  Error
       Error: ValueError
       Percentage: 0
       Parameters: ()
       File name: searx/engines/qwant.py:159
       Function: response
       Code: pub_date = datetime.fromtimestamp(item['date'], None)

    Error
        Error: TypeError
        Percentage: 0
        Parameters: ('an integer is required (got type NoneType)',)
        File name: searx/engines/qwant.py:196
        Function: response
       Code: pub_date = datetime.fromtimestamp(item['date'])

Fix timedelta from seconds to milliseconds [1], error message::

    Error
        Error: TypeError
        Percentage: 0
        Parameters: ('unsupported type for timedelta seconds component: NoneType',)
        File name: searx/engines/qwant.py:195
        Function: response
        Code: length = timedelta(seconds=item['duration'])

[1] https://github.com/searxng/searxng/issues/222

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 14:48:14 +02:00
Alexandre Flament
b0a12924a0 [fix] peertube: update _fetch_supported_languages
update the regex to match the changes in peertube source code
fix "make data.languages"
2021-07-23 12:03:16 +02:00
Markus Heiser
ea0f4346b1
Merge pull request #220 from dalf/fix-category-order
[fix] /preferences: remove the empty engine category "social medias"(it is "social media")
2021-07-23 08:52:39 +00:00
Alexandre Flament
b1100e2b7d [fix] /preferences: remove the empty engine category "social medias" (it is "social media") 2021-07-23 10:43:16 +02:00
Markus Heiser
6869119f5e
Merge pull request #218 from searxng/dependabot/pip/master/twine-3.4.2
Bump twine from 3.4.1 to 3.4.2
2021-07-23 07:14:19 +00:00
Markus Heiser
c25df4bb17
Merge pull request #219 from searxng/dependabot/pip/master/pylint-2.9.5
Bump pylint from 2.9.3 to 2.9.5
2021-07-23 07:12:13 +00:00
dependabot[bot]
6f084db41b
Bump pylint from 2.9.3 to 2.9.5
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.9.3 to 2.9.5.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.9.3...v2.9.5)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-23 07:04:41 +00:00
dependabot[bot]
e289ae277b
Bump twine from 3.4.1 to 3.4.2
Bumps [twine](https://github.com/pypa/twine) from 3.4.1 to 3.4.2.
- [Release notes](https://github.com/pypa/twine/releases)
- [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst)
- [Commits](https://github.com/pypa/twine/compare/3.4.1...3.4.2)

---
updated-dependencies:
- dependency-name: twine
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-23 07:04:37 +00:00
Markus Heiser
28c874bf3f [brand] searx.sh & morty.sh - renamed 'searx' into 'SearXNG'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:39:02 +02:00
Markus Heiser
7167a7ee36 [mod] utils/.searx.sh: add commands: install [dot-config|init-src]
This patch implements two new commands:

install dot-config :
  - copy ./config.sh to ${SEARX_SRC}
  - implemented in new function install_DOT_CONFIG

install init-src:
  - sync files (SEARX_SRC_INIT_FILES) with ${SEARX_SRC}
  - implemented in new function init_SEARX_SRC()

new functions: verify_continue_install() and prompt_installation_status()
  These functions are used in installation procedures to get the status of the
  installation procedure.  The status is based on the existing function:

    usage: install_searx_get_state

        Prompts a string indicating the status of the installation procedure

    missing-searx-clone:
       There is no clone at ${SEARX_SRC}
    missing-searx-pyenv:
       There is no pyenv in ${SEARX_PYENV}
    installer-modified:
       There are files modified locally in the installer (clone),
       see ${SEARX_SRC_INIT_FILES} description.
    python-installed:
       Scripts can be executed in instance's environment
       - user:  ${SERVICE_USER}
       - pyenv: ${SEARX_PYENV}

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:59 +02:00