Commit graph

8030 commits

Author SHA1 Message Date
Markus Heiser
8df1f0c47e [mod] add 'Accept-Language' HTTP header to online processores
Most engines that support languages (and regions) use the Accept-Language from
the WEB browser to build a response that fits to the language (and region).

- add new engine option: send_accept_language_header

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-01 17:01:59 +02:00
Markus Heiser
a2badb4fe4 [doc] add description of method EngineProcessor.get_params()
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-01 16:42:33 +02:00
Markus Heiser
1fbb514a4e
Merge pull request #1595 from tiekoetter/fix-uwsgi-buffer-size
[fix] uWSGI: increase buffer-size
2022-07-31 13:13:54 +02:00
Markus Heiser
48968bf46a [doc] list of changes that affect the infrastructure
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-31 13:08:06 +02:00
Léon Tiekötter
2b94fef7ec [fix] uWSGI: increase buffer-size
Increase max size of a request, by default it is 4k [1].  4096 as buffer-size is
too small and will result in the preference urls not working.

[1] https://uwsgi-docs.readthedocs.io/en/latest/Options.html#buffer-size
2022-07-31 12:40:06 +02:00
Markus Heiser
88733c0ce6
Merge pull request #1569 from dalf/fix-pyright-reported-errors
Fix pyright errors
2022-07-30 18:40:16 +02:00
Alexandre Flament
2babf59adc [fix] pyright repported errors
The errors make pyright usage useless since a new error won't be seen [1].

[1] https://github.com/searxng/searxng/pull/1569

```
  searx/compat.py:11:27 - error: Expression of type "Type[cached_property[_T@cached_property]]" cannot be assigned to declared type "Type[cached_property]"
    "Type[cached_property[_T@cached_property]]" is incompatible with "Type[cached_property]"
    Type "Type[cached_property[_T@cached_property]]" cannot be assigned to type "Type[cached_property]" (reportGeneralTypeIssues)
  searx/utils.py:69:36 - error: Expression of type "None" cannot be assigned to parameter of type "str"
    Type "None" cannot be assigned to type "str" (reportGeneralTypeIssues)
  searx/utils.py:573:85 - error: Expression of type "None" cannot be assigned to parameter of type "int"
    Type "None" cannot be assigned to type "int" (reportGeneralTypeIssues)
  searx/webapp.py:1306:22 - error: Argument of type "str" cannot be assigned to parameter "__a" of type "BytesPath" in function "join"
    Type "str" cannot be assigned to type "BytesPath"
      "str" is incompatible with "bytes"
      "str" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:68 - error: Argument of type "Literal['themes']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "Literal['themes']" cannot be assigned to type "BytesPath"
      "Literal['themes']" is incompatible with "bytes"
      "Literal['themes']" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:78 - error: Argument of type "str | Any | None" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "str | Any | None" cannot be assigned to type "BytesPath"
      Type "str" cannot be assigned to type "BytesPath"
        "str" is incompatible with "bytes"
        "str" is incompatible with protocol "PathLike[bytes]"
          "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:85 - error: Argument of type "Literal['img']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "Literal['img']" cannot be assigned to type "BytesPath"
      "Literal['img']" is incompatible with "bytes"
      "Literal['img']" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/engines/mongodb.py:8:6 - warning: Import "pymongo" could not be resolved (reportMissingImports)
  searx/engines/mysql_server.py:9:8 - warning: Import "mysql.connector" could not be resolved (reportMissingImports)
  searx/engines/postgresql.py:9:8 - warning: Import "psycopg2" could not be resolved from source (reportMissingModuleSource)
  searx/engines/xpath.py:187:28 - warning: "categories" is not defined (reportUndefinedVariable)
  searx/search/__init__.py:184:82 - warning: "flask" is not defined (reportUndefinedVariable)
  searx/search/checker/background.py:19:26 - error: Type of "schedule" is partially unknown
    Type of "schedule" is "(delay: Any, func: Any, *args: Any) -> Literal[True]" (reportUnknownVariableType)
  searx/shared/__init__.py:8:12 - warning: Import "uwsgi" could not be resolved (reportMissingImports)
  searx/shared/shared_uwsgi.py:5:8 - warning: Import "uwsgi" could not be resolved (reportMissingImports)
```
2022-07-30 18:04:44 +02:00
Markus Heiser
ededaab807
Merge pull request #1589 from return42/searxng-install-additions
[mod] fix minor leftovers from PR #1332
2022-07-30 16:25:32 +02:00
Markus Heiser
bded8750d5 [mod] fix minor leftovers from PR #1332
Related: https://github.com/searxng/searxng/pull/1332
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 16:13:47 +02:00
Markus Heiser
645c2a2ca1
Merge pull request #1332 from return42/searxng-install
Upgrade installation scripts and documentation
2022-07-30 13:45:12 +02:00
Léon Tiekötter
e5323b8aa2 [docs] corrections from @tiekoetter's review 2022-07-30 13:39:35 +02:00
Markus Heiser
6fbffe9d20 [docs] add section "Migrate and stay tuned!"
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser
692708aa77 [clean up] drop obsolete searx, filtron and morty install scripts
Since ./utils/searxng.sh is implemented, the old installation procedures from
filtron, morty and searx can be removed.

For users who want to upgrade, the procedures for removing old installations
have still been retained.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser
ed8a169029 [doc] update documentation of the installation procedures
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser
782f73540e [utils/searxng.sh] implement new script to install SearXNG
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser
81bba44869 [install scripts] rename SEARX_<name> variables to SEARXNG_<name>
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Alexandre Flament
0cf5310933
Merge pull request #1581 from searxng/dependabot/pip/master/setproctitle-1.3.0
Bump setproctitle from 1.2.3 to 1.3.0
2022-07-29 10:35:49 +02:00
Markus Heiser
ca2ddc615f
Merge pull request #1583 from searxng/translations_update
Update translations
2022-07-29 09:31:45 +02:00
searxng-bot
953aaec6a3 [translations] update from Weblate
c0c9107c - 2022-07-27 - Sangha Lee <totoriato@gmail.com>
5b48bce6 - 2022-07-24 - Linerly <linerly@protonmail.com>
79669e65 - 2022-07-29 - Markus Heiser <markus.heiser@darmarit.de>
520e9284 - 2022-07-25 - Markus Heiser <markus.heiser@darmarit.de>
7cf52ff5 - 2022-07-25 - Markus Heiser <markus.heiser@darmarit.de>
9d3ebe72 - 2022-07-24 - Markus Heiser <markus.heiser@darmarit.de>
2d03c097 - 2022-07-24 - Markus Heiser <markus.heiser@darmarit.de>
388af012 - 2022-07-27 - Markus Heiser <markus.heiser@darmarit.de>
a4bcf098 - 2022-07-25 - Miguel Silva <miguelcabeca.dev@gmail.com>
93fd0b72 - 2022-07-27 - Markus Heiser <markus.heiser@darmarit.de>
8f68b206 - 2022-07-26 - tents <remendne@pentrens.jp>
9007c99c - 2022-07-24 - Markus Heiser <markus.heiser@darmarit.de>
aeec96f2 - 2022-07-26 - Matija Kromar <matija.kromar@gmail.com>
69084863 - 2022-07-25 - Markus Heiser <markus.heiser@darmarit.de>
b48190ab - 2022-07-24 - alexfs2015 <alex04fs@gmail.com>
b6bbc0a5 - 2022-07-23 - Markus Heiser <markus.heiser@darmarit.de>
1a503806 - 2022-07-29 - Markus Heiser <markus.heiser@darmarit.de>
c960cb93 - 2022-07-27 - Markus Heiser <markus.heiser@darmarit.de>
8a2bd34b - 2022-07-25 - Markus Heiser <markus.heiser@darmarit.de>
1064cea0 - 2022-07-23 - LagManCZ <lagmen@post.cz>
67423045 - 2022-07-24 - alexfs2015 <alex04fs@gmail.com>
56c87fda - 2022-07-24 - Markus Heiser <markus.heiser@darmarit.de>
36a64f1c - 2022-07-24 - Ankit Gupta <guptaa.ankitt@gmail.com>
2022-07-29 07:17:55 +00:00
Markus Heiser
e08072def5
Merge pull request #1582 from searxng/dependabot/pip/master/sphinx-5.1.1
Bump sphinx from 5.0.2 to 5.1.1
2022-07-29 09:13:44 +02:00
dependabot[bot]
f0f195c043
Bump sphinx from 5.0.2 to 5.1.1
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 5.0.2 to 5.1.1.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/5.x/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v5.0.2...v5.1.1)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-29 07:04:21 +00:00
dependabot[bot]
9e3ae571d0
Bump setproctitle from 1.2.3 to 1.3.0
Bumps [setproctitle](https://github.com/dvarrazzo/py-setproctitle) from 1.2.3 to 1.3.0.
- [Release notes](https://github.com/dvarrazzo/py-setproctitle/releases)
- [Changelog](https://github.com/dvarrazzo/py-setproctitle/blob/master/HISTORY.rst)
- [Commits](https://github.com/dvarrazzo/py-setproctitle/compare/version-1.2.3...version-1.3.0)

---
updated-dependencies:
- dependency-name: setproctitle
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-29 07:04:16 +00:00
Markus Heiser
14085cc0a7
Merge pull request #1580 from searxng/update_data_update_engine_descriptions.py
Update searx.data - update_engine_descriptions.py
2022-07-29 07:23:02 +02:00
Markus Heiser
7602c23d71
Merge pull request #1579 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2022-07-29 07:22:12 +02:00
Markus Heiser
0bad4fb931
Merge pull request #1578 from searxng/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2022-07-29 07:21:06 +02:00
Markus Heiser
7dd75ae080
Merge pull request #1576 from searxng/update_data_update_languages.py
Update searx.data - update_languages.py
2022-07-29 07:20:28 +02:00
Markus Heiser
6c78e522ff
Merge pull request #1577 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2022-07-29 07:15:51 +02:00
dalf
2bdb367e8c Update searx.data - update_engine_descriptions.py 2022-07-29 02:13:07 +00:00
dalf
fc019fb296 Update searx.data - update_currencies.py 2022-07-29 02:01:30 +00:00
dalf
c2662ec3c8 Update searx.data - update_firefox_version.py 2022-07-29 02:01:08 +00:00
dalf
385542e6c5 Update searx.data - update_ahmia_blacklist.py 2022-07-29 02:01:04 +00:00
dalf
90068b3d0b Update searx.data - update_languages.py 2022-07-29 02:00:53 +00:00
Markus Heiser
50ec821e8c
Merge pull request #1556 from tiekoetter/fix-google-results-xpath
[fix] google engine: results XPath
2022-07-26 07:21:37 +02:00
Markus Heiser
c72d70d45c Revert "Quick fix for google engine for EU countries"
This reverts commit 747cf1a246.
2022-07-26 06:39:44 +02:00
Léon Tiekötter
950f036c03
[fix] google engine: results XPath 2022-07-26 00:24:15 +02:00
Émilien Devos
747cf1a246
Quick fix for google engine for EU countries
This revert part of the commit of 5fb2071cb2
2022-07-25 20:48:50 +00:00
Markus Heiser
7fab950284
Merge pull request #1554 from return42/fix-demo-engine
[fix] demo_online.py - fixed typo
2022-07-25 20:10:31 +02:00
Markus Heiser
0be0e63117 [fix] demo_online.py - fixed typo
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-25 20:04:00 +02:00
Markus Heiser
a3cfa20696
Merge pull request #1553 from return42/fix-1548
[fix] update documentation of Search-API (/dev)
2022-07-25 17:35:40 +02:00
Markus Heiser
5795c1971f [fix] update documentation of Search-API (/dev)
This patch fixes a leftover from [#1548], the list of the plugins was not
up-to-date:

- HTTPS_rewrite has been removed (247c46c6b)
- DOAI_rewrite is named 'Open_Access_DOI_rewrite' (575159b)

[#1548] https://github.com/searxng/searxng/pull/1548

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-25 17:22:31 +02:00
Markus Heiser
b14ec7286d
Merge pull request #1552 from unixfox/fix-consent-google
ucbcb bypass method to CONSENT yes for google
2022-07-25 13:51:52 +02:00
Emilien Devos
5fb2071cb2 [fix] google & youtube - set EU consent cookie
This change the previous bypass method for Google consent using
``ucbcb=1`` (6face215b8) to accept the consent using ``CONSENT=YES+``.

The youtube_noapi and google have a similar API, at least for the consent[1].

Get CONSENT cookie from google reguest::

    curl -i "https://www.google.com/search?q=time&tbm=isch" \
         -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \
         | grep -i consent
    ...
    location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1
    set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure
    ...

PENDING & YES [2]:

  Google change the way for consent about YouTube cookies agreement in EU
  countries. Instead of showing a popup in the website, YouTube redirects the
  user to a new webpage at consent.youtube.com domain ...  Fix for this is to
  put a cookie CONSENT with YES+ value for every YouTube request

[1] https://github.com/iv-org/invidious/pull/2207
[2] https://github.com/TeamNewPipe/NewPipeExtractor/issues/592

Closes: https://github.com/searxng/searxng/issues/1432
2022-07-25 13:27:06 +02:00
Markus Heiser
07aa2bfd98
Merge pull request #1549 from return42/fix-1544
[fix] sjp engine - convert enginename to a latin1 compliance name
2022-07-24 21:15:52 +02:00
Markus Heiser
4231a5770b [fix] sjp engine - convert enginename to a latin1 compliance name
The engine name is not only a *name* its also a identifier that is used in
logs, HTTP headers and more.  Unicode characters in the name of an engine could
cause various issues.

Closes: https://github.com/searxng/searxng/issues/1544
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-24 21:10:55 +02:00
Markus Heiser
55d1936bdc
Merge pull request #1548 from return42/fix-1541
[fix] update documentation of the Search-API (/dev)
2022-07-24 20:21:18 +02:00
Markus Heiser
eb85474920 [fix] update documentation of the Search-API (/dev)
Closes: https://github.com/searxng/searxng/issues/1541
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-24 20:15:03 +02:00
Markus Heiser
6a479f41c5
Merge pull request #1546 from james-still/master
[fix] Update xPath value for Emojipedia engine
2022-07-24 19:21:49 +02:00
james-still
2516e21c58 [fix] emojipedia - update XPath to be relative 2022-07-24 19:14:26 +02:00
Markus Heiser
3ff576ea72
Merge pull request #1533 from searxng/dependabot/pip/master/nose2-coverage_plugin--0.12.0
Bump nose2[coverage_plugin] from 0.11.0 to 0.12.0
2022-07-23 16:21:39 +02:00
Markus Heiser
cd58b7aee4
Merge pull request #1511 from searxng/dependabot/pip/master/flask-2.1.3
Bump flask from 2.1.2 to 2.1.3
2022-07-23 16:10:20 +02:00