Commit graph

567 commits

Author SHA1 Message Date
Bnyro
13d801b75d [feat] results: show source of suggested answer 2023-08-18 19:07:14 +02:00
Markus Heiser
fa1ef9a07b [mod] move some code from webapp module to webutils module (no functional change)
Over the years the webapp module became more and more a mess.  To improve the
modulaization a little this patch moves some implementations from the webapp
module to webutils module.

HINT: this patch brings non functional change

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-19 19:49:44 +02:00
Markus Heiser
1ec325adcc [mod] limiter -> botdetection: modularization and documentation
In order to be able to meet the outstanding requirements, the implementation is
modularized and supplemented with documentation.

This patch does not contain functional change, except it fixes issue #2455

----

Aktivate limiter in the settings.yml and simulate a bot request by::

    curl -H 'Accept-Language: de-DE,en-US;q=0.7,en;q=0.3' \
         -H 'Accept: text/html'
         -H 'User-Agent: xyz' \
         -H 'Accept-Encoding: gzip' \
         'http://127.0.0.1:8888/search?q=foo'

In the LOG:

    DEBUG   searx.botdetection.link_token : missing ping for this request: .....

Since ``BURST_MAX_SUSPICIOUS = 2`` you can repeat the query above two time
before you get a "Too Many Requests" response.

Closes: https://github.com/searxng/searxng/issues/2455
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29 14:54:56 +02:00
Markus Heiser
5226044c13 [mod] limiter: add random token to the limiter URL
By adding a random component in the limiter URL a bot can no longer send a ping
by request a static URL.

Related: https://github.com/searxng/searxng/pull/2357#issuecomment-1518525094
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29 14:54:56 +02:00
Markus Heiser
dba569462d [mod] limiter: reduce request rates for requests without a ping
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29 14:54:56 +02:00
Markus Heiser
d5ecda9930 [mod] move language recognition to get_search_query_from_webapp
To set the language from language recognition and hold the value selected by the
client, the previous implementation creates a copy of the SearchQuery object and
manipulates the SearchQuery object by calling function replace_auto_language().

This patch tries to implement a similar functionality in a more central place,
in function get_search_query_from_webapp() when the SearchQuery object is build
up.

Additional this patch uses the language preferred by the client, if language
recognition does not have a match / the existing implementation does not care
about client preferences and uses 'all' in case of no match.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 22:23:33 +02:00
Markus Heiser
c03b0ea650 [mod] add a Preferences.client property to store client prefs
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 21:41:57 +02:00
Markus Heiser
7d17d37ac7 [fix] don't show a category if there is no active engine in
When deactivate all the engines of a category, this category should disappeare.
This feature has been lost in commit 8e9ad1cc.

For better readability, webapp.get_enabled_categories() has been rewritten with
identical functionality.

Related:

- https://github.com/searxng/searxng/issues/1020
- https://github.com/searxng/searxng/issues/1604

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-10 09:22:21 +02:00
Markus Heiser
2ffd446e5c [mod] clarify the difference of the default category and subgrouping
This PR does no functional change it is just an attempt to make more clear in
the code, what a default category is and what a subcategory is.  The previous
name 'others' leads to confusion with the **category 'other'**.

If a engine is not assigned to a category, the default is assigned::

    DEFAULT_CATEGORY = 'other'

If an engine has only one category and this category is shown as tab in the user
interface, this engine has no further subgrouping::

    NO_SUBGROUPING = 'without further subgrouping'

Related:

- https://github.com/searxng/searxng/issues/1604
- https://github.com/searxng/searxng/pull/1545

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-07 11:03:25 +02:00
Markus Heiser
16f0db4493 [mod] replace utils.match_language by locales.match_locale
This patch replaces the *full of magic* ``utils.match_language`` function by a
``locales.match_locale``.  The ``locales.match_locale`` function is based on the
``locales.build_engine_locales`` introduced in 9ae409a0 [1].

In the past SearXNG did only support a search by a language but not in a region.
This has been changed a long time ago and regions have been added to SearXNG
core but not to the engines.  The ``utils.match_language`` was the function to
handle the different aspects of language/regions in SearXNG core and the
supported *languages* in the engine.  The ``utils.match_language`` did it with
some magic and works good for most use cases but fails in some edge case.

To replace the concurrence of languages and regions in the SearXNG core the
``locales.build_engine_locales`` was introduced in 9ae409a0 [1].  With the last
patches all engines has been migrated to a ``fetch_traits`` and a
language/region concept that is based on ``locales.build_engine_locales``.

To summarize: there is no longer a need for the ``locales.match_language``.

[1] https://github.com/searxng/searxng/pull/1652

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
4d4aa13e1f [mod] remove obsolete EngineTraits.supported_languages
All engines has been migrated from ``supported_languages`` to the
``fetch_traits`` concept.  There is no longer a need for the obsolete code that
implements the ``supported_languages`` concept.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
c9cd376186 [mod] replace searx.languages by searx.sxng_locales
With the language and region tags from the EngineTraitsMap the handling of
SearXNG's tags of languages and regions has been normalized and is no longer
a *mystery*.  The "languages" became "locales" that are supported by babel and
by this, the update_engine_traits.py can be simplified a lot.

Other code places can be simplified as well, but these simplifications
should (respectively can) only be done when none of the engines work with the
deprecated EngineTraits.supported_languages interface anymore.

This commit replaces searx.languages by searx.sxng_locales and fix the naming of
some names from "language" to "locale" (e.g. language_codes --> sxng_locales).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser
6e5f22e558 [mod] replace engines_languages.json by engines_traits.json
Implementations of the *traits* of the engines.

Engine's traits are fetched from the origin engine and stored in a JSON file in
the *data folder*.  Most often traits are languages and region codes and their
mapping from SearXNG's representation to the representation in the origin search
engine.

To load traits from the persistence::

    searx.enginelib.traits.EngineTraitsMap.from_data()

For new traits new properties can be added to the class::

    searx.enginelib.traits.EngineTraits

.. hint::

   Implementation is downward compatible to the deprecated *supported_languages
   method* from the vintage implementation.

   The vintage code is tagged as *deprecated* an can be removed when all engines
   has been ported to the *traits method*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Alexandre Flament
6748e8e2d5 Add "Auto-detected" as a language.
When the user choose "Auto-detected", the choice remains on the following queries.
The detected language is displayed.

For example "Auto-detected (en)":
* the next query language is going to be auto detected
* for the current query, the detected language is English.

This replace the autodetect_search_language plugin.
2023-02-17 15:17:36 +00:00
Markus Heiser
415f6398d6 [fix] time_range in HTML form should not be a string 'None'
Before this patch SearXNG returns 'time_range': 'None'::

    <form method="GET" action="/searx/search">
      ...
      <input type="hidden" name="time_range" value="None">
      ...
    </form>

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-10 15:32:21 +01:00
Markus Heiser
4c06837a50 [mod] make python code pylint 2.16.1 compliant
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-10 13:59:21 +01:00
dependabot[bot]
70fff2ebf1 Bump flask-babel from 2.0.0 to 3.0.0
Bumps [flask-babel](https://github.com/python-babel/flask-babel) from 2.0.0 to 3.0.0.
- [Release notes](https://github.com/python-babel/flask-babel/releases)
- [Changelog](https://github.com/python-babel/flask-babel/blob/master/CHANGELOG)
- [Commits](https://github.com/python-babel/flask-babel/compare/v2.0.0...v3.0.0)

---
updated-dependencies:
- dependency-name: flask-babel
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-20 09:38:30 +00:00
Alexandre Flament
b971167ced move searx.shared.redisdb to searx.redisdb 2022-12-10 09:26:38 +01:00
Alexandre Flament
fe419e355b The checker requires Redis
Remove the abstraction in searx.shared.SharedDict.
Implement a basic and dedicated scheduler for the checker using a Redis script.
2022-11-05 12:04:50 +01:00
Alexandre Flament
32e8c2cf09 searx.network: add "verify" option to the networks
Each network can define a verify option:
* false to disable certificate verification
* a path to existing certificate.

SearXNG uses SSL_CERT_FILE and SSL_CERT_DIR when they are defined
see https://www.python-httpx.org/environment_variables/#ssl_cert_file
2022-10-14 13:59:22 +00:00
Markus Heiser
ba8959ad7c [fix] typos / reported by @kianmeng in searx PR-3366
[PR-3366] https://github.com/searx/searx/pull/3366

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-27 18:32:14 +02:00
Markus Heiser
3198c906af [mod] paper.html: add links to doi resolver
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-24 14:19:51 +02:00
Alexandre Flament
d6446be38f [mod] science category: various update of about PR 1705 2022-09-23 20:52:55 +02:00
Alexandre FLAMENT
a96f503d7b Add searx.webutils.searxng_format_date
* Move the datetime to str code from searx.webapp.search to searx.webutils.searxng_format_date
* When the month, day, hour, day and second are zero, the function returns only the year.
2022-09-23 20:45:58 +02:00
Alexandre Flament
691c0ed6b9
Merge pull request #1743 from dalf/update_about_metrics
Update about the metrics
2022-09-04 11:29:28 +02:00
Markus Heiser
ad8ffd222c [mod] option 'ui: cache_url:' to configure internet cache or archive
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-09-04 09:42:20 +02:00
Alexandre FLAMENT
66f72a006f template: when enable_metrics are disabled, hide the empty stats
when metrics are disabled:
* hide the link to /stats at the bottom of the pages
* in /preferences, hide the columns "Response time" and "Reliability"
2022-09-02 08:52:23 +00:00
Alexandre FLAMENT
94a28ebe53 Stats: display only the score per result, no anymore the score 2022-09-02 08:30:38 +00:00
Alexandre FLAMENT
341ad46303 settings.yml: set default values for result_proxy
* initialize result_proxy with searx/settings_defaults.py
* allow result_proxy.key to be a string

this commit supersedes #1522
2022-08-28 09:27:53 +00:00
Markus Heiser
3b0f9c07b2 [fix] improve OpenSearch description
Some HTTP-Clients do have issues with the ``opensearch.xml`` from SearXNG
(related [1][2]) while other OpenSearch descriptions[3] (e.g. from qwant) work
flawles.

Inspired by the OpenSearch description from qwant and with informations from the
specification[4] the ``opensearch.xml`` has been *improved*.

- convert `<Url>` methods from lower case to upper case (`POST`|`GET`)
- add `<moz:SearchForm>` and `xmlns:moz="http://www.mozilla.org/2006/browser/search/"`
- add `<Query role="example" searchTerms="SearXNG" />`  [4]

  OpenSearch description documents should include at least one Query element of
  `role="example"` that is expected to return search results. Search clients may
  use this example query to validate that the search engine is working properly.

- modified `<LongName>` to SearXNG
- modified `<Description>` the word 'hackable' scares uninitiated users and was removed
- add the `type="image/png"` to `<Image>`

Test can be done by::

    make run

Visit http://127.0.0.1:8888/ and add the search engine to your WEB-Browser /
test with different WEB-Browser from desktop and Smartphones (are there any iOS
user here, please test on Safari and Chrome).

[1] https://app.element.io/#/room/#searxng:matrix.org/$xN_abdKhNqUlgXRBrb_9F3pqOxnSzGQ1TG0s0G9hQVw
[2] https://github.com/searxng/searxng/issues/431
[3] https://developer.mozilla.org/en-US/docs/Web/OpenSearch
[4] https://github.com/dewitt/opensearch/blob/master/opensearch-1-1-draft-6.md#the-query-element

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-11 19:04:36 +02:00
Alexandre Flament
2babf59adc [fix] pyright repported errors
The errors make pyright usage useless since a new error won't be seen [1].

[1] https://github.com/searxng/searxng/pull/1569

```
  searx/compat.py:11:27 - error: Expression of type "Type[cached_property[_T@cached_property]]" cannot be assigned to declared type "Type[cached_property]"
    "Type[cached_property[_T@cached_property]]" is incompatible with "Type[cached_property]"
    Type "Type[cached_property[_T@cached_property]]" cannot be assigned to type "Type[cached_property]" (reportGeneralTypeIssues)
  searx/utils.py:69:36 - error: Expression of type "None" cannot be assigned to parameter of type "str"
    Type "None" cannot be assigned to type "str" (reportGeneralTypeIssues)
  searx/utils.py:573:85 - error: Expression of type "None" cannot be assigned to parameter of type "int"
    Type "None" cannot be assigned to type "int" (reportGeneralTypeIssues)
  searx/webapp.py:1306:22 - error: Argument of type "str" cannot be assigned to parameter "__a" of type "BytesPath" in function "join"
    Type "str" cannot be assigned to type "BytesPath"
      "str" is incompatible with "bytes"
      "str" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:68 - error: Argument of type "Literal['themes']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "Literal['themes']" cannot be assigned to type "BytesPath"
      "Literal['themes']" is incompatible with "bytes"
      "Literal['themes']" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:78 - error: Argument of type "str | Any | None" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "str | Any | None" cannot be assigned to type "BytesPath"
      Type "str" cannot be assigned to type "BytesPath"
        "str" is incompatible with "bytes"
        "str" is incompatible with protocol "PathLike[bytes]"
          "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:85 - error: Argument of type "Literal['img']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "Literal['img']" cannot be assigned to type "BytesPath"
      "Literal['img']" is incompatible with "bytes"
      "Literal['img']" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/engines/mongodb.py:8:6 - warning: Import "pymongo" could not be resolved (reportMissingImports)
  searx/engines/mysql_server.py:9:8 - warning: Import "mysql.connector" could not be resolved (reportMissingImports)
  searx/engines/postgresql.py:9:8 - warning: Import "psycopg2" could not be resolved from source (reportMissingModuleSource)
  searx/engines/xpath.py:187:28 - warning: "categories" is not defined (reportUndefinedVariable)
  searx/search/__init__.py:184:82 - warning: "flask" is not defined (reportUndefinedVariable)
  searx/search/checker/background.py:19:26 - error: Type of "schedule" is partially unknown
    Type of "schedule" is "(delay: Any, func: Any, *args: Any) -> Literal[True]" (reportUnknownVariableType)
  searx/shared/__init__.py:8:12 - warning: Import "uwsgi" could not be resolved (reportMissingImports)
  searx/shared/shared_uwsgi.py:5:8 - warning: Import "uwsgi" could not be resolved (reportMissingImports)
```
2022-07-30 18:04:44 +02:00
Alexandre Flament
f8f239fe1f Donation link: default value to searxng.org, can be hidden or custom
Add a new setting: general.donation_url

By default the value is https://docs.searxng.org/donate.html

When the value is false, the link is hidden

When the value is true, the link goes to the infopage donation,
the administrator can create a custom page.
2022-07-02 11:29:21 +02:00
samsaptidev
31005595c9
Add privacypolicy_url option 2022-06-16 11:56:24 +02:00
Alexandre Flament
ee3f230d93 Autocomplete: don't suggest the current query.
Example of minor issue before this commit:
the autocompletion can suggest "Test" if the query is "Test".
2022-06-12 11:21:24 +02:00
Markus Heiser
ad964562ce [fix] move locale code from webapp.py to locales.py and fix #1303
To improve modularization this patch:

- moves *locale* related implementation from the webapp.py application to the
  locale.py module.

- The initialization of the locales is now done in the application (webapp) and
  is no longer done while importing searx.locales.

In the searx.locales module a new dictionary named `LOCALE_BEST_MATCH` has been
added.  In this dictionary we can map languages without a translation to
languages we have a translation for.

To fix #1303 zh-HK has been mapped to zh-Hant-TW (we do not need additional
translations of traditional Chinese)

Closes: https://github.com/searxng/searxng/issues/1303
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-06-12 10:52:26 +02:00
Markus Heiser
2de007138c [fix] prepare for pylint 2.14.0
Remove issue reported by Pylint 2.14.0:

- no-self-use: has been moved to optional extension [1]
- The refactoring checker now also raises 'consider-using-generator' messages
  for max(), min() and sum(). [2]

.pylintrc:
  - <option name>-hint has been removed since long, Pylint 2.14.0 raises an
    error on invalid options
  - bad-continuation and bad-whitespace have been removed [3]

[1] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/summary.html#removed-checkers
[2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/full.html#what-s-new-in-pylint-2-14-0
[2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.6/summary.html#summary-release-highlights

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-06-03 15:41:52 +02:00
Léon Tiekötter
7e95d6bb79 [mod] image proxy: allow binary/octet-stream mime type
The Petal Search Images engine sends the thumbnails with the binary/octet-stream mime type.
2022-06-02 14:32:37 +02:00
Julian Fairfax
7d443dfea3 Preferences URL in preferences
Revert translation changes

Auto-save preferences from URL

Preferences URL in preferences

Fix that

Preferences URL in preferences
2022-05-15 20:34:04 +02:00
mrpaulblack
c69c61838b [fix] autocomplete and infinite scroll persistence in preferences
* fixes a regression from https://github.com/searxng/searxng/pull/1186
2022-05-10 22:25:42 +02:00
Léon Tiekötter
18b8bbc017
[fix] PR 1186: GET method
PR https://github.com/searxng/searxng/pull/1186 prevented the GET method from being selected.
2022-05-10 17:47:53 +02:00
Alexandre Flament
9b3efa6d8a theme: remove __common__ 2022-05-07 19:40:48 +02:00
Markus Heiser
4326009d00 [format.python] based on bugfix in 9ed626130 2022-05-07 18:23:10 +02:00
Markus Heiser
444b1e70db [mod] add setting: search.autocomplete_min
Minimun characters to type before autocompleter starts.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-05-07 17:58:30 +02:00
Markus Heiser
ed2a4c8087 [mod] client_settings: pass settings from server to JS client
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-05-07 17:58:30 +02:00
Alexandre FLAMENT
550f70b1aa Add support for the Silesian language 2022-05-06 09:40:45 +00:00
HZH
c7de5c043c
fix: %2F in path will be converted to " › " 2022-04-07 23:15:58 +08:00
HZH
57a3252ff1
Unquote path on result page. 2022-04-05 02:37:18 +08:00
Austin Huang
96b8aac826
Extra null safety
Co-Authored-By: Alexandre Flament <alex.andre@al-f.net>
2022-03-29 15:17:36 -04:00
Austin Huang
6353b850b0
[fix] merge fake "plugins" endpoint back to "static"
https: //matrix.to/#/!vxScbLNEAmRvOraXBn:matrix.org/$0zqTtH3s-eVKlMwiAgiEJdetOkRx27EIRs-C0o8FzSg?via=libera.chat&via=matrix.org&via=tchncs.de
Co-Authored-By: Alexandre Flament <alex.andre@al-f.net>
2022-03-29 13:02:21 -04:00
Austin Huang
e09d2ad4a7
[fix] return correct plugin path
closes #1021
2022-03-28 15:24:01 -04:00