Commit graph

113 commits

Author SHA1 Message Date
fd93d0c602 Merge branch 'refs/heads/upstream'
# Conflicts:
#	.github/dependabot.yml
#	searx/settings.yml
#	searx/static/themes/simple/img/searxng.png
#	searx/static/themes/simple/img/searxng.svg
#	utils/brand.env
2024-04-28 14:36:37 +02:00
Markus Heiser
542f7d0d7b [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION
In the past, some files were tested with the standard profile, others with a
profile in which most of the messages were switched off ... some files were not
checked at all.

- ``PYLINT_SEARXNG_DISABLE_OPTION`` has been abolished
- the distinction ``# lint: pylint`` is no longer necessary
- the pylint tasks have been reduced from three to two

  1. ./searx/engines -> lint engines with additional builtins
  2. ./searx ./searxng_extra ./tests -> lint all other python files

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-11 14:55:38 +01:00
Markus Heiser
8205f170ff [mod] pylint all engines without PYLINT_SEARXNG_DISABLE_OPTION
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-11 14:55:38 +01:00
Markus Heiser
c197c0e35e [fix] remove twine from requirements-dev
SearXNG is a rolling release / we do not deploy packages on PyPi

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-02-16 18:29:40 +01:00
Markus Heiser
a8aaf5719b [test.robot] update gecko driver / v0.34.0 required by firefox 121.*
Update gecko driver to v0.34.0 [1]

[1] https://github.com/mozilla/geckodriver/releases/tag/v0.34.0

Closes: https://github.com/searxng/searxng/issues/3141
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-01-29 17:37:30 +01:00
Markus Heiser
f9c5727ddc [mod] get rid of ./utils/brand.env and its workflow
All the environments defined in ./utils/brand.env are generated on the fly, so
there is no longer a need to define the brand environment in this file and all
the workflows to handle this file.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-01-09 16:31:19 +01:00
Markus Heiser
70521339d9 [fix] black --exclude option is a regular expression
When ``searx/static/themes/simple/node_modules`` are installed the test from
black runs into this path / Error::

    $ make test
    TEST      [yamllint] $YAMLLINT_FILES
    TEST      [black] $BLACK_TARGETS
    --- searx/static/themes/simple/node_modules/flatted/python/test.py	2023-10-13 10:09:50.460026 +0000
    +++ searx/static/themes/simple/node_modules/flatted/python/test.py	2023-10-13 10:10:04.974335 +0000
    @@ -1,9 +1,11 @@
    from flatted import stringify as _stringify, parse
    ...

black --exclude option is not a comma separated list, its a regular expression /
see:

    $ black --help
    ...
    --exclude TEXT   A regular expression that matches files and
                     directories that should be excluded on recursive
                     searches ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-13 12:44:18 +02:00
Alexandre Flament
fb88721532 Upgrade dependencies
* geckodriver: 0.33.0
* node 16.20.2
* pyright 1.1.329
* eslint 8.50.0
2023-10-01 08:25:00 +02:00
Markus Heiser
64100db904 [doc] improve documentation of make targets and ./manage script
BTW force modularization of the ./mange script into sub modules:

- utils/lib_sxng_data.sh
- utils/lib_sxng_node.sh
- utils/lib_sxng_static.sh
- utils/lib_sxng_test.sh
- utils/lib_sxng_themes.sh

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-06 19:37:12 +02:00
Azure Star
a4788f8d90
Merge branch 'searxng:master' into master 2023-04-13 08:36:38 +02:00
Markus Heiser
cd26f3ea2f [fix] ./manage: replace data.languages by data.traits
Follow up of https://github.com/searxng/searxng/pull/2269

Related:

- https://github.com/searxng/searxng/issues/2314

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-05 07:44:29 +02:00
Azure Star
42bc05744b
Merge branch 'searxng:master' into master 2023-03-29 09:58:10 +02:00
Markus Heiser
6e5f22e558 [mod] replace engines_languages.json by engines_traits.json
Implementations of the *traits* of the engines.

Engine's traits are fetched from the origin engine and stored in a JSON file in
the *data folder*.  Most often traits are languages and region codes and their
mapping from SearXNG's representation to the representation in the origin search
engine.

To load traits from the persistence::

    searx.enginelib.traits.EngineTraitsMap.from_data()

For new traits new properties can be added to the class::

    searx.enginelib.traits.EngineTraits

.. hint::

   Implementation is downward compatible to the deprecated *supported_languages
   method* from the vintage implementation.

   The vintage code is tagged as *deprecated* an can be removed when all engines
   has been ported to the *traits method*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Fauli1221
e01362fee2
Merge branch 'searxng:master' into master 2023-02-17 18:12:21 +01:00
Markus Heiser
3eed80a074 [mod] install developer tools by: make node.env.dev
- rename ``./manage node.env.devtools`` to ``./manage node.env.dev``
- add ``make node.env.dev``
- remove local developer tools by ``make node.clean``

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-11 13:33:43 +01:00
Fauli1221
67dba29442
Merge branch 'searxng:master' into master 2023-02-01 08:42:39 +01:00
Alexandre Flament
a9d6f7532a weblate: migration to https://translate.codeberg.org/ 2023-01-21 15:45:12 +00:00
Alexandre Flament
f759a84af4 SearXNG version: fix make docker
continuation of #2117
related to #2111

This commit:
* fixes the Docker tag using an additional variable DOCKER_TAG, see searx/version.py
* fixes the Docker labels org.label-schema.vcs-ref and org.opencontainers.image.revision
* adds searx/version_frozen to .gitignore
2023-01-20 10:46:00 +00:00
Fauli1221
fa82fc17ec
Merge branch 'searxng:master' into master 2022-12-21 22:06:25 +01:00
Alexandre Flament
34e260f88f [fix] follow up of PR-1856
- the environment variable SEARXNG_REDIS_URL overrides the setting value redis.url
- ./manage sets SEARXNG_REDIS_URL to unix:///usr/local/searxng-redis/run/redis.sock if:
   - the socket exists
   - SEARXNG_REDIS_URL is not already defined

Update of PR #1856

Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2022-12-10 09:26:38 +01:00
Fauli1221
f60e26cdbc
Merge branch 'searxng:master' into master 2022-08-03 08:18:00 +02:00
Alexandre Flament
2babf59adc [fix] pyright repported errors
The errors make pyright usage useless since a new error won't be seen [1].

[1] https://github.com/searxng/searxng/pull/1569

```
  searx/compat.py:11:27 - error: Expression of type "Type[cached_property[_T@cached_property]]" cannot be assigned to declared type "Type[cached_property]"
    "Type[cached_property[_T@cached_property]]" is incompatible with "Type[cached_property]"
    Type "Type[cached_property[_T@cached_property]]" cannot be assigned to type "Type[cached_property]" (reportGeneralTypeIssues)
  searx/utils.py:69:36 - error: Expression of type "None" cannot be assigned to parameter of type "str"
    Type "None" cannot be assigned to type "str" (reportGeneralTypeIssues)
  searx/utils.py:573:85 - error: Expression of type "None" cannot be assigned to parameter of type "int"
    Type "None" cannot be assigned to type "int" (reportGeneralTypeIssues)
  searx/webapp.py:1306:22 - error: Argument of type "str" cannot be assigned to parameter "__a" of type "BytesPath" in function "join"
    Type "str" cannot be assigned to type "BytesPath"
      "str" is incompatible with "bytes"
      "str" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:68 - error: Argument of type "Literal['themes']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "Literal['themes']" cannot be assigned to type "BytesPath"
      "Literal['themes']" is incompatible with "bytes"
      "Literal['themes']" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:78 - error: Argument of type "str | Any | None" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "str | Any | None" cannot be assigned to type "BytesPath"
      Type "str" cannot be assigned to type "BytesPath"
        "str" is incompatible with "bytes"
        "str" is incompatible with protocol "PathLike[bytes]"
          "__fspath__" is not present (reportGeneralTypeIssues)
  searx/webapp.py:1306:85 - error: Argument of type "Literal['img']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join"
    Type "Literal['img']" cannot be assigned to type "BytesPath"
      "Literal['img']" is incompatible with "bytes"
      "Literal['img']" is incompatible with protocol "PathLike[bytes]"
        "__fspath__" is not present (reportGeneralTypeIssues)
  searx/engines/mongodb.py:8:6 - warning: Import "pymongo" could not be resolved (reportMissingImports)
  searx/engines/mysql_server.py:9:8 - warning: Import "mysql.connector" could not be resolved (reportMissingImports)
  searx/engines/postgresql.py:9:8 - warning: Import "psycopg2" could not be resolved from source (reportMissingModuleSource)
  searx/engines/xpath.py:187:28 - warning: "categories" is not defined (reportUndefinedVariable)
  searx/search/__init__.py:184:82 - warning: "flask" is not defined (reportUndefinedVariable)
  searx/search/checker/background.py:19:26 - error: Type of "schedule" is partially unknown
    Type of "schedule" is "(delay: Any, func: Any, *args: Any) -> Literal[True]" (reportUnknownVariableType)
  searx/shared/__init__.py:8:12 - warning: Import "uwsgi" could not be resolved (reportMissingImports)
  searx/shared/shared_uwsgi.py:5:8 - warning: Import "uwsgi" could not be resolved (reportMissingImports)
```
2022-07-30 18:04:44 +02:00
Markus Heiser
692708aa77 [clean up] drop obsolete searx, filtron and morty install scripts
Since ./utils/searxng.sh is implemented, the old installation procedures from
filtron, morty and searx can be removed.

For users who want to upgrade, the procedures for removing old installations
have still been retained.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Markus Heiser
ed8a169029 [doc] update documentation of the installation procedures
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-30 13:39:35 +02:00
Fauli1221
fb7fd7a897
Update manage
reading comments can be helpfull
2022-07-12 15:28:36 +02:00
Fauli1221
ba7d1abb5c
Update manage
set docker name manualy
2022-07-12 13:30:42 +02:00
Markus Heiser
c377679103 [fix] test.yamllint: return non-zero exit code on warnings
SearXNG's YAML files should be free of any warnings. This will stop the test
when there are warnings like::

     [warning] truthy value should be one of [false, true] (truthy)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-06-15 13:17:24 +02:00
Markus Heiser
9ed6261308 [fix] format.python: regexp argument '--include' in BLACK_TARGETS
--include TEXT

  A regular expression that matches files and directories that should be
  included on recursive searches. An empty value means all files are included
  regardless of the name.  ...  Exclusions are calculated first, inclusions
  later.  [default: \.pyi?$]

Closes: https://github.com/searxng/searxng/issues/1142
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-05-07 18:21:21 +02:00
Markus Heiser
e8541b6006 [theme] peel out oscar from SearXNG development
This is the first step of removing oscar theme

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-30 13:20:27 +02:00
Markus Heiser
50a4e30830 [mod] add test.pybabel to cover issues in translation tasks
We need to cover issue related to pybabel/translation tasks.  By example there
was an issue [1] (fixed [2]) when upgrading jinja2 from 3.0.3 to 3.1.0 [3] that
has not been covered by the CI tests.

[1] https://github.com/searxng/searxng/runs/5688624325?check_suite_focus=true#step:6:348
[2] https://github.com/searxng/searxng/pull/1011
[3] https://github.com/searxng/searxng/pull/1008

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-23 10:20:22 +02:00
Markus Heiser
cf644b413e [test.pyright] suppress unneeded error & warning messages
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-22 11:16:41 +02:00
Markus Heiser
b9cf3c82a1 [mod] add i18n infrastructure for SearXNG message files (searxng.msg)
With this patch ``searxng.msg`` files can be added to SearXNG.  In
``searxng.msg`` files messages can be defined which are not captured by babel's
gettext, like the generic names of the categories or messages that are stored in
constants.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-16 09:55:53 +01:00
Markus Heiser
b1912607ae [mod] replace /help by /info pages and include pages in project docs
This patch implements a bolierplate to share content from info-pages of the
SearXNG instance (URL /info) with the project documentation (path /docs/user).

The info pages are using Markdown (CommonMark), to include them in the project
documentation (reST) the myst-parser [1] is used in the Sphinx-doc build chain.

If base_url is known (defined in settings.yml) links to the instance are also
inserted into the project documentation::

    searxng_extra/docs_prebuild

[1] https://www.sphinx-doc.org/en/master/usage/markdown.html

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-03-12 11:36:31 +01:00
Markus Heiser
73a6da4dd9 [test.pyright] suppress warnings about intentional monkey patching
The warnings:

- "logger" is not defined'
- "supported_languages" is not defined'
- "language_aliases" is not defined'

are very verbose and superfluous, since these messages are related to
intentional monkey patching.

[1] https://github.com/searxng/searxng/pull/783#issuecomment-1019818178

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-25 23:26:53 +01:00
Markus Heiser
309147d86f [mod] introduce node.env.devtools function 2022-01-24 07:51:31 +01:00
Martin Fischer
6d701d2fea [fix] ensure that test.pyright installs pyright
nodejs.ensure only sets up NVM if there isn't a system-wide
installation of Node that matches our NODE_MINIMUM_VERSION.

The ubuntu image in the CI comes with an up to date node version,
so pyright from .nvm_packages is never installed.

This commit fixes this by introducing a package.json file.
2022-01-23 08:00:39 +01:00
Martin Fischer
96450b17d4 [mod] add test.pyright to test & ci.test targets
Since we currently have many type checking errors,
we for now only test with typeCheckingMode: off
which makes pyright only check files that contain a comment:

    # pyright: basic

to enable basic type checking, or

    # pyright: strict

to enable strict type checking.
2022-01-23 08:00:39 +01:00
Martin Fischer
3cd5ce55e8 [enh] clarify why we run pyright in the pyenv 2022-01-23 08:00:39 +01:00
Markus Heiser
8342773216 [mod] manage test.pyright: a commandline to run pyright tests
This patch implements the command and Makefile target::

    ./manage test.pyright
    make test.pyright

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-23 08:00:39 +01:00
Markus Heiser
fea9ad7927 [fix] issue reported by: make test.shell
$ make test.shell
    ./manage line 716:
    build_msg TEST "[reST markup] ${RST_FILES[@]}""
                                  ^-------------^ SC2145: Argument mixes string and array. Use * or separate argument.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-18 12:27:42 +01:00
Markus Heiser
27e3e56800 [mod] CI target test.rst: test reST markup of README.rst
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-17 09:35:42 +01:00
Markus Heiser
977e9a4330
Merge pull request #686 from return42/lib_redis
Add redis DB and connector
2022-01-11 19:55:14 +01:00
Markus Heiser
c9cfa6181b [fix] debug console (pdb) in a session started by make run
Commit c7f27404 moves the `python -m searx.webapp` process to the background.  A
background job can't open a simple python-debugger (pdb) console.

This patch moves all processes to the background except the searx.webapp.

Insert a break point somewhere in the webapp application::

    import pdb
    pdb.set_trace()

And start a debug session by::

    make run

and test you break point.

To test that the entire 'make run' stops in the case of an error in the
themes.live [1] background process try:

    make LIVE_THEME=typo-theme-name run

[1] https://github.com/searxng/searxng/pull/664#discussion_r776419585

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-05 10:09:17 +01:00
Markus Heiser
00d3a7477c
Merge pull request #664 from return42/themes.live
[mod] manage: add themes.live command (rebuild on modification)
2022-01-04 19:44:51 +01:00
Markus Heiser
fe8b88c8a4 [mod] script to build & install a redis instance
A script to build & install a simple & isolated redis service, dedicated to
SearXNG and connected via Unix socket.

    $ ./manage redis.help
    redis.:
      devpkg    : install essential packages to compile redis
      build     : build redis binaries at /800GBPCIex4/share/SearXNG/dist/redis/6.2.6/amd64
      install   : create user (searxng-redis) and install systemd service (searxng-redis)
      remove    : delete user (searxng-redis) and remove service (searxng-redis)
      shell     : start bash interpreter from user searxng-redis
      src       : clone redis source code to <path> and checkput 6.2.6
      useradd   : create user (searxng-redis) at /usr/local/searxng-redis
      userdel   : delete user (searxng-redis)
      addgrp    : add <user> to group (searxng-redis)
      rmgrp     : remove <user> from group (searxng-redis)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-03 14:34:26 +01:00
Markus Heiser
c7f2740414 [mod] live build of a theme: LIVE_THEME=simple make run
Environment variable to get live builds while modifying CSS & JS of a theme::

    LIVE_THEME=simple make run

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser
1706b130a4 [fix] remove leftover from 59f4c792
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser
edfd069d16 [mod] manage: add themes.live command (rebuild on modification)
To have a live build while editing the HTML client of a theme.  The first
argument of the command is the theme name:

    ./manage themes.live simple

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser
c849731eb1 [mod] test.black: show issues (diff) reported by python-black
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 08:59:47 +01:00
Alexandre Flament
3c77412d33 [mod] replace pycodestyle by black
"make test.black" checks for the code style
"make format.python" format the python code
2021-12-27 08:58:59 +01:00