Commit graph

5151 commits

Author SHA1 Message Date
Markus Heiser
d9ffb09011 [filtron.sh] make filtron rule file configurable
To select a different file with filtron rules, set environment

    FILTRON_RULES_TEMPLATE

the default is

    utils/templates/etc/filtron/rules.json

The installation is done by the new function install_rules() which offers a
multiple choice in case of collisions (known from searx.sh install setup).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28 12:48:49 +02:00
Alexandre Flament
fe3831db1a
Merge pull request #37 from return42/drop-req-idna
[fix] drop 'idna' from requirements.txt
2021-04-27 20:08:52 +02:00
Alexandre Flament
b06a15376c
Merge pull request #32 from searxng/fix-network
[fix] searx.network: fix rare cases where LOOP is None
2021-04-27 20:06:54 +02:00
Alexandre Flament
283ae7bfad [fix] searx.network: fix rare cases where LOOP is None
* searx.network.client.LOOP is initialized in a thread
* searx.network.__init__ imports LOOP which may happen
  before the thread has initialized LOOP

This commit adds a new function "searx.network.client.get_loop()"
to fix this issue
2021-04-27 17:47:36 +02:00
Markus Heiser
f724d6f6f1
Merge pull request #33 from searxng/remove-deprecated-files
[clean] drop deprecated files .codecov.yml, .landscape.yaml & tox.ini
2021-04-27 14:58:22 +00:00
Alexandre Flament
5781d75ca2 [clean] drop deprecated files .codecov.yml, .landscape.yaml & tox.ini
.codecov.yml :
  code coverage reports are attached on each build. See for example:
  https://github.com/searxng/searxng/actions/runs/788360104 (Artifacts section)

.landscape.yaml :
  landscape.io is offline

tox.ini :
  not in use
2021-04-27 16:52:53 +02:00
Alexandre Flament
87e914e398
Merge pull request #28 from searxng/mod-processors-error-message
[mod] processors: show identical error messages on /search and /stats
2021-04-27 16:50:41 +02:00
Markus Heiser
4afe5a8075
Merge pull request #34 from searxng/error_recorder_relative_filename
[mod] searx.metrics.error_recorder: store relative file name instead of the full absolute file name.
2021-04-27 14:36:34 +00:00
Markus Heiser
924f9afea3 [lint] pylint searx/search/processors files / BTW add some doc-strings
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-27 15:13:39 +02:00
Alexandre Flament
f45729ae72
Merge pull request #35 from return42/fix-manage
[fix] manage - fix miss usage of 'set -e'
2021-04-27 14:39:43 +02:00
Alexandre Flament
b1557b5443 [mod] processors: show identical error messages on /search and /stats 2021-04-27 14:20:07 +02:00
Markus Heiser
bdf3de4c00 [fix] drop 'idna' from requirements.txt
Requirement idna was added in 181c12ae04 but I don't know why.  This package
is not directly used by searxng but its a sub-requirement of some other packages
using package `requests` (with different range of supported versions, see
below).  In summary one can say: the version of idna should be depend on package
`requests`::

    ...
    Pallets-Sphinx-Themes==1.2.3
      ...
      - Sphinx [required: Any, installed: 3.5.4]
        ...
        - requests [required: >=2.5.0, installed: 2.25.1]
	  ...
          - idna [required: >=2.5,<3, installed: 2.10]
        ...
    ...
    transifex-client==0.14.2
      - requests [required: >=2.19.1,<3.0.0, installed: 2.25.1]
        ...
        - idna [required: >=2.5,<3, installed: 2.10]
    twine==3.4.1
      ...
      - requests [required: >=2.20, installed: 2.25.1]
        ...
        - idna [required: >=2.5,<3, installed: 2.10]

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-27 14:03:08 +02:00
Markus Heiser
3bd2f3dc81 [fix] manage - fix miss usage of 'set -e'
The philosophy of set -e is typically that it only exits upon uncaught
errors. Here, the presence of || outside the subshell seems to tell the shell
that the error inside the subshell is 'caught' and therefore set -e does not
cause an exit after false [1].

The shell does not exit if the command that fails is ... part of any command
executed in a && or || list except the command following the final && or ||, any
command in a pipeline but the last, or if the command’s return status is being
inverted with ! [2]

[1] https://unix.stackexchange.com/questions/296526/set-e-in-a-subshell
[2] https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html#The-Set-Builtin

BTW: fix error reported by 'make test.shell'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-27 13:02:55 +02:00
Alexandre Flament
b8f5adf500 [mod] searx.metrics.error_recorder: store relative file name instead of the full absolute file name. 2021-04-27 10:42:00 +02:00
Alexandre Flament
9b482e8fcc
Merge pull request #31 from return42/mod-pylint
[pylint] get PYLINT_FILES from tag '# lint: pylint'
2021-04-26 22:58:44 +02:00
Markus Heiser
a92a4cb443 [pylint] get PYLINT_FILES from tag '# lint: pylint'
These py files are linted by test.pylint(), all other files are linted by
test.pep8()

close: https://github.com/searxng/searxng/issues/21
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 20:52:52 +02:00
Markus Heiser
dc29f1d826 [pylint] tag PYLINT_FILES by comment # lint: pylint
These py files are linted by `test.pylint`, all other files are linted by
`test.pep8`.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 20:18:20 +02:00
Markus Heiser
e6deddbc70
Merge pull request #2 from searxng/dependabot/pip/master/pygments-2.8.1
Bump pygments from 2.8.0 to 2.8.1 and run

   ./manage pyenv.cmd searx_extra/update/update_pygments.py
2021-04-26 17:19:59 +00:00
Markus Heiser
6b92e8c6fd [upd] ./manage pyenv.cmd searx_extra/update/update_pygments.py
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 19:11:58 +02:00
Markus Heiser
ec10ba7fcb [mod] update_pygments -- prompt filenames of updated files
BTW: remove trailing whitespace

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 19:09:40 +02:00
dependabot[bot]
3cf97774da
Bump pygments from 2.8.0 to 2.8.1
Bumps [pygments](https://github.com/pygments/pygments) from 2.8.0 to 2.8.1.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/2.8.1/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.8.0...2.8.1)

Signed-off-by: dependabot[bot] <support@github.com>
2021-04-26 15:32:17 +00:00
dependabot[bot]
26de9f5c2b Bump sphinx from 3.5.3 to 3.5.4
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 3.5.3 to 3.5.4.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/commits/v3.5.4)

Signed-off-by: dependabot[bot] <support@github.com>
2021-04-26 15:28:00 +00:00
Alexandre Flament
f4ab1e1251
Merge pull request #30 from return42/brand-readme
[brand] README.rst fix img.shields.io and searxng.github.io URLs
2021-04-26 16:13:08 +02:00
Markus Heiser
eac8e41017 [brand] README.rst fix img.shields.io and searxng.github.io URLs
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 15:47:56 +02:00
Alexandre Flament
12bdf7146b
Merge pull request #27 from return42/fix-debug-log
[fix] debug log: UnicodeEncodeError: 'ascii' codec can't encode
2021-04-26 13:50:29 +02:00
Markus Heiser
1a402be175 [fix] debug log: UnicodeEncodeError: 'ascii' codec can't encode
The issue exists only in the debug log::

     --- Logging error ---
     Traceback (most recent call last):
       File "/usr/lib/python3.9/logging/__init__.py", line 1086, in emit
	 stream.write(msg + self.terminator)
     UnicodeEncodeError: 'ascii' codec can't encode characters in position 79-89: ordinal not in range(128)
     Call stack:
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 2464, in __call__
	 return self.wsgi_app(environ, start_response)
       File "/usr/local/searx/searx-src/searx/webapp.py", line 1316, in __call__
	 return self.app(environ, start_response)
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__
	 return self.app(environ, start_response)
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
	 response = self.full_dispatch_request()
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
	 rv = self.dispatch_request()
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
	 return self.view_functions[rule.endpoint](**req.view_args)
       File "/usr/local/searx/searx-src/searx/webapp.py", line 766, in search
	 number_of_results=format_decimal(number_of_results),
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask_babel/__init__.py", line 458, in format_decimal
	 locale = get_locale()
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask_babel/__init__.py", line 226, in get_locale
	 rv = babel.locale_selector_func()
       File "/usr/local/searx/searx-src/searx/webapp.py", line 249, in get_locale
	 logger.debug("%s uses locale `%s` from %s", request.url, locale, locale_source)
     Unable to print the message and arguments - possible formatting error.
     Use the traceback above to help find the error.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 12:55:26 +02:00
Alexandre Flament
51494849fc
Merge pull request #23 from searxng/metrics-stats-engines
Metrics stats engines
2021-04-25 14:31:52 +02:00
Alexandre Flament
d20f2822d0
Merge pull request #25 from return42/brand-001
SearxNG brand
2021-04-25 14:30:18 +02:00
Alexandre Flament
df41b77121 [mod] /stats : detail per engine
allow to submit a github issue including the technical details
(exceptions, errors, warning, checker result)
2021-04-25 14:28:06 +02:00
Markus Heiser
28b25185c5 [brand] searxng -- fix links to issue tracker & WEB-GUI
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 14:25:08 +02:00
Alexandre Flament
eebac31ccd
Merge pull request #26 from return42/brand-002
[github] searxng does not use github's discussion
2021-04-25 12:59:11 +02:00
Markus Heiser
272c9d6b04 [github] searxng does not use github's discussion
Remove gihtub's discussion template and add a issue template used for questions
& discussions.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 12:37:15 +02:00
Markus Heiser
c6a5cc019a [brand] searxng is a fork from searx
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 11:42:42 +02:00
Alexandre Flament
9937a908c0
Merge pull request #24 from return42/fix-doc
[fix doc] manage.sh update_packages does not exists anymore
2021-04-25 10:22:32 +02:00
Markus Heiser
3aa153f08e [fix doc] manage.sh update_packages does not exists anymore
Reported-by: https://github.com/searx/searx/issues/2776
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-24 17:03:10 +02:00
Alexandre Flament
0603b043ce
Merge pull request #16 from return42/add-core.ac
Add a search engine for core.ac.uk
2021-04-24 09:11:20 +02:00
Markus Heiser
6d41255eb1 [fix] Springer-Nature & core.ac need API key to work
It is the whole engine definition that should be comment out : without the
api_key nothing works.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
Markus Heiser
8efabd3ab7 [mod] core.ac.uk engine
- add to list of pylint scripts
- add debug log messages
- move API key int `settings.yml`
- improved readability
- add some metadata to results

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
spongebob33
7528e38c8a add core.ac.uk engine 2021-04-24 08:55:45 +02:00
Alexandre Flament
4863c1933e Rerevert c54bf42 2021-04-24 07:59:16 +02:00
Alexandre Flament
525bfe7830
Fix PEP8 2021-04-24 07:19:44 +02:00
Alexandre Flament
a7b9eca98a
Merge pull request #8 from return42/manage-script
Replace Makefile boilerplate by shell scripts
2021-04-24 07:14:35 +02:00
Alexandre Flament
fe064a5c39 temporary partial revert of c54bf42cb9 2021-04-24 07:13:04 +02:00
Alexandre Flament
3cdd6a6a50
Merge pull request #12 from searxng/metrics-stats-page
[mod] update /stats
2021-04-24 07:02:52 +02:00
Alexandre Flament
7032f7521c [mod] /stats: simple theme implementation 2021-04-24 06:58:49 +02:00
Alexandre Flament
09e7ecdce2 [mod] /stats : add reliability column and sort by column links 2021-04-24 06:58:49 +02:00
Markus Heiser
c54bf42cb9 [mod] metrics: add to pylint and justify indentations
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-24 06:58:49 +02:00
Alexandre Flament
65c29081cc [mod] update /stats 2021-04-24 06:58:22 +02:00
Alexandre Flament
461c6fb21e
Merge pull request #20 from return42/fix-commit-msg
[CI] JamesIves/github-pages-deploy-action COMMIT_MESSAGE
2021-04-23 14:38:07 +02:00
Markus Heiser
360266bca9 [CI] JamesIves/github-pages-deploy-action COMMIT_MESSAGE
remove rocket symbol from the commit message

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Suggested-by: https://github.com/JamesIves/github-pages-deploy-action/pull/576
2021-04-23 14:13:13 +02:00