Commit graph

4443 commits

Author SHA1 Message Date
Markus Heiser
924f9afea3 [lint] pylint searx/search/processors files / BTW add some doc-strings
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-27 15:13:39 +02:00
Alexandre Flament
f45729ae72
Merge pull request #35 from return42/fix-manage
[fix] manage - fix miss usage of 'set -e'
2021-04-27 14:39:43 +02:00
Alexandre Flament
b1557b5443 [mod] processors: show identical error messages on /search and /stats 2021-04-27 14:20:07 +02:00
Markus Heiser
bdf3de4c00 [fix] drop 'idna' from requirements.txt
Requirement idna was added in 181c12ae04 but I don't know why.  This package
is not directly used by searxng but its a sub-requirement of some other packages
using package `requests` (with different range of supported versions, see
below).  In summary one can say: the version of idna should be depend on package
`requests`::

    ...
    Pallets-Sphinx-Themes==1.2.3
      ...
      - Sphinx [required: Any, installed: 3.5.4]
        ...
        - requests [required: >=2.5.0, installed: 2.25.1]
	  ...
          - idna [required: >=2.5,<3, installed: 2.10]
        ...
    ...
    transifex-client==0.14.2
      - requests [required: >=2.19.1,<3.0.0, installed: 2.25.1]
        ...
        - idna [required: >=2.5,<3, installed: 2.10]
    twine==3.4.1
      ...
      - requests [required: >=2.20, installed: 2.25.1]
        ...
        - idna [required: >=2.5,<3, installed: 2.10]

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-27 14:03:08 +02:00
Markus Heiser
3bd2f3dc81 [fix] manage - fix miss usage of 'set -e'
The philosophy of set -e is typically that it only exits upon uncaught
errors. Here, the presence of || outside the subshell seems to tell the shell
that the error inside the subshell is 'caught' and therefore set -e does not
cause an exit after false [1].

The shell does not exit if the command that fails is ... part of any command
executed in a && or || list except the command following the final && or ||, any
command in a pipeline but the last, or if the command’s return status is being
inverted with ! [2]

[1] https://unix.stackexchange.com/questions/296526/set-e-in-a-subshell
[2] https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html#The-Set-Builtin

BTW: fix error reported by 'make test.shell'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-27 13:02:55 +02:00
Alexandre Flament
b8f5adf500 [mod] searx.metrics.error_recorder: store relative file name instead of the full absolute file name. 2021-04-27 10:42:00 +02:00
Alexandre Flament
9b482e8fcc
Merge pull request #31 from return42/mod-pylint
[pylint] get PYLINT_FILES from tag '# lint: pylint'
2021-04-26 22:58:44 +02:00
Markus Heiser
a92a4cb443 [pylint] get PYLINT_FILES from tag '# lint: pylint'
These py files are linted by test.pylint(), all other files are linted by
test.pep8()

close: https://github.com/searxng/searxng/issues/21
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 20:52:52 +02:00
Markus Heiser
dc29f1d826 [pylint] tag PYLINT_FILES by comment # lint: pylint
These py files are linted by `test.pylint`, all other files are linted by
`test.pep8`.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 20:18:20 +02:00
Markus Heiser
e6deddbc70
Merge pull request #2 from searxng/dependabot/pip/master/pygments-2.8.1
Bump pygments from 2.8.0 to 2.8.1 and run

   ./manage pyenv.cmd searx_extra/update/update_pygments.py
2021-04-26 17:19:59 +00:00
Markus Heiser
6b92e8c6fd [upd] ./manage pyenv.cmd searx_extra/update/update_pygments.py
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 19:11:58 +02:00
Markus Heiser
ec10ba7fcb [mod] update_pygments -- prompt filenames of updated files
BTW: remove trailing whitespace

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 19:09:40 +02:00
dependabot[bot]
3cf97774da
Bump pygments from 2.8.0 to 2.8.1
Bumps [pygments](https://github.com/pygments/pygments) from 2.8.0 to 2.8.1.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/2.8.1/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.8.0...2.8.1)

Signed-off-by: dependabot[bot] <support@github.com>
2021-04-26 15:32:17 +00:00
dependabot[bot]
26de9f5c2b Bump sphinx from 3.5.3 to 3.5.4
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 3.5.3 to 3.5.4.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/commits/v3.5.4)

Signed-off-by: dependabot[bot] <support@github.com>
2021-04-26 15:28:00 +00:00
Alexandre Flament
f4ab1e1251
Merge pull request #30 from return42/brand-readme
[brand] README.rst fix img.shields.io and searxng.github.io URLs
2021-04-26 16:13:08 +02:00
Markus Heiser
eac8e41017 [brand] README.rst fix img.shields.io and searxng.github.io URLs
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 15:47:56 +02:00
Alexandre Flament
12bdf7146b
Merge pull request #27 from return42/fix-debug-log
[fix] debug log: UnicodeEncodeError: 'ascii' codec can't encode
2021-04-26 13:50:29 +02:00
Markus Heiser
1a402be175 [fix] debug log: UnicodeEncodeError: 'ascii' codec can't encode
The issue exists only in the debug log::

     --- Logging error ---
     Traceback (most recent call last):
       File "/usr/lib/python3.9/logging/__init__.py", line 1086, in emit
	 stream.write(msg + self.terminator)
     UnicodeEncodeError: 'ascii' codec can't encode characters in position 79-89: ordinal not in range(128)
     Call stack:
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 2464, in __call__
	 return self.wsgi_app(environ, start_response)
       File "/usr/local/searx/searx-src/searx/webapp.py", line 1316, in __call__
	 return self.app(environ, start_response)
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__
	 return self.app(environ, start_response)
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
	 response = self.full_dispatch_request()
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
	 rv = self.dispatch_request()
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
	 return self.view_functions[rule.endpoint](**req.view_args)
       File "/usr/local/searx/searx-src/searx/webapp.py", line 766, in search
	 number_of_results=format_decimal(number_of_results),
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask_babel/__init__.py", line 458, in format_decimal
	 locale = get_locale()
       File "/usr/local/searx/searx-pyenv/lib/python3.9/site-packages/flask_babel/__init__.py", line 226, in get_locale
	 rv = babel.locale_selector_func()
       File "/usr/local/searx/searx-src/searx/webapp.py", line 249, in get_locale
	 logger.debug("%s uses locale `%s` from %s", request.url, locale, locale_source)
     Unable to print the message and arguments - possible formatting error.
     Use the traceback above to help find the error.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-26 12:55:26 +02:00
Alexandre Flament
51494849fc
Merge pull request #23 from searxng/metrics-stats-engines
Metrics stats engines
2021-04-25 14:31:52 +02:00
Alexandre Flament
d20f2822d0
Merge pull request #25 from return42/brand-001
SearxNG brand
2021-04-25 14:30:18 +02:00
Alexandre Flament
df41b77121 [mod] /stats : detail per engine
allow to submit a github issue including the technical details
(exceptions, errors, warning, checker result)
2021-04-25 14:28:06 +02:00
Markus Heiser
28b25185c5 [brand] searxng -- fix links to issue tracker & WEB-GUI
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 14:25:08 +02:00
Alexandre Flament
eebac31ccd
Merge pull request #26 from return42/brand-002
[github] searxng does not use github's discussion
2021-04-25 12:59:11 +02:00
Markus Heiser
272c9d6b04 [github] searxng does not use github's discussion
Remove gihtub's discussion template and add a issue template used for questions
& discussions.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 12:37:15 +02:00
Markus Heiser
c6a5cc019a [brand] searxng is a fork from searx
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 11:42:42 +02:00
Alexandre Flament
9937a908c0
Merge pull request #24 from return42/fix-doc
[fix doc] manage.sh update_packages does not exists anymore
2021-04-25 10:22:32 +02:00
Markus Heiser
3aa153f08e [fix doc] manage.sh update_packages does not exists anymore
Reported-by: https://github.com/searx/searx/issues/2776
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-24 17:03:10 +02:00
Alexandre Flament
0603b043ce
Merge pull request #16 from return42/add-core.ac
Add a search engine for core.ac.uk
2021-04-24 09:11:20 +02:00
Markus Heiser
6d41255eb1 [fix] Springer-Nature & core.ac need API key to work
It is the whole engine definition that should be comment out : without the
api_key nothing works.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
Markus Heiser
8efabd3ab7 [mod] core.ac.uk engine
- add to list of pylint scripts
- add debug log messages
- move API key int `settings.yml`
- improved readability
- add some metadata to results

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
spongebob33
7528e38c8a add core.ac.uk engine 2021-04-24 08:55:45 +02:00
Alexandre Flament
4863c1933e Rerevert c54bf42 2021-04-24 07:59:16 +02:00
Alexandre Flament
525bfe7830
Fix PEP8 2021-04-24 07:19:44 +02:00
Alexandre Flament
a7b9eca98a
Merge pull request #8 from return42/manage-script
Replace Makefile boilerplate by shell scripts
2021-04-24 07:14:35 +02:00
Alexandre Flament
fe064a5c39 temporary partial revert of c54bf42cb9 2021-04-24 07:13:04 +02:00
Alexandre Flament
3cdd6a6a50
Merge pull request #12 from searxng/metrics-stats-page
[mod] update /stats
2021-04-24 07:02:52 +02:00
Alexandre Flament
7032f7521c [mod] /stats: simple theme implementation 2021-04-24 06:58:49 +02:00
Alexandre Flament
09e7ecdce2 [mod] /stats : add reliability column and sort by column links 2021-04-24 06:58:49 +02:00
Markus Heiser
c54bf42cb9 [mod] metrics: add to pylint and justify indentations
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-24 06:58:49 +02:00
Alexandre Flament
65c29081cc [mod] update /stats 2021-04-24 06:58:22 +02:00
Alexandre Flament
461c6fb21e
Merge pull request #20 from return42/fix-commit-msg
[CI] JamesIves/github-pages-deploy-action COMMIT_MESSAGE
2021-04-23 14:38:07 +02:00
Markus Heiser
360266bca9 [CI] JamesIves/github-pages-deploy-action COMMIT_MESSAGE
remove rocket symbol from the commit message

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Suggested-by: https://github.com/JamesIves/github-pages-deploy-action/pull/576
2021-04-23 14:13:13 +02:00
Alexandre Flament
1a6e3b3ed0
Merge pull request #19 from return42/fix-gh-pages
[CI] JamesIves/github-pages-deploy-action SINGLE_COMMIT
2021-04-23 13:00:52 +02:00
Markus Heiser
5ecd539c6a [CI] JamesIves/github-pages-deploy-action SINGLE_COMMIT
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 12:01:11 +02:00
Markus Heiser
abd423cbf8 [fix] Makefile fix target 'test' & 'test.sh'
Renamed 'test.sh' to 'test.shell' seems to solve this issue on my ubu204::

    $ make test.sh
    ...
    TEST      robot
    ...
    INSTALL   gecko.driver
    ...
    Running 7 tests
    cat test.sh >test
    cat: test.sh: Datei oder Verzeichnis nicht gefunden
    make: *** [<eingebaut>: test] Fehler 1

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
c69fc20be2 [fix] ./manage pyenv.install - die when pyenv build fails
ERROR: When pyenv build fails, the pyenv command runs into a recursion [1]:

    ....
    BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  ERROR: Cannot install idna==3.1 and requests[socks]==2.25.1 because these package versions have conflicting dependencies.
    BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV
    BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  The conflict is caused by:

[1] https://github.com/return42/searx-next/pull/4#issuecomment-821999497

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
34badee467 [doc] modify docs to fit to the new build boilerplate
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
d0e371f474 [mod] replace makefile boilerplate by 'manage' script
Replaces the make targets with the bash scripts

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
720d0d48ad [enh] implement all build task in a bash script (manage)
note: in further patches script 'manage' will replace 'manage.sh'

pyenv.* :
  assert    : build virtualenv if not exists
  intsall   : developer install of searx into virtualenv
  unintsall : uninstall developer installation
  cmd ...   : run command ... in virtualenv
  OK        : test if virtualenv is OK
pypi.upload:
  Upload python packages to PyPi (to test use pypi.upload.test)
pybuild :
  Build python packages at ./${PYDIST}
pyclean :
  delete virtualenv and intermediate py files
test.* :
  pylint    : lint PYLINT_FILES, searx/engines, searx & tests
  pep8      : pycodestyle (pep8) for all files except PYLINT_FILES
  unit      : run unit tests
  coverage  : run unit tests with coverage
  robot     : run robot test
  clean     : clean intermediate test stuff
node.* :
  env       : download & install npm dependencies locally
  clean     : drop npm installations
buildenv :
  rebuild ./utils/brand.env
data.* :
  all       : update searx/languages.py and ./data/*
  languages : update searx/data/engines_languages.json & searx/languages.py
  useragents: update searx/data/useragents.json with the most recent versions of Firefox.
themes.* :
  all       : build all themes
  oscar     : build oscar theme
  simple    : build simple theme
  bootstrap : less compile bootstrap.min.css CSS
babel.compile :
  pybabel compile ./searx/translations
docs.* :
  html      : build HTML documentation
  gh-pages  : deploy on gh-pages branch
  autobuild : autobuild HTML documentation while editing
  prebuild  : build reST include files (./${DOCS_BUILD}/includes)
  clean     : clean documentation build
docker.build [push] :
  build (and push) docker image
gecko.driver :
  download & install geckodriver if not already installed (required for
  robot_tests)

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
438d35c386 [fix] do not spit out error messages for utils/lxc.sh --help
Error::

  $ utils/lxc.sh --help
  ERROR: missing command lxc
  ...

This breaks also docs build::

  $ make docs
  ...
  WARNING: Unexpected return code 42 from command '../utils/lxc.sh --help'
  ...

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:14:29 +02:00