Commit graph

5115 commits

Author SHA1 Message Date
Markus Heiser
6d41255eb1 [fix] Springer-Nature & core.ac need API key to work
It is the whole engine definition that should be comment out : without the
api_key nothing works.

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
Markus Heiser
8efabd3ab7 [mod] core.ac.uk engine
- add to list of pylint scripts
- add debug log messages
- move API key int `settings.yml`
- improved readability
- add some metadata to results

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-24 09:00:53 +02:00
spongebob33
7528e38c8a add core.ac.uk engine 2021-04-24 08:55:45 +02:00
Alexandre Flament
4863c1933e Rerevert c54bf42 2021-04-24 07:59:16 +02:00
Alexandre Flament
525bfe7830
Fix PEP8 2021-04-24 07:19:44 +02:00
Alexandre Flament
a7b9eca98a
Merge pull request #8 from return42/manage-script
Replace Makefile boilerplate by shell scripts
2021-04-24 07:14:35 +02:00
Alexandre Flament
fe064a5c39 temporary partial revert of c54bf42cb9 2021-04-24 07:13:04 +02:00
Alexandre Flament
3cdd6a6a50
Merge pull request #12 from searxng/metrics-stats-page
[mod] update /stats
2021-04-24 07:02:52 +02:00
Alexandre Flament
7032f7521c [mod] /stats: simple theme implementation 2021-04-24 06:58:49 +02:00
Alexandre Flament
09e7ecdce2 [mod] /stats : add reliability column and sort by column links 2021-04-24 06:58:49 +02:00
Markus Heiser
c54bf42cb9 [mod] metrics: add to pylint and justify indentations
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-24 06:58:49 +02:00
Alexandre Flament
65c29081cc [mod] update /stats 2021-04-24 06:58:22 +02:00
Alexandre Flament
461c6fb21e
Merge pull request #20 from return42/fix-commit-msg
[CI] JamesIves/github-pages-deploy-action COMMIT_MESSAGE
2021-04-23 14:38:07 +02:00
Markus Heiser
360266bca9 [CI] JamesIves/github-pages-deploy-action COMMIT_MESSAGE
remove rocket symbol from the commit message

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Suggested-by: https://github.com/JamesIves/github-pages-deploy-action/pull/576
2021-04-23 14:13:13 +02:00
Alexandre Flament
1a6e3b3ed0
Merge pull request #19 from return42/fix-gh-pages
[CI] JamesIves/github-pages-deploy-action SINGLE_COMMIT
2021-04-23 13:00:52 +02:00
Markus Heiser
5ecd539c6a [CI] JamesIves/github-pages-deploy-action SINGLE_COMMIT
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 12:01:11 +02:00
Markus Heiser
abd423cbf8 [fix] Makefile fix target 'test' & 'test.sh'
Renamed 'test.sh' to 'test.shell' seems to solve this issue on my ubu204::

    $ make test.sh
    ...
    TEST      robot
    ...
    INSTALL   gecko.driver
    ...
    Running 7 tests
    cat test.sh >test
    cat: test.sh: Datei oder Verzeichnis nicht gefunden
    make: *** [<eingebaut>: test] Fehler 1

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
c69fc20be2 [fix] ./manage pyenv.install - die when pyenv build fails
ERROR: When pyenv build fails, the pyenv command runs into a recursion [1]:

    ....
    BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  ERROR: Cannot install idna==3.1 and requests[socks]==2.25.1 because these package versions have conflicting dependencies.
    BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV
    BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  BUILDENV  The conflict is caused by:

[1] https://github.com/return42/searx-next/pull/4#issuecomment-821999497

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
34badee467 [doc] modify docs to fit to the new build boilerplate
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
d0e371f474 [mod] replace makefile boilerplate by 'manage' script
Replaces the make targets with the bash scripts

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:26:22 +02:00
Markus Heiser
720d0d48ad [enh] implement all build task in a bash script (manage)
note: in further patches script 'manage' will replace 'manage.sh'

pyenv.* :
  assert    : build virtualenv if not exists
  intsall   : developer install of searx into virtualenv
  unintsall : uninstall developer installation
  cmd ...   : run command ... in virtualenv
  OK        : test if virtualenv is OK
pypi.upload:
  Upload python packages to PyPi (to test use pypi.upload.test)
pybuild :
  Build python packages at ./${PYDIST}
pyclean :
  delete virtualenv and intermediate py files
test.* :
  pylint    : lint PYLINT_FILES, searx/engines, searx & tests
  pep8      : pycodestyle (pep8) for all files except PYLINT_FILES
  unit      : run unit tests
  coverage  : run unit tests with coverage
  robot     : run robot test
  clean     : clean intermediate test stuff
node.* :
  env       : download & install npm dependencies locally
  clean     : drop npm installations
buildenv :
  rebuild ./utils/brand.env
data.* :
  all       : update searx/languages.py and ./data/*
  languages : update searx/data/engines_languages.json & searx/languages.py
  useragents: update searx/data/useragents.json with the most recent versions of Firefox.
themes.* :
  all       : build all themes
  oscar     : build oscar theme
  simple    : build simple theme
  bootstrap : less compile bootstrap.min.css CSS
babel.compile :
  pybabel compile ./searx/translations
docs.* :
  html      : build HTML documentation
  gh-pages  : deploy on gh-pages branch
  autobuild : autobuild HTML documentation while editing
  prebuild  : build reST include files (./${DOCS_BUILD}/includes)
  clean     : clean documentation build
docker.build [push] :
  build (and push) docker image
gecko.driver :
  download & install geckodriver if not already installed (required for
  robot_tests)

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
438d35c386 [fix] do not spit out error messages for utils/lxc.sh --help
Error::

  $ utils/lxc.sh --help
  ERROR: missing command lxc
  ...

This breaks also docs build::

  $ make docs
  ...
  WARNING: Unexpected return code 42 from command '../utils/lxc.sh --help'
  ...

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
d5eaa3fb31 [enh] utils/lib.sh - commands to build Sphinx-doc & deploy gh-pages
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
036933599b [enh] utils/lib.sh - commands pyenv, pyenv.drop pyenv.(un)install
Implement a boilerplate to manage performance optimized virtualenv builds.
Shell scripts can use (e.g.) 'pyenv.cmd' to execute command in the virtualenv
without having to worry about whether and how the environment is provided. ::

  pyenv.cmd which python
  ..../local/py3/bin/python

  pyenv.cmd which pip
  ..../local/py3/bin/pip

If pyenv.cmd released multiple times the installation will only rebuild if the
function 'pyenv.OK' fails.  Function 'pyenv.OK' make some test to validate that
the virtualenv exists and works as expected.  The check also fails if
requirements listed requirements-dev.txt and requirements.txt has been edited.
Among these tests 'pyenv.OK' calls 'pyenv.check' which implements a python
script that validate the python installation.  Here is an example how a
'pyenv.check' implementation could look like::

    pyenv.check() {
       cat  <<EOF
    import yaml
    print('import yaml --> OK')
    EOF
    }

Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23 11:14:29 +02:00
Markus Heiser
f9b05a6c44 [fix] utils/searx.sh - Double quote to prevent globbing
Fix issues reported by `make test.sh`

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:14:29 +02:00
Alexandre Flament
3755cf2eba More branding 2021-04-22 21:10:22 +02:00
Alexandre Flament
04b34c70b8 More branding 2021-04-22 20:14:16 +02:00
Alexandre Flament
e8c2fbc433
Merge pull request #17 from searxng/searxng
searxng brand
2021-04-22 20:01:02 +02:00
Alexandre Flament
9d84025e4f searxng brand 2021-04-22 19:57:02 +02:00
Alexandre Flament
d01741c9a2
Merge pull request #15 from return42/add-springer
Add a search engine for Springer Nature
2021-04-22 13:23:31 +02:00
Alexandre Flament
cdcfd74f39
Merge pull request #14 from return42/add-artic
[enh] add engine using API from "The Art Institute of Chicago"
2021-04-22 13:22:49 +02:00
Markus Heiser
213041adc7 [enh] themes - add result.metadata to result sub header
Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-22 12:37:18 +02:00
Pierre Chevalier
a80bf1ba97 [enh] Add Springer Nature engine
Springer Nature is a global publisher dedicated to providing service to research
community [1] with official API [2].

To test this PR, first get your API key following this page:

   https://dev.springernature.com/signup

In searx/engines/springer.py at line 24, add this API key.  I left my own key,
commented out in the line aboce.  Feel free to use it, if needed.

[1] https://www.springernature.com/
[2] https://dev.springernature.com/
2021-04-22 12:35:25 +02:00
habsinn
41a2e3785e [enh] add engine using API from "The Art Institute of Chicago" 2021-04-22 12:25:43 +02:00
Alexandre Flament
981cdb7acf
Merge pull request #10 from return42/fix-youtube
[fix] youtube - send CONSENT Cookie to not be redirected
2021-04-22 12:21:31 +02:00
Alexandre Flament
e9508cc897
Merge pull request #13 from return42/fix-morty-doc
[fix] Fix typo in morty installation instruction
2021-04-22 12:20:52 +02:00
Alexandre Flament
a0e55a587a
Merge pull request #11 from return42/add-doi
[enh] add DOI resolver from sci-hub / replace default DOI
2021-04-22 12:20:37 +02:00
Zackptg5
bed7c3c614 [fix] Fix typo in morty installation instruction 2021-04-22 12:17:49 +02:00
Markus Heiser
9a05ad4c73 [enh] add DOI resolver from sci-hub / replace default DOI
The new sci-hub URLs are comming from @aurora-vasiliev [1].

[1] https://github.com/searx/searx/pull/2706

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-22 12:13:01 +02:00
Markus Heiser
e9a6ab4015 [fix] youtube - send CONSENT Cookie to not be redirected
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking.  To get the consent
from the user, youtube requests are redirected to confirm and get a CONSENT
Cookie from https://consent.youtube.com

This patch adds a CONSENT Cookie to the youtube request to avoid redirection.

[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Reported-by: https://github.com/searx/searx/issues/2774
2021-04-22 12:09:09 +02:00
Alexandre Flament
f251a9718e
Merge pull request #9 from return42/drop-https-rewrite
[drop] plugin HTTPS rewrite - rule base is to old (update is to big)
2021-04-22 11:55:58 +02:00
Markus Heiser
247c46c6b0 [drop] plugin HTTPS rewrite - rule base is to old (update is to big)
The usefulness of the _HTTPS rewrite_ plugin is questionable:

- the 36 rule files have not been updated since 2015 [1]
- actual there are 23760 rule files in the https-everywhere repo [2]

For the first, we can remove this plugin.  For a complete new implementation, it
might be good to know that there is a project "https-everywhere : Privacy for
Pythons" [3]

related:  https://github.com/return42/searx-next/issues/8

[1] https://github.com/return42/searx-next/tree/d187a1d/searx/plugins/https_rules
[2] https://github.com/EFForg/https-everywhere/tree/master/src/chrome/content/rules
[3] https://github.com/jayvdb/https-everywhere-py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-22 11:47:16 +02:00
Alexandre Flament
c6d5605d27
Merge pull request #7 from searxng/metrics
Metrics
2021-04-22 08:34:17 +02:00
Alexandre Flament
baff1cbbab fix issues from review 2021-04-21 18:49:13 +02:00
Alexandre Flament
b7848e3422 [fix] searxng fix: sjp engine 2021-04-21 16:31:29 +02:00
Alexandre Flament
7cfd8d900a [mod] oscar: /preferences , engines tab: report engine times
* display the median time instead of the average.
* add a "Reliability" column (sum up the metrics and the checker results).
* the "selected language", "SafeSearch", "Time range" values are displayed as "broken" when the checker tests fail.
2021-04-21 16:24:46 +02:00
Alexandre Flament
c27fef1cde [mod] metrics: add secondary parameter
Some error won't stop the engine:
* additional HTTP redirects for example
* some invalid results

secondary=True allows to flag these errors as not important.
2021-04-21 16:24:46 +02:00
Alexandre Flament
7acd7ffc02 [enh] rewrite and enhance metrics 2021-04-21 16:24:46 +02:00
Alexandre Flament
aae7830d14 [mod] refactoring: processors
Report to the user suspended engines.

searx.search.processor.abstract:
* manages suspend time (per network).
* reports suspended time to the ResultContainer (method extend_container_if_suspended)
* adds the results to the ResultContainer (method extend_container)
* handles exceptions (method handle_exception)
2021-04-21 16:24:46 +02:00
Alexandre Flament
ae5954f2da Merge remote-tracking branch 'return42/fix-preference-save' 2021-04-21 16:24:22 +02:00