Commit graph

100 commits

Author SHA1 Message Date
Markus Heiser
acb1577202 [docs] move blog article "Offline engines" to dev/offline_engines.rst
The article "Offline engines" should be in developer's documentation next to
chapter "Engine overview".

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-04 15:04:38 +02:00
Markus Heiser
ebbd5f7366 [docs] add new rubric 'Source-Code'
- document namespace: searx.engines
- move docs/dev/xpath_engine.rst -> docs/src/searx.engines.xpath.rst

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01 16:37:20 +02:00
Markus Heiser
8cd544b2a6 [doc] add documentation about the XPath engine
- pylint searx/engines/xpath.py
- fix indentation of some long lines
- add logging
- add doc-strings

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 11:48:21 +02:00
Markus Heiser
28b25185c5 [brand] searxng -- fix links to issue tracker & WEB-GUI
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25 14:25:08 +02:00
Markus Heiser
34badee467 [doc] modify docs to fit to the new build boilerplate
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23 11:26:22 +02:00
Alexandre Flament
99e0651cea [mod] by default allow only HTTPS, not HTTP
Related to https://github.com/searx/searx/pull/2373
2021-03-08 11:35:08 +01:00
Alexandre Flament
c22d4c764c [fix] duckduckgo engine: "!ddg !g" do not redirect to google
* searx understand "!ddg !g time" as : send "!g time" to DDG
* !g a DDG bang for Google: DDG return a HTTP redirect to Google

This commit adds a the allows_redirect param not to follow HTTP redirect.

The DDG engine returns a empty result as before without HTTP redirect.
2021-02-12 11:10:08 +01:00
Alexandre Flament
ca93a01844 [mod] dynamically set language_support variable
The language_support variable is set to True by default,
and set to False in only 5 engines.

Except the documentation and the /config URL, this variable is not used.

This commit remove the variable definition in the engines, and
set value according to supported_languages length: False when the length is 0,
True otherwise.

Close #2485
2021-02-01 17:10:37 +01:00
Alexandre Flament
9c55d772e9
Merge pull request #2408 from return42/rm-brand-make
[mod] move brand options from Makefile to settings.yml
2021-01-12 10:52:42 +01:00
Archie
ff2370eead
Fixed typo 2021-01-12 01:33:43 +00:00
Markus Heiser
9485179064 [mod] move brand options from Makefile to settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:12:38 +01:00
Alexandre Flament
eb1b5289d5 [mod] documentation: change the jinja context doesn't depend on searx.webapp
Before this commit, in the documentation, the jinja context is 'webapp' and contains
the global variable in the searx.webapp module.

This commit changes this to include only the mandatory variables to build the
documentation.
2020-12-27 10:00:35 +01:00
Markus Heiser
3beede7ab7 [doc] describe 'make books/{name}.html' and 'books/{name}.pdf'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 13:55:22 +01:00
Alexandre Flament
02fc4147ce [mod] dictzone, translated, currency_convert: use engine_type online_curency and online_dictionnary 2020-12-17 11:39:36 +01:00
Alexandre Flament
7ec8bc3ea7 [mod] split searx.search into different processors
see searx.search.processors.abstract.EngineProcessor

First the method searx call the get_params method.

If the return value is not None, then the searx call the method search.
2020-12-17 11:39:36 +01:00
Alexandre Flament
9bc1856e2b [mod] themes: remove legacy, courgette and pix-art themes 2020-12-17 11:33:28 +01:00
Markus Heiser
ea62df1dc0 [doc] update developer quickstart guide
See https://github.com/searx/searx/issues/2369#issuecomment-740652878

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-12 20:31:49 +01:00
Alexandre Flament
d703119d3a [enh] add raise_for_httperror
check HTTP response:
* detect some comme CAPTCHA challenge (no solving). In this case the engine is suspended for long a time.
* otherwise raise HTTPError as before

the check is done in poolrequests.py (was before in search.py).

update qwant, wikipedia, wikidata to use raise_for_httperror instead of raise_for_status
2020-12-11 14:37:08 +01:00
Alexandre Flament
925bb561a2
Merge pull request #2352 from dalf/no_http
Remove HTTP connections as much as possible
2020-12-06 10:18:49 +01:00
Alexandre Flament
f0054d67f1 [fix] wikipedia engine: don't raise an error when the query is not found
Add a new parameter "raise_for_status", set by default to True.
When True, any HTTP status code >= 300 raise an exception ( #2332 )
When False, the engine can manage the HTTP status code by itself.
2020-12-04 20:04:39 +01:00
Alexandre Flament
38d32337e7 [mod] documentations & comments: update http://* URL to https://*.
About http://lesscss.org see https://github.com/less/less-docs/issues/520
2020-12-04 16:52:25 +01:00
Alexandre Flament
1d0c368746 [enh] record details exception per engine
add an new API /stats/errors
2020-12-03 10:22:48 +01:00
Noémi Ványi
4a36a3044d
Add recoll engine (#2325)
recoll is a local search engine based on Xapian:
http://www.lesbonscomptes.com/recoll/

By itself recoll does not offer web or API access,
this can be achieved using recoll-webui:
https://framagit.org/medoc92/recollwebui.git

This engine uses a custom 'files' result template

set `base_url` to the location where recoll-webui can be reached
set `dl_prefix` to a location where the file hierarchy as indexed by recoll can be reached
set `search_dir` to the part of the indexed file hierarchy to be searched, use an empty string to search the entire search domain
2020-11-30 08:35:15 +01:00
Pierre Chevalier
482da5c6c7 Correct what was obviously a typo in vir(t)ualenv. 2020-11-13 16:06:16 +01:00
Markus Heiser
e697d78d5a [mod] make docs-live - remove --port option from sphinx-autobuild
Setting port fixed to 8080 has drawbacks when this port is already in use from
another application [1].  Removing this option from the command line offers the
use of the sphinx-autobuild defaults which are much more flexible [2].

[1] https://github.com/searx/searx/issues/2282
[2] https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-10-30 20:30:20 +01:00
Alexandre Flament
f204e4903d [fix] migration from github.com/asciimoo/searx to github.com/searx/searx : fix URLs 2020-09-28 16:44:14 +02:00
Adam Tauber
4a913247b2 [enh] add option to configure proxies per engine - closes #1827 2020-09-01 15:59:22 +02:00
Adam Tauber
1f2dc6c647 [enh] add external plugin support 2020-07-28 13:10:40 +02:00
Lukas van den Berk
4829a76aae
Created new plugin type custom_results. Added new plugin bang_redirect (#2027)
* Made first attempt at the bangs redirects plugin.

* It redirects. But in a messy way via javascript.

* First version with custom plugin

* Added a help page and a operator to see all the bangs available.

* Changed to .format because of support

* Changed to .format because of support

* Removed : in params

* Fixed path to json file and changed bang operator

* Changed bang operator back to &

* Made first attempt at the bangs redirects plugin.

* It redirects. But in a messy way via javascript.

* First version with custom plugin

* Added a help page and a operator to see all the bangs available.

* Changed to .format because of support

* Changed to .format because of support

* Removed : in params

* Fixed path to json file and changed bang operator

* Changed bang operator back to &

* Refactored getting search query. Also changed bang operator to ! and is now working.

* Removed prints

* Removed temporary bangs_redirect.js file. Updated plugin documentation

* Added unit test for the bangs plugin

* Fixed a unit test and added 2 more for bangs plugin

* Changed back to default settings.yml

* Added myself to AUTHORS.rst

* Refacored working of custom plugin.

* Refactored _get_bangs_data from list to dict to improve search speed.

* Decoupled bangs plugin from webserver with redirect_url

* Refactored bangs unit tests

* Fixed unit test bangs. Removed dubbel parsing in bangs.py

* Removed a dumb print statement

* Refactored bangs plugin to core engine.

* Removed bangs plugin.

* Refactored external bangs unit tests from plugin to core.

* Removed custom_results/bangs documentation from plugins.rst

* Added newline in settings.yml so the PR stays clean.

* Changed searx/plugins/__init__.py back to the old file

* Removed newline search.py

* Refactored get_external_bang_operator from utils to external_bang.py

* Removed unnecessary import form test_plugins.py

* Removed _parseExternalBang and _isExternalBang from query.py

* Removed get_external_bang_operator since it was not necessary

* Simplified external_bang.py

* Simplified external_bang.py

* Moved external_bangs unit tests to test_webapp.py. Fixed return in search with external_bang

* Refactored query parsing to unicode to support python2

* Refactored query parsing to unicode to support python2

* Refactored bangs plugin to core engine.

* Refactored search parameter to search_query in external_bang.py
2020-07-03 13:25:04 +00:00
Noémi Ványi
2addee3893 Use gender neutral pronouns
Closes #2031
2020-06-28 11:40:04 +02:00
Markus Heiser
aa5f947157 [enh] add 'kernel-include' reST-directive
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-18 18:51:31 +02:00
Markus Heiser
6fc26775e4 Merge branch 'master' of https://github.com/asciimoo/searx into filtron 2020-06-01 09:25:30 +02:00
Noémi Ványi
e3282748d0 add display_error_messages option to engine settings
A new option is added to engines to hide error messages from users. It
is called `display_error_messages` and by default it is set to `True`.
If it is set to `False` error messages do not show up on the UI.

Keep in mind that engines are still suspended if needed regardless of
this setting.

Closes #1828
2020-05-31 19:17:48 +02:00
Markus Heiser
4e307edb2d docs: document Makefile setup and 'buildenv' (aka brand)
Add documentation of the Makefile environment (aka brand, buildenv), introduced
in PR #1900.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-30 13:31:13 +02:00
Markus Heiser
0742c2504d build environment: add GIT_BRANCH variable
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-29 16:28:17 +02:00
Markus Heiser
7b4cf2eb48 tooling box: simplify build enviroments
- no more need for a .config.mk
- docs: use searx.brands environment
- searx.sh, filtron.sh & morty.sh are sourcing utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-29 15:09:34 +02:00
Markus Heiser
65b22be1f4 Merge PR #1900 of https://github.com/asciimoo/searx into filtron
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-29 13:03:27 +02:00
Markus Heiser
d471ec86dd Makefile: add target node.env - download & install npm dependencies
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-25 16:38:52 +01:00
Markus Heiser
757ebb5d9f [fix] brands: add variables from build env to grunt process
We have some variables in the build environment which are also needed in the
grunt process when building themes.  Theses variables are relavant if one
creates a fork with its own branding.  We treat these variables under the term
'brands'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-25 14:09:47 +01:00
Markus Heiser
04ad648105 makefile.python: remove python2 support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-08 18:35:38 +01:00
Markus Heiser
4d67164918 Merge branch 'master' of https://github.com/asciimoo/searx into filtron
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-23 13:50:22 +01:00
Markus Heiser
a9d184473f docs: fix dead Searx-instance links to https://searx.space
Related to #1853 / BTW update year of copyright note

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-20 18:11:34 +01:00
Markus Heiser
a99c65dd01 docs: misc marginal fixes
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-05 07:37:26 +01:00
Markus Heiser
eedd63ccd5 docs: revision of the installation instructions
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-04 16:42:13 +01:00
Markus Heiser
9278f0fb45 docs: add some documentation about the tooling box ./utils/*
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-04 13:13:17 +01:00
Markus Heiser
bdf392093a build requirements: add a shell script static analysis tool
ShellCheck: https://github.com/koalaman/shellcheck

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-08 18:09:36 +01:00
Markus Heiser
f602cb8e4d docs(admin): moved settings description from wiki to docs
Move wiki entry https://github.com/asciimoo/searx/wiki/settings.yml
into admin section of the docs (#1785).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2019-12-30 21:56:09 +01:00
Markus Heiser
f9be534b2a docs(dev): fix minor markup typos
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2019-12-30 21:55:15 +01:00
Markus Heiser
d6f2802e4b docs(dev): add more markups to reST primer
- Literal blocks
- Unicode substitution
- Horizontal list
- Math equations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2019-12-28 01:01:11 +01:00
Markus Heiser
92afe68d65 doc(dev): reST/sphinx add tabbed views extension (sphinx_tabs.tabs)
See issue #1785:

  idea: in the doc, provide installation instructions with one tab per
  distrubution

preview (don't bookmark):

  https://return42.github.io/searx/dev/reST.html#tabbed-views

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2019-12-26 10:26:12 +01:00