Commit graph

290 commits

Author SHA1 Message Date
3nprob
0fb423ea59 Allow overriding env vars SEARX_SETTINGS_PATH,UWSGI_SETTINGS_PATH 2021-04-08 12:56:15 +09:00
Markus Heiser
0305775e29 [mod] LXC switch to Fedora 33 / Fedora 31 reached its EOL
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-05 13:17:47 +01:00
Alexandre Flament
aaae9a209e
Merge pull request #2600 from dalf/searx-extra
Add searx_extra package
2021-03-05 09:43:39 +01:00
Alexandre Flament
b8cd326464 Add searx_extra package
Split the utils directory into:
* searx_extra contains update scripts, standalone_searx.py
* utils contains the files to build and setup searx.
2021-03-04 11:59:14 +01:00
Alexandre Flament
1d10ae175c
Merge pull request #2618 from thezeroalpha/master
Fix security vulnerabilities in provided nginx configuration
2021-03-04 11:27:03 +01:00
Alexandre Flament
6ba37777f7
Merge pull request #2623 from return42/fix-centos7
[fix] add package `which` to CentOS-7 boilerplate
2021-03-04 11:19:05 +01:00
Alexandre Flament
d26261c5ab
Merge pull request #2619 from return42/drop-ubu1604
[mod] Drop Ubuntu 16.04 (Xenial Xerus) support
2021-03-04 11:18:21 +01:00
Alexandre Flament
4c2a8aea39
Merge pull request #2620 from return42/fix-git
[fix] support git versions <v2.22
2021-03-04 11:17:24 +01:00
Alex Balgavy
8736f5bd70 Use $host in nginx morty.conf template 2021-03-04 11:16:27 +01:00
Markus Heiser
c355bc3481 [fix] add package 'which' to CentOS-7 boilerplate
Newer CentOS-7 images from https://images.linuxcontainers.org do no longer
include the which command.

Issue:

    $ sudo -H ./utils/lxc.sh cmd searx-centos7 ./utils/filtron.sh install all
    INFO:  [searx-centos7] ./utils/filtron.sh install all
    ...
    Install Go in user's HOME
    -------------------------

    download and install go binary ..
    ...
    -bash: line 1: which: command not found
    -->|ERROR - Go Installation not found in PATH!?!
    -bash: line 2: which: command not found

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 19:37:03 +01:00
Markus Heiser
2e58988191 [fix] utils/serax.sh create_pyenv() - drop duplicate 'pip install .'
The wrong and unnecessary `pip install .` is executed in /usr/local/searx and is
responsible for the error message:

    ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /usr/local/searx

The correct pip-install comes right after changing to `cd ${SEARX_SRC}`.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 18:57:20 +01:00
Markus Heiser
237b1cfdb3 [fix] support git versions <v2.22
LTS distros like Ubuntu 18.04 do not ship a up-to-date version of git.::

    $ sudo -H ./utils/lxc.sh cmd searx-ubu1804 git --version
    ...
    git version 2.17.1

The option `--show-current` was added in git v2.22, the alternative to this
option is::

    git rev-parse --abbrev-ref HEAD

Issue when using option `--show-current`::

    [searx-ubu1804] Clone searx sources
    [searx-ubu1804] -------------------
    [searx-ubu1804]
    [searx-ubu1804] error: unknown option `show-current'
    [searx-ubu1804] usage: git branch [<options>] [-r | -a] [--merged | --no-merged]
    ....

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 18:22:41 +01:00
Markus Heiser
daa4b676df [mod] Drop Ubuntu 16.04 (Xenial Xerus) support
EOL of Ubuntu 16.04 (Xenial Xerus) is in April 2021 but we do not support 16.04
since a longer time.  Issues are comming from dependencies (e.g. pip)::

    [searx-ubu1604]   |searx| SyntaxError: invalid syntax
    [searx-ubu1604]   |searx| Traceback (most recent call last):
    [searx-ubu1604]   |searx|   File "/usr/local/searx/searx-pyenv/bin/pip", line 7, in <module>
    [searx-ubu1604]   |searx|     from pip._internal.cli.main import main
    [searx-ubu1604]   |searx|   File "/usr/local/searx/searx-pyenv/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60
    [searx-ubu1604]   |searx|     sys.stderr.write(f"ERROR: {exc}")
    [searx-ubu1604]   |searx|                                    ^
    [searx-ubu1604]   |searx| SyntaxError: invalid syntax

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03 17:52:50 +01:00
Alex Balgavy
6b59800dc6 Fix security vulnerabilities in suggested nginx configuration
The suggested configurations for nginx found in the documentation and
templates lead to vulnerabilities allowing host spoofing [1] and path
traversal [2], as reported by Gixy [3]. This commit fixes those issues.

[1] https://github.com/yandex/gixy/blob/master/docs/en/plugins/hostspoofing.md
[2] https://github.com/yandex/gixy/blob/master/docs/en/plugins/aliastraversal.md
[3] https://github.com/yandex/gixy
2021-03-03 12:34:22 +01:00
Marc Abonce Seguin
d6681fd33b remove articles number from engines_languages.json 2021-02-25 23:54:21 -07:00
Alexandre Flament
7c1847d5f2 [mod] add utils/fetch_external_bangs.py
Based on duckduckgo bangs
Store bangs on a trie to allow autocomplete (not in this commit)
2021-02-24 18:48:36 +01:00
Alexandre Flament
5f4a085fc4
Merge pull request #2595 from dalf/update-wikidata-units
[mod] update wikidata_units.json and fetch_wikidata_units.py
2021-02-23 17:22:37 +01:00
Alexandre Flament
46ca32c3cc [mod] update currencies.json and fetch_currencies.py
use a sparql request on wikidata to get the list of currencies.

currencies.json contains the translation for all supported searx languages.

Supersede #993
2021-02-23 16:42:28 +01:00
Alexandre Flament
93d1da4906 [mod] update wikidata_units.json and fetch_wikidata_units.py
The fetch_wikidata_units.py result won't change randomly.
See comments in the script.
2021-02-23 13:10:38 +01:00
Alexandre Flament
90b9d0d6a8 [mod] CI: minor changes
* utils/makefile.python: travis-gh-pages renamed ci-gh-pages
2021-02-02 08:53:57 +01:00
Alexandre Flament
34de715e62
Merge pull request #2500 from dalf/github-action-data
[enh] every Sunday, call utils/fetch_*.py scripts and create a PR automatically
2021-02-01 17:16:58 +01:00
Alexandre Flament
99244440e4
Merge pull request #2514 from return42/fix-gh-pages
[fix] Makefile target gh-pages & flatten history of branch gh.pages
2021-02-01 17:07:08 +01:00
Markus Heiser
38b39ef0ae [fix] re-add 'pip-exe' target - partial revert 9b48ae47
Target pip-exe is a prerequisite of the targets:

  - pyinstall
  - pyuninstall

and was accidentally deleted in commit 9b48ae47.

HINT:
  do not confuse pyinstall with penvinstall

pyinstall & pyuninstall
    Installing into user's HOME using pip from OS,
    therefore the message is needed.

pyenvinstall & pyenvuninstall
    Installing into virtualenv (./local) using pip which is provided by
    prerequisite 'pyenv' in the virtualenv.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-02-01 16:58:04 +01:00
Alexandre Flament
d70c5a621a [mod] more robust make pyenv / make pyenvinstall
"make pyenv" ensures that ./local/py3/bin/python is an executable
2021-02-01 16:58:04 +01:00
Alexandre Flament
806af50738
Merge pull request #2494 from return42/rm-fabfile
[fix] remove Fabric file
2021-02-01 15:09:35 +01:00
Markus Heiser
40d2a116e1 [fix] Makefile target gh-pages & flatten history of branch gh.pages
1. This patch fixes error:

    rm -rf gh-pages/
    make V=1 gh-pages
    make[1]: Leaving directory '/800GBPCIex4/share/searx'
    [ -d "gh-pages/.git" ] || git clone  gh-pages
    fatal: repository 'gh-pages' does not exist

2. The gh-page build has been moved to ./build/gh-pages this also affects
   'travis-gh-pages'

3. The gh-pages commit messages now includes a ref to the repository and commit

4. Since a gh-pages history has only the drawback that the reposetory grows
   fast, this patch also flattens the history:

    cd build/gh-pages/; git log --oneline
    bash: cd: build/gh-pages/: Datei oder Verzeichnis nicht gefunden
    026126be (HEAD -> gh-pages, origin/gh-pages) make gh-pages: from https://github.com/return42/searx.git@71d66979c2935312e0aed7fc7c3cf6199fbe88a2

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-29 11:41:48 +01:00
Alexandre Flament
6047087aac [mod] utils/fetch_languages.py: write files at the right location 2021-01-24 14:25:27 +01:00
Markus Heiser
ea5c992d4f [fix] lxc.sh - SC2034: ubu2010_boilerplate appears unused.
$ make test.sh
  In utils/lxc.sh line 42:
  ubu2010_boilerplate="$ubu1904_boilerplate"
  ^-----------------^ SC2034: ubu2010_boilerplate appears unused. Verify use (or export if used externally).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-24 08:29:13 +01:00
Markus Heiser
a8544798ec [fix] remove Fabric file
The fabfile.py has not been updated since 5 years.  I also asked [1] if someone
still use Fabric wtihout any response.  Lets drop outdated Fabric file.

[1] https://github.com/searx/searx/discussions/2400

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-22 17:57:55 +01:00
Alexandre Flament
aa887eb375 [mod] checker : replace pycld3 by langdetect
pycld3 requires the native library cld3
langdetect is a pure python package
2021-01-19 21:26:04 +01:00
Alexandre Flament
709dd960f1
Merge pull request #2473 from return42/fix-setup.py
[fix] setup.py requires pyyaml installed
2021-01-16 19:05:36 +01:00
Markus Heiser
c4a98862bf [fix] setup.py requires pyyaml installed
pip install -e .
...
Obtaining file:///usr/local/searx/searx-src
    ERROR: Command errored out with exit status 1:
     command: /usr/local/searx/searx-pyenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/usr/local/searx/searx-src/setup.py'"'"'; __file__='"'"'/usr/local/searx/searx-src/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vzer91m2
         cwd: /usr/local/searx/searx-src/
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/local/searx/searx-src/setup.py", line 10, in <module>
        from searx.version import VERSION_STRING
      File "/usr/local/searx/searx-src/searx/__init__.py", line 19, in <module>
        import searx.settings_loader
      File "/usr/local/searx/searx-src/searx/settings_loader.py", line 8, in <module>
        import yaml
    ModuleNotFoundError: No module named 'yaml'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-16 08:58:13 +01:00
Alexandre Flament
a4dcfa025c [enh] engines: add about variable
move meta information from comment to the about variable
so the preferences, the documentation can show these information
2021-01-14 20:57:17 +01:00
Alexandre Flament
6e2872f436 [enh] add searx.shared
shared dictionary between the workers (UWSGI or werkzeug)
scheduler: run a task once every x seconds (UWSGI or werkzeug)
2021-01-12 11:47:17 +01:00
Alexandre Flament
8cbc9f2d58 [enh] add checker 2021-01-12 11:47:17 +01:00
Alexandre Flament
d5882147e0 [fix] utils/build_env.py: make sure to always use searx/settings.yml
ignore SEARX_SETTINGS_PATH
2021-01-11 22:59:35 +01:00
Markus Heiser
a31cb68668 [fix] utils/build_env.py: make sure to always use searx/settings.yml
ignore SEARX_SETTINGS_PATH

[1] https://github.com/searx/searx/pull/2408#pullrequestreview-565247895

Suggested-by: @dalf [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:56:15 +01:00
Markus Heiser
9e53470b4c [mod] get rid of searx/brand.py
Removes module searx/brand.py and creates a namespace at searx.brand.

This patch is a first 'proof of concept'.  Later we can decide to remove the
brand namespace entirely or not.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:12:38 +01:00
Markus Heiser
9485179064 [mod] move brand options from Makefile to settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-11 22:12:38 +01:00
Markus Heiser
dc544498d0 [mod] add target upload-pypi-test https://test.pypi.org/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-10 17:05:37 +01:00
Markus Heiser
89e6ba5af7 [mod] add Ubuntu 20.10 image to the LXC suite
Ubuntu 20.10 image [1] is avialable from https://images.linuxcontainers.org

BTW: replace builhost package 'ttf-dejavu' by more common meta package
'fonts-dejavu' ('ttf-dejavu' is no longer available in 20.10).

[1] d161de601b

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-02 17:27:16 +01:00
Markus Heiser
c7d92d6475 [fix] remove Ubuntu 19.10 from the LXC suite (EOL)
Official support for Ubuntu 19.10 'Eoan Ermine' ended on July 17, 2020.

The image has been dropped [1] from https://images.linuxcontainers.org

[1] d161de601b

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-02 17:23:18 +01:00
Alexandre Flament
5fb9a5c7c1
Merge pull request #2411 from dalf/update-secret-key-check
Update secret key check
2020-12-27 18:04:53 +01:00
Markus Heiser
af584f2c40 [perf] pylint: Use multiple processes to speed up Pylint.
Specifying '-j 0' will auto-detect the number of processors available to use.

Suggested-by: @dalf / https://github.com/searx/searx/pull/2421#issuecomment-751444257
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-27 12:04:39 +01:00
Alexandre Flament
568b9465e9 [mod] check secret_key when searx.webapp is imported
Without this commit the module searx checks the secret_key value.

With this commit, make docs, utils/standalone_searx.py,
utils/fetch_firefox_version.py works without SEARX_DEBUG=1

For reference see https://github.com/searx/searx/pull/2386
2020-12-27 10:30:20 +01:00
Alexandre Flament
1ba53e6129
Merge pull request #2420 from return42/fix-sphinx-book-targets
fix makefile `books/{name}.*` targets and add documentation about these targets
2020-12-27 09:47:49 +01:00
Alexandre Flament
1956ab4b50
Merge pull request #2412 from dalf/update-buildenv
[fix] update buildenv
2020-12-27 08:31:23 +01:00
Markus Heiser
d175a0fb7b [mod] LXC_ENV_FOLDER moved from ./lxc to ./lxc-env
BTW:

- add target $(LXC_ENV_FOLDER)
- fix indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-26 11:43:18 +01:00
Markus Heiser
bc7bd1d80a [fix] utils/lib.sh: various typos in messages and comments
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-26 11:38:47 +01:00
Markus Heiser
6316a8d49a [fix] message to install different python versions from apt PPA
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 23:17:21 +01:00
Markus Heiser
809bf85426 [perf] pyenvinstall: reinstall only when requirements are change
Create a (PY_ENV)/requirements.sha256 file with sha256 sums of the
./requirements*.txt files and reinstall only if the sha256 sums change.

Suggested-by: @dalf / https://github.com/searx/searx/pull/2418#pullrequestreview-558426579
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 16:25:09 +01:00
Markus Heiser
d85012b49b [perf] virtualenv: install boilerplate only once
Install boilerplate packages (pip, wheel, setuptools and requirements.txt) only
once, when virtualenv is created.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 15:59:57 +01:00
Markus Heiser
aace837665 [fix] makefile.sphinx: additions to commit c2a6f145
In commit c2a6f145 the sphinx-doc and sphinx-live targets has been removed,
since Sphinx is already installed by the pyenvinstall target.

This commit removes the targets from the remaining prerequisites of the 'BOOK'
targets

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 13:54:07 +01:00
Alexandre Flament
2fc2d17a27
Merge pull request #2418 from return42/fix-pyenv-install
[fix] makefile.python: remove duplicate pyenv-(un)install targets
2020-12-24 10:23:09 +01:00
Markus Heiser
af49bd2bd2 [fix] makefile.python: remove duplicate pyenv-(un)install targets
Makefile targets 'pyenv-install' and 'pyenv-uninstall' are unused duplicates of
'pyenvinstall' and 'pyenvuninstall'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-24 09:18:22 +01:00
Markus Heiser
5fc9273e23 [mod] utils/searx.sh install: create branch if not already exists
This patch keeps an eye on:

1. When installing searx, the branch to install needs to be a local branch.
2. The branch to install can be different from the current branch.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-23 16:06:06 +01:00
Markus Heiser
0fc3759d74 [fix] use SEARX_SETTINGS_TEMPLATE from .config environment
In commit a70b9b9f the SEARX_SETTINGS_TEMPLATE environment was added to the
.config file, but was not use in utils/searx.sh.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-23 14:10:07 +01:00
Alexandre Flament
db5b060455 [fix] update buildenv
CONTACT_URL is unset in Makefile, but searx/brand.py and
utils/brand.env are not updated.

This commit fixes this issue.
2020-12-21 10:55:28 +01:00
Alexandre Flament
72d2bb2289
Merge pull request #2399 from return42/fix-lxc
[fix] determine path to makefile.lxc in a LXC
2020-12-20 09:32:00 +01:00
Markus Heiser
be08e5bfab [fix] yum installation of 'Development Tools'
'yum install' does not support '@development-tools' notation for group
installation.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-18 22:32:05 +01:00
Markus Heiser
1a4524e03c [mod] remove obsolete virtualenv command
Installation and use of the command 'virtualenv' was only needed in py2 and py2
is no longer suported by searx.  In py3 the command is replaced by 'python -m
venv'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-18 22:31:13 +01:00
Markus Heiser
c89df6739f [fix] determine path to makefile.lxc in a LXC
The path to the makefile.lxc is not always static and
'/share/searx/utils/makefile.lxc' can be wrong.

When target 'lxc.activate' is called within the LXC container the path to
makefile.lxc depends on LXC_SHARE_FOLDER and the basename of the folder where
searx was cloned inital on the HOST system.

This patch determines the path of 'makefile.lxc' next to 'makefile.include'
itself.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-18 17:10:49 +01:00
BBaoVanC
19fce74443
Add link to contact instance maintainer to footer of each page (#2391) 2020-12-18 09:53:28 +01:00
Alexandre Flament
7ec8bc3ea7 [mod] split searx.search into different processors
see searx.search.processors.abstract.EngineProcessor

First the method searx call the get_params method.

If the return value is not None, then the searx call the method search.
2020-12-17 11:39:36 +01:00
Markus Heiser
a70b9b9f61 [doc] recommend to use 'use_default_settings=True'
Since #2291 is merged, it is recommend to use::

  use_default_settings=True

1. Add a template file use_default_settings.yml::

    SEARX_SETTINGS_TEMPLATE="${REPO_ROOT}/utils/templates/etc/searx/use_default_settings.yml"

2. In Chapter "Configuration" recommend to make use of
   'use_default_settings=True' and describe it

3. Rewrite of docs/admin/settings.rst
   - move chapter 'settings.yml location' to the top
   - update and split chapter 'Global Settings'

4. Add environment SEARX_SETTINGS_TEMPLATE to .config.sh

5. Use environment $SEARX_SETTINGS_TEMPLATE in the utils/searx.sh script

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-08 20:19:10 +01:00
Alexandre Flament
38d32337e7 [mod] documentations & comments: update http://* URL to https://*.
About http://lesscss.org see https://github.com/less/less-docs/issues/520
2020-12-04 16:52:25 +01:00
rachmadani haryono
c03e4c86bc
Feature/standalone searx update (#1591)
* chg: dev: update standalone_searx

parent d8a5df721b33dd8a7cc9e21dba4060f21d629f69
author rachmadaniHaryono <foreturiga@gmail.com> 1603896594 +0800
committer rachmadaniHaryono <foreturiga@gmail.com> 1603896619 +0800

chg: dev: debug engine_shortcuts
chg: dev: only initilize if engine is given
chg: dev: split main
chg: dev: standalone_searx
chg: dev: update standalone_searx
chg: doc: remove unnecessary log
chg: test: differentiate travis
chg: test: disable shortcut
chg: test: use default engine settings
fix: dev: category choices
fix: dev: duplicate engine shortcut
fix: dev: travis python3
fix: test:  use empty string as shortcut
fix: test: apkm
fix: test: engine shortcut
fix: test: mypy
fix: test: parameter
fix: test: pep8
fix: test: py2 compatibilities
fix: test: searx settings
fix: test: travis engines
new: dev: deduplicate engine
new: dev: main receive engines parameter
new: dev: parse_argument accept engines parameter
new: dev: split search query from get_result func
new: test: basic result case
Suggestions: use RawTextQuery to make the suggestions URLs. Update all themes accordingly.

* new: doc: searx import and init

* chg: dev: parse_argument

- doc
- run on __main__
- simple parse_args

* chg: doc: module

* chg: dev: import section

- remove unused python path modification
- new required package

* chg: dev: script run

- parse_argument func return directly parsed results
- main func return dict instead json text
- dump directly on sys.stdout.write

* chg: dev: get_search_query and get_search_query func

* chg: dev: main func

- move inner function outside
- return dict instead of json text

* new: dev: add utils to doc sys path

* new: doc: standalone_searx

* fix: doc: run script

* chg: dev: mypy type hint

* chg: dev: SearchQuery don't have attr engines

* chg: dev: reset engines __init__

* chg: test: unit test update

* chg: dev: pylint and flake8

* new: test: standalone_searx

* chg: dev: main func and doc

* chg: dev: import and type hint

* new: dev: main func

- remove get_result func
- single func which just translate dict

* chg: test: put mypy on dev requirement

* chg: doc: update

* new: doc: add standalone_searx module member

* chg: doc: shell command line

* chg: dev: remove mypy

* chg: doc: module docstring
2020-11-04 12:38:54 +00:00
Markus Heiser
e697d78d5a [mod] make docs-live - remove --port option from sphinx-autobuild
Setting port fixed to 8080 has drawbacks when this port is already in use from
another application [1].  Removing this option from the command line offers the
use of the sphinx-autobuild defaults which are much more flexible [2].

[1] https://github.com/searx/searx/issues/2282
[2] https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-10-30 20:30:20 +01:00
Alexandre Flament
54afe37e18 [mod] .travis.yml: build documentation automatically on the master branch 2020-10-28 22:48:16 +01:00
Alexandre Flament
ed6696e6bf [mod] add external_urls.json and wikidata_units.json 2020-10-28 08:09:25 +01:00
Marc Abonce Seguin
32957cdf49 add Ahmia filter plugin for onion results 2020-10-25 17:59:43 -07:00
renyhp
b386a815da
Fix typo chmod searx:searx > chown searx:searx 2020-10-19 17:31:02 +02:00
Alexandre Flament
f204e4903d [fix] migration from github.com/asciimoo/searx to github.com/searx/searx : fix URLs 2020-09-28 16:44:14 +02:00
Adam Tauber
8d47142f35
Merge pull request #2189 from dalf/architecture-clean-up
Architecture clean up
2020-09-28 14:56:23 +02:00
Alexandre Flament
c2a6f14516 [fix] Update Sphinx package versions
* Update call to sphinx-autobuild (remove --poll -B options, add --open-browser)
* Upgrade the Sphinx and sphinx-tabs versions.
* utils/makefile.sphinx: remove the SPHINXVERS variable.
* utils/makefile.sphinx: remove the sphinx-doc and sphinx-live targets, Sphinx is already installed by the pyenvinstall target.
2020-09-28 13:27:50 +02:00
Alexandre Flament
485a502b88 [mod] add typing and __slots__ 2020-09-22 19:05:05 +02:00
Alexandre Flament
691d12726b [mod] check the engine tokens in searx/webadapter.py instead of searx/search.py 2020-09-22 18:59:51 +02:00
Alexandre Flament
2dbc0de0cd [mod] add searx/webadapter.py
* move searx.search.get_search_query_from_webapp to searx.webadapter
* move searx.query.SearchQuery to searx.search
2020-09-22 18:59:51 +02:00
Marc Abonce Seguin
c86504b47a fix utils/fetch_languages to work with new languages 2020-09-22 11:37:44 +02:00
Markus Heiser
69a61ce333 [fix] remove doc-build for centos-7 from './utils/searx.sh doc'
Since there is no centos-7 documentation 'searx.sh doc' produce an error
message::

    ERROR: centos-: uWSGI not yet implemented

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-09-20 11:11:14 +02:00
Dalf
1022228d95 Drop Python 2 (1/n): remove unicode string and url_utils 2020-09-10 10:39:04 +02:00
Markus Heiser
623fb87065
[enh] add centos-7 to LXC images (#2118)
## What does this PR do?

To test centos-7 on non centos systems add centos-7 image to the LXC suite.

## Why is this change important?

PR #2112 adds centos-7 support to the install scripts

## How to test this PR locally?

`sudo ./utils/lxc.sh build searx-centos7`
2020-08-08 19:50:11 +02:00
Eliesemoule
1986b5ecac
Utility scripts adapted to run on CentOS 7 (#2112) 2020-08-01 22:12:44 +02:00
Markus Heiser
f9faafa896
[fix] external_bang - UnicodeDecodeError: 'ascii' codec can't decode (#2043)
Python's default encoding depends on the platform, set (python) default encoding
UTF-8 in uwsgi ini files:

    LANG=C.UTF-8
    LANGUAGE=C.UTF-8
    LC_ALL=C.UTF-8

Error pattern:

    Traceback (most recent call last):
      File "/usr/local/searx/searx-src/searx/webapp.py", line 74, in <module>
        from searx.search import SearchWithPlugins, get_search_query_from_webapp
      File "/usr/local/searx/searx-src/searx/search.py", line 32, in <module>
        from searx.external_bang import get_bang_url
      File "/usr/local/searx/searx-src/searx/external_bang.py", line 13, in <module>
        for bang in json.load(json_file)['bang']:
      File "/usr/lib/python3.8/json/__init__.py", line 293, in load
        return loads(fp.read(),
      File "/usr/lib/python3.8/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
      UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 31341: ordinal not in range(128)

close: https://github.com/asciimoo/searx/issues/2041

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-07-11 12:17:06 +02:00
Markus Heiser
8d318ee142
Merge branch 'master' into gigablast 2020-06-29 16:09:59 +00:00
Markus Heiser
18805ee352 [fix] markup of auto generated installation instruction (uWSGI part)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-24 15:15:15 +02:00
Markus Heiser
4eab5cf854
Merge branch 'master' into gigablast 2020-06-19 04:51:48 +00:00
Markus Heiser
15924bcb67 [fix] buildprocess of the docs by adding sphinx-doc-prebuilds target
The sphinx-doc-prebuilds is the single point of definition for prerequisites
each doc build needs

BTW: remove $(LXC_ENV_FOLDER) offset from BOOKS_FOLDER and DOCS_FOLDER path.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-18 18:44:22 +02:00
Markus Heiser
51428de0fe [fix] add missing latexmk package to debian's BUILD_PACKAGES
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-18 18:35:59 +02:00
Markus Heiser
6ff20cef73 [fix] indentation of filtron's rules (json)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-18 18:31:46 +02:00
Markus Heiser
1510382558 utils/fetch_languages.py: print more meaningfull messages
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-05-28 15:18:47 +02:00
Markus Heiser
f576cadfba [fix] installation instructions: pip install -e searx (setup.py)
In the past we did not really install searx into the virtualenv.  This is a
problem, since entry-points and other python installation stuff is not trigger.

See discussion: https://github.com/asciimoo/searx/pull/1938#issuecomment-632056508

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-05-21 15:46:43 +02:00
Markus Heiser
e50ccc99c0 archlinux: add package 'inetutils' to boilerplate
Even it is a core component, some hoster do not have pre installed the
'inetutils' package.  We do need this package for the 'hostname' command.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-05-04 16:15:23 +02:00
Markus Heiser
0973255aff [fix] fatal: repository '/root/searx' does not exist
The $SERVICE_USER (searx) needs read permission to clone repository.  The
installation script stops if permission is not granted.

- https://github.com/asciimoo/searx/pull/1803#issuecomment-616406528

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-30 17:49:26 +02:00
Markus Heiser
8829796f76 utils/morty.sh option new-key: set new morty key
A command to set new morty key in morty & searx service (settings.yml) and to
restart both services with new key::

  utils/morty.sh option new-key

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-28 16:54:42 +02:00
Markus Heiser
ea3255835a utils/morty.sh: set morty key to avoid service abuse
- https://github.com/asciimoo/searx/issues/1871#issuecomment-592459798

    make install all

generates random MORTY_KEY, install service with that key and sets option in the
searx settingy.yml file.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-28 16:21:45 +02:00
Markus Heiser
9504acaca6 Merge branch 'master' of https://github.com/asciimoo/searx into filtron 2020-04-23 12:49:56 +02:00
Markus Heiser
d21e0b0ef8 [fix] old images need an initial 'apt update'
see https://github.com/asciimoo/searx/pull/1803#issuecomment-616684353

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-22 01:11:25 +02:00
Markus Heiser
697dd6c6d3 [fix] ffox useragent: use Windows NT 10.0 and not Windows NT 10
This is an addittion to PR #1934:

  The .0 change really counts on some engines like Bing which seems to allow
  Windows NT 10.0 but not Windows NT 10.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-22 00:43:42 +02:00
Markus Heiser
99ff16c465 tooling box: added nginx + polished bash scripts and environment
- add installation method for nginx sites, morty and filtron
- clean up PUBLIC_URL environment in and outside of containers
- clean up comand lines
- handle uWSGI quirks on fedora (emperor mode)
- handle Python quirks on debian (there is no 'python' command anymore)
- lib.sh: add die and die_caller functions
- lxc_suite_install_info is now a function
- lint: shellcheck

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-13 11:34:28 +02:00