Commit graph

189 commits

Author SHA1 Message Date
Markus Heiser
67afa6b1fb bash scripts: source code cleaned up (shellcheck)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-10 14:00:52 +02:00
Markus Heiser
44edc92569 [fix] morty: PUBLIC_URL_PATH_MORTY needs trailing slash
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-09 14:28:34 +02:00
Markus Heiser
f693149cde Changes from the installation tests on (all) LXC containers.
Tested and fixed HTTP & uWSGI installation on:

  ubu1604 ubu1804 ubu1910 ubu2004 fedora31 archlinux

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-08 18:38:36 +02:00
Markus Heiser
ee39a098ac apache: normalize installation (docs and script)s over all distros
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-07 18:31:51 +02:00
Markus Heiser
eb0d4646d8 docs: rework of chapter "Install with apache"
BTW: normalize installation-nginx.rst
2020-04-06 17:59:06 +02:00
Markus Heiser
c81849cb5a filtron.sh & morty.sh: improve usage message (if used in containers)
BTW: normalize soma variable names

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-05 17:40:37 +02:00
Markus Heiser
c2caf9569c [fix] creation of /.lxcenv in containers
In utils/lib.sh there are two functions with the same name, but different tasks.
Rename one of them from lxc_init_container() into lxc_init_container_env().

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-05 15:15:28 +02:00
Markus Heiser
c748047e92 utils/searx.sh: add cmd to (un)set option image_proxy and result_proxy
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-04 17:54:20 +02:00
Markus Heiser
e530e20ae6 misc: fix variuous marginals
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-04 17:53:16 +02:00
Markus Heiser
3c19f2f1ce utils/lxc.sh install: support installation on selected containers
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-04 12:40:14 +02:00
Markus Heiser
e32b08f616 utils/morty.sh: add generic documentation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-03 20:26:20 +02:00
Markus Heiser
853be4e11c utils/filtron.sh: add generic documentation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-03 20:24:40 +02:00
Markus Heiser
af988dbf71 utils/lxc.sh: support build of selected containers
$ sudo -H ./utils/lxc.sh build  <name>
  $ sudo -H ./utils/lxc.sh show   [images|suite|info|config [<name>]]

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-03 17:08:42 +02:00
Markus Heiser
f32b4fcedd LXC: add virtualenv to LXC_BASE_PACKAGES
Commit 09a40625 adds virtualenv dependency.  BTW remove deprecated
--no-site-packages.  Not having access to global site-packages is now the
default behavior.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-02 20:27:44 +02:00
Markus Heiser
2441e24288 Merge branch 'master' of https://github.com/asciimoo/searx into filtron 2020-04-02 18:59:43 +02:00
Markus Heiser
09a4062505 makefile.python: partial revert off 9b48ae47, adds py2 support back
revert "makefile.python: ... drop build support py2" to get back Py2 support.
TPy2 support need virtualenv installed by the OS.

BTW: log environment and python version in travis's install phase

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-01 17:21:26 +02:00
Markus Heiser
31437d04e8 makefile.python: hardening - start pylint by module
- replace: $(PY_ENV_BIN)/pylint --> $(PY_ENV_BIN)/python -m pylint

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-01 16:07:59 +02:00
Markus Heiser
9b48ae47ec makefile.python: hardening - drop virtualenv, drop build support py2
From py3.3 on a subset of virtualenv is built-in, so you can run '-m venv' ot of
the box.

- replace: $(PY_ENV_BIN)/pip --> $(PY_ENV_BIN)/python -m pip
- remove obsolete virtualenv-exe target and adjust VTENV_OPTS
- remove obsolete msg-pip-exe target
- print list of py launchers available from $(PY_ENV_BIN) to the log
- fix hard coded ./local

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-01 14:16:49 +02:00
Markus Heiser
3bbcfe6ab3 travis: make travis build more verbose
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-01 11:57:48 +02:00
Markus Heiser
6007bd7fc8 makefile.python: fix (old) typo
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-01 10:59:51 +02:00
Markus Heiser
f27f6c6c5d docs: use make and uitls/searx.sh in containers
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-31 18:25:40 +02:00
Markus Heiser
dd53c45a2c docs: add utils/lxc.sh docs, normalize filtron, morty & searx docs
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-30 18:47:01 +02:00
Markus Heiser
3e4d022d04 [fix] make test.sh: fix various shellcheck error messages
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-30 11:03:46 +02:00
Markus Heiser
5e2b161379 Makefile: marginal fix
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-29 16:46:07 +02:00
Markus Heiser
0742c2504d build environment: add GIT_BRANCH variable
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-29 16:28:17 +02:00
Markus Heiser
7b4cf2eb48 tooling box: simplify build enviroments
- no more need for a .config.mk
- docs: use searx.brands environment
- searx.sh, filtron.sh & morty.sh are sourcing utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-29 15:09:34 +02:00
Markus Heiser
65b22be1f4 Merge PR #1900 of https://github.com/asciimoo/searx into filtron
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-29 13:03:27 +02:00
Markus Heiser
442b27ac1b bash: use build environment from Makefile
To Makefile target brand, add creation of bash environment in::

  utils/brand.env

In bash scripts (manage.sh) source env by::

  . utils/brand.env

manage.sh help: show GIT_URL and more environment

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-29 11:41:59 +02:00
Markus Heiser
1fcec0bbda lxc.sh install base: add bash and python3 to LXC_BASE_PACKAGES
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-23 18:06:35 +01:00
Markus Heiser
baf1ffd1fa LXC: set LXC_ENV_FOLDER=lxc/$(shell hostname)/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-23 18:05:46 +01:00
Markus Heiser
21ba25ebb8 Merge branch 'filtron' of https://github.com/return42/searx into filtron 2020-03-23 15:02:32 +01:00
Markus Heiser
2d956696e9 docs: building (PDF) books / build user book
BTW: cleaned up Makefile target help

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-21 18:45:38 +01:00
Markus Heiser
62da769952 ./utils/lxc.sh: add command 'install base'
to install basic packages like git and *build essentials*::

  $ ./utils/lxc.sh install base

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-18 17:47:48 +01:00
Markus Heiser
d2cfe9ce5b LXC: add /.lxcenv
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-18 15:34:46 +01:00
Markus Heiser
86e79488aa LXC: utils/makefile.lxc (inital) add /.lxcenv.mk to contaiiners
Get LXC environment when building make targets.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-15 17:01:36 +01:00
Markus Heiser
80f7b65804 searx.sh: add buildhost cmd, installs OS packages for builds
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-09 01:37:26 +01:00
Markus Heiser
f5f83b8a99 shellcheck: minor fixes to support all dsitros
checked with::

  sudo ./utils/lxc.sh cmd -- make test.sh

on ubu1804, ubu1910, fedora31 and archlinux.  All checks OK except:

ubu1604: uses shellcheck v0.3.7 (from 04/2015) which is no longer supported!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-08 18:37:16 +01:00
Markus Heiser
04ad648105 makefile.python: remove python2 support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-08 18:35:38 +01:00
Markus Heiser
938bb02550 searx.sh: add commandline 'install packages'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-08 18:32:30 +01:00
Markus Heiser
d48c7bf678 LXC: fixed raw.idmap & removed obsolete uid/gid subordinate
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-08 18:30:25 +01:00
Markus Heiser
e7f69b63f1 LXC: from searx.sh, morty.sh and filtron.sh tests
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-08 02:41:45 +01:00
Markus Heiser
b1e90cff23 LXC: separate lxc-suite from lxc & improved command line.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-07 20:24:08 +01:00
Markus Heiser
c15337850e fix: minor typos
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-06 22:06:19 +01:00
Markus Heiser
387c6a7769 docs: improve description of uwsgi & ngingx setup
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-06 14:47:00 +01:00
Markus Heiser
cbc08fdc26 docs: describe uwsgi setup of all suported distributions
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-04 19:56:40 +01:00
Markus Heiser
3aff2c19d1 docs: complement uwsgi documentation for all distros
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-04 12:22:20 +01:00
Markus Heiser
9f1cb7a943 Merge branch 'master' of https://github.com/asciimoo/searx into filtron 2020-03-04 12:11:58 +01:00
Markus Heiser
3cb7daedb7 docs: generic doocumentation for searx-uwsgi
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-03 19:57:15 +01:00
Markus Heiser
f25da6070e docs: generic documentation get templating
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-03 16:26:02 +01:00
Markus Heiser
7af991acda LXC: add ubuntu 19.10 and 20.04 to the list of test images
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-02 19:01:08 +01:00
Markus Heiser
c3e4753ce9 docs: generic documentation from the installation scripts
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-02 19:00:19 +01:00
Markus Heiser
491cb95a1f utils/lxc.env: separate environment that is used in containers
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-01 18:28:10 +01:00
Markus Heiser
40843fe95a searx/data/engines_languages.json: sort json file (no content change)
To get meaningfull diffs, the json file has to be sorted.  Before applying any
further content patch, the json file needs a inital sort (without changing any
content).

Sorted by::

  import sys, json

  with open('engines_languages.json') as f:
      j = json.load(f)

  with open('engines_languages.json', 'w') as f:
      json.dump(j, f, indent=2, sort_keys=True)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-01 07:56:46 +01:00
Markus Heiser
37c135f2ce LXC: improved UX when working with a bunch of containers
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-27 19:13:03 +01:00
Markus Heiser
af6acd3417 LXC: install searx-suite installs searx, filtron & morty on all containers
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-26 19:07:55 +01:00
Markus Heiser
d5917cc029 utils/lib.sh: make uWSGI installation available for all distros
support: ubuntu, debin, fedora, archlinux

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-25 20:20:17 +01:00
Markus Heiser
f17b28743a utils/serx.sh: make uWSGI installation available for all distros (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-23 21:05:22 +01:00
Markus Heiser
5fb6d4f508 LXC: normalize package installation & user creation.
utils/lib.sh:
- get DIST_ID & DIST_VERSION from /etc/os-release
- pkg_[install|remove|...] supports ubuntu, debian, archlinux & fedora

utils/lxc.sh
- Workaround for the "setrlimit(RLIMIT_CORE): Operation not permitted" error::
    'Set disable_coredump false' >> /etc/sudo.conf

utils/[searx.sh|filtron.sh|morty.sh]
- switched user creation from 'adduser' perl script to 'useradd' built-in
  command

utils/searx.sh
- install packages for ubuntu, debian, archlinux & fedora

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-23 12:10:45 +01:00
Markus Heiser
e36e0f80ae LXC: added archlinux and fedora31 containers
BTW: update all packages when installing conatiner's boilerplate

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-18 18:40:34 +01:00
Markus Heiser
ccb96544f7 LXC: improved console messages & prompt
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-18 18:20:03 +01:00
Markus Heiser
59e4026762 searx.sh: install settings at /etc/searx/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-17 18:58:59 +01:00
Markus Heiser
de58f02f6b filtron: add missing log action to the filtron rules
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-17 15:36:10 +01:00
Markus Heiser
0bfc61dbe9 tooling box: misc fixes from lxc tests
Tested by:

  sudo -H ./utils/lxc.sh build containers
  sudo -H ./utils/lxc.sh cmd /share/searx/utils/searx.sh    install all
  sudo -H ./utils/lxc.sh cmd /share/searx/utils/filtron.sh  install all
  sudo -H ./utils/lxc.sh cmd /share/searx/utils/morty.sh    install all

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-16 22:29:06 +01:00
Markus Heiser
a1f5f2ced8 LXC: minor fixes and renaming
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-16 22:26:03 +01:00
Markus Heiser
ad32739860 shellcheck: fix usse -n instead of ! -z (SC2236 SC2237)
- https://www.shellcheck.net/wiki/SC2236 -- Use -n instead of ! -z.
 - https://www.shellcheck.net/wiki/SC2237 -- Use [ -n .. ] instead of ! [ -z ....

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-16 20:07:37 +01:00
Markus Heiser
e8cf225046 LXC: tooling box - add boilerplate to containers
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-16 18:18:15 +01:00
Markus Heiser
0b2b325552 LXC: tooling box - add HOST shares and commandline to containers (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-13 18:25:03 +01:00
Markus Heiser
e6bf2038c3 filtron.sh: add 'install rules' command
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-11 15:58:35 +01:00
Markus Heiser
0d6153db12 filtron.sh: updated rules from production
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-11 15:57:42 +01:00
Markus Heiser
7751b29559 LXC: add LXC tooling box (initial, WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-08 19:12:28 +01:00
Markus Heiser
5c0f9c8c79 tooling box: varius fix from tests
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-08 17:12:01 +01:00
Markus Heiser
ed4cb4f160 tooling box: varius fix from tests 2020-02-08 13:24:08 +01:00
Markus Heiser
a014fa0ccd upload-pypi: twine is needed to upload to PyPi 'make upload-pypi'
To build & deploy packages on PyPi call make target:

  make upload-pypi

- https://github.com/asciimoo/searx/issues/1829

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-05 15:26:57 +01:00
Markus Heiser
71d7550dbe tooling box ./utils/*: minor fix from production test 2020-02-04 19:47:33 +01:00
Markus Heiser
2f40f61f83 /etc/filtron/rules.json: normalize rules from docs & tooling box
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-04 17:59:58 +01:00
Markus Heiser
eedd63ccd5 docs: revision of the installation instructions
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-04 16:42:13 +01:00
Markus Heiser
79e1f837ef utils & Makefile: add .config.mk & .config.sh for searx brands
By isolating the environment of makefiles and bash scripts into .config.mk and
.config.sh it is simple to maintain searx brands by setting some central
environments.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-04 10:39:42 +01:00
Markus Heiser
94ac560dcc doc: add descriptions of our tooling box in ./utils
Initial started with the scripts for searx, filtron and morty installations.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-03 13:25:51 +01:00
Markus Heiser
e7401796f6 utils/morty.sh: fixed public url
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-02 18:20:20 +01:00
Markus Heiser
a4437c47ac utils/morty.sh: add script to install morty result proxy
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-02 18:14:10 +01:00
Markus Heiser
709ac51d33 utils/filtron.sh: generalize systemd, accounts and golang tasks
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-01 16:59:27 +01:00
Markus Heiser
56a93ee770 utils/lib.sh: colorize messages
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-31 18:24:31 +01:00
Markus Heiser
0bb8847087 utils/filtron.sh: add option to debug filtron requests
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-31 17:25:38 +01:00
Markus Heiser
91a55e159e apache: reverse proxy, set ProxyPreserveHost On
related discussions:

- https://github.com/asciimoo/searx/issues/1822
- https://github.com/asciimoo/searx/issues/1819#issuecomment-580400259

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-31 15:54:07 +01:00
Markus Heiser
cd9b877c8c utils/searx.sh & filtron.sh: misc changes from first tests (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-31 15:53:53 +01:00
Markus Heiser
6274a54746 utils/searx.sh & filtron.sh: misc changes from first tests (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-30 19:55:51 +01:00
Markus Heiser
924bf65517 utils/searx.sh & filtron.sh: misc changes from first tests (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-29 20:00:50 +01:00
Markus Heiser
971a8264b2 utils/searx.sh: add apache site searx.conf:uwsgi (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-27 19:08:40 +01:00
Markus Heiser
d171fcd56e utils/searx.sh: add apache site searx.conf:uwsgi
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-21 18:38:57 +01:00
Markus Heiser
af2f58fc58 utils/filtron.sh: add script to install filtron middleware (apache)
Preparation for the installation of an apache site.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-20 19:08:56 +01:00
Markus Heiser
9b5a7f7559 utils/searx.sh: add script to install isolated searx service
First version which serves searx over uwsgi at http://127.0.0.1:8888

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-20 16:55:05 +01:00
Markus Heiser
89df9d9141 utils/searx.sh: add script to install isolated searx service (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-16 14:01:38 +01:00
Markus Heiser
3cf31528f3 utils/searx.sh: add script to install isolated searx service (WIP)
WIP: written from scratch / linted but untested

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-14 19:26:54 +01:00
Markus Heiser
c2b27d9120 Merge branch 'filtron' of https://github.com/return42/searx into filtron 2020-01-13 18:39:46 +01:00
Markus Heiser
b5449ec47c filtron: log suspiciously frequent queries (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-13 18:37:05 +01:00
Markus Heiser
3ed82a97af utils/filtron.sh: add command 'update filtron'
BTW: tidy up some code style and fix some issues reported by the linter.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-12 15:51:04 +01:00
Markus Heiser
39feb141bc docs(admin): add description of the utils/filtron.sh script
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-11 12:50:40 +01:00
Markus Heiser
4990b07b4b utils/filtron.sh: various fix from first installation test (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-09 16:25:05 +01:00
Markus Heiser
4139c63d23 utils/filtron.sh: add script to install filtron middleware
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-08 19:21:07 +01:00
Markus Heiser
28dacee288 utils: add lib.sh containing common shell script
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-08 18:13:22 +01:00