OwlBurst/Ponysearch
1
0
Fork 0
forked from Ponysearch/Ponysearch
Code Pull requests Activity

docs: use make and uitls/searx.sh in containers

Browse source 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
Markus Heiser 2020-03-31 18:25:40 +02:00
parent 8634eeb9de
commit f27f6c6c5d
6 changed files with 145 additions and 54 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
docs/admin/buildhosts.rst
View file

@ -4,9 +4,6 @@
Buildhosts
==========
To get best results from build, its recommend to install additional packages
on build hosts.
.. sidebar:: This article needs some work
If you have any contribution send us your :pull:`PR <../pulls>`, see
@ -17,6 +14,23 @@ on build hosts.
:local:
:backlinks: entry
To get best results from build, its recommend to install additional packages
on build hosts (see :ref:`searx.sh`).::
sudo -H ./utils/searx.sh buildhost
This will install packages needed by searx:
.. include:: ../../build/docs/includes/searx.rst
:start-after: START distro-packages
:end-before: END distro-packages
and packages needed to build docuemtation and run tests:
.. include:: ../../build/docs/includes/searx.rst
:start-after: START build-packages
:end-before: END build-packages
.. _docs build:
Build docs

26
docs/build-templates/searx.rst
View file

@ -28,6 +28,32 @@ ${fedora}
.. END distro-packages
.. START build-packages
.. tabs::
.. group-tab:: Ubuntu / debian
.. code-block:: sh
$ sudo -H apt-get install -y \\
${debian_build}
.. group-tab:: Arch Linux
.. code-block:: sh
$ sudo -H pacman -S --noconfirm \\
${arch_build}
.. group-tab:: Fedora / RHEL
.. code-block:: sh
$ sudo -H dnf install -y \\
${fedora_build}
.. END build-packages
.. START create user

115
docs/utils/lxc.sh.rst
View file

@ -1,5 +1,4 @@
.. _snap: https://snapcraft.io
.. _snapcraft LXD: https://snapcraft.io/lxd
.. _LXC/LXD Image Server: https://uk.images.linuxcontainers.org/
@ -7,6 +6,8 @@
.. _LXD: https://linuxcontainers.org/lxd/introduction/
.. _`LXD@github`: https://github.com/lxc/lxd
.. _archlinux: https://www.archlinux.org/
.. _lxc.sh:
================
@ -21,24 +22,31 @@
- `LXD@github`_
With the use of *Linux Containers* (LXC_) we can scale our tasks over a stack of
containers, what we call the: *lxc suite*. Before you can start with
containers, you need to install and initiate LXD_ once::
containers, what we call the: *lxc suite*. The *searx suite*
(:origin:`lxc-searx.env <utils/lxc-searx.env>`) is loaded by default, every time
you start the ``lxc.sh`` script (*you do not need to care about*).
Before you can start with containers, you need to install and initiate LXD_
once::
$ snap install lxd
$ lxd init --auto
The *searx suite* (:origin:`lxc-searx.env <utils/lxc-searx.env>`) is loaded by
default, every time you start the ``lxc.sh`` script (you do not need to care
about). To make use of the containers from the *searx suite*, you have to build
the :ref:`LXC suite containers <lxc.sh --help>` first. But be warned, this
might take some time::
To make use of the containers from the *searx suite*, you have to build the
:ref:`LXC suite containers <lxc.sh help>` initial. But be warned, this might
take some time::
$ sudo -H ./utils/lxc.sh build
A cup of coffee later, your LXC suite is build up and you can run whatever task
you want / in a selected or even in all :ref:`LXC suite containers <lxc.sh
--help>`. Each container shares the root folder of the repository and the
command ``utils/lxc.sh cmd`` handles relative path names *transparent*::
help>`.
*Good to know ...*
Eeach container shares the root folder of the repository and the
command ``utils/lxc.sh cmd`` **handles relative path names transparent**,
compare output of::
$ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile
...
@ -46,29 +54,70 @@ command ``utils/lxc.sh cmd`` handles relative path names *transparent*::
[searx-fedora31] -rw-r--r-- 1 root root 7603 Mar 30 11:54 Makefile
[searx-archlinux] -rw-r--r-- 1 root root 7603 Mar 30 11:54 Makefile
With this in mind, you can run :ref:`searx.sh` and install packages, needed by
searx::
$ sudo -H ./utils/lxc.sh cmd -- ./utils/searx.sh install packages
And run one of the :origin:`Makefile` targets::
$ sudo -H ./utils/lxc.sh cmd -- make test.sh
You can install a *buildhost environment* into the containers (time for another
cup of coffee)::
$ sudo -H ./utils/lxc.sh install buildhost
If you want to get rid off all the containers, just type::
If there comes the time you want to **get rid off all** the containers and
**clean up local images** just type::
$ sudo -H ./utils/lxc.sh remove
To clean up your local images use::
$ sudo -H ./utils/lxc.sh remove images
.. _lxc.sh --help:
Install suite
=============
To install the complete :ref:`searx suite (includes searx, morty & filtron)
<lxc-searx.env>` into all LXC_ use::
$ sudo -H ./utils/lxc.sh install suite
The command above installs a searx suite (see :ref:`installation scripts`). To
get the IP (URL) of the filtron service in the containers use ``show suite``
command. To test instances from containers just open the URLs in your
WEB-Browser::
$ sudo ./utils/lxc.sh show suite | grep filtron
[searx-ubu1604] INFO: (eth0) filtron: http://n.n.n.135:4004/
[searx-ubu1804] INFO: (eth0) filtron: http://n.n.n.141:4004/
[searx-ubu1910] INFO: (eth0) filtron: http://n.n.n.137:4004/
[searx-ubu2004] INFO: (eth0) filtron: http://n.n.n.127:4004/
[searx-fedora31] INFO: (eth0) filtron: http://n.n.n.18:4004/
[searx-archlinux] INFO: (eth0) filtron: http://n.n.n.12:4004/
Running commands
================
**Inside containers, you can use make or run scripts** from the
:ref:`toolboxing`. By example: to setup a :ref:`buildhosts` and run the
Makefile target ``test`` in the archlinux_ container::
sudo -H ./utils/lxc.sh cmd searx-archlinux ./utils/searx.sh buildhost
sudo -H ./utils/lxc.sh cmd searx-archlinux make test
Setup searx buildhost
=====================
You can **install the searx buildhost environment** into one or all containers.
The installation procedure to set up a :ref:`build host<buildhosts>` takes its
time. Installation in all containers will take more time (time for another cup
of coffee).::
sudo -H ./utils/lxc.sh cmd -- ./utils/searx.sh buildhost
To build (live) documentation inside a archlinux_ container::
sudo -H ./utils/lxc.sh cmd searx-archlinux make docs-clean docs-live
...
[I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080
To get IP of the container and the port number *live docs* is listening::
$ sudo ./utils/lxc.sh show suite | grep docs-live
...
[searx-archlinux] INFO: (eth0) docs-live: http://n.n.n.12:8080/
.. _lxc.sh help:
Overview
========
@ -77,3 +126,11 @@ The ``--help`` output of the script is largely self-explanatory:
.. program-output:: ../utils/lxc.sh --help
.. _lxc-searx.env:
searx suite
===========
.. literalinclude:: ../../utils/lxc-searx.env
:language: bash

12
utils/lxc-searx.env
View file

@ -49,18 +49,6 @@ lxc_suite_install() {
)
}
lxc_suite_prepare_buildhost() {
(
lxc_set_suite_env
FORCE_TIMEOUT=0
export FORCE_TIMEOUT
# "${LXC_REPO_ROOT}/utils/searx.sh* install packages
"${LXC_REPO_ROOT}/utils/searx.sh" install buildhost
rst_title "buildhost installation finished ($(hostname))" part
echo
)
}
lxc_suite_info() {
(
lxc_set_suite_env

10
utils/lxc.sh
View file

@ -82,7 +82,7 @@ usage::
$_cmd [start|stop] [containers|<name>]
$_cmd show [info|config|suite|images]
$_cmd cmd [--|<name>] '...'
$_cmd install [suite|base|buildhost]
$_cmd install [suite|base]
build
:containers: build, launch and 'install basic' packages on 'containers'
@ -106,7 +106,6 @@ cmd
install
:suite: install LXC suite; ${LXC_SUITE_INSTALL_INFO}
:base: prepare LXC; install basic packages
:buildhost: prepare LXC; install buildhost packages
EOF
usage_images
@ -150,7 +149,9 @@ main() {
# don't check prerequisite when in recursion
if [[ ! $1 == __* ]]; then
! required_commands lxc && lxd_info && exit 42
if ! in_container; then
! required_commands lxc && lxd_info && exit 42
fi
[[ -z $LXC_SUITE ]] && err_msg "missing LXC_SUITE" && exit 42
fi
@ -240,7 +241,7 @@ main() {
install)
sudo_or_exit
case $2 in
suite|base|buildhost)
suite|base)
lxc_exec "${LXC_REPO_ROOT}/utils/lxc.sh" __install "$2"
;;
*) usage "$_usage"; exit 42 ;;
@ -252,7 +253,6 @@ main() {
case $2 in
suite) lxc_suite_install ;;
base) FORCE_TIMEOUT=0 lxc_install_base_packages ;;
buildhost) lxc_suite_prepare_buildhost ;;
esac
;;
doc)

16
utils/searx.sh
View file

@ -48,7 +48,7 @@ git build-essential libxslt-dev zlib1g-dev libffi-dev libssl-dev
shellcheck"
BUILD_PACKAGES_debian="\
graphviz imagemagick texlive-xetex librsvg2-bin
firefox graphviz imagemagick texlive-xetex librsvg2-bin
texlive-latex-recommended texlive-extra-utils ttf-dejavu"
# pacman packages
@ -59,7 +59,7 @@ git base-devel libxml2
shellcheck"
BUILD_PACKAGES_arch="\
graphviz imagemagick texlive-bin extra/librsvg
firefox graphviz imagemagick texlive-bin extra/librsvg
texlive-core texlive-latexextra ttf-dejavu"
# dnf packages
@ -70,7 +70,7 @@ git @development-tools libxml2
ShellCheck"
BUILD_PACKAGES_fedora="\
graphviz graphviz-gd ImageMagick librsvg2-tools
firefox graphviz graphviz-gd ImageMagick librsvg2-tools
texlive-xetex-bin texlive-collection-fontsrecommended
texlive-collection-latex dejavu-sans-fonts dejavu-serif-fonts
dejavu-sans-mono-fonts"
@ -116,7 +116,7 @@ usage() {
usage::
$(basename "$0") shell
$(basename "$0") install [all|user|searx-src|pyenv|uwsgi|apache|packages]
$(basename "$0") install [all|user|searx-src|pyenv|uwsgi|apache|packages|buildhost]
$(basename "$0") update [searx]
$(basename "$0") remove [all|user|pyenv|searx-src]
$(basename "$0") activate [service]
@ -135,7 +135,7 @@ install / remove
:uwsgi: install searx uWSGI application
:settings: reinstall settings from ${REPO_ROOT}/searx/settings.yml
:packages: install needed packages from OS package manager
:buildhost: install packages from OS package manager needed by buildhost
:buildhost: install packages from OS package manager needed by buildhosts
update searx
Update searx installation ($SERVICE_HOME)
activate service
@ -688,9 +688,15 @@ rst-doc() {
local debian="${SEARX_PACKAGES_debian}"
local arch="${SEARX_PACKAGES_arch}"
local fedora="${SEARX_PACKAGES_fedora}"
local debian_build="${BUILD_PACKAGES_debian}"
local arch_build="${BUILD_PACKAGES_arch}"
local fedora_build="${BUILD_PACKAGES_fedora}"
debian="$(echo "${debian}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
arch="$(echo "${arch}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
fedora="$(echo "${fedora}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
debian_build="$(echo "${debian_build}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
arch_build="$(echo "${arch_build}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
fedora_build="$(echo "${fedora_build}" | sed 's/.*/ & \\/' | sed '$ s/.$//')"
eval "echo \"$(< "${REPO_ROOT}/docs/build-templates/searx.rst")\""