diff --git a/README.rst b/README.rst index a7b75d6e2..55deb674e 100644 --- a/README.rst +++ b/README.rst @@ -1,63 +1,73 @@ -searx -===== +.. SPDX-License-Identifier: AGPL-3.0-or-later -A privacy-respecting, hackable `metasearch -engine `__. +.. figure:: https://raw.githubusercontent.com/asciimoo/searx/master/searx/static/themes/oscar/img/logo_searx_a.png + :target: https://asciimoo.github.io/searx/ + :alt: searX + :width: 100% + :align: center -Pronunciation: səːks - -List of `running -instances `__. - -See the `documentation `__ and the `wiki `__ for more information. +------- +|searx install| +|searx homepage| +|searx wiki| +|AGPL License| +|Issues| +|commits| |OpenCollective searx backers| |OpenCollective searx sponsors| -Installation -~~~~~~~~~~~~ +Privacy-respecting, hackable `metasearch engine`_ / *pronunciation* **səːks**. -With Docker ------------ -Go to the `searx-docker `__ project. +.. _metasearch engine: https://en.wikipedia.org/wiki/Metasearch_engine -Without Docker --------------- -For all of the details, follow this `step by step installation `__. +.. |searx install| image:: https://img.shields.io/badge/-install-blue + :target: https://asciimoo.github.io/searx/admin/installation.html -Note: the documentation needs to be updated. +.. |searx homepage| image:: https://img.shields.io/badge/-homepage-blue + :target: https://asciimoo.github.io/searx -If you are in a hurry ---------------------- -- clone the source: - ``git clone https://github.com/asciimoo/searx.git && cd searx`` -- install dependencies: ``./manage.sh update_packages`` -- edit your - `settings.yml `__ - (set your ``secret_key``!) -- run ``python searx/webapp.py`` to start the application +.. |searx wiki| image:: https://img.shields.io/badge/-wiki-blue + :target: https://github.com/asciimoo/searx/wiki +.. |AGPL License| image:: https://img.shields.io/badge/license-AGPL-blue.svg + :target: https://github.com/asciimoo/searx/blob/master/LICENSE -Bugs -~~~~ +.. |Issues| image:: https://img.shields.io/github/issues/asciimoo/searx?color=yellow&label=issues + :target: https://github.com/asciimoo/searx/issues -Bugs or suggestions? Visit the `issue -tracker `__. - -`License `__ -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -More about searx -~~~~~~~~~~~~~~~~ - -- `openhub `__ -- `twitter `__ -- IRC: #searx @ freenode +.. |PR| image:: https://img.shields.io/github/issues-pr-raw/asciimoo/searx?color=yellow&label=PR + :target: https://github.com/asciimoo/searx/pulls +.. |commits| image:: https://img.shields.io/github/commit-activity/y/asciimoo/searx?color=yellow&label=commits + :target: https://github.com/asciimoo/searx/commits/master .. |OpenCollective searx backers| image:: https://opencollective.com/searx/backers/badge.svg :target: https://opencollective.com/searx#backer - .. |OpenCollective searx sponsors| image:: https://opencollective.com/searx/sponsors/badge.svg :target: https://opencollective.com/searx#sponsor + + +If you are looking for running instances, ready to use, then visit searx.space_. + +Otherwise jump to the user_, admin_ and developer_ handbooks you will find on +our homepage_. + +.. _searx.space: https://searx.space +.. _user: https://asciimoo.github.io/searx/user +.. _admin: https://asciimoo.github.io/searx/admin +.. _developer: https://asciimoo.github.io/searx/dev +.. _homepage: https://asciimoo.github.io/searx + +contact: + openhub_ // twitter_ // IRC: #searx @ freenode + +.. _openhub: https://www.openhub.net/p/searx +.. _twitter: https://twitter.com/Searx_engine + +------- + +|gluten free| + +.. |gluten free| image:: https://forthebadge.com/images/featured/featured-gluten-free.svg diff --git a/docs/admin/installation-uwsgi.rst b/docs/admin/installation-uwsgi.rst index ac4c463b9..7b482975d 100644 --- a/docs/admin/installation-uwsgi.rst +++ b/docs/admin/installation-uwsgi.rst @@ -108,6 +108,7 @@ restart the uwsgi application. :start-after: START searx uwsgi-description ubuntu-20.04 :end-before: END searx uwsgi-description ubuntu-20.04 + .. hotfix: a bug group-tab need this comment .. group-tab:: Arch Linux @@ -115,6 +116,7 @@ restart the uwsgi application. :start-after: START searx uwsgi-description arch :end-before: END searx uwsgi-description arch + .. hotfix: a bug group-tab need this comment .. group-tab:: Fedora / RHEL @@ -128,22 +130,21 @@ restart the uwsgi application. .. group-tab:: Ubuntu / debian .. kernel-include:: $DOCS_BUILD/includes/searx.rst - :code: ini :start-after: START searx uwsgi-appini ubuntu-20.04 :end-before: END searx uwsgi-appini ubuntu-20.04 + .. hotfix: a bug group-tab need this comment + .. group-tab:: Arch Linux .. kernel-include:: $DOCS_BUILD/includes/searx.rst - :code: ini :start-after: START searx uwsgi-appini arch :end-before: END searx uwsgi-appini arch + .. hotfix: a bug group-tab need this comment + .. group-tab:: Fedora / RHEL .. kernel-include:: $DOCS_BUILD/includes/searx.rst - :code: ini :start-after: START searx uwsgi-appini fedora :end-before: END searx uwsgi-appini fedora - - diff --git a/docs/blog/index.rst b/docs/blog/index.rst index 5a8135fdc..04d95350d 100644 --- a/docs/blog/index.rst +++ b/docs/blog/index.rst @@ -6,6 +6,7 @@ Blog :maxdepth: 2 :caption: Contents + lxcdev-202006 python3 admin intro-offline diff --git a/docs/blog/lxcdev-202006.rst b/docs/blog/lxcdev-202006.rst new file mode 100644 index 000000000..f9ca3c2cc --- /dev/null +++ b/docs/blog/lxcdev-202006.rst @@ -0,0 +1,413 @@ +.. _blog-lxcdev-202006: + +======================================= +Developing in Linux containers [202006] +======================================= + +.. _LXC: https://linuxcontainers.org/lxc/introduction/ + +.. sidebar:: Audience + + This blog post is written for experienced admins and developers / readers + should have a serious meaning about: *distributed*, *merge* and *linux + container*. + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +In PR :PR:`1803` we added a lot of scripts to Searx's boilerplate. In this blog +post I will show you, how you can make use of them in *distributed and +heterogeneous development cycles* (TL;DR; jump to the :ref:`blog-lxcdev-202006 +abstract`). + +Motivation +========== + +Normally in our development cycle, we edit the sources and run some test and/or +builds by using ``make`` before we commit. This cycle is simple and perfect but +might fail in some aspects we should not overlook. + + The environment in which we run all our development processes matters! + +The :ref:`makefile` and the :ref:`make pyenv` encapsulate a lot for us, but they +do not have access to all prerequisites. For example, there may have +dependencies on packages that are installed on the developer's desktop, but +usually are not preinstalled on a server or client system. Another examples +are; settings have been made to the software on the developer's host that would +never be set on a *production* system. + +*Linux Containers* (LXC_) are isolate environments and not to mix up on +developer's all the prerequisites of all the projects he contribute to, is +always a good choice. + +The scripts from PR :PR:`1803` can divide in those to install and maintain +software: + +- :ref:`searx.sh` +- :ref:`filtron.sh` +- :ref:`morty.sh` + +and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or +even development tasks over a stack of containers, what we call: *Searx's lxc +suite*. + +Gentlemen, start your engines! +============================== + +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _archlinux: https://www.archlinux.org/ + +Before you can start with containers, you need to install and initiate LXD_ +once: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ snap install lxd + $ lxd init --auto + +And you need to clone from origin or if you have your own fork, clone from your +fork: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ cd ~/Downloads + $ git clone https://github.com/asciimoo/searx.git + $ cd searx + +The :ref:`lxc-searx.env` consists of several images, see ``export +LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post +we exercise on a archlinux_ image. The container of this image is named +``searx-archlinux``. Lets build the container, but be sure that this container +does not already exists, so first lets remove possible old one: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh remove searx-archlinux + $ sudo -H ./utils/lxc.sh build searx-archlinux + +.. sidebar:: The ``searx-archlinux`` container + + is the base of all our exercises here. + +In this container we install all services :ref:`including searx, morty & filtron +` in once: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh install suite searx-archlinux + +To proxy HTTP from filtron and morty in the container to the outside of the +container, install nginx into the container. Once for the bot blocker filtron: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/filtron.sh nginx install + ... + INFO: got 429 from http://10.174.184.156/searx + +and once for the content sanitizer (content proxy morty): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/morty.sh nginx install + ... + INFO: got 200 from http://10.174.184.156/morty/ + +.. sidebar:: Fully functional searx suite + + From here on you have a fully functional searx suite running with bot blocker + (filtron) and Web content sanitizer (content proxy morty) needed for a + *privacy protecting* search engine. + +On your system, the IP of your ``searx-archlinux`` container differs from +http://10.174.184.156/searx, just open the URL reported in your installation +protocol in your WEB browser from the desktop to test the instance from outside +of the container. + +In such a searx suite admins can maintain and access the debug log of the +different services quite easy. + +.. _working in containers: + +In containers, work as usual +============================ + +Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers +open the root-bash in the container using ``./utils/lxc.sh cmd +searx-archlinux``: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux bash + INFO: [searx-archlinux] bash + [root@searx-archlinux searx]# pwd + /share/searx + +The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in +the searx-container. To debug the running searx instance use: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/searx.sh inspect service + ... + use [CTRL-C] to stop monitoring the log + ... + +Back in the browser on your desktop open the service http://10.174.184.156/searx +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug +option"* in searx's settings file and restarts the searx uwsgi application. To +debug services from filtron and morty analogous use: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/filtron.sh inspect service + $ ./utils/morty.sh inspect service + +Another point we have to notice is that each service (:ref:`searx `, +:ref:`filtron ` and :ref:`morty `) runs under dedicated +system user account with the same name (compare :ref:`create searx user`). To +get a shell from theses accounts, simply call one of the scripts: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/searx.sh shell + $ ./utils/filtron.sh shell + $ ./utils/morty.sh shell + +To get in touch, open a shell from the service user (searx@searx-archlinux): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh shell + // exit with [CTRL-D] + (searx-pyenv) [searx@searx-archlinux ~]$ ... + +The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system +user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv* +``(searx-pyenv)`` environment is activated. + +.. tabs:: + + .. group-tab:: searx@searx-archlinux + + .. code:: sh + + (searx-pyenv) [searx@searx-archlinux ~]$ pwd + /usr/local/searx + + + +Wrap production into developer suite +==================================== + +In this section we will see how to change the *"Fully functional searx suite"* +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the :ref:`installation basic`: + +- searx setup in: ``/etc/searx/settings.yml`` +- searx user's home: ``/usr/local/searx`` +- virtualenv in: ``/usr/local/searx/searx-pyenv`` +- searx software in: ``/usr/local/searx/searx-src`` + +The searx software is a clone of the ``GIT_URL`` (see :ref:`makefile setup`) and +the working tree is checked out from the ``GIT_BRANCH``. With the use of the +:ref:`searx.sh` the searx service was installed as :ref:`uWSGI application +`. To maintain this service, we can use ``systemctl`` (compare +:ref:`service architectures on distributions `). + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + systemctl stop uwsgi@searx + +With the command above, we stopped the searx uWSGI-App in the archlinux +container. + +The uWSGI-App for the archlinux dsitros is configured in +:origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at +least you should attend the settings of ``uid``, ``chdir``, ``env`` and +``http``:: + + env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml + http = 127.0.0.1:8888 + + chdir = /usr/local/searx/searx-src/searx + virtualenv = /usr/local/searx/searx-pyenv + pythonpath = /usr/local/searx/searx-src + +If you have read the :ref:`"Good to know section" ` you remember, that +each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the +searx installation into a developer one, we simple have to create a smylink to +the **transparent** reposetory from the desktop. Now lets replace the +repository at ``searx-src`` in the container with the working tree from outside +of the container: + +.. tabs:: + + .. group-tab:: container becomes a developer suite + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ln -s /share/searx/ /usr/local/searx/searx-src + +Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the searx service (in the +conatiner): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + systemctl restart uwsgi@searx + + +Remember: :ref:`working in containers` .. here are just some examples from my +daily usage: + +.. tabs:: + + .. group-tab:: desktop + + To *inspect* the searx instance (already described above): + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh inspect service + + Run :ref:`makefile`, e.g. to test inside the container: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + make test + + To install all prerequisites needed for a :ref:`buildhosts`: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh install buildhost + + To build the docs on a buildhost :ref:`buildhosts`: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + make docs + +.. _blog-lxcdev-202006 abstract: + +Abstract +======== + +We build up a fully functional searx suite in a archlinux container: + +.. code:: sh + + $ sudo -H ./utils/lxc.sh install suite searx-archlinux + +To access HTTP from the desktop we installed nginx for the services inside the +conatiner: + +.. tabs:: + + .. group-tab:: [root@searx-archlinux] + + .. code:: sh + + $ ./utils/filtron.sh nginx install + $ ./utils/morty.sh nginx install + +To wrap the suite into a developer one, we created a symbolic link to the +repository which is shared **transparent** from the desktop's file system into +the container : + +.. tabs:: + + .. group-tab:: [root@searx-archlinux] + + .. code:: sh + + $ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + $ ln -s /share/searx/ /usr/local/searx/searx-src + $ systemctl restart uwsgi@searx + +To get remarks from the suite of the archlinux container we can use: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh show suite searx-archlinux + ... + [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx + [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ + [searx-archlinux] INFO: (eth0) docs-live: http://10.174.184.156:8080/ + [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] + ... + diff --git a/docs/utils/lxc.sh.rst b/docs/utils/lxc.sh.rst index ae1412870..114d0911e 100644 --- a/docs/utils/lxc.sh.rst +++ b/docs/utils/lxc.sh.rst @@ -47,9 +47,9 @@ one**:: *Good to know ...* -Eeach container shares the root folder of the repository and the -command ``utils/lxc.sh cmd`` **handles relative path names transparent**, -compare output of:: +Each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` **handles relative path names transparent**, compare output +of:: $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile ... @@ -66,6 +66,7 @@ If there comes the time you want to **get rid off all** the containers and $ sudo -H ./utils/lxc.sh remove $ sudo -H ./utils/lxc.sh remove images +.. _lxc.sh install suite: Install suite ============= diff --git a/searx/templates/oscar/base.html b/searx/templates/oscar/base.html index 5bdfc5190..a3bfa5249 100644 --- a/searx/templates/oscar/base.html +++ b/searx/templates/oscar/base.html @@ -42,7 +42,7 @@ diff --git a/searx/templates/oscar/macros.html b/searx/templates/oscar/macros.html index 809c9f595..58f966997 100644 --- a/searx/templates/oscar/macros.html +++ b/searx/templates/oscar/macros.html @@ -8,34 +8,34 @@ {{ favicon }} {%- endmacro %} -{%- macro result_link(url, title, classes='') -%} -{{ title }} +{%- macro result_link(url, title, classes='', id='') -%} +{{ title }} {%- endmacro -%} -{% macro result_header(result, favicons) -%} -

{% if result.engine~".png" in favicons %}{{ draw_favicon(result.engine) }} {% endif %}{% if result.url %}{{ result_link(result.url, result.title|safe) }}{% else %}{{ result.title|safe}}{% endif %}

+{% macro result_header(result, favicons, id) -%} +

{% if result.engine~".png" in favicons %}{{ draw_favicon(result.engine) }} {% endif %}{% if result.url %}{{ result_link(result.url, result.title|safe, id=id) }}{% else %}{{ result.title|safe}}{% endif %}

{%- endmacro %} -{% macro result_sub_header(result) -%} +{% macro result_sub_header(result, id) -%} {% if result.publishedDate %}{% endif %} - {% if result.magnetlink %} • {{ result_link(result.magnetlink, icon('magnet') + _('magnet link'), "magnetlink") }}{% endif %} - {% if result.torrentfile %} • {{ result_link(result.torrentfile, icon('download-alt') + _('torrent file'), "torrentfile") }}{% endif %} + {% if result.magnetlink %} • {{ result_link(result.magnetlink, icon('magnet') + _('magnet link'), "magnetlink", id) }}{% endif %} + {% if result.torrentfile %} • {{ result_link(result.torrentfile, icon('download-alt') + _('torrent file'), "torrentfile", id) }}{% endif %} {%- endmacro %} -{% macro result_footer(result) -%} +{% macro result_footer(result, id) -%}
{{- "" -}}
{%- for engine in result.engines -%} {{ engine }} {%- endfor -%} {%- if result.url -%} - {{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info") }} + {{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info", id) }} {%- endif -%} {%- if proxify -%} - {{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info") }} + {{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info", id) }} {%- endif -%}
{%- if result.pretty_url -%} @@ -44,16 +44,16 @@ {%- endmacro %} -{% macro result_footer_rtl(result) -%} +{% macro result_footer_rtl(result, id) -%}
{{- "" -}} {% for engine in result.engines -%} {{ engine }} {%- endfor %} {%- if result.url -%} - {{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info") }} + {{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info", id) }} {%- endif -%} {% if proxify -%} - {{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info") }} + {{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info", id) }} {%- endif %} {%- if result.pretty_url -%}
{{ result.pretty_url }}
diff --git a/searx/templates/oscar/result_templates/code.html b/searx/templates/oscar/result_templates/code.html index a1c18a6b7..c7d93e2b3 100644 --- a/searx/templates/oscar/result_templates/code.html +++ b/searx/templates/oscar/result_templates/code.html @@ -1,11 +1,11 @@ -{% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon %} +{% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon with context%} {{ result_header(result, favicons) }} {{ result_sub_header(result) }} {% if result.content %}

{{ result.content|safe }}

{% endif %} -{% if result.repository %}

{{ icon('file') }} {{ result.repository }}

{% endif %} +{% if result.repository %}

{{ icon('file') }} {{ result.repository }}

{% endif %}
{{ result.codelines|code_highlighter(result.code_language)|safe }} diff --git a/searx/templates/oscar/result_templates/default.html b/searx/templates/oscar/result_templates/default.html index 57365f34e..bc71d190d 100644 --- a/searx/templates/oscar/result_templates/default.html +++ b/searx/templates/oscar/result_templates/default.html @@ -1,10 +1,10 @@ {% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon with context %} -{{- result_header(result, favicons) -}} -{{- result_sub_header(result) -}} +{{- result_header(result, favicons, loop.index) -}} +{{- result_sub_header(result, loop.index) -}} {%- if result.embedded -%} - • + • {%- endif -%} {%- if result.embedded -%} @@ -25,7 +25,7 @@ {%- endif -%} {%- if rtl -%} -{{ result_footer_rtl(result) }} +{{ result_footer_rtl(result, loop.index) }} {%- else -%} -{{ result_footer(result) }} +{{ result_footer(result, loop.index) }} {%- endif -%} diff --git a/searx/templates/oscar/result_templates/images.html b/searx/templates/oscar/result_templates/images.html index d0a3b7b83..96cdd53bb 100644 --- a/searx/templates/oscar/result_templates/images.html +++ b/searx/templates/oscar/result_templates/images.html @@ -1,6 +1,6 @@ -{%- from 'oscar/macros.html' import draw_favicon -%} +{%- from 'oscar/macros.html' import draw_favicon with context -%} -{{- "" -}} +{{- "" -}} {{ result.title|striptags }}{{- "" -}}