From e7788980ad1bb82e9ae084428d05a79b235c17ff Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Tue, 12 May 2020 14:17:47 +0200 Subject: [PATCH 01/11] [fix] about page- 'How to set as the default search engine?' Searx supports OpenSearch, remove the inline JS from the about page. close: https://github.com/asciimoo/searx/issues/1949 close: https://github.com/asciimoo/searx/issues/1616 Signed-off-by: Markus Heiser --- searx/templates/__common__/about.html | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/searx/templates/__common__/about.html b/searx/templates/__common__/about.html index d3e8d06a6..9741b5162 100644 --- a/searx/templates/__common__/about.html +++ b/searx/templates/__common__/about.html @@ -59,12 +59,16 @@

How to set as the default search engine?

-
Firefox
+

+ Searx supports OpenSearch. + For more information on changing your default search engine, see your browser's documentation: +

-
- Install - searx as a search engine on any version of Firefox! (javascript required) -
+

Where to find anonymous usage statistics of this instance ?

From 250e05e826c5aa072c0f05b0c1f0a8f66ba47631 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Tue, 12 May 2020 14:57:45 +0200 Subject: [PATCH 02/11] [README] update and polish up a little by using badges about markup & badges - https://github.com/ralsina/rst-cheatsheet/blob/master/rst-cheatsheet.rst - https://github.com/Naereen/badges/blob/master/README.rst - https://shields.io/ - https://gist.github.com/tterb/982ae14a9307b80117dbf49f624ce0e8 Signed-off-by: Markus Heiser --- README.rst | 90 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 47 insertions(+), 43 deletions(-) diff --git a/README.rst b/README.rst index a7b75d6e2..c0008acde 100644 --- a/README.rst +++ b/README.rst @@ -1,63 +1,67 @@ -searx -===== +.. SPDX-License-Identifier: AGPL-3.0-or-later -A privacy-respecting, hackable `metasearch -engine `__. +.. figure:: https://raw.githubusercontent.com/asciimoo/searx/master/searx/static/themes/oscar/img/logo_searx_a.png + :target: https://asciimoo.github.io/searx/ + :alt: searX + :width: 100% + :align: center -Pronunciation: səːks - -List of `running -instances `__. - -See the `documentation `__ and the `wiki `__ for more information. +------- +|searx install| +|searx homepage| +|searx wiki| +|AGPL License| +|Issues| +|commits| |OpenCollective searx backers| |OpenCollective searx sponsors| -Installation -~~~~~~~~~~~~ +Privacy-respecting, hackable `metasearch engine`_ / *pronunciation* **səːks**. -With Docker ------------ -Go to the `searx-docker `__ project. +.. _metasearch engine: https://en.wikipedia.org/wiki/Metasearch_engine -Without Docker --------------- -For all of the details, follow this `step by step installation `__. +.. |searx install| image:: https://img.shields.io/badge/-install-blue + :target: https://asciimoo.github.io/searx/admin/installation.html -Note: the documentation needs to be updated. +.. |searx homepage| image:: https://img.shields.io/badge/-homepage-blue + :target: https://asciimoo.github.io/searx -If you are in a hurry ---------------------- -- clone the source: - ``git clone https://github.com/asciimoo/searx.git && cd searx`` -- install dependencies: ``./manage.sh update_packages`` -- edit your - `settings.yml `__ - (set your ``secret_key``!) -- run ``python searx/webapp.py`` to start the application +.. |searx wiki| image:: https://img.shields.io/badge/-wiki-blue + :target: https://github.com/asciimoo/searx/wiki +.. |AGPL License| image:: https://img.shields.io/badge/license-AGPL-blue.svg + :target: https://github.com/asciimoo/searx/blob/master/LICENSE -Bugs -~~~~ +.. |Issues| image:: https://img.shields.io/github/issues/asciimoo/searx?color=yellow&label=issues + :target: https://github.com/asciimoo/searx/issues -Bugs or suggestions? Visit the `issue -tracker `__. - -`License `__ -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -More about searx -~~~~~~~~~~~~~~~~ - -- `openhub `__ -- `twitter `__ -- IRC: #searx @ freenode +.. |PR| image:: https://img.shields.io/github/issues-pr-raw/asciimoo/searx?color=yellow&label=PR + :target: https://github.com/asciimoo/searx/pulls +.. |commits| image:: https://img.shields.io/github/commit-activity/y/asciimoo/searx?color=yellow&label=commits + :target: https://github.com/asciimoo/searx/commits/master .. |OpenCollective searx backers| image:: https://opencollective.com/searx/backers/badge.svg :target: https://opencollective.com/searx#backer - .. |OpenCollective searx sponsors| image:: https://opencollective.com/searx/sponsors/badge.svg :target: https://opencollective.com/searx#sponsor + + +If you are looking for running instances, ready to use, then visit searx.space_. + +Otherwise jump to the user_, admin_ and developer_ handbooks you will find on +our homepage_. + +.. _searx.space: https://searx.space +.. _user: https://asciimoo.github.io/searx/user +.. _admin: https://asciimoo.github.io/searx/admin +.. _developer: https://asciimoo.github.io/searx/dev +.. _homepage: https://asciimoo.github.io/searx + +contact: + openhub_ // twitter_ // IRC: #searx @ freenode + +.. _openhub: https://www.openhub.net/p/searx +.. _twitter: https://twitter.com/Searx_engine From f8b16a6635d553f91cc3d2e81b816e10517e9a72 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Tue, 12 May 2020 15:02:53 +0200 Subject: [PATCH 03/11] [README] searx, world's first gluten free search engine ;) Signed-off-by: Markus Heiser --- README.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.rst b/README.rst index c0008acde..55deb674e 100644 --- a/README.rst +++ b/README.rst @@ -65,3 +65,9 @@ contact: .. _openhub: https://www.openhub.net/p/searx .. _twitter: https://twitter.com/Searx_engine + +------- + +|gluten free| + +.. |gluten free| image:: https://forthebadge.com/images/featured/featured-gluten-free.svg From 385e9b5c9e2d1caa73f99dac0bf1be1c46505121 Mon Sep 17 00:00:00 2001 From: Gordon Quad Date: Sat, 13 Jun 2020 22:42:16 +0100 Subject: [PATCH 04/11] add correction support for duckduckgo --- searx/engines/duckduckgo.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/searx/engines/duckduckgo.py b/searx/engines/duckduckgo.py index 0d2c0af2d..6e07b5021 100644 --- a/searx/engines/duckduckgo.py +++ b/searx/engines/duckduckgo.py @@ -50,6 +50,7 @@ result_xpath = '//div[@class="result results_links results_links_deep web-result url_xpath = './/a[@class="result__a"]/@href' title_xpath = './/a[@class="result__a"]' content_xpath = './/a[@class="result__snippet"]' +correction_xpath = '//div[@id="did_you_mean"]//a' # match query's language to a region code that duckduckgo will accept @@ -125,6 +126,11 @@ def response(resp): 'content': content, 'url': res_url}) + # parse correction + for correction in eval_xpath(doc, correction_xpath): + # append correction + results.append({'correction': extract_text(correction)}) + # return results return results From 18805ee35296e6e34be59f37083190990f4afcbb Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Wed, 24 Jun 2020 15:15:15 +0200 Subject: [PATCH 05/11] [fix] markup of auto generated installation instruction (uWSGI part) Signed-off-by: Markus Heiser --- docs/admin/installation-uwsgi.rst | 11 +++--- utils/searx.sh | 66 +++++++++++++++++++------------ 2 files changed, 46 insertions(+), 31 deletions(-) diff --git a/docs/admin/installation-uwsgi.rst b/docs/admin/installation-uwsgi.rst index ac4c463b9..7b482975d 100644 --- a/docs/admin/installation-uwsgi.rst +++ b/docs/admin/installation-uwsgi.rst @@ -108,6 +108,7 @@ restart the uwsgi application. :start-after: START searx uwsgi-description ubuntu-20.04 :end-before: END searx uwsgi-description ubuntu-20.04 + .. hotfix: a bug group-tab need this comment .. group-tab:: Arch Linux @@ -115,6 +116,7 @@ restart the uwsgi application. :start-after: START searx uwsgi-description arch :end-before: END searx uwsgi-description arch + .. hotfix: a bug group-tab need this comment .. group-tab:: Fedora / RHEL @@ -128,22 +130,21 @@ restart the uwsgi application. .. group-tab:: Ubuntu / debian .. kernel-include:: $DOCS_BUILD/includes/searx.rst - :code: ini :start-after: START searx uwsgi-appini ubuntu-20.04 :end-before: END searx uwsgi-appini ubuntu-20.04 + .. hotfix: a bug group-tab need this comment + .. group-tab:: Arch Linux .. kernel-include:: $DOCS_BUILD/includes/searx.rst - :code: ini :start-after: START searx uwsgi-appini arch :end-before: END searx uwsgi-appini arch + .. hotfix: a bug group-tab need this comment + .. group-tab:: Fedora / RHEL .. kernel-include:: $DOCS_BUILD/includes/searx.rst - :code: ini :start-after: START searx uwsgi-appini fedora :end-before: END searx uwsgi-appini fedora - - diff --git a/utils/searx.sh b/utils/searx.sh index 4009fddb0..886b38f8c 100755 --- a/utils/searx.sh +++ b/utils/searx.sh @@ -816,47 +816,61 @@ rst-doc() { case $DIST_ID-$DIST_VERS in ubuntu-*|debian-*) cat < /usr/share/doc/uwsgi/README.Debian.gz -# For uWSGI debian uses the LSB init process, this might be changed -# one day, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833067 -create ${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP} -enable: sudo -H ln -s ${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP} ${uWSGI_APPS_ENABLED}/ -start: sudo -H service uwsgi start ${SEARX_UWSGI_APP%.*} -restart: sudo -H service uwsgi restart ${SEARX_UWSGI_APP%.*} -stop: sudo -H service uwsgi stop ${SEARX_UWSGI_APP%.*} -disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} +.. code:: bash + + # init.d --> /usr/share/doc/uwsgi/README.Debian.gz + # For uWSGI debian uses the LSB init process, this might be changed + # one day, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833067 + + create ${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP} + enable: sudo -H ln -s ${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP} ${uWSGI_APPS_ENABLED}/ + start: sudo -H service uwsgi start ${SEARX_UWSGI_APP%.*} + restart: sudo -H service uwsgi restart ${SEARX_UWSGI_APP%.*} + stop: sudo -H service uwsgi stop ${SEARX_UWSGI_APP%.*} + disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + EOF ;; arch-*) cat < /usr/lib/systemd/system/uwsgi@.service -# For uWSGI archlinux uses systemd template units, see -# - http://0pointer.de/blog/projects/instances.html -# - https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html#one-service-per-app-in-systemd -create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} -enable: sudo -H systemctl enable uwsgi@${SEARX_UWSGI_APP%.*} -start: sudo -H systemctl start uwsgi@${SEARX_UWSGI_APP%.*} -restart: sudo -H systemctl restart uwsgi@${SEARX_UWSGI_APP%.*} -stop: sudo -H systemctl stop uwsgi@${SEARX_UWSGI_APP%.*} -disable: sudo -H systemctl disable uwsgi@${SEARX_UWSGI_APP%.*} +.. code:: bash + + # systemd --> /usr/lib/systemd/system/uwsgi@.service + # For uWSGI archlinux uses systemd template units, see + # - http://0pointer.de/blog/projects/instances.html + # - https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html#one-service-per-app-in-systemd + + create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + enable: sudo -H systemctl enable uwsgi@${SEARX_UWSGI_APP%.*} + start: sudo -H systemctl start uwsgi@${SEARX_UWSGI_APP%.*} + restart: sudo -H systemctl restart uwsgi@${SEARX_UWSGI_APP%.*} + stop: sudo -H systemctl stop uwsgi@${SEARX_UWSGI_APP%.*} + disable: sudo -H systemctl disable uwsgi@${SEARX_UWSGI_APP%.*} + EOF ;; fedora-*) cat < /usr/lib/systemd/system/uwsgi.service -# The unit file starts uWSGI in emperor mode (/etc/uwsgi.ini), see -# - https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html -create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} -restart: sudo -H touch ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} -disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} +.. code:: bash + + # systemd --> /usr/lib/systemd/system/uwsgi.service + # The unit file starts uWSGI in emperor mode (/etc/uwsgi.ini), see + # - https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html + + create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + restart: sudo -H touch ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + EOF ;; esac echo -e ".. END searx uwsgi-description $DIST_NAME" echo -e "\n.. START searx uwsgi-appini $DIST_NAME" - eval "echo \"$(< "${TEMPLATES}/${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP}")\"" + echo ".. code:: bash" + echo + eval "echo \"$(< "${TEMPLATES}/${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP}")\"" | prefix_stdout " " echo -e "\n.. END searx uwsgi-appini $DIST_NAME" ) From 9c71f620a7c29ea22e4f1743e733bd267cdb4b69 Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Wed, 24 Jun 2020 17:49:38 +0200 Subject: [PATCH 06/11] [add blog post] Developing in Linux containers [202006] Signed-off-by: Markus Heiser --- docs/blog/index.rst | 1 + docs/blog/lxcdev-202006.rst | 413 ++++++++++++++++++++++++++++++++++++ docs/utils/lxc.sh.rst | 7 +- 3 files changed, 418 insertions(+), 3 deletions(-) create mode 100644 docs/blog/lxcdev-202006.rst diff --git a/docs/blog/index.rst b/docs/blog/index.rst index 5a8135fdc..04d95350d 100644 --- a/docs/blog/index.rst +++ b/docs/blog/index.rst @@ -6,6 +6,7 @@ Blog :maxdepth: 2 :caption: Contents + lxcdev-202006 python3 admin intro-offline diff --git a/docs/blog/lxcdev-202006.rst b/docs/blog/lxcdev-202006.rst new file mode 100644 index 000000000..f9ca3c2cc --- /dev/null +++ b/docs/blog/lxcdev-202006.rst @@ -0,0 +1,413 @@ +.. _blog-lxcdev-202006: + +======================================= +Developing in Linux containers [202006] +======================================= + +.. _LXC: https://linuxcontainers.org/lxc/introduction/ + +.. sidebar:: Audience + + This blog post is written for experienced admins and developers / readers + should have a serious meaning about: *distributed*, *merge* and *linux + container*. + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +In PR :PR:`1803` we added a lot of scripts to Searx's boilerplate. In this blog +post I will show you, how you can make use of them in *distributed and +heterogeneous development cycles* (TL;DR; jump to the :ref:`blog-lxcdev-202006 +abstract`). + +Motivation +========== + +Normally in our development cycle, we edit the sources and run some test and/or +builds by using ``make`` before we commit. This cycle is simple and perfect but +might fail in some aspects we should not overlook. + + The environment in which we run all our development processes matters! + +The :ref:`makefile` and the :ref:`make pyenv` encapsulate a lot for us, but they +do not have access to all prerequisites. For example, there may have +dependencies on packages that are installed on the developer's desktop, but +usually are not preinstalled on a server or client system. Another examples +are; settings have been made to the software on the developer's host that would +never be set on a *production* system. + +*Linux Containers* (LXC_) are isolate environments and not to mix up on +developer's all the prerequisites of all the projects he contribute to, is +always a good choice. + +The scripts from PR :PR:`1803` can divide in those to install and maintain +software: + +- :ref:`searx.sh` +- :ref:`filtron.sh` +- :ref:`morty.sh` + +and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or +even development tasks over a stack of containers, what we call: *Searx's lxc +suite*. + +Gentlemen, start your engines! +============================== + +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _archlinux: https://www.archlinux.org/ + +Before you can start with containers, you need to install and initiate LXD_ +once: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ snap install lxd + $ lxd init --auto + +And you need to clone from origin or if you have your own fork, clone from your +fork: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ cd ~/Downloads + $ git clone https://github.com/asciimoo/searx.git + $ cd searx + +The :ref:`lxc-searx.env` consists of several images, see ``export +LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post +we exercise on a archlinux_ image. The container of this image is named +``searx-archlinux``. Lets build the container, but be sure that this container +does not already exists, so first lets remove possible old one: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh remove searx-archlinux + $ sudo -H ./utils/lxc.sh build searx-archlinux + +.. sidebar:: The ``searx-archlinux`` container + + is the base of all our exercises here. + +In this container we install all services :ref:`including searx, morty & filtron +` in once: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh install suite searx-archlinux + +To proxy HTTP from filtron and morty in the container to the outside of the +container, install nginx into the container. Once for the bot blocker filtron: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/filtron.sh nginx install + ... + INFO: got 429 from http://10.174.184.156/searx + +and once for the content sanitizer (content proxy morty): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/morty.sh nginx install + ... + INFO: got 200 from http://10.174.184.156/morty/ + +.. sidebar:: Fully functional searx suite + + From here on you have a fully functional searx suite running with bot blocker + (filtron) and Web content sanitizer (content proxy morty) needed for a + *privacy protecting* search engine. + +On your system, the IP of your ``searx-archlinux`` container differs from +http://10.174.184.156/searx, just open the URL reported in your installation +protocol in your WEB browser from the desktop to test the instance from outside +of the container. + +In such a searx suite admins can maintain and access the debug log of the +different services quite easy. + +.. _working in containers: + +In containers, work as usual +============================ + +Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers +open the root-bash in the container using ``./utils/lxc.sh cmd +searx-archlinux``: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux bash + INFO: [searx-archlinux] bash + [root@searx-archlinux searx]# pwd + /share/searx + +The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in +the searx-container. To debug the running searx instance use: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/searx.sh inspect service + ... + use [CTRL-C] to stop monitoring the log + ... + +Back in the browser on your desktop open the service http://10.174.184.156/searx +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug +option"* in searx's settings file and restarts the searx uwsgi application. To +debug services from filtron and morty analogous use: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/filtron.sh inspect service + $ ./utils/morty.sh inspect service + +Another point we have to notice is that each service (:ref:`searx `, +:ref:`filtron ` and :ref:`morty `) runs under dedicated +system user account with the same name (compare :ref:`create searx user`). To +get a shell from theses accounts, simply call one of the scripts: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/searx.sh shell + $ ./utils/filtron.sh shell + $ ./utils/morty.sh shell + +To get in touch, open a shell from the service user (searx@searx-archlinux): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh shell + // exit with [CTRL-D] + (searx-pyenv) [searx@searx-archlinux ~]$ ... + +The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system +user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv* +``(searx-pyenv)`` environment is activated. + +.. tabs:: + + .. group-tab:: searx@searx-archlinux + + .. code:: sh + + (searx-pyenv) [searx@searx-archlinux ~]$ pwd + /usr/local/searx + + + +Wrap production into developer suite +==================================== + +In this section we will see how to change the *"Fully functional searx suite"* +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the :ref:`installation basic`: + +- searx setup in: ``/etc/searx/settings.yml`` +- searx user's home: ``/usr/local/searx`` +- virtualenv in: ``/usr/local/searx/searx-pyenv`` +- searx software in: ``/usr/local/searx/searx-src`` + +The searx software is a clone of the ``GIT_URL`` (see :ref:`makefile setup`) and +the working tree is checked out from the ``GIT_BRANCH``. With the use of the +:ref:`searx.sh` the searx service was installed as :ref:`uWSGI application +`. To maintain this service, we can use ``systemctl`` (compare +:ref:`service architectures on distributions `). + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + systemctl stop uwsgi@searx + +With the command above, we stopped the searx uWSGI-App in the archlinux +container. + +The uWSGI-App for the archlinux dsitros is configured in +:origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at +least you should attend the settings of ``uid``, ``chdir``, ``env`` and +``http``:: + + env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml + http = 127.0.0.1:8888 + + chdir = /usr/local/searx/searx-src/searx + virtualenv = /usr/local/searx/searx-pyenv + pythonpath = /usr/local/searx/searx-src + +If you have read the :ref:`"Good to know section" ` you remember, that +each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the +searx installation into a developer one, we simple have to create a smylink to +the **transparent** reposetory from the desktop. Now lets replace the +repository at ``searx-src`` in the container with the working tree from outside +of the container: + +.. tabs:: + + .. group-tab:: container becomes a developer suite + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ln -s /share/searx/ /usr/local/searx/searx-src + +Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the searx service (in the +conatiner): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + systemctl restart uwsgi@searx + + +Remember: :ref:`working in containers` .. here are just some examples from my +daily usage: + +.. tabs:: + + .. group-tab:: desktop + + To *inspect* the searx instance (already described above): + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh inspect service + + Run :ref:`makefile`, e.g. to test inside the container: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + make test + + To install all prerequisites needed for a :ref:`buildhosts`: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh install buildhost + + To build the docs on a buildhost :ref:`buildhosts`: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + make docs + +.. _blog-lxcdev-202006 abstract: + +Abstract +======== + +We build up a fully functional searx suite in a archlinux container: + +.. code:: sh + + $ sudo -H ./utils/lxc.sh install suite searx-archlinux + +To access HTTP from the desktop we installed nginx for the services inside the +conatiner: + +.. tabs:: + + .. group-tab:: [root@searx-archlinux] + + .. code:: sh + + $ ./utils/filtron.sh nginx install + $ ./utils/morty.sh nginx install + +To wrap the suite into a developer one, we created a symbolic link to the +repository which is shared **transparent** from the desktop's file system into +the container : + +.. tabs:: + + .. group-tab:: [root@searx-archlinux] + + .. code:: sh + + $ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + $ ln -s /share/searx/ /usr/local/searx/searx-src + $ systemctl restart uwsgi@searx + +To get remarks from the suite of the archlinux container we can use: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh show suite searx-archlinux + ... + [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx + [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ + [searx-archlinux] INFO: (eth0) docs-live: http://10.174.184.156:8080/ + [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] + ... + diff --git a/docs/utils/lxc.sh.rst b/docs/utils/lxc.sh.rst index ae1412870..114d0911e 100644 --- a/docs/utils/lxc.sh.rst +++ b/docs/utils/lxc.sh.rst @@ -47,9 +47,9 @@ one**:: *Good to know ...* -Eeach container shares the root folder of the repository and the -command ``utils/lxc.sh cmd`` **handles relative path names transparent**, -compare output of:: +Each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` **handles relative path names transparent**, compare output +of:: $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile ... @@ -66,6 +66,7 @@ If there comes the time you want to **get rid off all** the containers and $ sudo -H ./utils/lxc.sh remove $ sudo -H ./utils/lxc.sh remove images +.. _lxc.sh install suite: Install suite ============= From 02c6710564bf2f4f246db64e09ba380badf65671 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Sat, 27 Jun 2020 16:37:27 +0200 Subject: [PATCH 07/11] Make Oscar theme scalable --- searx/templates/oscar/base.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/searx/templates/oscar/base.html b/searx/templates/oscar/base.html index 2ccadd700..5bdfc5190 100644 --- a/searx/templates/oscar/base.html +++ b/searx/templates/oscar/base.html @@ -8,7 +8,7 @@ - + {% block meta %}{% endblock %} {% block title %}{% endblock %}{{ instance_name }} From 2addee3893d236d2be08d28a4fb61d6f24b478bf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?No=C3=A9mi=20V=C3=A1nyi?= Date: Sun, 28 Jun 2020 11:40:04 +0200 Subject: [PATCH 08/11] Use gender neutral pronouns Closes #2031 --- docs/blog/private-engines.rst | 2 +- docs/dev/search_api.rst | 4 ++-- docs/user/own-instance.rst | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/blog/private-engines.rst b/docs/blog/private-engines.rst index c26b3ed1c..796f0fc69 100644 --- a/docs/blog/private-engines.rst +++ b/docs/blog/private-engines.rst @@ -13,7 +13,7 @@ Private engines To solve this issue private engines were introduced in :pull:`1823`. A new option was added to engines named `tokens`. It expects a list of strings. If the user making a request presents one of the tokens -of an engine, he/she is able to access information about the engine +of an engine, they can access information about the engine and make search requests. Example configuration to restrict access to the Arch Linux Wiki engine: diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst index 922548ffb..960d2fb37 100644 --- a/docs/dev/search_api.rst +++ b/docs/dev/search_api.rst @@ -81,7 +81,7 @@ Parameters Theme of instance. Please note, available themes depend on an instance. It is possible that an - instance administrator deleted, created or renamed themes on his/her instance. + instance administrator deleted, created or renamed themes on their instance. See the available options in the preferences page of the instance. ``oscar-style`` : default ``logicodev`` @@ -91,7 +91,7 @@ Parameters ``oscar``. Please note, available styles depend on an instance. It is possible that an - instance administrator deleted, created or renamed styles on his/her + instance administrator deleted, created or renamed styles on their instance. See the available options in the preferences page of the instance. ``enabled_plugins`` : optional diff --git a/docs/user/own-instance.rst b/docs/user/own-instance.rst index 4876abe99..af415b61d 100644 --- a/docs/user/own-instance.rst +++ b/docs/user/own-instance.rst @@ -44,9 +44,9 @@ hidden from visited result pages. What are the consequences of using public instances? ---------------------------------------------------- -If someone uses a public instance, he/she has to trust the administrator of that +If someone uses a public instance, they have to trust the administrator of that instance. This means that the user of the public instance does not know whether -his/her requests are logged, aggregated and sent or sold to a third party. +their requests are logged, aggregated and sent or sold to a third party. Also, public instances without proper protection are more vulnerable to abusing the search service, In this case the external service in exchange returns From b51d4ec041c59001f8365904d799b5188ec60c3c Mon Sep 17 00:00:00 2001 From: Adam Tauber Date: Sun, 28 Jun 2020 20:40:48 +0200 Subject: [PATCH 09/11] [fix] add aria labels to result links - #350 --- searx/templates/oscar/macros.html | 26 +++++++++---------- .../oscar/result_templates/code.html | 4 +-- .../oscar/result_templates/default.html | 10 +++---- .../oscar/result_templates/images.html | 4 +-- .../templates/oscar/result_templates/map.html | 8 +++--- .../oscar/result_templates/torrent.html | 8 +++--- .../oscar/result_templates/videos.html | 12 ++++----- 7 files changed, 36 insertions(+), 36 deletions(-) diff --git a/searx/templates/oscar/macros.html b/searx/templates/oscar/macros.html index 809c9f595..58f966997 100644 --- a/searx/templates/oscar/macros.html +++ b/searx/templates/oscar/macros.html @@ -8,34 +8,34 @@ {{ favicon }} {%- endmacro %} -{%- macro result_link(url, title, classes='') -%} -{{ title }} +{%- macro result_link(url, title, classes='', id='') -%} +{{ title }} {%- endmacro -%} -{% macro result_header(result, favicons) -%} -

{% if result.engine~".png" in favicons %}{{ draw_favicon(result.engine) }} {% endif %}{% if result.url %}{{ result_link(result.url, result.title|safe) }}{% else %}{{ result.title|safe}}{% endif %}

+{% macro result_header(result, favicons, id) -%} +

{% if result.engine~".png" in favicons %}{{ draw_favicon(result.engine) }} {% endif %}{% if result.url %}{{ result_link(result.url, result.title|safe, id=id) }}{% else %}{{ result.title|safe}}{% endif %}

{%- endmacro %} -{% macro result_sub_header(result) -%} +{% macro result_sub_header(result, id) -%} {% if result.publishedDate %}{% endif %} - {% if result.magnetlink %} • {{ result_link(result.magnetlink, icon('magnet') + _('magnet link'), "magnetlink") }}{% endif %} - {% if result.torrentfile %} • {{ result_link(result.torrentfile, icon('download-alt') + _('torrent file'), "torrentfile") }}{% endif %} + {% if result.magnetlink %} • {{ result_link(result.magnetlink, icon('magnet') + _('magnet link'), "magnetlink", id) }}{% endif %} + {% if result.torrentfile %} • {{ result_link(result.torrentfile, icon('download-alt') + _('torrent file'), "torrentfile", id) }}{% endif %} {%- endmacro %} -{% macro result_footer(result) -%} +{% macro result_footer(result, id) -%}
{{- "" -}}
{%- for engine in result.engines -%} {{ engine }} {%- endfor -%} {%- if result.url -%} - {{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info") }} + {{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info", id) }} {%- endif -%} {%- if proxify -%} - {{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info") }} + {{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info", id) }} {%- endif -%}
{%- if result.pretty_url -%} @@ -44,16 +44,16 @@ {%- endmacro %} -{% macro result_footer_rtl(result) -%} +{% macro result_footer_rtl(result, id) -%}
{{- "" -}} {% for engine in result.engines -%} {{ engine }} {%- endfor %} {%- if result.url -%} - {{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info") }} + {{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info", id) }} {%- endif -%} {% if proxify -%} - {{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info") }} + {{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info", id) }} {%- endif %} {%- if result.pretty_url -%}
{{ result.pretty_url }}
diff --git a/searx/templates/oscar/result_templates/code.html b/searx/templates/oscar/result_templates/code.html index a1c18a6b7..c7d93e2b3 100644 --- a/searx/templates/oscar/result_templates/code.html +++ b/searx/templates/oscar/result_templates/code.html @@ -1,11 +1,11 @@ -{% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon %} +{% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon with context%} {{ result_header(result, favicons) }} {{ result_sub_header(result) }} {% if result.content %}

{{ result.content|safe }}

{% endif %} -{% if result.repository %}

{{ icon('file') }} {{ result.repository }}

{% endif %} +{% if result.repository %}

{{ icon('file') }} {{ result.repository }}

{% endif %}
{{ result.codelines|code_highlighter(result.code_language)|safe }} diff --git a/searx/templates/oscar/result_templates/default.html b/searx/templates/oscar/result_templates/default.html index 57365f34e..bc71d190d 100644 --- a/searx/templates/oscar/result_templates/default.html +++ b/searx/templates/oscar/result_templates/default.html @@ -1,10 +1,10 @@ {% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon with context %} -{{- result_header(result, favicons) -}} -{{- result_sub_header(result) -}} +{{- result_header(result, favicons, loop.index) -}} +{{- result_sub_header(result, loop.index) -}} {%- if result.embedded -%} - • + • {%- endif -%} {%- if result.embedded -%} @@ -25,7 +25,7 @@ {%- endif -%} {%- if rtl -%} -{{ result_footer_rtl(result) }} +{{ result_footer_rtl(result, loop.index) }} {%- else -%} -{{ result_footer(result) }} +{{ result_footer(result, loop.index) }} {%- endif -%} diff --git a/searx/templates/oscar/result_templates/images.html b/searx/templates/oscar/result_templates/images.html index d0a3b7b83..96cdd53bb 100644 --- a/searx/templates/oscar/result_templates/images.html +++ b/searx/templates/oscar/result_templates/images.html @@ -1,6 +1,6 @@ -{%- from 'oscar/macros.html' import draw_favicon -%} +{%- from 'oscar/macros.html' import draw_favicon with context -%} -{{- "" -}} +{{- "" -}} {{ result.title|striptags }}{{- "" -}}