Commit graph

5151 commits

Author SHA1 Message Date
Markus Heiser
7ef6ffa803 [fix] tidy up ignore lists .gitignore & .dockerignore
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 16:55:30 +02:00
Alexandre Flament
2530576f24
Merge pull request #157 from searxng/mod_static_build
static build: move generated files to searx/static/themes/*/src/generated
2021-06-22 10:36:19 +02:00
Markus Heiser
c1f72b71ba [fix] .gitattributes: bootstrap.css & .js has been moved to oscar
see 86912e22 [mod] oscar: get bootstrap and typeahead from NPM

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 10:24:05 +02:00
Markus Heiser
1559ed386e [fix] pygments.less - remove pyenv_OK condition
BTW: add pygments.less to the help message

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 10:22:33 +02:00
Alexandre Flament
04492b76de [themes] ignore packages-lock.json 2021-06-22 08:17:06 +02:00
Alexandre Flament
2bc9b00cd7 Static build 2021-06-22 08:15:17 +02:00
Alexandre Flament
dca3bcca9e [mod] simple theme: include fonts
"npm run webfont" to build the fonts directory.
It requires fontforge and ttfautohint distro packages.

partial revert of commit 7137d2893f
2021-06-22 08:15:17 +02:00
Alexandre Flament
d20f6a1f19 [mod] make themes.all update pygments*.less
Add a searx/static/themes/*/src/generated empty folder
2021-06-22 08:15:11 +02:00
Alexandre Flament
fb2f4ba902
Merge pull request #164 from searxng/docker_upgrade_alpine
Docker: upgrade to alpine 3.14
2021-06-22 07:27:19 +02:00
Markus Heiser
089b97156a
Merge pull request #167 from return42/fix-166
[fix] UnicodeEncodeError for preferences ULR
2021-06-21 23:37:09 +00:00
Markus Heiser
5973491701 [fix] UnicodeEncodeError for preferences ULR
Closes: https://github.com/searxng/searxng/issues/166

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 01:35:54 +02:00
Alexandre Flament
f3e56836d6
Merge pull request #165 from return42/patch-google
improve & document google engine
2021-06-21 20:12:07 +02:00
Markus Heiser
0ef6aa5126 [docs] add documentation from the sources of the google engines
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 18:25:52 +02:00
Alexandre Flament
ef12c41206
Docker: upgrade to alpine 3.14
Uses Python version 3.9 instead of 3.8
2021-06-21 18:03:54 +02:00
Markus Heiser
05e90f2e57 [fix] google answers: normalize space of the answers.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 16:50:25 +02:00
Markus Heiser
f096d68ec6 [mod] google engine: reduce mobile UI parameters to what is needed
Reverse engineering shows that not all of the parameters used by google's mobile
UI (aka "more results" button) are needed [1].

[1] https://github.com/searxng/searxng/pull/160#issuecomment-865013625

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 16:50:23 +02:00
Alexandre Flament
f4da4bafa7
Merge pull request #160 from searxng/google_mobile_ui
[experimental] google: use the mobile UI
2021-06-21 14:56:50 +02:00
Alexandre Flament
bf5f505d74
Merge pull request #161 from return42/fix-typo-manage
[fix] py.build - fix small typo in the build message
2021-06-21 14:55:28 +02:00
Alexandre Flament
7a5c36408a [mod] google: add "use_mobile_ui" parameter to use mobile endpoint.
disable by default, it has to be enabled in settings.yml

related to  #159
2021-06-21 14:52:04 +02:00
Markus Heiser
f41b7a7204 [fix] py.build - fix small typo in the build message
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 12:52:26 +02:00
Alexandre Flament
bb3e67f72b [mod] simple theme: normalize.css becomes a packages.json dependency 2021-06-21 08:28:29 +02:00
Alexandre Flament
0e3a87b73a
Merge pull request #158 from return42/fix-node.clean
[fix] node.clean - ignore npm dependencies when npm is not installed
2021-06-20 07:20:38 +02:00
Markus Heiser
a62e1123e2 [fix] node.clean - ignore npm dependencies when npm is not installed
error pattern::

    $ make clean
    CLEAN     pyenv
    PYENV     [virtualenv] drop local/py3
    CLEAN     docs -- build/docs dist/docs
    CLEAN     locally installed npm dependencies
    ./manage: line 318: npm: command not found
    ERROR: node.clean exit with error (127)
    make: *** [Makefile:90: node.clean] Error 127

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-19 16:58:18 +02:00
Alexandre Flament
eb6832eb95
Merge pull request #156 from return42/fix-google-news
Fix google news
2021-06-18 14:01:03 +02:00
Markus Heiser
9328c66e93 [fix] google news - send CONSENT Cookie to not be redirected
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking.  To get the consent
from the user, google-news requests are redirected to confirm and get a CONSENT
Cookie from https://consent.google.de/s?continue=...

This patch adds a CONSENT Cookie to the google-news request to avoid
redirection.

The behavior of the CONTENTS cookies over all google engines seems similar but
the pattern is not yet fully clear to me, here are some random samples from my
analysis ..

Using common google search from different domains::

    google.com:        CONSENT=YES+cb.{{date}}-14-p0.de+FX+816
    google.de:         CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    google.fr:         CONSENT=YES+srp.gws-{{date}}-0-RC2.fr+FX+826

When searching about videos (google-videos)::

    google.es:         CONSENT=YES+srp.gws-{{date}}-0-RC2.es+FX+076
    google.de:         CONSENT=YES+srp.gws-{{date}}-0-RC2.de+FX+171

Google news has only one domain for all languages::

    news.google.com:   CONSENT=YES+cb.{{date}}-14-p0.de+FX+816

Using google-scholar search from different domains::

    scholar.google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    scholar.google.fr: does not use such a cookie / did not ask the user
    scholar.google.es: does not use such a cookie / did not ask the user

Interim summary:

  Pattern is unclear and I won't apply the CONSENT cookie to all google engines.
  More experience is need before we generalize the CONSENT cookies over all
  google engines.

Related:

- e9a6ab401 [fix] youtube - send CONSENT Cookie to not be redirected
- https://github.com/benbusby/whoogle-search/issues/311
- https://github.com/benbusby/whoogle-search/issues/243

[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-18 13:21:20 +02:00
Markus Heiser
dd7b53d369 [fix] google-news engine - KeyError: 'hl in request
Since we added

- 1c67b6aec [enh] google engine: supports "default language"

there is a KeyError: 'hl in request,error pattern::

    ERROR:searx.searx.search.processor.online:engine google news : exception : 'hl'
    Traceback (most recent call last):
      File "searx/search/processors/online.py", line 144, in search
        search_results = self._search_basic(query, params)
      File "searx/search/processors/online.py", line 118, in _search_basic
        self.engine.request(query, params)
      File "searx/engines/google_news.py", line 97, in request
        if lang_info['hl'] == 'en':
      KeyError: 'hl'

Closes: https://github.com/searxng/searxng/issues/154
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-18 11:34:11 +02:00
Markus Heiser
4417d2b922
Merge pull request #153 from searxng/dependabot/pip/master/sphinx-tabs-3.1.0
Bump sphinx-tabs from 3.0.0 to 3.1.0
2021-06-18 08:19:38 +00:00
dependabot[bot]
4b59d72af8
Bump sphinx-tabs from 3.0.0 to 3.1.0
Bumps [sphinx-tabs](https://github.com/executablebooks/sphinx-tabs) from 3.0.0 to 3.1.0.
- [Release notes](https://github.com/executablebooks/sphinx-tabs/releases)
- [Changelog](https://github.com/executablebooks/sphinx-tabs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/executablebooks/sphinx-tabs/compare/v3.0.0...v3.1.0)

---
updated-dependencies:
- dependency-name: sphinx-tabs
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-06-18 07:05:28 +00:00
Markus Heiser
7ef33c91e0
Merge pull request #136 from searxng/issue-115
/preferences: use simple layout for the oscar theme (Issue 115)
2021-06-17 13:38:50 +00:00
Alexandre Flament
881659ca9d [mod] oscar theme: /preferences : HTML detail order match visual tabs
First details about the general tab, then detail about UI tab, etc...
No functionnal change
2021-06-17 15:29:07 +02:00
Alexandre Flament
f83b64270c [mod] oscar theme /preferences: reorganize the preferences
close #115
2021-06-17 15:29:07 +02:00
Alexandre Flament
92c68fe636
Merge pull request #151 from searxng/mod_oscar
make themes: 2 node_modules directories instead of 3
2021-06-17 10:46:50 +02:00
Markus Heiser
ae677cb64b [enh] ./manage node.env - check build tools first
The node.env build environment require npm, ttfautohint and fontforge installed
in the OS.  These tools can be installed by::

    sudo -H ./utils/searx.sh install buildhost

If one of the tools is not installed, the script node.env stops with a
appropriate message.

BTW: We ignore CentOS-7 as developer & build platform

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-17 09:23:02 +02:00
Alexandre Flament
a447c58218
Merge pull request #152 from return42/fix-typo
[fix] typo in a searx/settings.yml
2021-06-16 17:08:35 +02:00
Markus Heiser
96d4dbf74d [fix] typo in a searx/settings.yml
remove the erroneously leading underline

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-16 16:51:28 +02:00
Alexandre Flament
14ba56fd1a [mod] make node.clean: call the "clean" script from the packages.json
The dependencies and build process are described in the gruntfiles.js and packages.json files.
It makes sense to move the clean up to these files too.
2021-06-16 16:04:58 +02:00
Alexandre Flament
c013256881 [mod] remove common grunt install
Before this commit, there are 3 node_modules directory:
* one in .
* two others in ./searx/statics/themes/*

This is no desirable:
* it declares the npm depdenencies in the shell script.
* dependabot can't updates theses dependencies.
* this is a not standard way to build a package (two different locations for the dependencies).

With this commit and the PR  #150 there is one unique node_modules directory per theme.
2021-06-16 15:39:32 +02:00
Alexandre Flament
fdc367d58b
Merge pull request #149 from return42/doc-install-plg
[doc] admin manual - describe installation of external plugins
2021-06-16 14:44:56 +02:00
Alexandre Flament
85033f3d09 Static build 2021-06-16 14:30:46 +02:00
Alexandre Flament
7137d2893f [mod] simple theme: remove src/less/ion.less
This file is generated by webfont.
* It is now generated as searx/static/themes/simple/ion.less
* It is generated before the .less compilation.
* .gitignore includes this file

Add two new package depedencies: fontforge ttfautohint
See utils/searx.sh
2021-06-16 14:30:09 +02:00
Alexandre Flament
e3ebfd41d7 [mod] simple theme: autocomplete-js becomes a packages.json dependency
Update to version 2.7.1
2021-06-16 12:44:07 +02:00
Alexandre Flament
b4a47cacd8 [mod] simple theme: leaflet becomes a packages.json dependency
the build of the themes updates:
* js/leaflet.js ( was leaflet/leaflet.js )
* css/leaflet.css ( was leaflet/leaflet.css )
* css/images ( was leaflet/images )
2021-06-16 12:44:07 +02:00
Alexandre Flament
6288b549e2 [mod] simple theme: move magnet.svg to src/fonts
see https://github.com/searx/searx/pull/1109
2021-06-16 12:44:07 +02:00
Alexandre Flament
6b80c57a3c [mod] simple theme: move source files to the src directory 2021-06-16 12:38:06 +02:00
Alexandre Flament
49ea5b7644 Static build 2021-06-16 09:16:37 +02:00
Alexandre Flament
9108c90f0e [mod] simple theme: update dependencies 2021-06-16 09:16:02 +02:00
Markus Heiser
d70ec8826d [doc] admin manual - describe installation of external plugins
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-15 18:00:35 +02:00
Markus Heiser
127552f67b
Merge pull request #146 from return42/ddg-definitions
[pylint] searx/engines/duckduckgo_definitions.py
2021-06-15 15:08:59 +00:00
Markus Heiser
1b4208b946
Merge pull request #148 from return42/fix-yamllint
[fix] test.yamllint - fixed typo
2021-06-15 15:07:53 +00:00
Markus Heiser
9b0151eaa8 [fix] test.yamllint - fixed typo
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-15 17:04:51 +02:00