OwlBurst/Ponysearch
1
0
Fork 0
forked from Ponysearch/Ponysearch
Code Pull requests Activity
5497 commits 4 branches 17 tags 96 MiB
Commit graph

149 commits

Author SHA1 Message Date
marc
af35eee10b tests for _fetch_supported_languages in engines 
and refactor method to make it testable without making requests
 2016-12-15 00:40:21 -06:00
marc
f62ce21f50 [mod] fetch supported languages for several engines 
utils/fetch_languages.py gets languages supported by each engine and
generates engines_languages.json with each engine's supported language.
 2016-12-13 19:58:10 -06:00
marc
149802c569 [enh] add supported_languages on engines and auto-generate languages.py 2016-12-13 19:32:00 -06:00
Alexandre Flament
e48f07a367 Merge branch 'master' into searchpy2 2016-12-09 23:11:45 +01:00
Adam Tauber
55dc538398 [mod] move load_module function to utils 2016-11-19 17:51:19 +01:00
Alexandre Flament
01e2648e93 Simplify search.py, basically updated PR #518 
The timeouts in settings.yml is about the total time (not only the HTTP request but also the prepare the request and parsing the response)
It was more or less the case before since the threaded_requests function ignores the thread after the timeout even the HTTP request is ended.

New / changed stats :
* page_load_time : record the HTTP request time
* page_load_count: the number of HTTP request
* engine_time : the execution total time of an engine
* engine_time_count : the number of "engine_time" measure

The avg response times in the preferences are the engine response time (engine_load_time / engine_load_count)

To sum up :
* Search.search() filters the engines that can't process the request
* Search.search() call search_multiple_requests function
* search_multiple_requests creates one thread per engine, each thread runs the search_one_request function
* search_one_request calls the request function, make the HTTP request, calls the response function, extends the result_container
* search_multiple_requests waits for the the thread to finish (or timeout)
 2016-11-05 13:45:20 +01:00
Adam Tauber
86daef2063 [fix] do not allow underscore in engine names - closes #708 2016-09-28 22:30:05 +02:00
Adam Tauber
7d9c898170 Merge pull request #634 from kvch/advanced-search 
support time range search
 2016-07-26 00:06:16 +02:00
Adam Tauber
54d987636e [fix] do not load engines which cannot be initialized - closes #585 2016-07-25 23:36:52 +02:00
Noemi Vanyi
93c0c49e9a add time range search with yahoo 2016-07-25 23:19:46 +02:00
Adam Tauber
257e52954a [enh][fix] update to latest dependencies ++ fix tests & travis test runner 
WARNING: dependency changes
 2016-07-04 22:46:43 +02:00
Adam Tauber
10947536aa [enh] suspend engines after error 
The duration is based on the number of continuous errors, but maximized in one minute
 2016-02-20 00:21:56 +01:00
Adam Tauber
17b0c9f74a [mod] engine load refactor 2016-02-19 15:13:01 +01:00
Adam Tauber
1fcf066a81 [mod] change settings file structure according to #314 2015-08-02 20:32:22 +02:00
Alexandre Flament
6765823c56 [fix] set engine.safesearch to False if not defined 2015-05-30 12:37:18 +02:00
Alexandre Flament
4689fe341c update versions.cfg to use the current up-to-date packages 2015-05-02 15:45:17 +02:00
Adam Tauber
b19e681cce [enh] default disabled engines - closes #109 2015-01-31 23:11:45 +01:00
Adam Tauber
299a80a1eb [enh] using the logger 2015-01-09 04:13:05 +01:00
Cqoicebordel
5d977056f7 Flake8 and Twitter corrections 
Lots of Flake8 corrections
Maybe we should change the rule to allow lines of 120 chars. It seems more usable.

Big twitter correction : now it outputs the words in right order...
 2014-12-29 21:31:04 +01:00
Adam Tauber
2e41bfcbdb [enh] engine shortcut validation 2014-12-20 12:17:03 +01:00
Adam Tauber
2cfa760b57 [mod] engine load refactor 2014-12-13 19:26:40 +01:00
Adam Tauber
5740cfbf1c [fix] pep8 part II. 2014-10-19 12:41:04 +02:00
Thomas Pointhuber
fdad5f7cad fix float division by zero bug in score calculation 2014-10-01 19:16:34 +02:00
Adam Tauber
b0ba367a1a [enh][mod] search refactor 2014-07-07 13:59:27 +02:00
Adam Tauber
a07b2b514c [fix] url path unquoted check to avoid duplications 2014-07-05 17:33:19 +02:00
Adam Tauber
b226e6462b [fix] www. domain duplications 2014-05-20 01:16:49 +02:00
Adam Tauber
2be94c2078 [enh] engine default category init 2014-03-08 13:59:50 +01:00
asciimoo
5b2680be4b [enh] new engine parameter: timeout 2014-02-13 17:08:22 +01:00
asciimoo
c1d7d30b8e [mod] len() removed from conditions 2014-02-11 13:13:51 +01:00
asciimoo
9eb31c0186 [fix] unicode and pep8 fixes 2014-02-07 02:45:12 +01:00
Gabor Nagy
22fd0746d9 Improves PEP8 compatibility. 2014-02-05 20:24:31 +01:00
asciimoo
88425f05d6 [enh] shortcut support added to engines 2014-01-31 15:45:18 +01:00
asciimoo
2a788c8f29 [enh] search language support init 2014-01-31 04:35:23 +01:00
asciimoo
8bb94e3dc4 [enh] paging support 2014-01-29 20:52:04 +01:00
asciimoo
2b1d2dbc84 [enh] i18n support 2014-01-22 00:17:49 +01:00
asciimoo
b2492c94f4 [fix] pep/flake8 compatibility 2014-01-20 02:31:20 +01:00
Matej Cotman
dd4662978d fix: robot fw, entry points, some flake8, package searx egg 2014-01-20 01:06:29 +01:00
asciimoo
3afdd1d994 [enh] settings unification - new dependency: pyyaml 2014-01-19 00:17:02 +01:00
asciimoo
39ebe1d519 [fix] weight configurable via engines.cfg 2014-01-18 21:55:42 +01:00
asciimoo
046e190721 [mod] function name 2014-01-18 21:53:59 +01:00
asciimoo
a65070a72d [enh] own useragent handling init 2014-01-12 20:13:14 +01:00
asciimoo
7b4ec5c5e9 [fix] highlighting only html 2014-01-10 23:38:08 +01:00
asciimoo
a5544ab95e [enh] score per result stat added 2014-01-05 23:13:53 +01:00
asciimoo
53878fde2d [fix] error percentage @ stats page 2014-01-05 01:00:32 +01:00
asciimoo
5f065886ff [fix] not highlighting Nonetype 2013-12-16 14:06:53 +01:00
asciimoo
3755f437e4 [fix] utf8 content highlighting 2013-11-19 11:27:17 +01:00
asciimoo
66ff28bb9d [enh] suggestion support added 2013-11-13 19:32:46 +01:00
asciimoo
9688495b9f [mod] result scoring separated 2013-11-12 21:34:28 +01:00
asciimoo
b9ebf18f59 [fix] query highlight fix 2013-11-10 21:47:42 +01:00
asciimoo
cbb397939d [enh] incasesensitive query highlighting 2013-11-10 21:41:01 +01:00
asciimoo
6d956a7d1d [enh] engine score weight support @ config and engines 2013-11-10 21:16:25 +01:00
asciimoo
14a53e3430 [enh] search query highlight in result texts 2013-11-09 18:37:25 +01:00
asciimoo
1a2cf205f9 [enh] skipping empty/nonetype categories 2013-11-04 21:46:23 +01:00
asciimoo
4cc391169a [enh] ignoring empty request urls 2013-11-04 21:06:53 +01:00
asciimoo
70278567ec [mod] searching by selected engines 2013-11-04 00:18:07 +01:00
asciimoo
d2a952ecdf [enh] request arg refactor 2013-11-03 00:40:45 +01:00
asciimoo
443af2ba83 [fix] varname 2013-11-03 00:34:09 +01:00
asciimoo
a8797839a1 [enh] request timeout added 2013-11-03 00:31:03 +01:00
asciimoo
326a33fafe [enh] better url comparison 2013-11-02 18:07:48 +01:00
asciimoo
cd268f90e2 [fix] correct scheme @ parsed url 2013-11-02 18:04:13 +01:00
asciimoo
ec95fddce2 [mod] scoring 2013-10-28 11:38:33 +01:00
asciimoo
e13fe033ed [enh] engine exception handling ++ stats 2013-10-27 20:45:21 +01:00
asciimoo
ca0800306f [mod] scoring mods 2013-10-27 19:06:30 +01:00
asciimoo
21e9c87305 [mod] pageload stat order 2013-10-27 14:55:47 +01:00
asciimoo
f01e67dbd4 [mod] scoring mods 2013-10-27 14:48:16 +01:00
asciimoo
0f301adf60 [enh] score stat mods 2013-10-27 14:39:23 +01:00
asciimoo
09154c1f01 [mod] results engines mods 2013-10-27 14:34:35 +01:00
asciimoo
22c94ec149 [fix] using https links 2013-10-27 14:23:53 +01:00
asciimoo
e2c1c10657 [enh] engine score stats 2013-10-27 14:21:23 +01:00
asciimoo
13a4582a40 [enh] search count 2013-10-27 13:55:18 +01:00
asciimoo
afe767d8e9 [enh] stat bar graphs 2013-10-27 02:59:32 +02:00
asciimoo
4f4ec44bb3 [enh] stat updates 2013-10-27 02:50:24 +02:00
asciimoo
43ac547bc1 [enh] new stat: average request duration 2013-10-27 01:23:13 +02:00
asciimoo
b8191f43e7 [enh] engine statistics 2013-10-27 01:02:28 +02:00
asciimoo
09e7353307 [enh] config checks 2013-10-26 18:44:58 +02:00
asciimoo
89b68242d3 [mod] var rename 2013-10-26 02:21:55 +02:00
asciimoo
34bdeda6d4 [enh] basic config check 2013-10-25 23:41:14 +02:00
asciimoo
5dcddbc0ab [enh] engine params to response callbacks 2013-10-25 22:00:56 +02:00
asciimoo
39d229e110 [enh][mod] engines.cfg support 2013-10-23 23:54:46 +02:00
asciimoo
5e358ea9a9 [enh] merging only results with same rendering templates 2013-10-23 12:24:51 +02:00
asciimoo
8520be3cd6 [enh] categories to request callback 2013-10-22 18:57:20 +02:00
asciimoo
ec28b77a93 [fix] filtering 2013-10-20 20:47:00 +02:00
asciimoo
95e4a9353a [fix] request header parametering 2013-10-20 20:20:10 +02:00
asciimoo
68bd8a19e9 [enh] cookie support 2013-10-19 22:34:46 +02:00
asciimoo
9ea84b285b [fix] better url comparison II. 2013-10-19 20:44:35 +02:00
asciimoo
ad2ff1fda8 [fix] better url comparison 2013-10-19 20:11:56 +02:00
asciimoo
38c003bf3d [enh] custom engine result weights 2013-10-19 19:04:46 +02:00
asciimoo
5ee0b7bb80 [enh] engine blacklist support 2013-10-19 19:01:06 +02:00
asciimoo
39a223091e [enh] displaying all search engines in results 2013-10-19 18:28:48 +02:00
asciimoo
70cbc09e93 [enh] better url comparison 2013-10-19 17:36:44 +02:00
asciimoo
90cec08e25 [fix] dict keys 2013-10-18 08:24:58 +02:00
asciimoo
f76d80de54 [enh] category support 2013-10-17 21:06:28 +02:00
asciimoo
cf36b20b5d [enh] license infos 2013-10-17 00:32:32 +02:00
asciimoo
fa9c9e090b [enh] result ordering and deduplication 2013-10-16 23:03:26 +02:00
asciimoo
a346327c6f [enh] result ordering 2013-10-15 23:20:26 +02:00
asciimoo
9e1ded688f [enh] engine selection support 2013-10-15 22:18:08 +02:00
asciimoo
d793c2733c [enh] engine types 2013-10-15 19:11:43 +02:00
asciimoo
d5ec0f43e4 [mod] search refactor 2013-10-15 18:19:06 +02:00
asciimoo
ae9fb1d7dc [enh] initial commit 2013-10-14 23:09:13 +02:00