Hi I configured sphinx search in my test server.
Now I am getting this kind of an error "Sphinx_Query failed: no enabled local indexes to search".
I am not getting why this error. Any body can help me plese.
This is my sphinx conf
source objectcollection
{
type = mysql
sql_host = localhost
sql_user = root
sql_pass = root
sql_db = mydatabase
sql_port = 3306
sql_query = \
SELECT id, id as mid obtype_id, searchtext from tab_objectcollection;
sql_attr_uint = mid
sql_attr_uint = obtype_id
sql_query_info = SELECT * FROM tab_objectcollection WHERE id=$id
}
index combinedobject
{
source = objectcollection
path = /usr/local/sphinx/var/data/objectcollection
morphology = stem_en
min_stemming_len = 4
stopwords = /usr/local/sphinx/var/data/stopwords.txt
min_word_len = 3
min_prefix_len = 3
min_infix_len = 0
enable_star = 1
phrase_boundary = ., ?, !, U+2026 # horizontal ellipsis
phrase_boundary_step = 100
html_strip = 1
}
indexer
{
# memory limit, in bytes, kiloytes (16384K) or megabytes (256M)
# optional, default is 32M, max is 2047M, recommended is 256M to 1024M
mem_limit = 256M
# maximum xmlpipe2 field length, bytes
# optional, default is 2M
#
max_xmlpipe2_field = 16M
# write buffer size, bytes
# several (currently up to 4) buffers will be allocated
# write buffers are allocated in addition to mem_limit
# optional, default is 1M
#
#write_buffer = 16M
}
searchd
{
listen = 3312
max_matches = 10000
log = /usr/local/sphinx/var/log/searchd.log
query_log = /usr/local/sphinx/var/log/query.log
pid_file = /usr/local/sphinx/var/log/searchd.pid
}
Thanks
Have you
Actully built the index - ie called 'indexer' program, to make the index files.
Started the Search Daemon - searchd
I think this error means that sphinx can't find the file(s) specified by "path" in your index. In my case I had:
path = /var/lib/sphinxsearch/data/delta
And I had ran the indexer successfully (or so I thought) like this:
indexer delta --rotate
It said there were some documents collected. HOWEVER it actually created these files:
/var/lib/sphinxsearch/data/delta.new.sp?
And searchd failed to rotate the files. Thus spake the log:
WARNING: rotating index 'delta': rename '/var/lib/sphinxsearch/data/delta.mvp' to '/var/lib/sphinxsearch/data/delta.old.mvp' failed: No such file or directory
The solution was: delete those new files and run indexer without --rotate the first time.
The fact that --rotate doesn't work the first time seems like a bit of a bug to me, but I can't really be bothered to submit a bug report. It probably requires me to register or some nonsense. Anyway, hope this helps.
What I Understand by your question is, in the configuration file you have to mention which table or data is to be Indexed into it. Also there could be a problem with Sphinx daemon, that it is not able to create the Indexed data and write it into files. Do check with the above.
Hope to be of help somehow.
This seems to be an issue with sphinx 2.0.5, it's filed here:
http://sphinxsearch.com/bugs/view.php?id=1268
Try using a different version (I tried 2.0.6 and the problem is gone)
Related
I have a PHP script that is launching an SQL query, which takes a few seconds and returns about 15K+ results, after making a complex calculation based on data found on a DB table on my server, named table.
After getting the results, it is supposed to Insert (using a Replace) these records onto another DB table on my server.
The script used to work flawlessly when table was a simple core table. I later modified the SQL query to use data from a VIEW table, we can call it view_table, which is based on table but has an extra column that is calculated on the fly.
This had made the script to start crashing my whole SQL server, every once in awhile, throwing this error:
PHP Warning: mysqli::query(): MySQL server has gone away in
/home/user/script.php on line 109
Below is line 109:
function getRecordsFromDB(){
logMemoryUtilizationDetails();
file_put_contents(LOG_DIRECTORY.'service.log', date("d/m:H:i").':'."getRecordsFromDB::Entry".PHP_EOL, FILE_APPEND);
global $sourceConn;
$selectQuery = file_get_contents(__DIR__.'/my-query.sql');
$items = array();
$result = $sourceConn->query($selectQuery); // LINE 109
if ($result){
if ($result->num_rows > 0) {
while($row = $result->fetch_assoc()) {
$item = new Item();
$item->id = $row['id'];
$item->itemId = $row['itemid'];
I tried to log create to see how much memory does script use on start and exit, and when it is successful, it is using just about 37MB of RAM at its peak.
My server has 6GB of RAM on 4 cores.
There is no other script running on my server that is causing SQL server crashes like this, so I'm sure that this script is causing the crash.
Here is the MY.CNF of my server:
[mysqld]
# disable mysql strict mode
sql_mode=""
log-error=/var/lib/mysql/host.myhost.com.err
performance_schema=1
query_cache_type=0
query_cache_size=0
query_cache_limit=0
key_buffer_size=16M
max_connections=200
max_tmp_tables=1
table_open_cache=2000
local-infile=0
thread_cache_size=4
innodb_file_per_table=1
default-storage-engine=MyISAM
innodb_use_native_aio=0
max_allowed_packet=1024M
innodb_buffer_pool_size=800M
open_files_limit=10000
#wait_timeout=500
tmp_table_size=256M
max_heap_table_size=256M
innodb_buffer_pool_instances=1
#general_log=on
#general_log_file = /var/lib/mysql/all-queries.log
slow-query-log=0
slow-query-log-file=/var/lib/mysql/slow_queries.log
long_query_time=1.0
#log_queries_not_using_indexes=1
And this is from my PHP.INI (for PHP 7.2 which I'm using):
max_execution_time = 240
max_input_time = 60
max_input_vars = 1000
memory_limit = 512M
[MySQLi]
mysqli.max_persistent = -1
;mysqli.allow_local_infile = On
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
I don't see any mysql.connect_timeout setting in those files.
I have many other scripts and they all work fine, so I wouldn't want to change something globally as I'm afraid it can cause other issues on my server.
Looks like a timed-out or failed query. Please paste the sql query you are using. You can also try and see yourself where the query might cause this by pasting the query in your mysql IDE (Navicat is my favorite) and prepend it with 'explain extended' (no quotes). So your query would look like 'explain extended select ...(all 300 lines)'
Look for keys higher than 4, no primary keys and rows queried with really high numbers for starters.
Also, it looks like instead of a view you may want to consider creating a stored procedure in which you can select everything into a temporary table and then do the on-the-fly calculated value in the next query. Of course, you need to configure my.cnf to recognize temporary table so it can destroy it once the session is complete. Also, if you have any replication or cluster of servers, make sure to stop binlog before creating the temporary table and then start it once your queries are completed and your session is about to close.
If you like, please paste your my.cnf (mysql config file) to make sure your config is optimal for the large query.
Also, for trouble-shooting purposes, you may want to temporarily increase the max execution time in php.ini.
I'm using Sphinx to provide a search webpage to a huge set of data, recently I upgraded Sphinx from v2.1.8 to v2.2.4
I had some troubles in config file, one of them is that 'enable_star' option has been removed, which affected the expected results in my search page, so if search for 'rea' it will not return 'real madrid' until I complete typing 'real', the same issue at words ends 'madrid'.
The expected results if I searched for 'mad' :
Real Madrid
Atlatico Madrid
Mad-Croc
Madila
mad bla
In my case I get 'Mad-Croc' and 'mad bla'.
Here is a part of my config file :
docinfo = extern
mlock = 0
morphology = stem_en
min_word_len = 1
expand_keywords = 1
dict = keywords
PHP Code :
$_sphinx = new SphinxClient();
$_sphinx->SetServer('............', '....');
$_sphinx->SetMatchMode(SPH_MATCH_ANY);
$_sphinx->SetFieldWeights(array('auther_name' => 50));
$_sphinx->SetArrayResult(true);
$_sphinx->SetSortMode(SPH_SORT_EXTENDED2, 'cat_priority DESC, #weight DESC');
//////////////////
$_result = $_sphinx->Query($searchTerm . '*');
could any body look for this.
You dont seem to have min_prefix_len setup on your index, suggest you add it.
Although not sure how your index would ever of worked, as min_prefix_len, would be required for enable_star=0 to have an effect.
That should allow expand_keywords to work its magic. At which poing suggest removing the * from the end of the query. Which would only affect the last word entered anyway, and * should autotmatically by added by expand_keywords setting anyway.
I got a very simple select statement for instance:
SELECT id, some_more_fields_that_do_not_matter
FROM account
WHERE status = '0'
LIMIT 2
Keep in mind that the above returns the following id's: 1,2
The next thing I do is loop these rows in a for each and update some records:
UPDATE account
SET connection = $_SESSION['account_id'],
status = '1'
WHERE id = $row_id
Now the rows in the table with id's 1,2 have the status '1' (which I do check to make sure the rows where correctly updated). If it failed to do such, I will undo everything. As soon as everything is OK I have a counter at the first place which is 2 in this case, so 2 rows should have been updated, and I check this with a simple COUNT(*).
This information will also be emailed with for instance the following data (which means everything was updated correctly):
- Time of update: 2013-09-30 16:30:02
- Total rows to be updated (selected) = 2
- Total rows successfully updated after completing queries = 2
The following id's should have been updated
(these where returned by the SELECT statement):
1,2
So far so good. Now comes the weird part.
The very next query made by a other user will however sometimes return for instance the id's 1,2 (but that's impossible because these should never be returned by the SELECT statement because they do not contain the status '0' anymore. So what happens is the following:
I now receive an email with for instance:
- Time of update: 2013-09-30 16:30:39
- Total rows to be updated (selected) = 10
- Total rows successfully updated after completing queries = 8
The following id's should have been updated
(these where returned by the SELECT statement):
1,2,3,4,5,6,7,8,9,10
Now it is really strange the 1 and 2 are selected by the update. In most cases it goes good, but very rarely it just doesn't and returns some same ID's which are already updated with a status '1'.
Notice the time between these updates. It's not even the same time. I first thought it would be something like that these queries would be executed at the exact same time (which is impossible right?). Or is this possible? Or could it somehow be that the query has been cached, and should I edit some settings at my mysql.conf file?
I have never had this problem, I tried every way of updating but it seems to keep happening. Is it maybe possible at some way to combine these 2 queries in one huge update query? All the data is just the same and doesn't do anything strange. I hope someone has a clue what this could cause the problem and why this is randomly (rarely) happening.
EDIT:
I updated the script, added the microtime to check how long the SELECT, UPDATE and CHECK-(SELECT) together takes.
First member (with ID 20468) makes a call at: 2013-10-01 08:30:10
2/2 rows have been updated correctly of the following 2 ID's:
33412,33395
Queries took together 0.878005027771 seconds
Second member (with ID 10123) makes a call at: 2013-10-01 08:30:14
20/22 rows have been updated correctly of the following 22 ID's:
33392,33412,33395,33396,41489,13011,12555,27971,22811 and some more but not important
Queries took together 3.3440849781036 seconds
Now you see that the 33412 and 33395 are again returned by the SELECT.
Third member (with ID 20951) makes a call at: 2013-10-01 08:30:16
9/9 rows have been updated correctly of the following 9 ID's:
33392,33412,33395,33396,41489,13011,12555,27971,22811
Queries took together Didn't return anything which concerns me a
little bit too
Since we do not know how long the last queries took, we only know that the first and second should work correctly without problems because if you look there are 4 seconds between them. and the execution time was 3.34 seconds. Besides that the first one started at 2013-10-01 08:30:17 because the time that is logged for the call (when emailing it) is at the end of the script. The check to see how long the queries took are from the start of the first query and the stop directly after the last query and this is before I send the email (of course).
Could it be something in my.cnf file that mysql is doing this weird?
Still I don't understand why id didn't return any execution time for the last (third) call.
A solution for this would be to Queue these actions by first saving them into a table and executing them one at a time by a cron job. But it's not really what I want, it should be instant when a member makes the call. Thanks for the help so far.
Anyway here is my my.cnf in case someone has suggestions for me (Server has 16GB RAM installed):
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam-recover = BACKUP
max_connections = 20
query_cache_type = 1
query_cache_limit = 1M
query_cache_size = 4M
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M
innodb_buffer_pool_size = 333M
join_buffer_size = 128K
tmp_table_size = 16M
max_heap_table_size = 16M
table_cache = 200
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
!includedir /etc/mysql/conf.d/
EDIT 2:
$recycle_available = $this->Account->Membership->query("
SELECT Account.id,
(SELECT COUNT(last_clicks.id) FROM last_clicks WHERE last_clicks.account = Account.id AND last_clicks.roulette = '0' AND last_clicks.date BETWEEN '".$seven_days_back."' AND '".$tomorrow."') AS total,
(SELECT COUNT(last_clicks.id)/7 FROM last_clicks WHERE last_clicks.account = Account.id AND last_clicks.roulette = '0' AND last_clicks.date BETWEEN '".$seven_days_back."' AND '".$tomorrow."') AS avg
FROM membership AS Membership
INNER JOIN account AS Account ON Account.id = Membership.account
WHERE Account.membership = '0' AND Account.referrer = '0' AND Membership.membership = '1'
HAVING avg > 0.9
ORDER BY total DESC");
foreach($referrals as $key => $value)
{
$this->Account->query("UPDATE account SET referrer='".$account_id."', since='".$since_date."', expires='".$value['Account']['expires']."', marker='0', kind='1', auction_amount='".$value['Account']['auction_amount']."' WHERE id='".$recycle_available[$key]['Account']['id']."'");
$new_referral_id[] = $recycle_available[$key]['Account']['id'];
$counter++;
}
$total_updated = $this->Account->find('count',array('conditions'=>array('Account.id'=>$new_referral_id, 'Account.referrer'=>$account_id, 'Account.kind'=>1)));
You indicate in the comments you are using transactions. However, I can't see any $dataSource->begin(); nor $dataSource->commit(); in the PHP snippet you posted. Therefore, you must be doing $dataSource->begin(); prior to the snippet and $dataSource->commit(); or $dataSource->rollback(); after the snippet.
The problem is that you're updating and then trying to select prior to committing. No implicit commit is created, so you don't see updated data: http://dev.mysql.com/doc/refman/5.7/en/implicit-commit.html
It is hard to tell the reason of this strange behaviour without having my hands on the DB. But much better way to do what you are doing would be to do it all in one query:
UPDATE account
SET connection = $_SESSION['account_id'],
status = '1'
WHERE status = '0'
Most likely this will solve the problem you are facing.
I suggest using this syntax:
$query="UPDATE account SET connection = {$_SESSION['account_id']}, status = 1 WHERE id=$row_id;";
The compiler throws an error when you use an integer with ''. And remember to use {} when you have an array.
I'm testing Sphinx with PHP (tutorial : http://www.siteduzero.com/informatique/tutoriels/creer-un-moteur-de-recherche-avec-sphinx-et-php)
If I run "search" like this :
search -i news navigateur
I have the following error :
index 'news': search error: .
The indexer works fine :
$ sudo indexer news
Sphinx 2.0.6-release (r3473) Copyright (c)
2001-2012, Andrew Aksyonoff Copyright (c) 2008-2012, Sphinx
Technologies Inc (http://sphinxsearch.com)
using config file '/usr/share/php/sphinx/etc/sphinx.conf'...
indexing index 'news'...
collected 57 docs, 0.7 MB sorted 0.1 Mhits, 100.0%
done total 57 docs, 689264 bytes total 0.050 sec, 13684561 bytes/sec,
1131.67 docs/sec
total 3 reads, 0.000 sec, 115.5 kb/call avg, 0.1 msec/call avg
total 9 writes, 0.000 sec, 83.5 kb/call avg, 0.0 msec/call avg
And, testing with the API, the PHP code return nothing. Anybody already have this error ?
Here the config file :
index news {
source = news
path = /usr/share/php/sphinx/var/data/news
}
indexer {
mem_limit = 32M
}
searchd {
port = 3312
log = /usr/share/php/sphinx/var/log/searchd/searchd.log
query_log = /usr/share/php/sphinx/var/log/searchd/query.log
pid_file = /usr/share/php/sphinx/var/log/searchd/searchd.pid
}
source news {
type = mysql
sql_host = localhost
sql_user = root
sql_pass =
sql_db = siteduzero_sphinx
sql_query_pre = SET NAMES utf8
sql_query = SELECT id\
, categorie\
, titre\
, contenu FROM news
sql_attr_uint = categorie
sql_query_info = SELECT titre FROM news WHERE id=$id }
Thanks.
Fabrice
Just ran into the same problem. The root of evil in the last config parameter. This command will work correctly:
search -i news --noinfo navigateur
Or you can remove this string from config:
sql_query_info = SELECT titre FROM news WHERE id=$id
If you don't need it.
It's a bug in the search utility.
The good news is the search utility, is just a testing tool, so its not critical. Its not actually used for anything other than testing.
As for your problem with the API, do you have the searchd daemon running?
Rather than just saying it returns 'nothing', you should use getLastError and getLastWarning to investigate why...
I have a problem with Sphinx. I have configuration like this:
sql_query = \
SELECT id, product_title, product_inf, product_code, ptype_name, title, cat, value, car \
FROM Catalog_View;
sql_attr_uint = car
sql_attr_uint = cat
Catalog_View is a view which collect data from several tables. It works good and haven't got any problem. I created index with this configuration:
index src1
{
source = src1
path = /var/data/src1
docinfo = extern
mlock = 0
morphology = stem_en, stem_ru
min_word_len = 3
charset_type = sbcs
min_prefix_len = 0
min_infix_len = 3
enable_star = 1
}
And indexer done his job perfect. But when I'm looking for empty query (like this '') and setup two filters
$cl->SetFilter('cat',array(9));
$cl->SetFilter('car',array(2));
I loose a lot of matches. For example when I use SQL-query to Catalog_View I have 76 rows, and the same in Sphinx gives me only 11 rows. I can't figure out what am i doing wrong. Everything seems fine except filter.
Actually I have the same problem with filters when I'm looking for non-empty query.
I stumbled upon this one too. My solution was making document IDs unique. If you have duplicated document IDs the possible result can be the same. I suppose Sphinx would take only first unique document thrashing all duplicated data.