I'm having problems with timeout using the DataSax php driver for Cassandra.
Whenever I execute a certain command it always throws this exception after 10s:
PHP Fatal error: Uncaught exception 'Cassandra\Exception\TimeoutException' with message 'Request timed out'
My php code is like this:
$cluster = Cassandra::cluster()->build();
$session = $cluster->connect("my_base");
$statement = new Cassandra\SimpleStatement("SELECT COUNT(*) as c FROM my_table WHERE my_colunm = 1 AND my_colunm2 >= '2015-01-01' ALLOW FILTERING")
$result = $session->execute($statement);
$row = $result->first();
My settings in cassandra.yaml is:
# How long the coordinator should wait for read operations to complete
read_request_timeout_in_ms: 500000
# How long the coordinator should wait for seq or index scans to complete
range_request_timeout_in_ms: 1000000
# How long the coordinator should wait for writes to complete
write_request_timeout_in_ms: 2000
# How long the coordinator should wait for counter writes to complete
counter_write_request_timeout_in_ms: 50000
# How long a coordinator should continue to retry a CAS operation
# that contends with other proposals for the same row
cas_contention_timeout_in_ms: 50000
# How long the coordinator should wait for truncates to complete
# (This can be much longer, because unless auto_snapshot is disabled
# we need to flush first so we can snapshot before removing the data.)
truncate_request_timeout_in_ms: 60000
# The default timeout for other, miscellaneous operations
request_timeout_in_ms: 1000000
I've already tried this:
$result = $session->execute($statement,new Cassandra\ExecutionOptions([
'timeout' => 120
])
);
and this:
$cluster = Cassandra::cluster()->withDefaultTimeout(120)->build();
and this:
set_time_limit(0)
And it always throws the TimeoutException after 10s..
I'm using Cassandra 3.6
Any idea?
Using withConnectTimeout (instead of, or together with withDefaultTimeout) might help avoid a TimeoutException (it did in my case)
$cluster = Cassandra::cluster()->withConnectTimeout(60)->build();
However, if you need such a long timeout, then there is probably an underlying problem that will need solving eventually.
Two things you are doing wrong.
ALLOW FILTERING : Be careful. Executing this query with allow filtering might not be a good idea as it can use a lot of your computing resources. Don't use allow filtering in production Read the
datastax doc about using ALLOW FILTERING
https://docs.datastax.com/en/cql/3.3/cql/cql_reference/select_r.html?hl=allow,filter
count() : It also a terrible idea to use count(). count() actually pages through all the data. So a select count() from userdetails without a limit would be expected to timeout with that many rows. Some details here: http://planetcassandra.org/blog/counting-key-in-cassandra/
How to Fix it ?
Instead of using ALLOW FILTERING, You should create index table of
your clustering column if you need query without partition key.
Instead of using count(*) you should create a counter table
Related
The company i work for uses Kayako to manage its support tickets. I was set to make an external webapp that takes all the tickets from one company and displays the history.
Im using mysqli_query to connect to the db.
$link = mysqli_connect($serverName, $userName, $password, $dbName);
$sql = "SELECT ticketid, fullname, creator, contents FROM swticketposts";
$result = mysqli_query($link, $sql) or die(mysqli::$error);
The problem is that the "contents" table in mySQL uses the datatype LONGTEXT.
Trying to read this data with php gives me either timeout or max memory usage errors.
Line 49 is the $result = mysqli_query etc line.
EDIT: Posted wrong error msg in original post
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8192 bytes)
Trying to work around the memory problem i added:
ini_set('memory_limit', '-1');
which gave me this timeout error instead:
Fatal error: Maximum execution time of 30 seconds exceeded in C:\xampp\htdocs\test\php\TicketCall.php on line 49
Thing is, when we view the tickets on the kayako platform, it reads the same contents table and does it instantly, so there must be a way to read these longtext's faster. How is beyond me.
Solutions i can't use:
Change the datatype to something smaller (would break our kayako system)
TLDR; Is there a way to read from longtext data types without killing memory and getting timout errors using php and mysql.
First of all, you might consider changing your query to something like this:
SELECT ticketid, fullname, creator, contents FROM swticketposts WHERE customer = 'something'
That will make your query return less data by filtering out the information from customers your report doesn't care about. It may save execution time on your query.
Second, you may wish to use
SUBSTRING(contents, 1, 1000) AS contents
in place of contents in your query. This will get you back part of the contents column (the first thousand characters); it may (or may not) be good enough for what you're trying to do.
Third, mysqli_ generally uses buffered querying. That means the entire result set gets slurped into your program's memory when you run the query. That's probably why your memory is blowing out. Read this. http://php.net/manual/en/mysqlinfo.concepts.buffering.php
You can arrange to handle your query row-by-row doing this:
$unbufResult = mysqli_query($link, $sql, MYSQLI_USE_RESULT) or trigger_error($link->error);
if ($unbufResult ) {
while ($row = $unbufResult ->fetch_assoc()) {
/* deal with the data from one row */
}
}
$unbufResult->close();
You need to be careful to use close() when you're done with these unbuffered result sets, because they use resources on your MySQL server until you do. This will probably prevent your memory blowout. (There's nothing special about using fetch_assoc() here; you can fetch each row with any method you wish.)
Fourth, php's memory and time limits are usually tuned for interactive web site operation. Sometimes report generation takes a long time. Put calls to set_time_limit() in and around your loop, something like this.
set_time_limit (300); /* five minutes */
$unbufResult = mysqli_query($link, $sql, MYSQLI_USE_RESULT) or trigger_error($link->error);
if ($unbufResult ) {
while ($row = $unbufResult ->fetch_assoc()) {
set_time_limit (30); /* reset time limit on each row */
/* deal with the data from one row */
}
}
$unbufResult->close();
set_time_limit (300); /* set time limit back to five minutes when done reading */
I was doing bulk inserts in the RealTime Index using PHP and by Disabling AUTOCOMIT ,
e.g.
// sphinx connection
$sphinxql = mysqli_connect($sphinxql_host.':'.$sphinxql_port,'','');
//do some other time consuming work
//sphinx start transaction
mysqli_begin_transaction($sphinxql);
//do 50k updates or inserts
// Commit transaction
mysqli_commit($sphinxql);
and kept the script running overnight, in the morning i saw
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate
212334 bytes) in
so when i checked the nohup.out file closely , i noticed , these lines ,
PHP Warning: mysqli_query(): MySQL server has gone away in /home/script.php on line 502
Warning: mysqli_query(): MySQL server has gone away in /home/script.php on line 502
memory usage before these lines was normal , but memory usage after these lines started to increase, and it hit the php mem_limit and gave PHP Fatal error and died.
in script.php , line 502 is
mysqli_query($sphinxql,$update_query_sphinx);
so my guess is, sphinx server closed/died after few hours/ minutes of inactivity.
i have tried setting in sphinx.conf
client_timeout = 3600
Restarted the searchd by
systemctl restart searchd
and still i am facing same issue.
So how can i not make sphinx server die on me ,when no activity is present for longer time ?
more info added -
i am getting data from mysql in 50k chunks at a time and doing while loop to fetch each row and update it in sphinx RT index. like this
//6mil rows update in mysql, so it takes around 18-20 minutes to complete this then comes this following part.
$subset_count = 50000 ;
$total_count_query = "SELECT COUNT(*) as total_count FROM content WHERE enabled = '1'" ;
$total_count = mysqli_query ($conn,$total_count_query);
$total_count = mysqli_fetch_assoc($total_count);
$total_count = $total_count['total_count'];
$current_count = 0;
while ($current_count <= $total_count){
$get_mysql_data_query = "SELECT record_num, views , comments, votes FROM content WHERE enabled = 1 ORDER BY record_num ASC LIMIT $current_count , $subset_count ";
//sphinx start transaction
mysqli_begin_transaction($sphinxql);
if ($result = mysqli_query($conn, $get_mysql_data_query)) {
/* fetch associative array */
while ($row = mysqli_fetch_assoc($result)) {
//sphinx escape whole array
$escaped_sphinx = mysqli_real_escape_array($sphinxql,$row);
//update data in sphinx index
$update_query_sphinx = "UPDATE $sphinx_index
SET
views = ".$escaped_sphinx['views']." ,
comments = ".$escaped_sphinx['comments']." ,
votes = ".$escaped_sphinx['votes']."
WHERE
id = ".$escaped_sphinx['record_num']." ";
mysqli_query ($sphinxql,$update_query_sphinx);
}
/* free result set */
mysqli_free_result($result);
}
// Commit transaction
mysqli_commit($sphinxql);
$current_count = $current_count + $subset_count ;
}
So there are a couple of issues here, both related to running big processes.
MySQL server has gone away - This usually means that MySQL has timed out, but it could also mean that the MySQL process crashed due to running out of memory. In short, it means that MySQL has stopped responding, and didn't tell the client why (i.e. no direct query error). Seeing as you said that you're running 50k updates in a single transaction, it's likely that MySQL just ran out of memory.
Allowed memory size of 134217728 bytes exhausted - means that PHP ran out of memory. This also leads credence to the idea that MySQL ran out of memory.
So what to do about this?
The initial stop-gap solution is to increase memory limits for PHP and MySQL. That's not really solving the root cause, and depending on t he amount of control you have (and knowledge you have) of your deployment stack, it may not be possible.
As a few people mentioned, batching the process may help. It's hard to say the best way to do this without knowing the actual problem that you're working on solving. If you can calculate, say, 10000 or 20000 records instad of 50000 in a batch that may solve your problems. If that's going to take too long in a single process, you could also look into using a message queue (RabbitMQ is a good one that I've used on a number of projects), so that you can run multiple processes at the same time processing smaller batches.
If you're doing something that requires knowledge of all 6 million+ records to perform the calculation, you could potentially split the process up into a number of smaller steps, cache the work done "to date" (as such), and then pick up the next step in the next process. How to do this cleanly is difficult (again, something like RabbitMQ could simplify that by firing an event when each process is finished, so that the next one can start up).
So, in short, there are your best two options:
Throw more resources/memory at the problem everywhere that you can
Break the problem down into smaller, self contained chunks.
You need to reconnect or restart the DB session just before mysqli_begin_transaction($sphinxql)
something like this.
<?php
//reconnect to spinx if it is disconnected due to timeout or whatever , or force reconnect
function sphinxReconnect($force = false) {
global $sphinxql_host;
global $sphinxql_port;
global $sphinxql;
if($force){
mysqli_close($sphinxql);
$sphinxql = #mysqli_connect($sphinxql_host.':'.$sphinxql_port,'','') or die('ERROR');
}else{
if(!mysqli_ping($sphinxql)){
mysqli_close($sphinxql);
$sphinxql = #mysqli_connect($sphinxql_host.':'.$sphinxql_port,'','') or die('ERROR');
}
}
}
//10mil+ rows update in mysql, so it takes around 18-20 minutes to complete this then comes this following part.
//reconnect to sphinx
sphinxReconnect(true);
//sphinx start transaction
mysqli_begin_transaction($sphinxql);
//do your otherstuff
// Commit transaction
mysqli_commit($sphinxql);
I have a query that uses stuff() when using it in PHP it fails due to it running out of memory.
the error occurs on this line;
$data = odbc_exec($conn, $query);
at this point php doesn't actually receive any data it is just assigning a network resource. the connection is fine, i use it everywhere else, and the query strlen is only ~400.
so what is actually going on here?
select oeename.id, oeename.Name, stuff(
(select distinct ', ' + departmentequipment.name from type join DepartmentEquipment on departmentequipmentfk = departmentequipment.id where type.OEENameFK = oeename.ID order by ', ' + departmentequipment.name for xml path('')) ,1,1,'') as departmentequipmentnames
from oeename order by oeename.name asc
even when i add top 1 to the query the query still exceeds the memory limit.
EDIT: the php script was working before, someone had detected a bug in a somewhat related part of my website and upon attempting to fault find the issue i came across this problem on my local development server. the table is now partly larger, but its failing on exec rather than reading the data in.
I believe you're running into this bug: https://bugs.php.net/bug.php?id=68964
To summarize the workaround, you need to cast any ntext fields to nvarchar(max) instead.
I have a weird problem.
I'm running a query:
SELECT IMIE, NAZWISKO, PESEL2, ADD_DATE, CONVERT(varchar, ADD_DATE, 121) AS XDATA, ID_ZLECENIA_XXX, * FROM XXX_KONWERSJE_HISTORIA AS EKH1
INNER JOIN XXX_DANE_PACJENTA EDP1 ON EKH1.ID_ZLECENIA_XXX=EDP1.ORDER_ID_XXX
WHERE EKH1.ID_KONWERSJE = (
SELECT MIN(ID_KONWERSJE)
FROM XXX_KONWERSJE_HISTORIA AS EKH2
WHERE EKH1.ID_ZLECENIA_XXX = EKH2.ID_ZLECENIA_XXX
)
AND EDP1.RECNO = (
SELECT MAX(RECNO)
FROM XXX_DANE_PACJENTA EDP2
WHERE EDP2.ORDER_ID_XXX = EDP1.ORDER_ID_XXX
)
AND EKH1.ID_ZLECENIA_XXX LIKE '%140000393%'
AND ADD_DATE>'20140419' AND ADD_DATE<='20140621 23:59:59.999'
ORDER BY EKH1.ID_KONWERSJE, EKH1.ID_ZLECENIA_XXX DESC
And the query works ok if I use a date limit around 2 months (63 days - it gives me 1015 results). If I extend the date limit query simply fails (Query failed blabla).
This happens under windows 64 bit php (apache, Xamp).
When I run this query directly from MS SQL SERWER Management Studio everything works fine, no matter what date limit I choose.
What is going on? Is there a limit of some kind under apache/php? (There is no information like "query time excessed", only "query failed")
And the query works ok if I use a date limit around 2 months (63 days
- it gives me 1015 results). If I extend the date limit query simply fails (Query failed blabla). ...
What is going on? Is there a limit of
some kind under apache/php? (There is no information like "query time
excessed", only "query failed")
This could happen because selectivity of ADD_DATE>'20140419' AND ADD_DATE<='20140621 23:59:59.999' is medium/low (there are [too] many rows that satisfy this predicate) and SQL Server have to scan (yes, scan) XXX_KONWERSJE_HISTORIA to many times to check following predicate:
WHERE EKH1.ID_KONWERSJE = (
SELECT ...
FROM XXX_KONWERSJE_HISTORIA AS EKH2
WHERE EKH1.ID_ZLECENIA_XXX = EKH2.ID_ZLECENIA_XXX
)
How many times have to scan SQL Server XXX_KONWERSJE_HISTORIA table to verify this predicate ? You can look at the properties of Table Scan [XXX_KONWERSJE_HISTORIA] data access operator: 3917 times
What you can do for the beginning ? You should create the missing index (see that warning with green above the execution plan):
USE [OptimedMain]
GO
CREATE NONCLUSTERED INDEX [<Name of Missing Index, sysname,>]
ON [dbo].[ERLAB_KONWERSJE_HISTORIA] ([ID_ZLECENIA_ERLAB])
INCLUDE ([ID_KONWERSJE])
GO
When I run this query directly from MS SQL SERWER Management Studio
everything works fine, no matter what date limit I choose.
SQL Server Management Studio has execution timeout set to 0 by default (no execution timeout).
Note: if this index will solve the problem then you should try (1) to create an index on ADD_DATE with all required (CREATE INDEX ... INCLUDE(...)) columns and (2) to create unique clustered indexes on these tables.
Try to set these php configurations in your php script via ini_set
ini_set('memory_limit', '512M');
ini_set('mssql.timeout', 60 * 20);
Not sure it will help you out.
This code takes ~0.1s
// find
$benchmark = Profiler::start ('Testing', 'find()');
$cursor = MongoBG::getInstance ( )->setDatabase ('test')->setCollection ('testcoll')->find();
Profiler::stop ($benchmark);
$benchmark = Profiler::start ('Testing', 'cursor walk');
while ($cursor->hasNext()) {
print_r($cursor->getNext());
}
Profiler::stop ($benchmark);
so "find()" took only 0.000017 seconds
but "cursor walk" 0.102812 seconds
Collection is about 100 rows, speed remains the same with 1000 or only 10 items in it.
Some server info:
FreeBSD 8.1, PHP 5.3.5 with (mongo/1.1.4), MongoDB version 1.6.6-pre
With such a quick time, it sounds like find didn't do anything but prepare an object (no communication with the database), and it's only upon using the cursor the actual query was executed and results were read. The cursor is doing the work which is why it's slower.
I know that's how the mongodb drivers for node.js worked. If you looked at it like that, the cursor speed isn't bad for opening a connection, authenticating, sending the query, receiving and buffering the response, then parsing/loading it into an object to return to you.