I have a MySQL database with a backend: PHP.
In one table I have to insert 60,000 rows.
We made a query into my PHP that returns 1,000 rows. For each rows we have to insert 60 rows. We thought make a loop but we don't know if is the best practices that way.
The part of the code that insert the data is:
$turnos = $db->query("SELECT * FROM turno t
WHERE t.canchaId = :cancha
AND t.fecha BETWEEN :fechaInicio AND :fechaFin
AND t.nombre LIKE :cadena
ORDER BY t.fecha,t.hora ASC",
array("cancha" => $cancha["idCancha"], "fechaInicio" => $fechaInicio, "fechaFin" => $fechaFin, "cadena" => "%turno fijo%"));
foreach($turnos as $turno) {
//turnos have 1000 rows
$fecha = date_create($turno["fecha"]);
date_add($fecha, date_interval_create_from_date_string('7 days'));
$anioAuxiliar = 2017;
while(2017 == $anioAuxiliar){
//60 times
$data1 = turno[data1];
$data2 = turno[data2];
...
$fechaAGuardar = (String) $fecha->format('Y-m-d');
$result = $db->query("INSERT INTO turno(fechaAGuardar, data2, data3, data4, data5, data6, data7, data8) VALUES(:fechaAGuardar, :data2, :data3, :data4, :data5, :data6, :data7, :data8)",
array("fechaAGuardar" => $fechaAGuardar, "data2" => $data2, "data3" => $data3, "data4" => $data4, "data5" => $data5, "data6" => $data6, "data7" => $data7, "data8" => $data8));
date_add($fecha, date_interval_create_from_date_string('7 days'));
$anioAuxiliar = (int) $fecha->format("Y");
}
$cantidad_turnos = $cantidad_turnos + 1;
}
This php is into a hosting with phpmyadmin.
So my questions are:
This is the best way to insert 60,000 rows?
Shall we considerer take into account another constraint? (eg: phpmyadmin don't allow you insert that amount of rows)
Thanks for helping me, Any suggestions are welcome
//EDIT//
The inserts data change, we have to insert datetime, and for each loop we have to add 7 day the last date inserted. So we can't use insert with select
As a bunch of fellows described in the comments, INSERT/SELECT is the way to go if this data is in the same server/database. There's no need to use PHP at all. Your year comment can be handled with DATE_ADD.
Anyway, if there is any other requirement and you can't use PHP, consider using Bulk Data Loading.
Analysing your code, the MOST IMPORTANT TIP would be: don't use multiple INSERT INTO TABLE expressions. Each INSERT INTO will cause a round trip do the database and things will get really slow. Instead of it, concat multiple values with one INSERT INTO (example from the link):
INSERT INTO yourtable VALUES (1,2), (5,5), ...;
Good luck!
Related
I have an app that is receiving sensor data. As an illustration let's say the expected range is between 0.01 and 10. In this case in the migration I may have something like:
$table->float('example',5, 2)
This way I am able to handle an order of magnitude outside the expected range. However it is possible for the sensor to glitch and send values of say 10 000. As the sensor is sending an array of values it is likely that not all the data is incorrect so it is preferred to still write the data to the DB. For the initial insert I am using the code below which is working as expected:
DB::table($tableName)->insertOrIgnore($insert_array);
However, in some circumstances the sensor can resend the data in which case the record needs to be updated. It's possible that the value(s) that are out of range can remain in the array, in which case the update statement below will throw an out of range error:
DB::table($tableName)->where('$some_id','=', $another_id)->update($insert_array);
I have not been able to find an something akin to an "updateorignore" functionality. What is the best way to handle updating this record? Note I cannot simply put it in a try catch since this table will be a parent table to some children and ignoring it will result in some orphaned entries.
Do you have some unique points of data to tie the records together, such as a timestamp or an id from the origin of the data? If so, you can use updateOrInsert
DB::table($tableName)
->updateOrInsert(
['remote_id' => $insert_array['remote_id'],
[
'example_field_1' => $insert_array['example_field_1'],
'example_field_2' => $insert_array['example_field_2'],
]
);
This is discussed at:
https://laravel.com/docs/master/queries#updates
Thanks to James Clark Developer to help me get to the solution. Below is some sample code to resolve the problem:
$values = array();
//set up values for binding to prevent SQL injection
foreach ($insert_array as $item) {
$values[] = '?';
}
//array values need to be in a "flat array"
$flat_values = implode(", ", $values);
//add in separators ` to column names
$columns = implode("`, `",array_keys($insert_array));
//write sql statement
$sql = "INSERT IGNORE INTO `example_table` (`$columns`) VALUES
($flat_values) ON DUPLICATE KEY UPDATE id = '$id'";
DB::insert($sql, array_values($insert_array));
}
I am trying to insert the data of a query, currently I am doing it with the eloquent ORM of laravel, but it takes a long time since there are approximately 120k of records, so I would like to know if they can help me as it can be done in the same query
This is the query:
$var = DB::select("
select selrs.*, sim.emin, sim.emax
from (select rac.*, sup.zona, sup.sitio, sup.manejo
from datos_rendim_actual rac
left join super sup on (sup.codigo = (rac.fundo::int8 * 1000000 + rac.rodal::int8 ))) selrs
left join sitman sim on (sim.sitio = selrs.sitio and sim.manejo = selrs.manejo)
where selrs.edad >= sim.emin and selrs.edad <= sim.emax
");
This is the dd of $var
array:123921 [▼
0 => {#813 ▼
+"id": 255214
+"fundo": 101
+"rodal": 196826
+"codigo": null
+"edad": 10
+"densidad": "1019"
+"vol_prod1": "0"
+"vol_prod2": "113.95"
+"created_at": null
+"updated_at": null
+"zona": 5
+"sitio": 1
+"manejo": 7
+"emin": 10
+"emax": 20
}
So This is how I insert them:
foreach ($var as $lista) {
$rendimA = new RendimActual;
$rendimA->codigo = $lista->fundo.$lista->rodal;
$rendimA->edad = $lista->edad;
$rendimA->densidad = $lista->densidad;
$rendimA->vol_prod1 = $lista->vol_prod1;
$rendimA->vol_prod2 = $lista->vol_prod2;
$rendimA->vol_prod3 = $lista->vol_prod3;
$rendimA->save();
}
The fields that I have to fill are in the rendim_actual table and are the following:
codigo = concat(fundo, rodal) from $var
edad= from $var
densidad = from $var
vol_prod1 to n (actually there are 36 but as an example just leave 3) from $var
in terms of time insert by Eloquent takes about 15 minutes, I hope you can help me, ty
I am using laravel-5.8 and postgresql
I don't have much experience dealing with large databases, but in your solution you will be making 120k separate inserts, which will not be efficient.
If the records in your $var array are already in the correct format from your table you can try using the query builder's insert method.
// Not certain what your table name is
DB::table('rendim_actual')->insert($var);
DB::commit();
If the $var array becomes soo big that you cannot insert all data in one insert, you can look into this SO Question about chunking inserts.
What you are currently doing is running an insert query for each RendimActual. What you should be doing is bulk insert via ::insert()
insert()
This will insert 1 query with 120K values
$insert = [];
foreach ($var as $lista) {
$insert[] = [
'codigo' => $lista->fundo . $lista->rodal,
'edad' => $lista->edad,
'densidad' => $lista->densidad,
'vol_prod1' => $lista->vol_prod1,
'vol_prod2' => $lista->vol_prod2,
'vol_prod3' => $lista->vol_prod3
];
}
RendimActual::insert($insert);
chunk()
Since you are inserting 120K values inside 1 query, there's a chance of consuming a lot of memory. So, it better to chunk() the data if 2000 rows and insert those 2K first. Even though you will be running 120K/2K = 60 queries ... at least this way each query will not be consuming memory compared to 1 big (120K) insert query.
foreach (array_chunk($insert, 2000) as $inst) {
RendimActual::insert($inst);
}
Keep me posted in the comments below. Cheers!
I am trying to make a PHP loop work for me in MySQL. Currently all visits to a website via a specific URL parameterare logged into a table along with the date and time of the visit. I am rebuilding the logging procedure to only count the visits via one specific parameter on one day, but I'll have to convert the old data first.
So here's what I'm trying to do: The MySQL table (let's call it my_visits) has 3 columns: parameter, visit_id and time.
In my PHP code, I've created the following loop to gather the data I need (all visits made via one paramter on one day, for all parameters):
foreach (range(2008, 2014) as $year) {
$visit_data = array();
$date_ts = strtotime($year . '-01-01');
while ($date_ts <= strtotime($year . '-12-31')) {
$date = date('Y-m-d', $date_ts);
$date_ts += 86400;
// count visit data
$sql = 'SELECT parameter, COUNT(parameter) AS total ' .
'FROM my_visits ' .
'WHERE time BETWEEN \''.$date.' 00:00\' AND \''.$date.' 23:59\' '.
'GROUP BY parameter ORDER BY total DESC';
$stmt = $db->prepare($sql);
$stmt->execute(array($date));
while ($row = $stmt->fetch()) {
$visit_data[] = array(
'param' => $row['parameter'],
'visit_count' => $row['total'],
'date' => $date);
}
$stmt->closeCursor();
}
}
Later on, the gathered data is inserted into a new table (basically eliminating visit_id) using a multiple INSERT (thanks to SO! :)).
The above code works, but due to the size of the table (roughly 3.4 million rows) it is very slow. Using 7 * 365 SQL queries just to gather the data seems just wrong to me and I fear the impact of just running the script will slow everything down substantially.
Is there a way to make this loop work in MySQL, like an equivalent query or something (on a yearly basis perhaps)? I've already tried a solution using GROUP BY, but since this eliminates either the specific dates or the parameters, I can't get it to work.
You can GROUP further.
SELECT `parameter`, COUNT(`parameter`) AS `total`, DATE(`time`) AS `date`
FROM `my_visits`
GROUP BY `parameter`, DATE(`time`)
You can then execute it once (instead of in a loop) and use $row['date'] instead of $date.
This also means you don't have to update your code when we reach 2015 ;)
Im trying to think of a way to form a query where I can query for each parameter I need, and get the results back on a per param basis. Ive tried a combination of AND/OR but not getting the expected results.
the basis of the query I wanna do is to the extent of
select fb_wall, fb_checkin, phone
from member_prefs where
memeberid = 21
OR memeberid = 22
OR memeberid = 23
OR memeberid = 24
The catch is, I need to either ignore when one of the memberid's doesnt have data and keep the query going (or catch the id, so if one is found I can add the row for it in another query after the fact but still keep the query going).
Right now my dummy data is 3 out of the 4 memberid's have data and one doesn't if I do AND in the query, the query stops and yields no results. Where as if I do the OR i seem to only be returning one of there sets of data.
What I want to do is after the query build an array to pass around that would look like
array(
[0] array("memberid"=>21, "fb_wall"=>0, "fb_checkin"=>1, "phone"=>0),
[1] array("memberid"=>22, "fb_wall"=>1, "fb_checkin"=>0, "phone"=>1),
[2] array("memberid"=>24, "fb_wall"=>1, "fb_checkin"=>1, "phone"=>1)
)
just seem to be having a little issue forming the initial query, and like I said if I could in this case pass 23 else where so I could run an insert command on that id for the same table that would be awesome but not needed for the over all need here.
How about:
//parameter data
$members = array(
22 => null,
23 => null,
24 => null,
25 => null
);
//make a comma separated list of ids
$ids = implode(',', array_keys($members));
//generate query text
$sql = '
select memberid, fb_wall, fb_checkin, phone
from member_prefs
where memberid in ('.$ids.')
';
$result = mysql_query($sql);
while ($row = mysql_fetch_row($result)) {
$members[$row[0]] = $row;
}
Start with an array with the relevant keys, and replace the elements as you read the rows.
$ares = array(21 => null, 22 => null, 23 => null, 24 => null);
I have a multi-dimensional array which contains ten thousands of data. A lot... The array is structured like this:
Array (
[0] => Array ( [0] => city [1] => code [2] => country )
[1] => Array ( [0] => city [1] => code [2] => country )
)
What I am trying to do is to insert the array values city, code and country into a table in a mysql database. I found posts that match exactly what I want to do, but for some reason it is not working with me. When I say it is not working I mean that the php doesn't even start. If I remove the here below code, the file runs normaly. So the problem really comes from that code portion. Hope someone will not mind helping me. Thank you in advance. Cheers. Marc.
//some code to build the array
//db_connect code
$sql = array();
foreach( $myarray as $row )
{
$sql[] = '("'.$row[0].'", "'.$row[1]).'","'.$row[2].'")';
}
mysql_query('INSERT INTO test (t_city, t_code, t_country) VALUES '.implode(',', $sql));
As said before, the error in building the sql array, is a surplus bracket. Change
$sql[] = '("'.$row[0].'", "'.$row[1]).'","'.$row[2].'")';
to
$sql[] = '("'.$row[0].'", "'.$row[1].'","'.$row[2].'")';
As ashein noted in comments, the query length is limited by the "max_allowed_paket" variable. If the query is larger than this, an error is raised and connection gets closed.
There is a bracket after $row[1] :)
Use this (remove the bracket):
$sql[] = '("'.$row[0].'", "'.$row[1].'","'.$row[2].'")';
You can try inserting every array record as separate sql-query.
foreach( $myarray as $row )
{
mysql_query('INSERT INTO test (t_city, t_code, t_country) VALUES ("'.$row[0].'", "'.$row[1]).'","'.$row[2].'");
}
but there would be a lot of queries
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
You want to dynamically build such a query by submitting multiple of the value pairs at once and not running into limits.
So what you do is to build the insert query while adding iteratively one row after the other. If adding a row would trigger the limit, the query is send and the query is reset:
# sample data
$data = array(
array('city1', 'code', 'country'),
array('city2', 'code', 'country'),
array('city3', 'code', 'country'),
array('city4', 'code', 'country'),
array('city5', 'code', 'country'),
array('city6', 'code', 'country'),
array('city7', 'code', 'country'),
);
$max_allowed_packet = 1048576; # mysql default value
$max_allowed_packet = 128; # for demonstration purposes
$sql = new SQLInsertQuery('INSERT INTO test (t_city, t_code, t_country) VALUES ', $max_allowed_packet);
foreach($data as $row) {
$sql->addRow($row);
}
$sql->query(); # manually query any potential left-over query.
This example outputs the following:
Running: INSERT INTO test (t_city, t_code, t_country) VALUES ('city1','code','country'),('city2','code','country');
Running: INSERT INTO test (t_city, t_code, t_country) VALUES ('city3','code','country'),('city4','code','country');
Running: INSERT INTO test (t_city, t_code, t_country) VALUES ('city5','code','country'),('city6','code','country');
Running: INSERT INTO test (t_city, t_code, t_country) VALUES ('city7','code','country');
Demo, Gist
You might want to add a counter for the queries run as well so you can validate after the loop and the final query if at all a query was sent (the limit can be too low so that no query is send at all - depending on your data - so it's worth to have a sanity check for this edge-case).
I hope this example is helpful.