I have the following string:
$result = "category,date,product,cost
car,10/10/2005,toyota,3000
bike,12/12/2007,yamaha,1000";
I need to arrange the string into the best format to insert into a database (needs to be efficient as there might be lots of data also first row will not be needed).
The string is on new lines, so I can:
$n = explode("\n", $result);
which puts it in this format:
[0] => category,date,product,cost
[1] => car,10/10/2005,toyota,3000
[2] => bike,12/12/2007,yamaha,1000
now how would i go about formatting this to insert in a db, once again the solution needs to be efficient (for performance), also I would like the first row to not be inserted and finally on the insert I wish for the date to not be inserted.
Hope it makes sense, thanks for the help in advance.
--------edit---------------
The table (item) is structured with the following fields:
category, product, cost
thanks
If you are going to have a similar string with the same columns every time, then you could do something like this:
$n = explode("\n", $result);
// skip the first row since it's just column names
for ($i = 1; $i <= count($n); $i++)
{
$cat = $n[$i][[0]; // category
$date = $n[$i][[1]; // date
$prod = $n[$i][[2]; // product
$cost = $n[$i][[3]; // cost
// and you can put it into a query like so
$sql = "insert into myTable ('category', 'date', 'product', 'cost') values ('$cat', '$date', '$prod', '$cost')";
}
Does this answer your question?
Improving from previously answered post:
$n = explode("\n", $result);
// skip the first row since it's just column names
$sql = "insert into myTable ('category', 'date', 'product', 'cost') values ";
for ($i = 1; $i <= count($n); $i++)
{
$cat = $n[$i][[0]; // category
$date = $n[$i][[1]; // date
$prod = $n[$i][[2]; // product
$cost = $n[$i][[3]; // cost
$sql .= "('$cat', '$date', '$prod', '$cost'),";
}
$sql = trim($sql, ",");
if(count($n) != 0)
mysql_query($sql)
This gives you just one single query which will insert multiple rows.
Related
I'm inserting/updating a mysql table from json URL, but some of the values in $points contain a dash and I want to amend to a zero (or null would be fine) when updating so that I can have the column set to INT.
I know how to simply replace all - to 0 in mysql but stuck when it comes to trying to do as part of update from a json URL. My code is:
for ($i = 0; $i < $length; $i++) {
$sus = $array['formguide'][$i]['SUSPENSION'];
$wpoints = $array['formguide'][$i]['WEEKPOINTS'];
$tcode = $array['formguide'][$i]['TEAMCODE'];
$val = $array['formguide'][$i]['VALUE'];
$points = $array['formguide'][$i]['POINTS'];
$pname = $array['formguide'][$i]['PLAYERNAME'];
$tname = $array['formguide'][$i]['TEAMNAME'];
$sixwpoints = $array['formguide'][$i]['SIXWEEKPOINTS'];
$injury = $array['formguide'][$i]['INJURY'];
$playerid = $array['formguide'][$i]['PLAYERID'];
$pos = $array['formguide'][$i]['POS'];
//insert or update values into mysql
$sql = "INSERT INTO formguide (suspension, weekpoints, teamcode,
value, points, playername, teamname,
sixweekpoints, injury, playerid, pos)
VALUES ('$sus', '$wpoints', '$tcode','$val','$points','$pname','$tname',
'$sixwpoints','$injury','$playerid','$pos')
ON DUPLICATE KEY UPDATE suspension='$sus', weekpoints='$wpoints',
teamcode='$tcode', value='$val', points='$points', playername='$pname',
teamname='$tname', sixweekpoints='$sixwpoints', injury='$injury',
playerid='$playerid', pos='$pos'";
I tried REPLACE() on points within the UPDATE() section i.e.:
...points=REPLACE('$points','-','0'), playername='$pname', ...
however that didn't work.
Any ideas how I can do this, or is there an alternative to have the points column as INT and accept a dash?
Thanks
REPLACE is for modifying values coming from SQL, whereas you want to modify a value coming from PHP.
Use str_replace instead.
$points = str_replace('-', '0', $points);
First, you need to escape or cast all values to avoid SQL injection vulnerabilities. Since you did not post your table schema, I'm guessing the types. mysqli_real_escape_string() is for strings, (int) and (float) for numbers of the respective type.
Doing that, your original problem is solved automatically, as PHP returns 0, if you cast '-' to integer.
Second, your query is a usecase for a proper use of REPLACE, which works exactly like INSERT, except that if an old row in the table has the same value as a new row for a PRIMARY KEY or a UNIQUE index, the old row is deleted before the new row is inserted.
for ($i = 0; $i < $length; $i++)
{
$sus = mysqli_real_escape_string($link, $array['formguide'][$i]['SUSPENSION']);
$wpoints = (int) $array['formguide'][$i]['WEEKPOINTS'];
$tcode = mysqli_real_escape_string($link, $array['formguide'][$i]['TEAMCODE']);
$val = (float) $array['formguide'][$i]['VALUE'];
$points = (int) $array['formguide'][$i]['POINTS'];
$pname = mysqli_real_escape_string($link, $array['formguide'][$i]['PLAYERNAME']);
$tname = mysqli_real_escape_string($link, $array['formguide'][$i]['TEAMNAME']);
$sixwpoints = (int) $array['formguide'][$i]['SIXWEEKPOINTS'];
$injury = mysqli_real_escape_string($link, $array['formguide'][$i]['INJURY']);
$playerid = (int) $array['formguide'][$i]['PLAYERID'];
$pos = (int) $array['formguide'][$i]['POS'];
// Insert or update values into mysql
$sql = "REPLACE INTO formguide SET
suspension='$sus',
weekpoints=$wpoints,
teamcode='$tcode',
value=$val,
points=$points,
playername='$pname',
teamname='$tname',
sixweekpoints=$sixwpoints,
injury='$injury',
playerid=$playerid,
pos=$pos
";
// ...
}
I've got a script that I needed to change since the data which is going to be inserted into the db got too big to do it at once. So I created a loop, that splits up the array in blocks of 6000 rows and then inserts it.
I don't know exactly if the data is to big for the server to process at once or if it's too big to upload, but atm I got both steps split up in these 6000s blocks.
Code:
for ($j = 0; $j <= ceil($alength / 6000); $j++){
$array = array_slice($arraysource, $j * 6000, 5999);
$sql = "INSERT INTO Ranking (rank, name, score, kd, wins, kills, deaths, shots, time, spree) VALUES ";
foreach($array as $a=>$value){
//transforming code for array
$ra = $array[$a][0];
$na = str_replace(",", ",", $array[$a][1]);
$na = str_replace("\", "\\\\", $na);
$na = str_replace("'", "\'", $na);
$sc = $array[$a][2];
$kd = $array[$a][3];
$wi = $array[$a][4];
$ki = $array[$a][5];
$de = $array[$a][6];
$sh = $array[$a][7];
$ti = $array[$a][8];
$sp = $array[$a][9];
$sql .= "('$ra',' $na ','$sc','$kd','$wi','$ki','$de','$sh','$ti','$sp'),";
}
$sql = substr($sql, 0, -1);
$conn->query($sql);
}
$conn->close();
Right now it only inserts the first 5999 rows, but not more as if it only executed the loop once. No error messages..
Don't know if this'll necessarily help, but what about using array_chunk, array_walk, and checking error codes (if any)? Something like this:
function create_query(&$value, $key) {
//returns query statements; destructive though.
$value[1] = str_replace(",", ",", $value[1]);
$value[1] = str_replace("\", "\\\\", $value[1]);
$value[1] = str_replace("'", "\'", $value[1]);
$queryvalues = implode("','",$value);
$value = "INSERT INTO Ranking (rank, name, score, kd, wins, kills, deaths, shots, time, spree) VALUES ('".$queryvalues."');";
}
$array = array_chunk($arraysource, 6000);
foreach($array as $key=>$value){
array_walk($value,'create_query');
if (!$conn->query($value)) {
printf("Errorcode: %d\n", $conn->errno);
}
}
$conn->close();
Secondly, have you considered using mysqli::multi_query? It'll do more queries at once, but you'll have to check the max allowed packet size (max_allowed_packet).
Another tip would be to check out the response from the query, which your code doesn't include.
Thanks for the tips but I figured it out. Didn't think about this ^^
it was the first line after the for loop that i didnt include in my question:
array_unshift($array[$a], $a + 1);
this adds an additional value infront of each user, the "rank". But the numbers would repeat after one loop finishes and it can't import users with the same rank.
now it works:
array_unshift($array[$a], $a + 1 + $j * 5999);
I am trying the use refine tools for a search on my website. The bit i'm stuck with is search by start letter. For example i could use a wildcard '%X%' but his would return anything that contained the letter 'x'.
I read on few sites that SUBSTRING can be used in mysql queries
http://dev.mysql.com/
http://www.kirupa.com/
https://stackoverflow.com/questions/6302027
This is what I have so far but returns nothing. There is data in the database that should return with the query.
public function refineUsersFollowers($user_id,$q){
if($this->databaseConnection()){
// get the users followers
$state = array(1,2);
$stmt = $this->db_connection->prepare("SELECT * FROM friends WHERE id_2 = :1 AND Friend_Request_State = :2 OR id_2 = :3 AND Friend_Request_State = :4");
$stmt->bindParam(':1', $user_id);
$stmt->bindParam(':2', $state[0]);
$stmt->bindParam(':3', $user_id);
$stmt->bindParam(':4', $state[1]);
$stmt->execute();
// format the SQL OR statements
$sql = '';
$ids = [];
while($rows = $stmt->fetch(\PDO::FETCH_ASSOC)){
array_push($ids,$rows['id_1']);
}
for($x = 0; $x < count($ids); $x++){
if(count($ids) == 1){
//if there is one result
$sql.= ' user_id = :'.$x." AND SUBSTRING('first_name',0,1) = :".$x.$x;
}else if($x == (count($ids) - 1)){
// last entry
$sql.= ' user_id = :'.$x." AND SUBSTRING('first_name',0,1) = :".$x.$x;
}else{
//continue loop
$sql.= ' user_id = :'.$x." AND SUBSTRING('first_name',0,1) = :".$x.$x." OR";
}
}
$stmt = $this->db_connection->prepare("SELECT * FROM account WHERE ".$sql);
for($x = 0; $x < count($ids); $x++){
$stmt->bindParam(':'.$x,$ids[$x]);
$insert = $x.$x.'';
$stmt->bindParam(':'.$insert,$q);
}
$stmt->execute();
$results = $stmt->fetch(\PDO::FETCH_ASSOC);
print_r($results);
// check for followers that start with letter
}
}
The first part of the function is fine, this gets an array of id's which is then placed together as an SQL string. Is the SQL not returning results because SUBSTRING is not supported in this way?
If so is there a way of producing a query like this or would it be easier to pull every result from the database then check them in a different function?
You have two issues with this expression:
SUBSTRING('first_name', 0, 1) = :".$x.$x;
First, substr() in SQL (in general) starts counting with 1 and not 0. So, the first argument should be 1.
Second, you have the first argument in single quotes. So, at best, this would return the letter 'f'. Here is a simple rule: Only use single quotes for string and date constants. Never use single quotes to refer to column names.
There are several way to write what you want. Here are three:
SUBSTRING(first_name, 1, 1) = $x
LEFT(first_name, 1) = $x
first_name like '$x%'
You query can be greatly simplified with the LIKE operator. This:
"AND SUBSTRING('first_name',0,1) = :".$x.$x;
can become this:
"AND first_name LIKE '".$x.$x."%'";
I'm not sure what the $x.$x is for, so I just left it in for illustrative purposes.
I'm working with a company that every day gives me a CSV file with about 25.000 rows.
The difference between CSV of one day and that of the day before is that in the newest one, some rows (many less than the total) are deleted and others are added. So the two files has about 24900 rows in common.
I have to store all the rows in the time.. so every day I have to update my table in DB with the current CSV.
I think about:
<?php
$fh = fopen($actual_csv, 'r');
$contents = fread($fh, filesize($actual_csv));
fclose($fh);
$fileLines = explode("\n", $contents);
for ($i = 1; $i < count($fileLines) - 1; $i++) {
$fieldList = explode(';', $fileLines[$i]);
//$fieldList[0] is my unique id
if(mysql_num_rows(mysql_query("SELECT * FROM table_where_i_store_all WHERE id='$fieldList[0]'"))<=0){
mysql_query("INSERT INTO table_where_i_store_all (column names..) VALUES ('$fieldList[0],........')"); // there are many column so i don't write it..
}
}
?>
I think this is not very powerful and fast. is there any better way? thanks!!!
create unique index on the id fields (maybe you've done in already) and use INSERT IGNORE or INSERT ... ON DUPLICATE KEY UPDATE
ALTER TABLE table_where_i_store_all ADD UNIQUE( id );
$fileLines = explode("\n", $contents);
$linemax = count( $fileLines )-1;
if( $linemax < 1 ) // empty file?
return;
$SQL = "INSERT IGNORE INTO table_where_i_store_all (column_names) VALUES ";
for ($i = 1; $i < $linemax; $i++) {
$fieldList = explode(';', $fileLines[$i]);
//$fieldList[0] is my unique id
$SQL .= "('$fieldList[0],........'),";
}
$SQL = substr( $SQL, 0, strlen($SQL)-1); // remove extra comma from end
$res = mysql_query($SQL);
I have a question that pertains to gathering events from tables in a calendar program where they are separated into "events" or "repeated events". I can get all individual events perfectly well now (thanks to Chris on this site), but if they are repeating events I have to calculate it from what is given in this particular db. If I change the types or the data in the db, it will probably trash the calendar so I have to use what I have.
The variables I have sorted out so far are:
$quid2 = The IDs for today's events that are classified as repeating events (needed earlier)
$quname = The repeated event names
$qucls = The date UNIX time for the last sent reminder of events dated today
$qutype = One of these words - daily, weekly, monthly or yearly
$qudesc = A description of the event
These variables all have the same number of items and are ordered correctly between each other. (I Hope)
Below is the logic I am trying to accomplish. It are most assuredly not proper syntax but I think it is understandable; I need to figure out what the syntax and form is. I am utterly and completely new at this... so please be gentle...
It needs to be put in an array (I think)
$arr1 = some type of array($quname, $qucls, $qutype, $qudesc)
update the array...
IF $qutype($row2) = "daily", then + 1440 to it's $qucls($row[1])
IF $qutype($row2) = "weekly", then + 10080 to it's $qucls($row[1])
IF $qutype($row2) = "monthly, then + 1 month to it's $qucls($row[1])
IF $qutype($row2) = "yearly", then + 1 year to it's $qucls($row[1])
Then final product...
while ($row = mysql_fetch_array($arr1, MYSQL_NUM)) {
$UxTime = $row[1];
date_default_timezone_set('America/Denver');
$Time = date("H:i", $UxTime);
$qufinal = sprintf("Event: %s \nTime: %s \nDesc: %s \n\n", $row[0], $Time, $row[3);
}
...
This is a big learning project for me. I don't know enough PHP and mysql to make this work but I know just enough to get me in trouble. Thanks!
EDIT: adding the queries from which I made these variables:
$today = date("Ymd");
mysql_select_db($dbname);
$query1 = "SELECT cal_id, cal_name, cal_date, cal_time, cal_type, cal_description FROM webcal_entry WHERE cal_type = "M" AND cal_date != " . $today;
$wequ1 = mysql_query($query1)
while ($row = mysql_fetch_array($wequ1, MYSQL_NUM)) {
$quid1 = $row[0], $quname = $row[1], $qutime = $row[2], $qudesc = $row[3];
}
$query2 = "SELECT cal_id, cal_type, cal_ByDay FROM webcal_entry_repeats WHERE cal_id = " . $quid1;
$wer1 = mysql_query($query2)
while ($row = mysql_fetch_array($wer1, MYSQL_NUM)) {
$quid2 = $row[0] $qutype = $row[1], $qubdy = $row[2];
}
$query3 = "SELECT cal_id, cal_last_sent FROM webcal_reminders WHERE cal_id = " . $quid2;
$wer2 = mysql_query($query3)
while ($row = mysql_fetch_array($wer2, MYSQL_NUM)) {
$qucls = $row[1];
}
The syntax for arrays is as follows:
$arrayName = array($quname, $qucls, $qutype, $qudesc);
Then you can access the values by their index on the array variable:
$arrayName[0] == $quname
$arrayName[1] == $qucls
...
You can also define it as associative array:
$arrayName = array(
"quname" => $quname,
"qucls" => $qucls,
"qutype" => $qutype,
"qudesc" => $qudesc
);
Using this syntax you can access the elements by their name:
$arrayName["quname"] == $quname
$arrayName["qucls"] == $qucls
...
More reading on this: Arrays
However, you don't really need an array for what you plan to do here. Arrays are very useful if you want to store data that is structurally equal. This applies e.g. to rows in a database: They always have the same number of entries, and the columns are of the same type.
If you have just one dataset at that point of code (one event in this case), then you need no array. Of course you have several events, but as they are processed in a loop (I assume) you handle only one event at a time, and then head to the next.
So, you want to modify a variable depending on the value $qutype. To do that, you can use a switch statement:
$dateObj = date_create("#$qucls");
switch($qutype) {
case "daily":
date_add($dateObj, date_interval_create_from_date_string("1 day"));
break;
case "weekly":
date_add($dateObj, date_interval_create_from_date_string("1 week"));
break;
case "monthly":
date_add($dateObj, date_interval_create_from_date_string("1 month"));
break;
case "yearly":
date_add($dateObj, date_interval_create_from_date_string("1 year"));
break;
}
$qucls = date_format($dateObj, "U");
I don't add the number of seconds, because that would work for days and weeks- but not for months and years, as they don't have a fixed number of seconds.
If you have questions about the functions I used above you can look up their documentation on php.net.
In the code you show you must not use mysql_fetch_array.
That function is only meant for result rows you got from a call to mysql_query, but not for normal arrays.
You don't need the while loop either. All you have to do is formatting $qucls to a readable format and produce the final string:
date_default_timezone_set('America/Denver');
$Time = date("H:i", $qucls);
$qufinal = sprintf("Event: %s \nTime: %s \nDesc: %s \n\n", $quname, $Time, $qudesc);
Edit:
Like discussed in the comments here is the revised and commented code you edited in:
$today = date("Ymd");
mysql_select_db($dbname);
// You need to use single quotes at the 'M'. Using double quotes will
// end the string and thus leading to incorrect syntax
$query1 = "SELECT cal_id, cal_name, cal_date, cal_time, cal_type, cal_description FROM webcal_entry WHERE cal_type = 'M' AND cal_date != " . $today;
$wequ1 = mysql_query($query1);
// This is a counter variable which is incremented in the loop
$i = 0;
// This is the outer while loop used to gather and store the events
while ($row = mysql_fetch_array($wequ1, MYSQL_NUM)) {
// Store the results in arrays
// Statements must be seperated by a ;
$quid1[$i] = $row[0];
$quname[$i] = $row[1];
$qutime[$i] = $row[2];
$qudesc[$i] = $row[3];
$query2 = "SELECT cal_id, cal_type, cal_ByDay FROM webcal_entry_repeats WHERE cal_id = " . $quid1[$i];
$wer1 = mysql_query($query2);
// Assuming that IDs are unique this query can only return one entry. Therefore no while is
// needed, but an if statement tests if the ID actually matched a result
if ($row = mysql_fetch_array($wer1, MYSQL_NUM)) {
//$quid2[$i] = $row[0]; <- the query above ensures that $quid1[$i] == $quid2[$i]. No need to store it again
$qutype[$i] = $row[1];
$qubdy[$i] = $row[2];
}
$query3 = "SELECT cal_id, cal_last_sent FROM webcal_reminders WHERE cal_id = " . $quid1[$i];
$wer2 = mysql_query($query3);
// Same as above; If the IDs are unique then you need no loop
if ($row = mysql_fetch_array($wer2, MYSQL_NUM)) {
// The $i++ used here is the short form. As this is the last time $i is
// used in the loop it needs to be increased before the next round. You can do
// this like this or in an extra statement. This way it's evaluated and then increased
$qucls[$i++] = $row[1];
}
// End outer while loop
}
// Now go through the results. $i holds the number of entries in the arrays + 1
// Secondary counter variable and for-loop
for ($j = 0; $j < $i; $j++) {
// Adding the dates to $qucls, formatting the string, ...
// Access them like above: $qucls[$j]
// Do not increase $j manually though - the for loop does that for you already
}
Please note that this code is untested. It's syntactically correct though.
On a side note: You are currently using three different database queries to gather the data.
You can easily merge them into a single query using SQL JOINs. If you want somebody to show you how to do that, you can show them in a seperate question and ask for them to be joined into one.