How to insert large amount of data in MySQL DB using JSON - php

I am using 7 JSON File, and I am trying to insert 95 coloums + 50-100 rows at a time.
This is how my code looks like: I removed some 6 JSON files and other rows&colums. This is a short version of my code
<?php
$requestsDone = 2;
$maxRequests = 100;
$currentnumber = 1;
while ($requestsDone <= $maxRequests) {
$response = file_get_contents("https://api.themoviedb.org/3/movie/".$requestsDone."?api_key=522cexxxxxxxxxxxxx9ax34a");
if ($response != FALSE) {
$response = json_decode($response, true);
}
if ( ($response["title"]) != "" )
{
$requestsDone++;
include 'admin/connectdb.php';
try {
// prepare sql and bind parameters
$stmt = $conn->prepare("INSERT INTO TABLE (title,image,money,tag1,tag2)
VALUES (:title, :image, :money,:tag1,:tag2)");
$stmt->bindParam(':title', $title);
$stmt->bindParam(':image', $image);
$stmt->bindParam(':money', $money);
$stmt->bindParam(':tag1', $tag1);
$stmt->bindParam(':tag1', $tag2);
// insert a row
$movie_title = "".$response["title"]."";
if ( $response["image"] != "" )
$image = "".$response["image"]."";
if ( $response["money"] != "" )
$money = "".$response["money"]."";
if ( ($response["tag"][0]["name"]) != "" )
$tag1 = "".$response["tag"][0]["name"]."";
if ( ($response["tag"][1]["name"]) != "" )
$tag2 = "".$response["tag"][1]["name"]."";
$stmt->execute();
echo "New records created successfully ".$currentnumber++." <br/>";
}
catch(PDOException $e)
{
echo "Error: " . $e->getMessage();
}
$conn = null;
}
}
?>
I am getting 2 type of errors: Sometime 1st error, sometime 2nd error.
1st Error: Cloudflare time-out: Error 504 Screenshot
2nd Error: Error: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
Please let me know, if you need more information.
Edit:
This is how one of the JSON file looks like
{"adult":false,"backdrop_path":"/87hTDiay2N2qWyX4Ds7ybXi9h8I.jpg","belongs_to_collection":null,"budget":63000000,"genres":[{"id":18,"name":"Drama"}],"homepage":"http://www.foxmovies.com/movies/fight-club","id":550,"imdb_id":"tt0137523","original_language":"en","original_title":"Fight Club","overview":"A ticking-time-bomb insomniac and a slippery soap salesman channel primal male aggression into a shocking new form of therapy. Their concept catches on, with underground \"fight clubs\" forming in every town, until an eccentric gets in the way and ignites an out-of-control spiral toward oblivion.","popularity":9.765478999999999,"poster_path":"/adw6Lq9FiC9zjYEpOqfq03ituwp.jpg","production_companies":[{"name":"Regency Enterprises","id":508},{"name":"Fox 2000 Pictures","id":711},{"name":"Taurus Film","id":20555},{"name":"Linson Films","id":54050},{"name":"Atman Entertainment","id":54051},{"name":"Knickerbocker Films","id":54052}],"production_countries":[{"iso_3166_1":"DE","name":"Germany"},{"iso_3166_1":"US","name":"United States of America"}],"release_date":"1999-10-15","revenue":100853753,"runtime":139,"spoken_languages":[{"iso_639_1":"en","name":"English"}],"status":"Released","tagline":"How much can you know about yourself if you've never been in a fight?","title":"Fight Club","video":false,"vote_average":8.199999999999999,"vote_count":8047}

Related

PHP Improve performance to execute multiple queries while reading a file with thousand lines

I'm trying to build a script where I need to read a txt file and execute some process with the lines on the file. For example, I need to check if the ID exists, if the information has updated, if yes, then update the current table, if no, then insert a new row on another temporary table to be manually checked later.
These files may contain more than 20,30 thousand lines.
When I just read the file and print some dummie content from the lines, it takes up to 40-50ms. However, when I need to connect to the database to do all those verifications, it stops before the end due to the timeout.
This is what I'm doing so far:
$handle = fopen($path, "r") or die("Couldn't get handle");
if ($handle) {
while (!feof($handle)) {
$buffer = fgets($handle, 4096);
$segment = explode('|', $buffer);
if ( strlen($segment[0]) > 6 ) {
$param = [':code' => intval($segment[0])];
$codeObj = Sql::exec("SELECT value FROM product WHERE code = :code", $param);
if ( !$codeObj ) {
$param = [
':code' => $segment[0],
':name' => $segment[1],
':value' => $segment[2],
];
Sql::exec("INSERT INTO product_tmp (code, name, value) VALUES (:code, :name, :value)", $param);
} else {
if ( $codeObj->value !== $segment[2] ) {
$param = [
':code' => $segment[0],
':value' => $segment[2],
];
Sql::exec("UPDATE product SET value = :value WHERE code = :code", $param);
}
}
}
}
fclose($handle);
}
And this is my Sql Class to connect with PDO and execute the query:
public static function exec($sql, $param = null) {
try {
$conn = new PDO('mysql:charset=utf8mb4;host= '....'); // I've just deleted the information to connect to the database (password, user, etc.)
$q = $conn->prepare($sql);
if ( isset($param) ) {
foreach ($param as $key => $value) {
$$key = $value;
$q->bindParam($key, $$key);
}
}
$q->execute();
$response = $q->fetchAll();
if ( count($response) ) return $response;
return false;
} catch(PDOException $e) {
return 'ERROR: ' . $e->getMessage();
}
}
As you can see, each query I do through Sql::exec(), is openning a new connection. I don't know if this may be the cause of such a delay on the process, because when I don't do any Sql query, the script run within ms.
Or what other part of the code may be causing this problem?
First of all, make your function like this,
to avoid multiple connects and also o get rid of useless code.
public static function getPDO() {
if (!static::$conn) {
static::$conn = new PDO('mysql:charset=utf8mb4;host= ....');
}
return static::$conn;
}
public static function exec($sql, $param = null) {
$q = static::getPDO()->prepare($sql);
$q->execute($param);
return $q;
}
then create unique index for the code field
then use a single INSERT ... ON DUPLICATE KEY UPDATE query instead of your thrree queries
you may also want to wrap your inserts in a transaction, it may speed up the inserts up to 70 times.

PHP+Apache2+Ubuntu Server: How to get all threads to work in parallel?

I usually work with web hosting companies but I decided to start learning working with servers to expand my knowledge.
I'll better give a real example to explain my question the best:
I have a web application that gathers data from a slow API that returns JSON data of products.
I have a function running every 1AM running a lot of queries on "id"s in my database.
Crontab:
0 1 * * * cd /var/www/html/tools; php index.php aso Cli_kas kas_alert
So this creates a process for the app (please correct me here if I'm wrong) and each process creates threads, and just to be more accurate, they are multi-threads since they do more than one thing: like pulling data from the DB to get the right variables and string them to the API queries, getting the data from the API, organizing it, searching the relevant data, and then inserting new data to the database.
The main PHP functions:
// MAIN: Cron Job Function
public function kas_alert() {
// 0. Deletes all the saved data from the `data` table 1 month+ ago.
// $this->kas_model->clean_old_rows();
// 1. Get 'prod' table
$data['table'] = $this->kas_model->prod_table();
// 2. Go through each row -
foreach ( $data['table'] as $row ) {
// 2.2. Gets all vars from the first query.
$last_row_query = $this->kas_model->get_last_row_of_tag($row->tag_id);
$last_row = $last_row_query[0];
$l_aaa_id = $last_row->prod_aaa_id;
$l_and_id = $last_row->prod_bbb_id;
$l_r_aaa = $last_row->dat_data1_aaa;
$l_r_and = $last_row->dat_data1_bbb;
$l_t_aaa = $last_row->dat_data2_aaa;
$l_t_and = $last_row->dat_data2_bbb;
$tagword = $last_row->tag_word;
$tag_id = $last_row->tag_id;
$country = $last_row->kay_country;
$email = $last_row->u_email;
$prod_name = $last_row->prod_name;
// For the Weekly report:
$prod_id = $last_row->prod_id;
$today = date('Y-m-d');
// 2.3. Run the tagword query again for today on each one of the tags and insert to DB.
if ( ($l_aaa_id != 0) || ( !empty($l_aaa_id) ) ) {
$aaa_data_today = $this->get_data1_aaa_by_id_and_kw($l_aaa_id, $tagword, $country);
} else{
$aaa_data_today['data1'] = 0;
$aaa_data_today['data2'] = 0;
$aaa_data_today['data3'] = 0;
}
if ( ($l_and_id != 0) || ( !empty($l_and_id) ) ) {
$bbb_data_today = $this->get_data1_bbb_by_id_and_kw($l_and_id, $tagword, $country);
} else {
$bbb_data_today['data1'] = 0;
$bbb_data_today['data2'] = 0;
$bbb_data_today['data3'] = 0;
}
// 2.4. Insert the new variables to the "data" table.
if ($this->kas_model->insert_new_tag_to_db( $tag_id, $aaa_data_today['data1'], $bbb_data_today['data1'], $aaa_data_today['data2'], $bbb_data_today['data2'], $aaa_data_today['data3'], $bbb_data_today['data3']) ){
}
// Kas Alert Outputs ($SEND is echoed in it's original function)
echo "<h1>prod Name: $prod_id</h1>";
echo "<h2>tag id: $tag_id</h2>";
var_dump($aaa_data_today);
echo "aaa old: ";
echo $l_r_aaa;
echo "<br> aaa new: ";
echo $aaa_data_today['data1'];
var_dump($bbb_data_today);
echo "<br> bbb old: ";
echo $l_r_and;
echo "<br> bbb new: ";
echo $bbb_data_today['data1'];
// 2.5. Check if there is a need to send something
$send = $this->check_if_send($l_aaa_id, $l_and_id, $l_r_aaa, $aaa_data_today['data1'], $l_r_and, $bbb_data_today['data1']);
// 2.6. If there is a trigger, send the email!
if ($send) {
$this->send_mail($l_aaa_id, $l_and_id, $aaa_data_today['data1'], $bbb_data_today['data1'], $l_r_aaa, $l_r_and, $tagword, $email, $prod_name);
}
}
}
For #Raptor, this is the function that get's the API data:
// aaa tag Query
// Gets aaa prod dataing by ID.
public function get_data_aaa_by_id_and_tg($id, $tag, $query_country){
$tag_for_url = rawurlencode($tag);
$found = FALSE;
$i = 0;
$data = array();
// Create a stream for Json. That's how the code knows what to expect to get.
$context_opts = array(
'http' => array(
'method' => "GET",
'header' => "Accepts: application/json\r\n"
));
$context = stream_context_create($context_opts);
while ($found == FALSE) {
// aaa Query
$json_query_aaa = "https://api.example.com:443/aaa/ajax/research_tag?app_id=$id&term=$tag_for_url&page_index=$i&country=$query_country&auth_token=666";
// Get the Json
$json_query_aaa = file_get_contents($json_query_aaa, false, $context);
// Turn Json to a PHP array
$json_query_aaa = json_decode($json_query_aaa, true);
// Get the data2
$data2 = $json_query_aaa['tag']['data2'];
if (is_null($data2)){ $data2 = 0; }
// Get data3
$data3 = $json_query_aaa['tag']['phone_prod']['data3'];
if (is_null($data3)){ $data3 = 0; }
// Finally, the main prod array.
$json_query_aaa = $json_query_aaa['tag']['phone_prod']['app_list'];
if ( count($json_query_aaa) > 2 ) {
for ( $j=0; $j<count($json_query_aaa); $j++ ) {
if ( $json_query_aaa[$j]['id'] == $id ) {
$found = TRUE;
$data = $json_query_aaa[$j]['data'] + 1;
break;
}
if ($found == TRUE){
break;
}
}
$i++;
} else {
$data = 0;
break;
}
}
$data['data1'] = $data;
$data['data2'] = $data2;
$data['data3'] = $data3;
return $data;
}
All threads are stacked one after an other, and when one thread is done, only then - the second thread can proceed, ect'.
And in technical view on this, all threads wait in the RAM until the one before them is done working "inside" the CPU. (correct me if I'm wrong again :] )
This doesn't even "tickle" the servers RAM or CPU when looking at it in the process manager (I use "htop"). RAM is at 400M/4.25G and CPU at ONLY 0.7%-1.3%.
Making me feel this isn't the best I can get from my current server, and getting slow results from my web app.
How do I get things done in a way that all threads work in parallel, but not to a point that my app crashes due to lacks of CPU or RAM?

Firebird ibase_trans() very slow PHP

When I'm use ibase_trans() the script start infinite loop and don't respond.
I'am requesting nusoap webservice.
If I just print the nusoap data the script execution time is 3 seconds, after I uncomment the line that start transaction it starts infinite loop.
$dbh1 = ibase_connect(DB_HOST1, DB_USER1, DB_PASS1) or die('die message');
$tr1 = ibase_trans($dbh1);
After the nusoap data loop I have those lines :
$query3 = ibase_query($tr1,"INSERT INTO EXECUTAR ( FLAG ) VALUES ( 2 )") or die('teste');
Here start the problem.
My php version : 5.6
Xampp version: 3.2.1
Firebird: 2.5
Full code :
if ($client->fault) {
echo "<h2>Fault</h2><pre>";
print_r($result);
echo "</pre>";
} else {
$error = $client->getError();
if ($error) {
echo "<h2>Error.</h2><pre>" . $error . "</pre>";
} else {
$dbh1 = ibase_connect(DB_HOST1, DB_USER1, DB_PASS1) or die('die message');
$tr1 = ibase_trans($dbh1);
$TotalProdutos = 0;
foreach($result['ConsultaEstoqueProdutosResult']['diffgram']['NewDataSet']['Estoques'] as $produto){
$TotalProdutos++;
$query1 = ibase_query($tr1,"INSERT INTO ATUALIZA_PRE ( CODIGO,
ESTOQUE,
ATIVO,
DISPONIVEL,
ESTOQUEDISPONIVEL,
PRECOSEMST,
PRECOREVENDA )
VALUES (
'{$produto['CODIGO']}',
'{$produto['ESTOQUE']}',
{$produto['ATIVO']},
{$produto['DISPONIVEL']},
{$produto['ESTOQUEDISPONIVEL']},
{$produto['PRECOSEMST']},
'".$produto['PRECOREVENDA']."')")or die('teste1');
}
$dbh1 = ibase_connect(DB_HOST1, DB_USER1, DB_PASS1) or die('die message');
$tr1 = ibase_trans($dbh1);
$query3 = ibase_query($tr1,"INSERT INTO EXECUTAR ( FLAG ) VALUES ( 2 )") or die('teste');
ibase_commit($tr1);
ibase_close($dbh1);
}
}
Can someone help me please?

MySQL INSERT INTO from PHP://INPUT multiple rows

I have data (exact) from this HTTP POST:
rowno=1.00000000&date_line=2014-10-07&name=Dan%20Volunteer&affiliation=Enterprise&checkno=1701&amount=20025.00000000&total=20250.00000000&notes=&date_deposit=&rowno=2.00000000&date_line=2014-10-07&name=Harper%20Lee&affiliation=Enterprise%20B&checkno=1702&amount=225
then this code to process
<?php
file_get_contents("php://input");
$db = null;
if (isset($_SERVER['SERVER_SOFTWARE']) &&
strpos($_SERVER['SERVER_SOFTWARE'],'Google App Engine') !== false) {
// Connect from App Engine.
try{
$db = new pdo('mysql:unix_socket=/cloudsql/wonder:bread;dbname=loaf', 'root', '');
}catch(PDOException $ex){
die(json_encode(
array('outcome' => false, 'message' => 'Unable to connect.')
)
);
}
};
try {
if (array_key_exists('name', $_POST)) {
$stmt = $db->prepare('INSERT INTO entries (name, affiliation) VALUES (:name, :affiliation)');
$stmt->execute(array(':name' => htmlspecialchars($_POST['name']), ':affiliation' => htmlspecialchars($_POST['affiliation'])));
$affected_rows = $stmt->rowCount();
// Log $affected_rows.
}
} catch (PDOException $ex) {
// Log error.
}
$db = null;
?>
<?php
header("Content-type: application/vnd.fdf");
// read and store the data however you want
// reply with some FDF data
echo <<<RESPONSE
%FDF-1.2
1 0 obj
<< /FDF <<
/Status (Wham bam! File sent.)
>>
>>
endobj
trailer
<< /Root 1 0 R >>
%%EOF
RESPONSE;
?>
This http post has two records (row/recount count always varies), but only data from the last row is being inserted. Need all rows.
I'm going to stab at this one....I think what is happening is that you are just processing the return post as is, so the first rowno is being skipped over (rather the second rowno is overwriting the first). If you receive that post back as a string, you need to split it by preg_match() or explode() so that you can loop over it with your try.
Try this class on your string. This class will split the string into arrays based on rows. Then you need to take the resulting array $insert then process each array in your an sql loop...does that make sense?
class ProcessPost
{
public static function Split($value = '',$splitVal = 'rowno=')
{
if(!empty($value)) {
// Explode by row values
$rows = explode($splitVal,$value);
$rows = array_filter($rows);
if(is_array($rows) && !empty($rows)) {
foreach($rows as $_row => $querystring) {
parse_str($splitVal.$querystring,$_array[]);
}
foreach($_array as $row_key => $row_val) {
if(empty($row_val))
unset($_array[$row_key]);
}
return $_array;
}
}
}
}
$test = 'rowno=1.00000000&date_line=2014-10-07&name=Dan%20Volunteer&affiliation=Enterprise&checkno=1701&amount=20025.00000000&total=20250.00000000&notes=&date_deposit=&rowno=2.00000000&date_line=2014-10-07&name=Harper%20Lee&affiliation=Enterprise%20B&checkno=1702&amount=225';
$insert = ProcessPost::Split($test);

Undefined offset: 0 in first PHP app

I'm currently building my first PHP application. I want to read in bus times from a csv file and give back ti users the next bus from their residence to our university. It is my first try with PHP, so be gentle:
<?php
// configuration
require("../includes/config.php");
if (!empty($_SESSION["id"]))
{
//lookup database entries for house
$users = query("SELECT * FROM users WHERE id = ?", $_SESSION["id"]);
if ($users === false)
{
apologize("Sorry, there was an error");
}
//lookup database entries for house
$residences = query("SELECT * FROM residences WHERE id = ?", $users[0]["residence"]);
if ($residences === false)
{
apologize("Sorry, there was an error");
}
//if user specified a residence in his profile
if($residences[0]["id"] != 0)
{
$times = array();
//if there is no bus today, in this case sat and sun
if(date( "w", $timestamp) == 0 || date( "w", $timestamp) == 6)
{
$times[0] = "There is no bus today";
}
//load the busplan for his residence
else
{
//open file and load in array if time is higher than date("His");
$timesList = file_get_contents($users[0]["residence"] . ".csv");
$nextbuses = explode(',', $timesList);
$hoursMins = date("Gi");
$num = 0;
for($i = 0; $i < count($nextbuses); $i++)
{
if($hoursMins < $nextbuses[$i])
{
$times[$num] = $nextbuses[$i];
$num++;
}
}
}
render("shuttle_show.php", ["title" => "Next Shuttle from your residence.", "times" => $times]);
}
}
This uses the function query:
function query(/* $sql [, ... ] */)
{
// SQL statement
$sql = func_get_arg(0);
// parameters, if any
$parameters = array_slice(func_get_args(), 1);
// try to connect to database
static $handle;
if (!isset($handle))
{
try
{
// connect to database
$handle = new PDO("mysql:dbname=" . DATABASE . ";host=" . SERVER, USERNAME, PASSWORD);
// ensure that PDO::prepare returns false when passed invalid SQL
$handle->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);
}
catch (Exception $e)
{
// trigger (big, orange) error
trigger_error($e->getMessage(), E_USER_ERROR);
exit;
}
}
// prepare SQL statement
$statement = $handle->prepare($sql);
if ($statement === false)
{
// trigger (big, orange) error
trigger_error($handle->errorInfo()[2], E_USER_ERROR);
exit;
}
// execute SQL statement
$results = $statement->execute($parameters);
// return result set's rows, if any
if ($results !== false)
{
return $statement->fetchAll(PDO::FETCH_ASSOC);
}
else
{
return false;
}
}
the other functions it uses are not relevant I guess. Now I cant seem to find why this keeps producing:
Notice: Undefined offset: 0 in /Applications/MAMP/htdocs/html/shuttle.php on line 16
Notice: Undefined offset: 0 in /Applications/MAMP/htdocs/html/shuttle.php on line 22
The relevant lines are
$residences = query("SELECT * FROM residences WHERE id = ?", $users[0]["residence"]);
and
if($residences[0]["id"] != 0)
Would appreciate some help! :)
Edit:
I transferred the same file to my linux system and without any changes I get a vardump. If I use the same vardump on MAMP on my Mac the array is empty. Now I get for:
var_dump($users);
array (size=1)
0 =>
array (size=5)
'id' => int 12
'username' => string 'frechdaxx' (length=9)
'mail' => string '*************#gmail.com' (length=23)
'hash' => string '$1$qr5axj4C$BET5zZGJza2DcHI8eD8fV0' (length=34)
'residence' => int 9573
Why is this a problem at all? the function query worked with the exact same syntax before when I accessed the user table and as we can see it gives back all other values correctly.
Why the difference between my environments? The array is empty on my MAMP server, but works on Linux. However, other code examples with the query function work perfectly on both environments
Why the big int of 9573? The value in the table is 2.
This is simply that $users[0]["residence"] doesn't exists (is not set). You should check it before attempting to use it. As you can see, it's only a notice, not a error. This means that in some case scenarios, the variable can be unset and PHP won't complain THAT much, but this is definitely not good as a general rule.
I would change it to:
$users = query("SELECT * FROM users WHERE id = ?", $_SESSION["id"]);
if (empty($users[0]["residence"]))
{
apologize("Sorry, there was an error");
}
Furthermore, that only "fixes" it. If you want to go to the root of the problem, it's in the query() function that is called in $users = query("SELECT * FROM users WHERE id = ?", $_SESSION["id"]);:
// execute SQL statement
$results = $statement->execute($parameters);
// return result set's rows, if any
if ($results !== false)
{
$fetched = $statement->fetchAll(PDO::FETCH_ASSOC);
// Check that ther was actually something fetched
if(!empty($fetched))
{
return $fetched;
}
}
/* You don't need to return false, null is the default returned value.
* If you *want* it, then delete the else. */
}
From the documentation, fetchall returns an empty array if there's nothing, so the function might return an empty array.
$users = query("SELECT * FROM users WHERE id = ?", $_SESSION["id"]);
if ($users === false) { }
You are strictly checking if query returned false, that only occurs when something was wrong with query or parameters. That does not check if any result was returned, and you are referring to the first returned record ($users[0]["residence"]) that is not present ($users is an empty array).
Since user id comes from $_SESSION["id"] it should be present and available in database, but there is possibility that somewhere in your code you are overwriting it with other id.

Categories