Simultaneous execution of many requests - php

I have an Instagram robot site
My program for each user makes a task that must be run every minute
Now my server has encountered a problem
I want my plan to be optimized
i am 427 user and my script for user create one cron jobs and repeat every minutes
all cron jobs send this formats
mysite.com/api/cron/[userpagename]
and my functions cron this code
/****************************************/
/* CRON */
/****************************************/
public function cron($type = ""){
$schedule_list = $this->db->select('activities.*, account.username, account.password, account.proxy, account.default_proxy')
->from($this->tb_activities." as activities")
->join($this->tb_accounts." as account", "activities.account = account.id")
->where("account.username = '{$type}' AND activities.status = 1 AND activities.time <= '".NOW."' AND account.status = 1")->order_by("time", "asc")->limit(2,0)->get()->result();
if(!empty($schedule_list)){
foreach ($schedule_list as $key => $schedule) {
if(!permission("instagram/activity", $schedule->uid)){
$this->db->delete($this->tb_activities, array("id" => $schedule->id));
$this->db->update($this->tb_activities, array("status" => 0, array("id" => $pid)));
$this->db->delete('instagram_activities_log', "uid = '{$schedule->uid}'");
$this->db->delete('instagram_activities', "uid = '{$schedule->uid}'");
$this->db->delete('instagram_accounts', "uid = '{$schedule->uid}'");
$this->deleteLineInFile("./cron.txt",$schedule->username);
exec('crontab ./cron.txt');
$this->model->send_email(get_option("email_renewal_reminders_subject", ""), get_option("email_renewal_reminders_content", ""), $schedule->uid);
}
$action = $schedule->action;
echo $schedule->action;
$mainId = $schedule->pid;
$settings = json_decode($schedule->data);
$speed = get_value($settings, "speed");
$next = get_time_next_schedule($schedule->settings, $speed->$action);
$stop = $this->stop_activity($mainId, $schedule->action , $schedule);
print_r(get_random_numbers(20));
if($stop){ break; }
if($next->task != 0){
$schedule->number = $next->task;
$proxy_data = get_proxy($this->tb_accounts, $schedule->proxy, $schedule);
try {
$ig = new InstagramAPI($schedule->username, $schedule->password, $proxy_data->use);
$result = $ig->activity->process($schedule);
//print_r($result);
if(is_array($result) || is_object($result)){
$this->save_log($schedule, $result);
$next->numbers = get_action_left($next->numbers, $result, $next->task);
}
} catch (Exception $e) {
print_r($e->getMessage());
$mess = Instagram_Get_Message($e->getMessage());
//ig_update_setting($schedule->action."_block", $e->getMessage(), $mainId);
if(strpos($mess, "This action was blocked")){
$next->numbers = get_random_numbers(0);
ig_update_setting($schedule->action."_block", "", $mainId);
}
else{
$next->numbers = get_action_left($next->numbers, array(), $next->task);
}
}
}
$this->db->update(INSTAGRAM_ACTIVITIES, array(
"time" => date("Y-m-d H:i:s", strtotime(NOW) + $next->minute*60),
"settings" => json_encode($next->numbers),
"changed" => date("Y-m-d H:i:s", strtotime(NOW) + $next->minute*60)
),
array("id" => $schedule->id)
);
}
}else{
echo "No activity";
}
I want to send all these requests together without error or failure

Related

Low speed of saving to the MySQL (PHP - Yii2)

I am trying to import data into MySQL from a JSON file.
public function importProductFile($file, $return = true)
{
$products = json_decode($file);
$dubTableName = Product::tableName() . "_dub";
$start = time();
if ($this->db->createDuplicateTable(Product::tableName(), $dubTableName)) {
$i = 0;
foreach ($products as $product) {
$i++;
$item = new Product_dub();
$item->id_1c_product = $product->id;
$category = Category_dub::findOne(['id_1c_category' => $product->category_id]);
if (!$category) {
Answer::failure("В этом товаре отсутствует категория или такой категории не существует: " . $product->title);
}
$item->category_id = $category->id;
$item->title = $product->title;
$brand = Brands_dub::findOne(['id_1c_brand' => $product->brand_id]);
if (!$brand) {
Answer::failure("В этом товаре отсутствует бренд/изготовитель: " . $product->title);
}
$item->brand_id = $brand->id;
// $item->shortdesc = $product->shortdesc;
$item->content1 = $product->content1;
$item->content2 = $product->content2;
$item->content3 = $product->content3;
$item->link_order = $product->link_order;
$item->img = $product->img;
$item->in_stock = $product->in_stock ? 1 : 0;
$item->is_popular = $product->is_popular ? 1 : 0;
if (!$item->save()) {
Answer::failure("Не удалось импортировать: Проверьте данные в " . $product->title);
}
if ($i == 200) {
break;
}
}
}
$finish = time();
$res = $finish - $start . "sec. ";
if ($return) {
echo $res;
Answer::success();
}
}
There are about 1100 objects in my JSON file. It takes 7 seconds to add 100 rows to the database. Adding 200 lines - 15 seconds. 300 = 33 sec, 400 = 58 sec. Why does it slow down over time and how to speed up this process?
I do everything on the local OpenServer server.
PHP 7.2 version, Xeon 2620v3 processor, 16 GB DDR4, HDD.
UPD 1.
"Can you try not importing and just determine the speed of reading" - I comment $item->save() and get 1-2 sec for all of JSON files. "In each iteration of your cycle you are running 2 DB queries to load category and brand." - I tried to delete these lines for test - but the result was 1-2 seconds faster than with 2 DB queries.
UPD 2.
I changed save() to insert() - the speed has increased. Now all JSON (1107 lines) is imported in 40 seconds.
Are there faster ways to load ready-made data from JSON into the database?
What if there are 100 thousand lines or a million? Is it normal practice to wait a few hours?
public function importProductFile($file, $return = true)
{
$products = json_decode($file);
$dubTableName = Product::tableName() . "_dub";
$start = time();
if ($this->db->createDuplicateTable(Product::tableName(), $dubTableName)) {
$start = time();
$categoryMap = Category_dub::find()->select(['id', 'id_1c_category'])->indexBy('id_1c_category')->column();
$brandMap = Brands_dub::find()->select(['id', 'id_1c_brand'])->indexBy('id_1c_brand')->column();
foreach ($products as $product) {
Yii::$app->db->createCommand()->insert('product_dub', [
'id_1c_product' => $product->id,
'category_id' => $categoryMap[$product->category_id] ?? '0',
'title' => $product->title,
'brand_id' => $brandMap[$product->brand_id] ?? 'No brand',
'content1' => $product->content1,
'content2' => $product->content2,
'content3' => $product->content3,
'link_order' => $product->link_order,
'img' => $product->img ?? 'no-image.png',
'in_stock' => $product->in_stock ? 1 : 0,
'is_popular' => $product->is_popular ? 1 : 0,
])->execute();
}
}
}
$finish = time();
$res = $finish - $start . "sec. ";
if ($return) {
echo $res;
Answer::success();
}
}
I changed save() to insert() - the speed has increased. Now all JSON (1107 lines) is imported in 40 seconds.
Are there faster ways to load ready-made data from JSON into the database?
What if there are 100 thousand lines or a million? Is it normal practice to wait a few hours?
public function importProductFile($file, $return = true)
{
$products = json_decode($file);
$dubTableName = Product::tableName() . "_dub";
$start = time();
if ($this->db->createDuplicateTable(Product::tableName(), $dubTableName)) {
$start = time();
$categoryMap = Category_dub::find()->select(['id', 'id_1c_category'])->indexBy('id_1c_category')->column();
$brandMap = Brands_dub::find()->select(['id', 'id_1c_brand'])->indexBy('id_1c_brand')->column();
foreach ($products as $product) {
Yii::$app->db->createCommand()->insert('product_dub', [
'id_1c_product' => $product->id,
'category_id' => $categoryMap[$product->category_id] ?? '0',
'title' => $product->title,
'brand_id' => $brandMap[$product->brand_id] ?? 'No brand',
'content1' => $product->content1,
'content2' => $product->content2,
'content3' => $product->content3,
'link_order' => $product->link_order,
'img' => $product->img ?? 'no-image.png',
'in_stock' => $product->in_stock ? 1 : 0,
'is_popular' => $product->is_popular ? 1 : 0,
])->execute();
}
}
}
$finish = time();
$res = $finish - $start . "sec. ";
if ($return) {
echo $res;
Answer::success();
}
}
You can use the bulk insert as mentioned in this answer and Yii2 docs. Using this bulk insert, you need to remember that the event will not be triggered.
Yii::$app->db->createCommand()->batchInsert('product_dub', array_keys(reset($products)), $products)->execute();

Google Task API PHP: Setting the TaskList ID & "Invalid task list ID" error

I'm trying to loop through task lists in order to generate a list of tasks using the Google Task PHP library.
I have:
Done all the credential stuff & can call the API
I can get task lists
A list of tasks for the respective task list output correctly using the ids generated from the point above & the tasklist parameter in Task API explorer
Where I'm stuck:
I'm not sure if I'm calling the 1) wrong method or 2) passing the wrong parameter to get a list of tasks for a respective tasklist id.
My code:
function getGcalTasks(){
$client = $this->getGcalTaskClient();
try {
$service = new Google_Service_Tasks($client);
$optParamLists = array(
'maxResults' => 10,
);
$result_lists = $service->tasklists->listTasklists($optParamLists);
if (
is_array($result_lists->getItems())
&& count($result_lists->getItems())
) {
foreach ($result_lists->getItems() as $tasklist) {
$taskListId = trim($tasklist->getId());
$taskListTitle = trim($tasklist->getTitle());
if(
$taskListId
){
$optParamsTasks = array(
// I've tried all of the below and still get: "Invalid task list ID",
'id' => $taskListId,
'kind' => 'tasks#taskList',
'title' => $taskListTitle,
//'tasklist' => $taskListId,
//'taskList' => $taskListId,
//'tasklistId' => $taskListId,
//'listName' => $taskListTitle,
);
$result_tasks = $service->tasks->listTasks($optParamsTasks);
}
}
}
} catch (Exception $e) {
log_message('error',$e->getMessage());
}
}
Welp, I took a look a few minutes later and realized that listTasks() only accepts one parameter, the id. The code below is working for me:
function getGcalTasks(){
$client = $this->getGcalTaskClient();
$tasks = array();
try {
$service = new Google_Service_Tasks($client);
$optParamLists = array(
'maxResults' => 10,
);
$result_lists = $service->tasklists->listTasklists($optParamLists);
if (
is_array($result_lists->getItems())
&& count($result_lists->getItems())
) {
foreach ($result_lists->getItems() as $tasklist) {
$taskListId = trim($tasklist->getId());
$taskListTitle = trim($tasklist->getTitle());
if(
$taskListId
){
$optParamsTasks = array(
'tasklist' => $taskListId,
);
$result_tasks = $service->tasks->listTasks($taskListId);
$tasks[] = $result_tasks->getItems();
}
}
return $tasks;
}
} catch (Exception $e) {
log_message('error',$e->getMessage());
}
}

phpscript stops after 4th sleep

I have a csv file containing about 3500 user's data. I want to import these users to my database and send them an email that they are registered.
I have the following code:
public function importUsers()
{
$this->load->model('perk/engagement_model');
$this->load->model('user/users_model');
$this->load->model('acl/aclUserRoles_model');
$allengagements = $this->engagement_model->getAll();
$filename = base_url() . 'assets/overdracht_users.csv';
$file = fopen($filename, "r");
$count = 0;
$totalImported = 0;
$importFails = array();
$mailFails = array();
while (($mappedData = fgetcsv($file, 10000, ";")) !== FALSE)
{
$count++;
//Skip first line because it is the header
if ($count > 1) {
if (!empty($mappedData[0])) {
$email = $mappedData[0];
$user = $this->users_model->getByEmail($email);
if (!$user) {
$user = new stdClass();
$user->email = $mappedData[0];
$user->first_name = $mappedData[1];
$user->family_name = $mappedData[2];
$user->address_line1 = $mappedData[3];
$user->address_postal_code = $mappedData[4];
$user->address_city = $mappedData[5];
$user->address_country = 'BE';
$user->volunteer_location = $mappedData[5];
$user->volunteer_location_max_distance = 50;
$user->phone = $mappedData[6];
if (!empty($mappedData[7])) {
$user->birthdate = $mappedData[7] . "-01-01 00:00:00";
} else {
$user->birthdate = null;
}
foreach ($allengagements as $eng) {
if ($eng->description == $mappedData[8]) {
$engagement = $eng->engagement_id;
}
}
$user->engagement = $engagement;
if (!empty($mappedData[9])) {
$date_created = str_replace('/', '-', $mappedData[9]);
$date_created = date('Y-m-d H:i:s', strtotime($date_created));
} else {
$date_created = date('Y-m-d H:i:s');
}
$user->created_at = $date_created;
if (!empty($mappedData[10])) {
$date_login = str_replace('/', '-', $mappedData[10]);
$date_login = date('Y-m-d H:i:s', strtotime($date_login));
} else {
$date_login = null;
}
$user->last_login = $date_login;
$user->auth_level = 1;
$user->is_profile_public = 1;
$user->is_account_active = 1;
$combinedname = $mappedData[1] . $mappedData[2];
$username = str_replace(' ', '', $combinedname);
if (!$this->users_model->isUsernameExists($username)) {
$uniqueUsername = $username;
} else {
$counter = 1;
while ($this->users_model->isUsernameExists($username . $counter)) {
$counter++;
}
$uniqueUsername = $username . $counter;
}
$user->username = $uniqueUsername;
$userid = $this->users_model->add($user);
if (!empty($userid)) {
$totalImported++;
//Add the user in the volunteer group in ACL
$aclData = [
'userID' => $userid,
'roleID' => 1,
'addDate' => date('Y-m-d H:i:s')
];
$this->aclUserRoles_model->add($aclData);
//Registration mail to volunteer
$mail_data['name'] = $user->first_name . ' ' . $user->family_name;
$mail_data['username'] = $user->username;
$this->email->from(GENERAL_MAIL, 'Test');
$this->email->to($user->email);
//$this->email->bcc(GENERAL_MAIL);
$this->email->subject('Test');
$message = $this->load->view('mail/register/registration',$mail_data,TRUE);
$this->email->message($message);
$mailsent = $this->email->send();
if (!$mailsent) {
array_push($mailFails, $mappedData);
}
} else {
array_push($importFails, $mappedData);
}
if ($count % 50 == 0) {
var_dump("count is " . $count);
var_dump("we are sleeping");
$min=20;
$max=40;
$randSleep = rand($min,$max);
sleep($randSleep);
var_dump("end of sleep (which is " . $randSleep . "seconds long)");
}
var_dump($user);
} else {
array_push($importFails, $mappedData);
}
}
}
}
var_dump("Totale aantal rijen in het bestand (met header) : " . $count);
var_dump("Totale aantal geimporteerd in de database : " . $totalImported);
var_dump("Totale aantal gefaalde imports in de database : " . count($importFails));
var_dump("Deze zijn gefailed : ");
var_dump($importFails);
}
If I do not add the users in the database, or send out a mail, and just var_dump() the $user, i can see all 3500+ users being correctly created in php objects (so they should be able to be inserted correctly).
The problem is that I want to add in a random sleep, between 20 and 40 seconds after every 50 mails that I sent.
So I started doing some testing and after commenting out the insert and mail code, I started running the script, noticing that after some amount (not 50 at all), it just stops for a bit, then continues and shows me the the var_dumps in my if case at the bottom, it can be shown here in the screenshots below.
The first screenshot shows the code stopping for a bit (note that I am only var_dumping stuff, i am not adding something in the database or sending out an email yet).
This screenshot shows what happens after the script reaches 200:
The script just completely stops from this point on. I have tried this 3 times, and every single time it stops exactly on 200.
What is happening here??
Most likely you're hitting default PHP limits. temporarily remove PHP default limits with:
ini_set('memory_limit',-1);
set_time_limit(0);
then, rerun the script and check your output.
There can be multiple reasons, but to know the exact one please enable error output with.
ini_set('display_errors',1);
ini_set('display_startup_errors',1);
and if limits are not the problem, the errors will tell you more.
HINT: using logs are still better than outputting errors to the visitors, but my guess is you're testing this on your computer.

Facebook Graph API Paging in Batches

I am struggling to understand how to make use of the paging mechanism with batch requests to Facebook's Graph API.
My issue with the code below is that despite attempting to track an offset for each facebook_id that is in my database, I'm unable to determine when a particular request has come up empty and therefore shouldn't be requested again.
I'm parsing the facebook_id's from the paging array in the responses. At the point when no more posts are available, FB returns an empty data array and therefore I'm unable to recognize which request has no more pages, and thus reduce the subsequent requests. Each time I'm querying FB with $fb_batch_limit urls despite the fact not all requests have returned a paging token from the previous call.
How would a real programmer tackle this problem?
$posts = array();
//$fb_batch_limit = 30;
$fb_batch_limit = 2;
$fb_req_limit=10;
$facebook = new Facebook(array(
'appId' => $fb_app_id,
'secret' => $fb_app_secret
));
$facebook->setAccessToken($fb_user_token);
$index = 0;
$since = '1Jan14';
$until = 'today';
$num_records_res = $oDB->select("SELECT count(*) as count FROM facebook f");
$num_records_row = mysqli_fetch_array($num_records_res);
$num_records = $num_records_row['count'];
while($index < $num_records) {
printf("# %d to %d\n", $index, $index+$fb_batch_limit);
$res = $oDB->select("SELECT facebook_id FROM facebook f ".
"LIMIT $index, $fb_batch_limit");
$index = $index + $fb_batch_limit;
$offsets = array();
while($rr = mysqli_fetch_array($res)){
$offsets[$rr['facebook_id']] = 0;
}
if (!count($offsets)) {
printf("no more accounts\n");
break;
}
$r=1;
do {
$urls = array();
foreach ($offsets as $fbid => $offset) {
$tmp["method"] = 'GET';
$tmp["relative_url"] = "/".$fbid."/feed?fields=likes.limit(1).summary(true)&offset=$offset&limit=$fb_req_limit";//&since=$since&until=$until";
$tmp["include_headers"] = false;
$urls[] = $tmp;
}
$numurls = count($urls);
printf("$r: fetching %d urls have %d posts\n", $numurls, count($posts));
$response = $facebook->batch($urls);
foreach ($response as $result) {
if (array_key_exists("error", $result)) {
//FIXME handle rate limits somehow
print_r($result);
die;
}
if (array_key_exists("paging", $result)) {
$fbid = substr(parse_url($result['paging']['previous'])['path'],6);
$fbid = substr($fbid, 0, -5);
if (($count = count($result['data'])) > 0) {
$posts = array_merge($posts, $result['data']);
if (array_key_exists("paging", $result) && array_key_exists("next", $result["paging"])) {
if (array_key_exists($fbid, $offsets)) {
$offsets[$fbid] += $count;
}
}
}
else {
printf("$fbid complete with ".
$offsets[$fbid]." posts\n");
unset($offsets[$fbid]);
}
}
else {
// how to stop requesting this single url?
print_r($result);
echo "--\n";
$numurls--;
}
}
} while (++$r && $numurls);
}

mysql to json using php. Nested objects

Good Afternoon,
I am trying to get these results into arrays in PHP so that I can encode them into json objects and send them to the client. The results of the query look like this:
id name hours cat status
3bf JFK Int 24 pass open
3bf JFK Int 24 std closed
3bf JFK Int 24 exp open
5t6 Ohm CA 18 pass closed
5t6 Ohm CA 18 std closed
5t6 Ohm CA 18 std2 open
5t6 Ohm CA 18 exp open
...
I would like for the json objects to look like this:
{ "id": "3bf", "name": "JFK Int", "cats":
{ [ { "cat": "pass", "status": "open" },
{ "cat": "std", "status": "closed" },
{ "cat": "exp", "status": "open" } ] }
{ "id": "5t6", "name": "Ohm CA", "cats":
{ [ { "cat": "pass", "status": "closed" },
{ "cat": "std", "status": "closed" },
{ "cat": "std2", "status": "open" } ],
{ "cat": "exp", "status": "open" } ] }
I have succesfully connected to mysql and exported using json_encode using flat tables but this part I do not know how to do in PHP. Thanks.
This is the code that I have. This returns an array of json objects but it is flat, not nested:
$SQL = "SELECT id, name, hours, cat, status FROM bwt.vewPortCats";
$result = mysql_query($SQL);
$arr = array();
while ($row = mysql_fetch_assoc($result)) {
$arr[] = $row;}
$json = json_encode($arr);
echo $json;
The data itself is from a view that combines the tables ports and cats.
what you could do (sorry, not the best code I could write... short on time, ideas, and energy ;-) is something like this (I hope it still conveys the point):
$SQL = "SELECT id, name, hours, cat, status FROM bwt.vewPortCats";
$result = mysql_query($SQL);
$arr = array();
while ($row = mysql_fetch_assoc($result)) {
// You're going to overwrite these at each iteration, but who cares ;-)
$arr[$row['id']]['id'] = $row['id'];
$arr[$row['id']]['name'] = $row['name'];
// You add the new category
$temp = array('cat' => $row['cat'], 'status' => $row['status']);
// New cat is ADDED
$arr[$row['id']]['cats'][] = $temp;
}
$base_out = array();
// Kind of dirty, but doesn't hurt much with low number of records
foreach ($arr as $key => $record) {
// IDs were necessary before, to keep track of ports (by id),
// but they bother json now, so we do...
$base_out[] = $record;
}
$json = json_encode($base_out);
echo $json;
Haven't had the time to test or think twice about it, but again, I hope it conveys the idea...
With thanks to #maraspin, I have got my below code:
function merchantWithProducts($id)
{
if (
!empty($id)
) {
//select all query
$query = "SELECT
m.id as 'mMerchantID', m.name as 'merchantName', m.mobile, m.address, m.city, m.province,
p.id as 'ProductID', p.merchantId as 'pMerchantID', p.category, p.productName, p.description, p.price, p.image, p.ratingCount
FROM " . $this->table_name . " m
JOIN by_product p
ON m.id = p.merchantId
WHERE m.id = :id
GROUP BY m.id";
// prepare query statement
$stmt = $this->conn->prepare($query);
// sanitize
// $this->id = htmlspecialchars(strip_tags($this->id));
// bind values
$stmt->bindParam(":id", $this->id);
try {
$success = $stmt->execute();
if ($success === true) {
$results = $stmt->fetchAll();
$this->resultToEncode = array();
foreach ($results as $row) {
$objItemArray = array(
"merchantID" => $row->mMerchantID,
"merchantName" => $row->merchantName,
"mobile" => $row->mobile,
"address" => $row->address,
"city" => $row->city,
"province" => $row->province,
"product" => array(
"productID" => $row->ProductID,
"pMerchantID" => $row->pMerchantID,
"category" => $row->category,
"productName" => $row->productName,
"description" => $row->description,
"price" => $row->price,
"image" => $this->baseUrl . 'imagesProducts/' . $row->image,
"ratingCount" => $row->ratingCount
)
);
array_push($this->resultToEncode, $objItemArray);
}
http_response_code(200);
$httpStatusCode = '200 OK';
$pass = true;
// return json_encode($resultToEncode);
} else {
http_response_code(204);
$httpStatusCode = '204 No Content';
$pass = false;
$this->resultToEncode = 'No Record Found';
}
} catch (PDOException $pdoEx) {
http_response_code(500); // internal server error.
$httpStatusCode = '500 Internal Server Error';
$pass = false;
$this->resultToEncode = $pdoEx->getCode();
} catch (Exception $ex) {
// return $ex->getMessage();
http_response_code(404); // 404 Not Found.
$httpStatusCode = '404 Not Found';
$pass = false;
$this->resultToEncode = $ex->getMessage();
}
} else {
http_response_code(400);
$httpStatusCode = '400 bad request';
$pass = false;
$this->resultToEncode = 'User id not specified';
}
echo json_encode(array('passed' => $pass, 'Response' => $httpStatusCode, 'result' => $this->resultToEncode));
}

Categories