Server keeps crashing on heavy load (facebook SDK + Laravel 5.2 + AWS) - php

I'm on the free tier of AWS and using the laravel framework and the facebook v.2.5 SDK (Web). I'm trying to get the latests 10 posts from facebook for approximately 600 users. Which would be 6000 posts max. Every time I run the query it runs through about 10 loops and then the app completely crashes and goes offline. Then returns after a few minutes. Laravel isn't showing me any errors.
My code is:
/**
* Get facebook users posts
* #return \SammyK\LaravelFacebookSdk\LaravelFacebookSdk;
*/
public function posts(\SammyK\LaravelFacebookSdk\LaravelFacebookSdk $fb)
{
// get posts
$profiles_to_get = DB::table('facebook_profiles')->distinct('username')->get();
$fb_admin_profile = DB::table('profiles')->where('social_media_type', "facebook")->first();
$admin_fb_access_token = $fb_admin_profile->oauth_token;
foreach ($profiles_to_get as $profile_to_get) {
try {
$response = $fb->get('/'.$profile_to_get->username.'?fields=posts.limit(10)', $admin_fb_access_token);
$userNode = $response->getGraphUser();
$posts = json_decode($userNode['posts']);
foreach ($posts as $post)
{
isset($post->message) ? $fb_posts[] = array('account_id' => $profile_to_get->id,
'facebook_id' => $userNode->getID(),
'message_id' => $post->id,
'message' => $post->message,
'created_time' => $post->created_time->date,
'created_at' => Carbon::now(),
'updated_at' => Carbon::now(),
) : null;
foreach ($fb_posts as $fb_post)
{
$postDuplicateChecker = DB::table('facebook_posts')->where('message_id', $fb_post['message_id'])->get();
if($postDuplicateChecker == !null)
{
DB::table('facebook_posts')->where('message_id', $fb_post['message_id'])->update($fb_post);
$notification = "First notification";
}
else
{
DB::table('facebook_posts')->insert( $fb_post );
$notification = "Second notification";
}
}
if ($post > 0 && $post % 10 == 0)
{
sleep(5);
}
}
} catch(\Facebook\Exceptions\FacebookSDKException $e) {
dd($e->getMessage());
}
}
return Redirect::route('someroute',[ 'notification' => $notification]);
}
I've tried setting the query timeout at 300 so it doesn't time out and also making the loop sleep after every 10 requests so that it reduces the load. Also I have other apps running on the same server but they never go offline when this app crashes.
My question is is there any way to optimize the code so that I don't have to upgrade the server or is my only choice to upgrade the server?

The answer was to batch the query using array_chunk and chunk the process into smaller pieces which could be handled easier.
array array_chunk ( array $array , int $size [, bool $preserve_keys = false ] )
Reference: http://php.net/manual/en/function.array-chunk.php

Related

Getting messages from multiple chats madelineproto

I am looking for some help here.
I am trying to get messages from multiple chats in Telegram using MadelineProto using the user IDs with code:
require 'vendor\autoload.php';
$settings[ 'logger' ][ 'logger' ] = 0;
$settings[ 'serialization' ][ 'serialization_interval' ] = 30;
$MadelineProto = new API('session.madeline', $settings);
$MadelineProto->async(true);
$MadelineProto->start();
$chat_one = 1234567;
$chat_two = 3254682;
$chat_three = 154789965;
function getMessagesById($u_id, $MadelineProto){
$offset_id = 0;
$limit = 100;
do {
$messages_Messages = $MadelineProto->messages->getHistory(
['peer' =>$u_id,
'offset_id' => $offset_id,
'offset_date' => 0,
'add_offset' => 0,
'limit' => $limit,
'max_id' => 0,
'min_id' => 0,
'hash' => 0 ]
);
if (count($messages_Messages['messages']) == 0) break;
print_r($messages_Messages);
$offset_id = end($messages_Messages['messages'])['id'];
yield $MadelineProto->sleep(3);
} while (true);
}
getMessagesById($chat_one , $MadelineProto);
getMessagesById($chat_two , $MadelineProto);
getMessagesById($chat_three , $MadelineProto);
But this does not return any message. I tried it by setting async to false and I get the messages only for the first function call and not two others.
Any idea here, what I am missing?
What I am actually after is only getting messages from specifics contacts and the code above works with async set to false for only one contact. I use ajax to refresh it every 10 seconds for new messages but sometimes it returns empty, so any better idea to do this would be appreciated. I went through the docs but nothing helped on getting messages, all sendMessage.
Thanks
It is necessary to transfer the object:
$inputChannel = ['_' => 'inputPeerChat', 'chat_id' => $u_id];
['peer' => inputChannel,

Array becoming multi-dimensional in AJAX request

So I am creating a function in WordPress which counts and sets a user session and storing its values in the user's local storage. I was able to make it work perfectly by using cookies and when the site is hosted locally, for some reason, it is not working when I uploaded it on the staging site. So I am trying implement this function using another approach and decided to use local storage instead.
There's a problem with the Array values that the function is generating and I have spent almost the entire day trying to debug the problem. It is generating multi-dimensional instead of a single one.
Here's my function code:
function monitor_post_views() {
$page_id = 'page' . $_POST['page_id'];
$timestamp = time();
// 30 minutes timeout
$timeout = 1800;
// Serves as my guide for debugging, will not include in the final code
$message = '';
if ( ! empty($_POST['page_id_array']) ) {
//Checks if values from local storage exist
//Gets the stored Array coming from AJAX call
$page_id_array[] = json_decode(stripslashes($_POST['page_id_array']), true);
if ( in_array_r($page_id_array, $page_id) ) {
//Check if current page is found in array
$message = 'FOUND IN ARRAY CHECKING !!!!';
$temp= [];
$page_id_array_temp = array('id' => $page_id, 'expiration' => $timestamp, 'message' => $message);
$temp = $page_id_array_temp;
//Pushes the generated array inside the $page_id_array
array_push($page_id_array, $temp);
print_r(json_encode($page_id_array));
foreach ( $page_id_array as $page ) {
//If page is in array, check if the session is expired, if not, do nothing, if expired, update and then run the view function
}
} else {
// ID Not found in Array, Insert a new entry
$message = 'ID NOT FOUND IN ARRAY, CREATING ENTRY !!!';
$temp = [];
$page_id_array_temp = array('id' => $page_id, 'expiration' => $timestamp, 'message' => $message);
$temp = $page_id_array_temp;
//Pushes the generated array inside the $page_id_array
array_push($page_id_array, $temp);
print_r(json_encode($page_id_array));
//Set post view function here base on $_POST['page_id']
}
} else {
//Not found in local storage, need to create one
$message = 'CREATING A NEW ENTRY !!!!';
$temp = [];
$page_id_array = array('id' => $page_id, 'expiration' => $timestamp, 'message' => $message);
$temp = $page_id_array;
print_r(json_encode($temp));
//Set post view function here base on $_POST['page_id']
}
wp_die();
}
add_action('wp_ajax_monitor_post_views', 'monitor_post_views');
add_action('wp_ajax_nopriv_monitor_post_views', 'monitor_post_views');
Here's a screenshot of what this function generates
Array
Here's a sample JSON
[[{"id":"page1202","expiration":1551125579,"message":"FOUND IN ARRAY CHECKING !!!!"},{"id":"page1206","expiration":1551125613,"message":"ID NOT FOUND IN ARRAY !!!! INSERTING ENTRY !!!"}],{"id":"page1296","expiration":1551125624,"message":"ID NOT FOUND IN ARRAY !!!! INSERTING ENTRY !!!"}]
I was trying to generate a one dimensional but ended up with this.
Any thoughts? Thanks in advance
The problem is you are creating arrays too many times:
Change $page_id_array and $page_id_array_temp to
$page_id_array=new \stdClass();//no need to declare as an array
replace
$page_id_array_temp = array('id' => $page_id, 'expiration' => $timestamp, 'message' => $message);
with
$page_id_array->id=$page_id;
$page_id_array->expiration=$timestamp;
$page_id_array->message=$message;
also change
$temp = [];
you can use it directly
//no need to declare $temp as an array
$temp=$page_id_array;

How to upload large files (around 10GB)

I want to transfer to my Amazon S3 bucket an archive of around 10GB, using a PHP script (it's a backup script).
I actually use the following code :
$uploader = new \Aws\S3\MultipartCopy($s3Client, $tmpFilesBackupDirectory, [
'Bucket' => 'MyBucketName',
'Key' => 'backup'.date('Y-m-d').'.tar.gz',
'StorageClass' => $storageClass,
'Tagging' => 'expiration='.$tagging,
'ServerSideEncryption' => 'AES256',
]);
try
{
$result = $uploader->copy();
echo "Upload complete: {$result['ObjectURL']}\n";
}
catch (Aws\Exception\MultipartUploadException $e)
{
echo $e->getMessage() . "\n";
}
My issue is that after few minutes (let's say 10mn), I receive an error message from the apache server : 504 Gateway timeout.
I understand that this error is related to the configuration of my Apache server, but I don't want to increase the timeout of my server.
My idea is to use the PHP SDK Low-Level API to do the following steps:
Use Aws\S3\S3Client::uploadPart() method in order to manually upload 5 parts, and store the response obtained in $_SESSION (I need the ETag values to complete the upload);
Reload the page using header('Location: xxx');
Perform again the first 2 steps for the next 5 parts, until all parts are uploaded;
Finalise the upload using Aws\S3\S3Client::completeMultipartUpload().
I suppose that this should work but before to use this method, I'd like to know if there is an easier way to achieve my goal, for example by using the high-level API...
Any suggestions?
NOTE : I'm not searching for some existing script : my main goal is to learn how to fix this issue :)
Best regards,
Lionel
Why not just use the AWS CLI to copy the file? You can create a script in the CLI and that way everything is AWS native. (Amazon has a tutorial on that.) You can use the scp command:
scp -i Amazonkey.pem /local/path/backupfile.tar.gz ec2-user#Elastic-IP-of-ec2-2:/path/backupfile.tar.gz
From my perspective, it would be easier to do the work within AWS, which has features to move files and data. If you'd like to use a shell script, this article on automating EC2 backups has a good one, plus more detail on backup options.
To answer my own question (I hope it might help someone one day!), he is how I fixed my issue, step by step:
1/ When I load my page, I check if the archive already exists. If not, I create my .tar.gz file and I reload the page using header().
I noticed that this step was quite slow since there is lot of data to archive. That's why I reload my page to avoid any timeout during the next steps!
2/ If the backup file exists, I use AWS MultipartUpload to send 10 chunks of 100MB each. Everytime that a chunk is sent successfully, I update a session variable ($_SESSION['backup']['partNumber']) to know what is the next chunk that needs to be uploaded.
Once my 10 chunks are sent, I reload the page again to avoid any timeout.
3/ I repeat the second step until the upload of all parts is done, using my session variable to know which part of the upload needs to be sent next.
4/ Finally, I complete the multipart upload and I delete the archive stored locally.
You can of course send more than 10 times 100MB before to reload your page. I chose this value to be sure that I won't reach a timeout even if the download is slow. But I guess I could send easilly around 5GB each time without issue.
Note: You cannot redirect you script to itself too much time. There is a limit (I think it's around 20 times for Chrome and Firefoxbefore to get an error, and more for IE). In my case (the archive is around 10GB), transfering 1GB per reload is fine (the page will be reloaded around 10 times). But it the archive size increases, I'll have to send more chunks each time.
Here is my full script. I could surely be improved, but it's working quite well for now and it may help someone having similar issue!
public function backup()
{
ini_set('max_execution_time', '1800');
ini_set('memory_limit', '1024M');
require ROOT.'/Public/scripts/aws/aws-autoloader.php';
$s3Client = new \Aws\S3\S3Client([
'version' => 'latest',
'region' => 'eu-west-1',
'credentials' => [
'key' => '',
'secret' => '',
],
]);
$tmpDBBackupDirectory = ROOT.'Var/Backups/backup'.date('Y-m-d').'.sql.gz';
if(!file_exists($tmpDBBackupDirectory))
{
$this->cleanInterruptedMultipartUploads($s3Client);
$this->createSQLBackupFile();
$this->uploadSQLBackup($s3Client, $tmpDBBackupDirectory);
}
$tmpFilesBackupDirectory = ROOT.'Var/Backups/backup'.date('Y-m-d').'.tar.gz';
if(!isset($_SESSION['backup']['archiveReady']))
{
$this->createFTPBackupFile();
header('Location: '.CURRENT_URL);
}
$this->uploadFTPBackup($s3Client, $tmpFilesBackupDirectory);
unlink($tmpDBBackupDirectory);
unlink($tmpFilesBackupDirectory);
}
public function createSQLBackupFile()
{
// Backup DB
$tmpDBBackupDirectory = ROOT.'Var/Backups/backup'.date('Y-m-d').'.sql.gz';
if(!file_exists($tmpDBBackupDirectory))
{
$return_var = NULL;
$output = NULL;
$dbLogin = '';
$dbPassword = '';
$dbName = '';
$command = 'mysqldump -u '.$dbLogin.' -p'.$dbPassword.' '.$dbName.' --single-transaction --quick | gzip > '.$tmpDBBackupDirectory;
exec($command, $output, $return_var);
}
return $tmpDBBackupDirectory;
}
public function createFTPBackupFile()
{
// Compacting all files
$tmpFilesBackupDirectory = ROOT.'Var/Backups/backup'.date('Y-m-d').'.tar.gz';
$command = 'tar -cf '.$tmpFilesBackupDirectory.' '.ROOT;
exec($command);
$_SESSION['backup']['archiveReady'] = true;
return $tmpFilesBackupDirectory;
}
public function uploadSQLBackup($s3Client, $tmpDBBackupDirectory)
{
$result = $s3Client->putObject([
'Bucket' => '',
'Key' => 'backup'.date('Y-m-d').'.sql.gz',
'SourceFile' => $tmpDBBackupDirectory,
'StorageClass' => '',
'Tagging' => '',
'ServerSideEncryption' => 'AES256',
]);
}
public function uploadFTPBackup($s3Client, $tmpFilesBackupDirectory)
{
$storageClass = 'STANDARD_IA';
$bucket = '';
$key = 'backup'.date('Y-m-d').'.tar.gz';
$chunkSize = 100 * 1024 * 1024; // 100MB
$reloadFrequency = 10;
if(!isset($_SESSION['backup']['uploadId']))
{
$response = $s3Client->createMultipartUpload([
'Bucket' => $bucket,
'Key' => $key,
'StorageClass' => $storageClass,
'Tagging' => '',
'ServerSideEncryption' => 'AES256',
]);
$_SESSION['backup']['uploadId'] = $response['UploadId'];
$_SESSION['backup']['partNumber'] = 1;
}
$file = fopen($tmpFilesBackupDirectory, 'r');
$parts = array();
//Reading parts already uploaded
for($i = 1; $i < $_SESSION['backup']['partNumber']; $i++)
{
if(!feof($file))
{
fread($file, $chunkSize);
}
}
// Uploading next parts
while(!feof($file))
{
do
{
try
{
$result = $s3Client->uploadPart(array(
'Bucket' => $bucket,
'Key' => $key,
'UploadId' => $_SESSION['backup']['uploadId'],
'PartNumber' => $_SESSION['backup']['partNumber'],
'Body' => fread($file, $chunkSize),
));
}
}
while (!isset($result));
$_SESSION['backup']['parts'][] = array(
'PartNumber' => $_SESSION['backup']['partNumber'],
'ETag' => $result['ETag'],
);
$_SESSION['backup']['partNumber']++;
if($_SESSION['backup']['partNumber'] % $reloadFrequency == 1)
{
header('Location: '.CURRENT_URL);
die;
}
}
fclose($file);
$result = $s3Client->completeMultipartUpload(array(
'Bucket' => $bucket,
'Key' => $key,
'UploadId' => $_SESSION['backup']['uploadId'],
'MultipartUpload' => Array(
'Parts' => $_SESSION['backup']['parts'],
),
));
$url = $result['Location'];
}
public function cleanInterruptedMultipartUploads($s3Client)
{
$tResults = $s3Client->listMultipartUploads(array('Bucket' => ''));
$tResults = $tResults->toArray();
if(isset($tResults['Uploads']))
{
foreach($tResults['Uploads'] AS $result)
{
$s3Client->abortMultipartUpload(array(
'Bucket' => '',
'Key' => $result['Key'],
'UploadId' => $result['UploadId']));
}
}
if(isset($_SESSION['backup']))
{
unset($_SESSION['backup']);
}
}
If someone has questions don't hesitate to contact me :)

Long TTFB from FB Graph API

I wrote a PHP script home-made in order to get last FB posts from a page, which is simply executed from an jQuery/AJAX code when my document is ready :
$("my_object").load("fb_feed.php");
Running on local PHP server as on a VPS, to Time To First Byte is very long : 10s at least, 20s on average, or more
ini_set("allow_url_fopen", 1);
function getGraphUrl($param){
return "https://graph.facebook.com/". $param ."access_token=THE_TOKEN";
}
function getContent($param){
$url = getGraphUrl($param);
return json_decode(file_get_contents($url));
}
$feed = getContent("ID_OF_THE_PAGE/feed?");
$posts = array();
foreach ($feed->data as $i => $post) {
if(isset($post->message)){
$object_id = getContent($post->id . "?fields=object_id&")->object_id;
if(isset($object_id)){
$img_url = "https://graph.facebook.com/" . $object_id . "/picture";
$short = (getimagesize($img_url)[1] < 170 ? true : false);
if($short){
$img_url = "images/empty.png";
}
array_push($posts, array('id' => $post->id,'message' => $post->message,
'img_url' => $img_url, 'date' => $post->created_time));
}
}
if(count($posts) == 6){
break;
}
}
The extract of the PHP script which load posts 👆
Everything works, the only problem is the loading duration which is very very long
Is there a way to load faster, rather than build my own cache ?
Is my request poorly made ?

Python script can retrieve value from memcache, but PHP script gets empty result

i run a python script whitch cache some data
self.cache.set('test', 'my sample data', 300)
data = self.cache.get('test')
self.p(data)
this program will result in print of 'my sample data' ... everything its good, but when i try to access this key from php
$data = $this->cache->get('test');
print_r($test);
i only get empty result
so i check the server stats
$list = array();
$allSlabs = $this->cache->getExtendedStats('slabs');
$items = $this->cache->getExtendedStats('items');
foreach($allSlabs as $server => $slabs) {
foreach($slabs AS $slabId => $slabMeta) {
$cdump = $this->cache->getExtendedStats('cachedump',(int)$slabId);
foreach($cdump AS $server => $entries) {
if($entries) {
foreach($entries AS $eName => $eData) {
$list[$eName] = array(
'key' => $eName,
'server' => $server,
'slabId' => $slabId,
'detail' => $eData,
'age' => $items[$server]['items'][$slabId]['age'],
);
}
}
}
}
}
ksort($list);
print_r($list);
and this key 'test' is there ... but i cannot access it
if i cache something in php i get the result everytime, but somehow this python + php cache wont work
if someone has an idea where could be a problem plese advice ... i try everything
Could it be that the hashes don't match between PHP and Python? A solution is here: http://www.ruturaj.net/python-php-memcache-hash
Add the following to your Python script to change how hashes are calculated...
import memcache
import binascii
m = memcache.Client(['192.168.28.7:11211', '192.168.28.8:11211
', '192.168.28.9:11211'])
def php_hash(key):
return (binascii.crc32(key) >> 16) & 0x7fff
for i in range(30):
key = 'key' + str(i)
a = m.get((php_hash(key), key))
print i, a

Categories