i am using PHP to read feed xml in every half an hour, as of now i am always reading whole feed file whether it is updated or not. But i want to check whether feed xml is update since last download, if updated then only read the xml otherwise not.
I am trying to achieve it using below code, but $lastmodifiedRSS is always empty. I am not sure what is going wrong in my code. If i get $lastmodifiedRSS then i can easily compare it with last loaded time and then decide what to do.
If any Expert can share some info that would be great. Thank you in advance.
//get feed information & content
$feed = new feed($rssurl);
$feed->load();
//get the last modified date (web server information)
$lastmodifiedRSS = $feed->http->get_header_value('Last-Modified');
function http($url)
{
$this->url($url);
$this->user_agent = 'posh';
$this->header = "";
$this->code = "";
$this->codeStr = "";
$this->max_d = 5;
$this->authorization = "";
$this->proxy_auth = "";
$this->url = $url;
$this->addProxy();
}
function feed($url)
{
if ($url) {
$this->url = $url;
$this->http = new http($url);
//$this->addProxy();
} else {
$this->http = new http('');
}
}
function load()
{
$content= $this->http->get();
if (!$content)
return false;
$content = $this->transcode($content);
return $content;
}
function get_header_value($name)
{
if (preg_match("/$name: ?([^\\r\\n]*)\\r\\n/m",$this->head,$matches) !== false)
{
if (count($matches) > 1)
{
return $matches[1];
}
}
return false;
}
Regards,
Mona
Make use of stat() in PHP
<?php
print_r(stat('users.json'));
OUTPUT:
Array ( [0] => 2 [1] => 0 [2] => 33206 [3] => 1 [4] => 0 [5] => 0 [6] => 2 [7] => 298 [8] => 1384073940 [9] => 1384073940 [10] => 1368626190 [11] => -1 [12] => -1 [dev] => 2 [ino] => 0 [mode] => 33206 [nlink] => 1 [uid] => 0 [gid] => 0 [rdev] => 2 [size] => 298 [atime] => 1384073940 [mtime] => 1384073940 [ctime] => 1368626190 [blksize] => -1 [blocks] => -1 )
Source
Keeping a track of the variables [atime] , [size] can help you achieve what you are trying to do.
This kind of checking are better handled with HTTP headers itself.
If the server sends Etag header in the response, you could use it with the If-None-Match header on the next request to know if the file has changed.
Similarly, if the server sends Last-Modified header in the response, you could use this timestamp with the If-Modified-Since header on the next request to know if the file got changed.
Just to let you know what you could do, I can't contribute any code cause I don't know which HTTP library you're using here.
Related
Short Question: The imap_fetchstructure is not returning the filename and size attributes only for .msg file types, why?
Explanation:
I am trying to download email attachments from an IMAP account using imap_fetchstructure in the SugarCRM application. It works fine for all the other types, but not for the .msg files.
I checked the logs in SugarCRM and found that the structure returned for .msg file by imap_fetchstructure($this->conn, $msgNo); doesn't have the filename or name attribute as in the following result,
stdClass Object
(
[type] => 2
[encoding] => 0
[ifsubtype] => 1
[subtype] => RFC822
[ifdescription] => 0
[ifid] => 0
[ifdisposition] => 1
[disposition] => attachment
[ifdparameters] => 1
[dparameters] => Array
(
[0] => stdClass Object
(
[attribute] => creation-date
[value] => Tue, 01 Nov 2022 04:31:50 GMT
)
[1] => stdClass Object
(
[attribute] => modification-date
[value] => Tue, 01 Nov 2022 05:52:52 GMT
)
)
[ifparameters] => 0
[parameters] => stdClass Object
(
)
[parts] => Array(...)
)
In SugarCRM, I can see that if the filename and name attribute is not found in dparameters and parameters respectively, they are skipping the record and not saving it.
$fname = $this->handleEncodedFilename($this->retrieveAttachmentNameFromStructure($part));
if(!empty($fname)) {//assign name to attachment
$attach->name = $fname;
} else {//if name is empty, default to filename
$attach->name = urlencode($this->retrieveAttachmentNameFromStructure($part));
}
$attach->filename = $attach->name;
if (empty($attach->filename)) {
continue;
}
function retrieveAttachmentNameFromStructure($part)
{
$result = "";
foreach ($part->dparameters as $k => $v)
{
if( strtolower($v->attribute) == 'filename')
{
$result = $v->value;
break;
}
}
if (empty($result)) {
foreach ($part->parameters as $k => $v) {
if (strtolower($v->attribute) == 'name') {
$result = $v->value;
break;
}
}
}
return $result;
}
So, why are the filename and size attributes empty that is returned from imap_fetchstructure only for .msg file types?
It used to work before, but stopped working recently. I also noted that all the previous .msg emails saved have 'octet-stream' as the mime type, whereas in the current received structure I see the subtype as RFC822. Is that a problem?
Is it a problem from PHP side? I tried setting the filename manually something like 'test.msg'. If I do that, it does save the file. But it seems to be not working if I try to open that file in Outlook.
Does anyone know what is the catch here? Am I missing something?
I have an url and could scrap it using file_get_contents() and collect the links inside :
http://www.140online.com/Class/450/%D8%B3%D9%88%D8%A8%D8%B1%20%D9%85%D8%A7%D8%B1%D9%83%D8%AA
this is the result array :
Array
(
[0] => /company/C79955/المصرية ماركت - امام منصور امام/
[1] => /company/C574901/مريم ماركت/
[2] => /company/C1131/سوبر ماركت اولاد رجب/
[3] => /company/C19699/البركه/
[4] => /company/C21707/سوبر ماركت بست باى/
[5] => /company/C23552/الشامل مى ماركت/
[6] => /company/C21714/سوبر ماركت التميمى/
[7] => /company/C171121/الجزيره/
[8] => /company/C172099/الحمد ماركت/
[9] => /company/C172325/الخشاب ماركت/
[10] => /company/C174805/الدكان ماركت/
[11] => /company/C183996/اسواق الشمس المركزيه/
[12] => /company/C184039/اسواق مرحبا/
[13] => /company/C189580/البحيرى ماركت/
[14] => /company/C216371/سوق زمزم/
[15] => /company/C216941/سيتى ارت/
[16] => /company/C226223/وجينات محمد سعيد محمد موسى/
[17] => /company/C297432/ماركت اولاد محمود/
[18] => /company/C326553/سوبر ماركت عمرو عبد السلام سالم محمد/
[19] => /company/C326566/سوبر ماركت عونى عزيز هنرى/
)
then I need to scrap each link to get some data but here is the problem !
to use any link I have to add the domain name before.
http://www.140online.com/company/C79955/المصرية ماركت - امام منصور امام/
but when I try to get it's contents it send me this error :
file_get_contents(http://www.140online.com/company/C79955/%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A%D8%A9%20%D9%85%D8%A7%D8%B1%D9%83%D8%AA%20-%20%D8%A7%D9%85%D8%A7%D9%85%20%D9%85%D9%86%D8%B5%D9%88%D8%B1%20%D8%A7%D9%85%D8%A7%D9%85/): failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error
although I encode the Arabic part as it shown by explode:
$url_ex = explode("/", $itm_url[$i]);
$i_url = 'http://'.$url_ex[2].'/'.$url_ex[3].'/'.$url_ex[4].'/'.rawurlencode($url_ex[5]).'/';
the another strange thing is I can get contents for some links but not for all
please help
I had some problems with file_get_contents for web scraping. I would recommend to use CURL. This library returns HTTP error codes to know what kind of problem you have.
http://php.net/manual/en/book.curl.php
This is an example how to use CURL, I hope it works.
$ch1 = curl_init('string_url');
$options1 = array(CURLOPT_POST => true,
CURLOPT_HEADER => $request_headers,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_POSTFIELDS => (if u need to send params)
);
curl_setopt_array($ch1, $options1);
$page = curl_exec($ch1);
if(curl_exec($ch1) === false)
{
\print_r(curl_error($ch1));
//HTTP and many other errors if you have it
}
curl_close($ch1);
$dom = new \DOMDocument();
$dom->validateOnParse = true;
$dom->preserveWhiteSpace = false;
$dom->substituteEntities = false;
//Load Document Object Model
$dom->loadHTML($page);
$domxPath = new \DOMXPath($dom);
$urls = $domxPath->query("/html/body//div[#class='urls']");
//Example here searchs all divs which has urls class
//You need to specify which element of DOM contains URLS
//$urls variable must contains an array with urls that you need
I want to save the statistics of the Mikrotik /simple queue using the PHP API. I have been able to pull the data but seems my implementation on the PHP side is a problem. The following is the code and the resulting array object.
foreach ($util->setMenu('/queue simple')->getAll() as $queueEntry) {
// $lasArray = $queueEntry;
print_r($queueEntry);
}
Excerpt for Result since its returning for all users in the office, I have choosen just to display for one user. Take it that PEAR2\Net\RouterOS\Response Object is retuned for all users, i.e all in this case over 50 users. I would like to save this data to database but only the relevant ones like [.id], [name], [target], [limit-at], [max-limit] and [bytes], any assistance here would be highly regarded.
PEAR2\Net\RouterOS\Response Object
(
[unrecognizedWords:protected] => Array
(
)
[_type:PEAR2\Net\RouterOS\Response:private] => !re
[attributes:protected] => Array
(
[.id] => *12
[name] => GikundaPhone
[target] => 192.168.1.108/32
[parent] => none
[packet-marks] =>
[priority] => 8/8
[queue] => default-small/default-small
[limit-at] => 128000/384000
[max-limit] => 384000/384000
[burst-limit] => 0/0
[burst-threshold] => 0/0
[burst-time] => 0s/0s
[bucket-size] => 0.1/0.1
[bytes] => 16515474/129310087
[total-bytes] => 0
[packets] => 127812/133712
[total-packets] => 0
[dropped] => 76/8667
[total-dropped] => 0
[rate] => 0/0
[total-rate] => 0
[packet-rate] => 0/0
[total-packet-rate] => 0
[queued-packets] => 0/0
[total-queued-packets] => 0
[queued-bytes] => 0/0
[total-queued-bytes] => 0
[invalid] => false
[dynamic] => false
[disabled] => false
)
[_tag:PEAR2\Net\RouterOS\Message:private] =>
)
Have found and answer to my own question. This is what I have done. The
foreach ($util->setMenu('/queue simple')->getAll() as $queueEntry) {
// $lasArray = $queueEntry;
print_r($queueEntry);
}
provided alot of information thats unnecessary, so I have found about the routeros_api.class.php downloadable from here and followed but modified information from here. Then just used
$address = 'IPV4_Address_of_router';
$user = 'username_of_router';
$pass = 'password_of_router';
require('routeros_api.class.php');
$API = new routeros_api();
$API->debug = false;
// router credentials and after including the routeros_api.cass.php
if ($API->connect($address, $user, $pass)) {
$results = $API->comm("/queue/simple/print");
foreach ($results as $row) {
$clientName = $row['name'];
$clientIP = $row['target'];
$clientMaxDown = $row['limit-at'];
$clientMaxUp = $row['max-limit'];
$clientDownloads = $row['bytes'];
}
}
Only thing remaining was to save to database which is simple. Maybe someone may get helped someday by this.
After all I'm sorry for my English.
I'm writing a script in PHP and I encountered a small problem.
The idea is to create an array per line of text and replace the fourth value with a string representing the status.
I have a line of text e.g.
2022,33422,0,1,0,22
With a single line it works to create the array, but with multiple lines it creates an unexpected result.
Example:
2022,33422,0,1,0,22
2024,3232,01,1,04,298762
$myArray = explode(',', $uploadServer);
$status = array(0 => "Unprocessed",
1 => "Processing",
2 => "Download aborted because the file became too big.",
3 => "Download aborted because the file downloaded too long.",
4 => "Download finished. Uploading to RapidShare.",
5 => "Upload to RapidShare finished. Job finished.",
7 => "Upload failed 3 times for unknown reasons. Job aborted.",
8 => "Upload failed because the file is black listed.",
9 => "Download failed for unknown reasons.",
11 => "Enqueued for later processing because this account already downloads 5 files at the same time.");
foreach ($myArray as $valor) {
if(array_key_exists($valor[3],$status)) {
return $passer[] = $status[$valor[3]];
} else {
return FALSE;
}
}
The result of $myArray is
Array
(
[0] => 2022
[1] => 33422
[2] => 0
[3] => 1
[4] => 0
[5] => 22
)
But I need this final array
Array
(
[0]=>array(
[0] => 2022
[1] => 33422
[2] => 0
[3] => Processing
[4] => 0
[5] => 22
)
[1]=>array(
[0] => 2022
[1] => 33422
[2] => 0
[3] => Processing
[4] => 0
[5] => 22
)
)
Any idea?
thanks
UPDATE: So you have a string with a linebreak in it. This linebreak is an invisible character (\n). First split your string on every linebreak. Then loop through this array. split the strings on every comma and change the value with index 3:
$myArray = explode("\n", $uploadServer);
$status = array(0 => "Unprocessed",
1 => "Processing",
2 => "Download aborted because the file became too big.",
3 => "Download aborted because the file downloaded too long.",
4 => "Download finished. Uploading to RapidShare.",
5 => "Upload to RapidShare finished. Job finished.",
7 => "Upload failed 3 times for unknown reasons. Job aborted.",
8 => "Upload failed because the file is black listed.",
9 => "Download failed for unknown reasons.",
11 => "Enqueued for later processing because this account already downloads 5 files at the same time.");
$passer = array(); //create array to fill
foreach( $myArray as $valor ) { //loop through lines
if( !empty($valor) ) { // check for empty line
$tmp = explode(',', $valor); //make array of strings
if(array_key_exists($tmp[3],$status)) { //check value [3]
$tmp[3] = $status[$tmp[3]];
}
$passer[] = $tmp; //append array to parent array
}
}
if I understand you correctly, you are overthinking the solution.
foreach ($myArray as $valor) {
if(isset($status[$valor[3]])) { # this checks if there is a status with that index/key
return $passer[] = $status[$valor[3]];
} else {
return FALSE;
}
}
From REST API of twitter I user get/followers function.
I pasted a code snippet below.
My problem is, modt of the time I get followers' ids successfully. But when a user has more than 5000-6000 followers then my results comes wrong.
When I check from user's profile page, I see that user has 5500 followers, but when I run following code, most of the time 5500 ids come, but sometimes 29994 followers come inside $ids variable. Now I'm logging the results having more then 29k followers. I saw some of the requests returned with 29994 followers, but I couldn't find the answer.
Do I miss something in get ids - cursor approach? Thank you
Edit: After some debugging I logged "$cursor_archieve" parameter and found out this:
* Sometimes next_cursor and previous_cursor comes same:
Array
(
[0] => -1
[1] => 1400573121087317741
[2] => 1400573121087317741
[3] => 1400573121087317741
[4] => 1400573121087317741
[5] => 1400573121087317741
[6] => 1400573121087317741
)
So in this situation, although user has 7100 followers I get only 5000 followers
Sometimes cursors come sequentially same:
Array
(
[0] => -1
[1] => 1404335879106773348
[2] => 1404341060469987526
[3] => 1404338682006540390
[4] => 1404341060469987526
[5] => 1404335879106773348
[6] => 1404338682006540390
)
My code is like this:
public function getIds($user = "someuser"){
$tmhOAuth = new tmhOAuth(array( 'consumer_key' => YOUR_CONSUMER_KEY,
'consumer_secret' => YOUR_CONSUMER_SECRET,
'user_token' => $atoken1, 'user_secret' => $atoken2, ));
$cursor = '-1';
$ids = array();
$cursor_archieve = array();
while(true):
$code=$tmhOAuth->request('GET', $tmhOAuth->url('1/followers/ids'),
array('screen_name' => $user, 'cursor' => $cursor));
if ($code == 200) {
$responseData = json_decode($tmhOAuth->response['response'],true);
$ids = array_merge($ids, $responseData['ids']);
$cursor = $responseData['next_cursor_str'];
$cursor_archieve[] = $cursor;
} else {
return 0;
}
if ( $cursor == '0' || count($ids) >= 29000 ) {
break;
}
endwhile;
return $ids;
}
edit2: Should I make "array_unique" to delete duplicate ids, or doesn't use next cursor if previous_cursor=next cursor or any other option?
In every case user has 5500-6500 followers. So If I take only one cursor, I only can get first 5000 followers.
The reason was a programatic error in my codes. I fixed it after a week's debug session