My code:
while ($row = $stmt -> fetch(PDO::FETCH_ASSOC)) {
$uri = 'https://www.themoviedb.org/movie/' . $row['tmdb_id'];
$get = file_get_contents($uri);
$pos1 = strpos($get, '<span class="genres">');
$pos2 = strpos($get, '</span>', $pos1);
$text = substr($get,$pos1,$pos2-$pos1+7);
if (strstr ($text, 'Animation'))
array_push ($array, $row['poster_path']);
}
it return me this error:
failed to open stream: HTTP request failed! HTTP/1.1 429 Too Many Requests
in "path/index.php" on line 128
line 128 is: $get = file_get_contents($uri);
Your target url is preventing this from happening. You're basically sending 70 requests in less than one second (probably). This is identified as an attack and at some point blocked.
You need to add some kind of timer to build some kind of queue that does this from time to time, instead of doing it all at once.
Related
i have following problem:
in a function, i put in an array with at least 700 names. I get out an array with all information about their releases from the last 10 days.
The function gets via iTunes API a json response, which i want to use for further analyzings.
Problem:
- while executing function, it takes about 3mins to finish it.
- homepage is not reachable for others, while i execute it:
(Error on Server: (70007)The timeout specified has expired: AH01075: Error dispatching request to : (polling)) --> Running out of memory?
Questions:
- how to code this function more efficient?
- how to code this function without using to much memory, shall i use unset(...) ??
Code:
function getreleases($artists){
# print_r($artists);
$releases = array();
foreach( $artists as $artist){
$artist = str_replace(" ","%20",$artist);
$ituneslink = "https://itunes.apple.com/search?term=".$artist."&media=music&entity=album&limit=2&country=DE";
$itunesstring = file_get_contents($ituneslink);
$itunesstring = json_decode($itunesstring);
/*Results being decoded from json to an array*/
if( ($itunesstring -> resultCount)>0 ){
foreach ( $itunesstring -> results as $value){
if( (date_diff(date_create('now'), date_create( ($value -> releaseDate )))->format('%a')) < 10) {
#echo '<br>Gefunden: ' . $artist;
$releases[] = $value;
}
}
}else{
echo '<br><span style="color:red">Nicht gefunden bei iTunes: ' . $artist .'</span>';
}
unset($ituneslink);
unset($itunesstring);
unset($itunesstring2);
}
return $releases;
}
The problem lies in the fact that every time that function is executed, your server needs to make 700+ API Calls, parse the data, and work your logic on it.
One potential solution is to use Wordpress's transients to 'cache' the value (or perhaps even the whole output), this way, it won't have to execute that strenuous function on every connection, it'll just pull the data from the transient. You can set an expiry date for a transient, so you can have it refetch the information every X days/hours.
Take a look at this article from CSS Tricks that walks you through a simple example using transients.
But the problem is not fixed. While updating the stuff and getting 700 items from iTunes API and while Running in the for-loop, the homepage is getting out of memory. although homepage is not reachable from my computer. I just tried for a "timeout" or "sleep" sothat the script is searching for stuff every few seconds. But it doesn't change it.
I just improved: Changed "foreach" to "for" because of memory reasons. Now variables are not being copied. Are there more problems :-/ ??
I've got to for-loops in there. Maybe $itunesstring is being copied ?
if(!function_exists('get_contents')){
function get_contents(&$url){
// if cURL is available, use it...
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$cache = curl_exec($ch);
curl_close($ch);
return $cache;
}
}
function getfromituneslink(&$link,&$name){
$name = str_replace("'","",$name);
$name = substr($name, 0, 14);
$result = get_transient("getlink_itunes_{$name}");
if(false === $result){
$result = get_contents($link);
set_transient("getlink_itunes_{$name}",$result, 12*HOUR_IN_SECONDS);
}
return $result;
}
function getreleases(&$artists){
$releases= array();
while( 0 < count($artists)){
$itunesstring = array();
$artist = array_shift($artists);
$artist = str_replace(" ","%20",$artist);
$ituneslink = "https://itunes.apple.com/search?term=".$artist."&media=music&entity=album&limit=2&country=DE";
$itunesstring = getfromituneslink($ituneslink,$artist);
unset($ituneslink);
$itunesstring = json_decode($itunesstring);
if( ($itunesstring -> resultCount)>0 ){
#for($i=0; $i< (count($itunesstring -> results))-1; ++$i)
while( 0 < count(($itunesstring -> results))){
$value = array_shift($itunesstring -> results);
#$value = &$itunesstring[results][$i];
#foreach ( $itunesstring -> results as $value)
if( (date_diff(date_create('now'), date_create( ($value -> releaseDate )))->format('%a')) < 6) {
$releases[] = array($value->artistName, $value->collectionName, $value->releaseDate, str_replace("?uo=4","",$value -> collectionViewUrl));
unset($value);
}
}
}else{
echo '<br><span style="color:red">Nicht gefunden bei iTunes: ' . str_replace("%20"," ",$artist) .'</span>';
}
unset($ituneslink);
unset($itunesstring);
}
return $releases;
}
I don't know, where the problem is. :-(
Any other possibilty to let the function run to get the information one by another
I'm a newbie and I have a url that contains Persian language characters.
For example this:
http://tabnak.ir/fa/news/577155/ویدیوی-درگیری-نیروهای-سیا-و-پنتاگون-در-سوریه-با-همدیگر-ویدیوهایی-از-جنجال-پاسخ-مشایخی-به-مجیدی-و-حرفهای-عجیب-الویس-پریسلی-ایران
When I want to get the html source of that url, with this line of code:
$source = file_get_contents($url);
I get this error:
Warning: file_get_contents(http://tabnak.ir/fa/news/577155/ویدیوی-درگیری-نیروهای-سیا-و-پنتاگون-در-سوریه-با-همدیگر-ویدیوهایی-از-جنجال-پاسخ-مشایخی-به-مجیدی-و-حرفهای-عجیب-الویس-پریسلی-ایران):
failed to open stream: HTTP request failed! HTTP/1.0 400 Bad request in C:\wamp\www\file.php on line 25
I wanted to solve this problem by using the urlencode, but it didn't work.
The urlencode output of that line becomes:
http%3A%2F%2Ftabnak.ir%2Ffa%2Fnews%2F577155%2F%D9%88%DB%8C%D8%AF%DB%8C%D9%88%DB%8C-%D8%AF%D8%B1%DA%AF%DB%8C%D8%B1%DB%8C-%D9%86%DB%8C%D8%B1%D9%88%D9%87%D8%A7%DB%8C-%D8%B3%DB%8C%D8%A7-%D9%88-%D9%BE%D9%86%D8%AA%D8%A7%DA%AF%D9%88%D9%86-%D8%AF%D8%B1-%D8%B3%D9%88%D8%B1%DB%8C%D9%87-%D8%A8%D8%A7-%D9%87%D9%85%D8%AF%DB%8C%DA%AF%D8%B1-%D9%88%DB%8C%D8%AF%DB%8C%D9%88%D9%87%D8%A7%DB%8C%DB%8C-%D8%A7%D8%B2-%D8%AC%D9%86%D8%AC%D8%A7%D9%84-%D9%BE%D8%A7%D8%B3%D8%AE-%D9%85%D8%B4%D8%A7%DB%8C%D8%AE%DB%8C-%D8%A8%D9%87-%D9%85%D8%AC%DB%8C%D8%AF%DB%8C-%D9%88-%D8%AD%D8%B1%D9%81%E2%80%8C%D9%87%D8%A7%DB%8C-%D8%B9%D8%AC%DB%8C%D8%A8-%D8%A7%D9%84%D9%88%DB%8C%D8%B3-%D9%BE%D8%B1%DB%8C%D8%B3%D9%84%DB%8C-%D8%A7%DB%8C%D8%B1%D8%A7%D9%86
Which is not a correct url address, and I can't get contents again.
What should I do?
Can you try this ? This way, you should be able to do the file_get_contents on the encoded url
$url = 'http://tabnak.ir/fa/news/577155/ویدیوی-درگیری-نیروهای-سیا-و-پنتاگون-در-سوریه-با-همدیگر-ویدیوهایی-از-جنجال-پاسخ-مشایخی-به-مجیدی-و-حرفهای-عجیب-الویس-پریسلی-ایران';
$url = mb_convert_encoding($url, 'HTML-ENTITIES', "UTF-8");
$source = file_get_contents($url);
EDIT (TESTED THIS AND WORKS) :
Try this, maybe by encoding the part of the URL where there are arabic character, it might work :
$link = 'http://tabnak.ir/fa/news/577155/ویدیوی-درگیری-نیروهای-سیا-و-پنتاگون-در-سوریه-با-همدیگر-ویدیوهایی-از-جنجال-پاسخ-مشایخی-به-مجیدی-و-حرفهای-عجیب-الویس-پریسلی-ایران';
$exploded = explode('/',$link);
$exploded[6] = urlencode($exploded[6]);
$urlimplode = implode($exploded,'/');
$source = file_get_contents($urlimplode);
echo $source;
In my case, where I wanted to pass a parameter with Get, I did as follows:
$message = "یک پیغام تست";
$link = "http://localhost:5000/myAPI_name?msg=".$message;
$exploded = explode('=',$link);
$exploded[1] = urlencode($exploded[1]);
$urlimplode = implode($exploded,'=');
$source = file_get_contents($urlimplode);
$source=json_decode($source,true);
As the content of $message is Persian, I had to perform encoding just for anything after =
I currentley trying to get data from a website, and use it in my PHP script.
The link I am trying to reach is:
http://steamcommunity.com/market/priceoverview/?country=US¤cy=3&appid=730&market_hash_name=★%20Bayonet
Here is my code:
<?php
$skin = $_GET['market_hash_name'];
$link = "http://steamcommunity.com/market/priceoverview/?country=US¤cy=3&appid=730&market_hash_name=".$skin."";
$FN = file_get_contents($link);
echo $FN;
?>
And this is how I use my link:
http://myhost.com/getPrice.php?market_hash_name=★ Bayonet
This is the error I am getting:
Warning: file_get_contents(http://steamcommunity.com/market/priceoverview/?country=US¤cy=3&appid=730&market_hash_name=★ Bayonet): failed to open stream: HTTP request failed! HTTP/1.0 500 Internal Server Error in F:\xampp\htdocs\getPrice.php on line 5
EDIT:
Okay, so I have found a solution for my problem, but now a new error is comming up:
Warning: file_get_contents(http://steamcommunity.com/market/priceoverview/?country=US¤cy=3&appid=730&market_hash_name=%E2%98%85+Bayonet+%7C+Stained (Factory New)): failed to open stream: HTTP request failed! HTTP/1.0 500 Internal Server Error in F:\xampp\htdocs\getPrice.php on line 7
This is how I am using my link now:
getPrice.php?market_hash_name=★ Bayonet | Stained
And this is my code:
function getPrice($string) {
$skin = urlencode($_GET['market_hash_name']);
$link = "http://steamcommunity.com/market/priceoverview/?country=US¤cy=3&appid=730&market_hash_name=".$skin." ".$string;
$content = file_get_contents($link);
$data = json_decode($content, true);
return $data['lowest_price'];
}
$FN = getPrice("(Factory New)");
$MW = getPrice("(Minimal Wear)");
$FT = getPrice("(Field-Tested)");
$WW = getPrice("(Well-Worn)");
$BS = getPrice("(Battle-Scarred)");
echo "Factory New: $FN; Minimal Wear: $MW; Field-Tested: $FT; Well-Worn: $WW; Battle-Scared: $BS";
Among other characters, non-ASCII characters must be URL-encoded (aka percent-encoded) in query strings.
$skin = url_encode($_GET['market_hash_name']);
$link = "http://steamcommunity.com/market/priceoverview/?country=US¤cy=3&appid=730&market_hash_name=".$skin."";
$FN = file_get_contents($link);
I have a file uploaded in AWS s3 bucket and set that file to public permission . i want to share that file in my Facebook .. the thing is i can just copy that public link and share it . but i also want the count of the downloads to stored .. in other way i want to host a php file in my web hosting where there will be a tab like bar in which that file name,file size, download link and total download count will be there . Please help me with the code
I tried the following code which i got from google search but no use
<?php
$aws_key = '_YOUR_AWS_KEY_000000';
$aws_secret = '_your_aws_secret_00000000000000000000000';
$aws_bucket = 'anyexample-test'; // AWS bucket
$aws_object = 'test.png'; // AWS object name (file name)
if (strlen($aws_secret) != 40) die("$aws_secret should be exactly 40 bytes long");
$dt = gmdate('r'); // GMT based timestamp
// preparing string to sign
$string2sign = "GET
{$dt}
/{$aws_bucket}/{$aws_object}";
// preparing HTTP query
$query = "GET /{$aws_bucket}/{$aws_object} HTTP/1.1
Host: s3.amazonaws.com
Connection: close
Date: {$dt}
Authorization: AWS {$aws_key}:".amazon_hmac($string2sign)."\n\n";
echo "Downloading: http://s3.amazonaws.com/{$aws_bucket}/{$aws_object}\n";
list($header, $resp) = downloadREST($fp, $query);
echo "\n\n";
if (strpos($header, '200 OK') === false) // checking for error
die($header."\r\n\r\n".$resp);
$aws_object_fs = str_replace('/', '_', $aws_object);
// AWS object may contain slashes. We're replacing them with underscores
#$fh = fopen($aws_object_fs, 'wb');
if ($fh == false)
die("Can't open file {$aws_object_fs} for writing. Fatal error!\n");
echo "Saving data to {$aws_object_fs}...\n";
fwrite($fh, $resp);
fclose($fh);
// Sending HTTP query, without keep-alive support
function downloadREST($fp, $q)
{
// opening HTTP connection to Amazon S3
// since there is no keep-alive we open new connection for each request
$fp = fsockopen("s3.amazonaws.com", 80, $errno, $errstr, 30);
if (!$fp) die("$errstr ($errno)\n"); // connection failed, pity
fwrite($fp, $q); // sending query
$r = ''; // buffer for result
$check_header = true; // header check flag
$header_end = 0;
while (!feof($fp)) {
$r .= fgets($fp, 256); // reading response
if ($check_header) // checking for header
{
$header_end = strpos($r, "\r\n\r\n"); // this is HTTP header boundary
if ($header_end !== false)
$check_header = false; // We've found it, no more checking
}
}
fclose($fp);
$header_boundary = $header_end+4; // 4 is length of "\r\n\r\n"
return array(substr($r, 0, $header_boundary), substr($r, $header_boundary));
}
// hmac-sha1 code START
// hmac-sha1 function: assuming key is global $aws_secret 40 bytes long
// http://en.wikipedia.org/wiki/HMAC
// warning: key is padded to 64 bytes with 0x0 after first function call
// hmac-sha1 function
function amazon_hmac($stringToSign)
{
if (!function_exists('binsha1'))
{ // helper function binsha1 for amazon_hmac (returns binary value of sha1 hash)
if (version_compare(phpversion(), "5.0.0", ">=")) {
function binsha1($d) { return sha1($d, true); }
} else {
function binsha1($d) { return pack('H*', sha1($d)); }
}
}
global $aws_secret;
if (strlen($aws_secret) == 40)
$aws_secret = $aws_secret.str_repeat(chr(0), 24);
$ipad = str_repeat(chr(0x36), 64);
$opad = str_repeat(chr(0x5c), 64);
$hmac = binsha1(($aws_secret^$opad).binsha1(($aws_secret^$ipad).$stringToSign));
return base64_encode($hmac);
}
// hmac-sha1 code END
?>
I would suggest using the official AWS SDK for PHP, because it has all of the request signing and handling logic implemented for you. Here is an article by one of the SDK's developers that is relevant to what you are doing: Streaming Amazon S3 Objects From a Web Server
Infact if you just need to see the number of downloads, you can achieve this without running yourown server with php.
This info is already available in the S3 bucket logs, if you enable. This will be more accurate, since the in the PHP approach there is no way to track download, if the user take the S3 link directly and share/download.
These logs are little difficult to parse though, but the services like https://qloudstat.com and http://www.s3stat.com/ help here.
Another point: Downloads will be considerably faster, if you enable CDN - Cloudfront in front of the S3 bucket.
I'm trying to send a huge amount of data using SSL/TLS connection in PHP. It works pretty well if the data chunk isn't very big or if I don't use TLS, but what I need (near 2MiB), the fwrite function shows the warning:
Warning: fwrite(): SSL operation failed with code 1. OpenSSL Error messages: error: 1409F07F: SSL routines: SSL3_WRITE_PENDING: bad write retry
The relevant code I'm using to connect clients:
$cntxt = stream_context_create(array('ssl' => array('local_cert' => 'certificate.pem')));
$server = stream_socket_server('tls://127.0.0.1:8080', $errno, $errstr, STREAM_SERVER_BIND|STREAM_SERVER_LISTEN, $cntxt);
// Wait for client connection //
$client = stream_socket_accept($server);
// Use non-blocking socket to allow answering many clients at a time
stream_set_blocking($client, 0);
$clients[] = $client;
When sending data, it's append to a buffer and, from time to time, this function is called for each client and linked buffer:
function trySend($client, &$buffer) {
if (strlen($buffer)) {
$len = fwrite($client, $buffer);
$buffer = substr($buffer, $len);
}
}
As I said, my code works for small ammount of data or for normal (non-TLS) connections. I've searched for this error and found http://www.openssl.org/docs/ssl/SSL_write.html:
SSL_write() will only return with success, when the complete contents of buf of length num has been written. This default behaviour can be changed with the SSL_MODE_ENABLE_PARTIAL_WRITE option of SSL_CTX_set_mode(3). When this flag is set, SSL_write() will also return with success, when a partial write has been successfully completed. In this case the SSL_write() operation is considered completed. The bytes are sent and a new SSL_write() operation with a new buffer (with the already sent bytes removed) must be started. A partial write is performed with the size of a message block, which is 16kB for SSLv3/TLSv1.
But how can I do this in PHP?
Any help appreciated :)
I have found I can get around this problem by restricting the length of the string passed to fwrite() to 8192, which prevents fwrite() warning.
So for the code in your example I would try changing the substr() call to:
$buffer = substr($buffer, $len, 8192);
The solution is:
$msg = chr(0) . pack('n', 32) . pack('H*', $deviceToken) . pack('n', strlen($payload)) . $payload;
try {
$result = fwrite($fp, $msg, strlen($msg));
}
catch (Exception $ex) {
sleep(1); //sleep for 5 seconds
$result = fwrite($fp, $msg, strlen($msg));
}