hi guys i have problem to get images with different parts in names, let me to explain more.
in directory i have many pictures with this format in name : 1-[1-9].jpg or 2-[1-9].jpg or ...
for example names can be 1-5.jpg or 1-14.jpg or 2-3.jpg so i dont know what is true way to get my files!!
this is an example what i want:
function get_data($url)
{
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,false);
curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$diff = [1-9];
$html = get_data('http://mysite.info/screenshots/1-' . $diff. '.jpeg' );
If it is one of your server, you should figure out what is the pattern of your image file names, for example, x-y.jpeg, where x is from 1 to 100, and y is from 1 to 9. Then process with this code:
foreach (range(1, 100) as $x) {
foreach (range(1, 9) as $y) {
$html = get_data('http://mysite.info/screenshots/'.$x.'-'.$y.'.jpeg' );
}
}
You cannot use wildcards or similar approaches in http requests. Such a request has to query exactly one specific object, except if you have a processing logic in the server side.
In your example you use urls requesting a specific image file. Using that approach you request exactly one single and specific image. You'd have to make a single request for each possible name pattern to retrieve all available image files matching your desired pattern.
Another approach would be to not request a specific image in your http request, but a processing logic, typically a script on the server side. Such script can accept things like a wildcard pattern (or similar) and return matches, even multiple images at once. But this obviously means that you have control over that server system, can implement the logic on that system. So it has to be your system you query the images from.
Related
I'm currently running a personal file upload site, which roughly does the following;
Select files to upload through the website
PHP uploads and zips the files together
The zipped file gets sent via cURL to a remote storage server
A URL is presented to share the files
I'm having an issue with step 3. I currently have a progress bar while the file uploads to the website and I'd like another percentage to show the cURL transfer to the storage server. I've seen that you that can use CURLOPT_PROGRESSFUNCTION - but I can not get it working, after a lot of searching.
My current code is as below;
// Prepare the POST data
$data = array(
'file' => new CurlFile('/path/to/local.zip', 'application/zip', 'uploaded_archive.zip'),
'upload_folder' => 'folder_name'
);
// Send the POST
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://example.com/upload.php");
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0");
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_NOPROGRESS, false);
curl_setopt($ch, CURLOPT_PROGRESSFUNCTION, 'progressCall');
curl_setopt($ch, CURLOPT_BUFFERSIZE, 128);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 160);
curl_setopt($ch, CURLOPT_TIMEOUT, 160);
$res = curl_exec($ch);
curl_close($ch);
This calls "progressCall" with the progress, which looks like the below to test;
// Function to save the output of progress from CURLOPT_PROGRESSFUNCTION
function progressCall($ch, $dltotal, $dlnow, $uptotal, $upnow) {
$progress = "$ch/$dltotal/$dlnow/$uptotal/$upnow";
file_put_contents('tmp/curlupload.txt', 'PERC: ' . $progress . "\n", FILE_APPEND);
}
The problem is, I get the output below;
https://gist.github.com/ialexpw/6160e32495d857cc0d8984373f3808ae
PERC: Resource id #15/0/0/0/65532
PERC: Resource id #15/0/0/0/65532
PERC: Resource id #15/0/0/0/131064
PERC: Resource id #15/0/0/0/131064
PERC: Resource id #15/0/0/0/196596
This seems to show the uploaded amount going up (which is OK), but has 0 for the total upload amount. I've tried a lot of example online, but I can not work out why the total amount to be uploaded is always 0. I tried setting a Content-Length, although it's the same.
Any help would be appreciated if something is wrong with the above.
Thanks!
Hello sorry for the response 2 years after, but just working on it !
Possible for Upload, but as mentionned we don't have in this example the total file size.
The easiest way if possible is to take the file size on the callback
function, with a global for example :
function progressCall($ch, $dltotal, $dlnow, $uptotal, $upnow) {
// You will get this before with | $fsize = filesize($thefile);
global $fsize;
$percent = round($upnow / $fsize * 100);
echo $percent . ' % downloaded';
}
I have a repetitive task that I do daily. Log in to a web portal, click a link that pops open a new window, and then click a button to download an Excel spreadsheet. It's a time consuming task that I would like to automate.
I've been doing some research with PHP and cUrl, and while it seems like it should be possible, I haven't found any good examples. Has anyone ever done something like this, or do you know of any tools that are better suited for it?
Are you familiar with the basics of HTTP requests? Like, do you know the difference between a POST and a GET request? If what you're doing amounts to nothing more than GET requests, then it's actually super simple and you don't need to use cURL at all. But if "clicking a button" means submitting a POST form, then you will need cURL.
One way to check this is by using a tool such as Live HTTP Headers and watching what requests happen when you click on your links/buttons. It's up to you to figure out which variables need to get passed along with each request and which URLs you need to use.
But assuming that there is at least one POST request, here's a basic script that will post data and get back whatever HTML is returned.
<?php
if ( $ch = curl_init() ) {
$data = 'field1=' . urlencode('somevalue');
$data .= '&field2[]=' . urlencode('someothervalue');
$url = 'http://www.website.com/path/to/post.asp';
$userAgent = 'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)';
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
$html = curl_exec($ch);
curl_close($ch);
} else {
$html = false;
}
// write code here to look through $html for
// the link to download your excel file
?>
try this >>>
$ch = curl_init();
$csrf_token = $this->getCSRFToken($ch);// this function to get csrf token from website if you need it
$ch = $this->signIn($ch, $csrf_token);//signin function you must do it and return channel
curl_setopt($ch, CURLOPT_HTTPGET, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 300);// if file large
curl_setopt($ch, CURLOPT_URL, "https://your-URL/anything");
$return=curl_exec($ch);
// the important part
$destination ="files.xlsx";
if (file_exists( $destination)) {
unlink( $destination);
}
$file=fopen($destination,"w+");
fputs($file,$return);
if(fclose($file))
{
echo "downloaded";
}
curl_close($ch);
this was working up until quite recently and i cannot seem to crack the case.
if you manually visit the url hit against in the script, the results are there..but if i do it in the code, i am having an issue.
you can see in my output test that i am no longer getting any output...
any ideas?
<?
//$ticker=urldecode($_GET["ticker"]);
$ticker='HYG~FBT~';
echo $ticker;
$tickerArray=preg_split("/\~/",$ticker);
// create curl resource
$ch = curl_init();
// set urlm
curl_setopt($ch, CURLOPT_URL, "http://www.batstrading.com/market_data/symbol_data/csv/");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$a='';
$output = curl_exec($ch);
echo "<br><br>OUTPUT TEST: ".($output);
$lineCt=0;
$spaceCt=0;
$splitOutput=preg_split("[\n|\r]",$output);
for($ii=0;$ii<sizeof($tickerArray);$ii++){
$i=0;
$matchSplit[$ii]=-1;
while($i<sizeof($splitOutput) && $matchSplit[$ii]==-1){
$splitOutput2=preg_split("/\,/",$splitOutput[$i]);
if($i>0){
if(strcasecmp($splitOutput2[0],strtoupper($tickerArray[$ii]))==0){
$matchSplit[$ii]=$splitOutput[$i]."#";
}
}
$i++;
}
if($matchSplit[$ii]==-1){
echo "notFound#";
}else{
echo $matchSplit[$ii];
}
}
//echo ($output);
curl_close($ch);
?>
I added a user agent to your script and it seems to work fine here:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.batstrading.com/market_data/symbol_data/csv/");
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15); //time out of 15 seconds
$output = curl_exec($ch);
curl_close($ch);
//Then your output paring code
The output I get:
HYG~FBT~
OUTPUT TEST: Name,Volume,Ask Size,Ask Price,Bid Size,Bid Price,Last Price,Shares Matched,Shares Routed SPY,35641091,0,0.0,0,0.0,134.38,34256509,1384582 BAC,22100508,0,0.0,0,0.0,7.78,20407265,1693243 QQQ,12085707,0,0.0,0,0.0,62.65,11703725,381982 XLF,11642347,0,0.0,0,0.0,14.47,11429581,212766 VXX,9838310,0,0.0,0,0.0,28.2,9525266,313044 EEM,9711498,0,0.0,0,0.0,43.28,9240820,470678 IWM,8272528,0,0.0,0,0.0,81.19,7930349,342179 AAPL,6145951,0,0.0,0,0.0,498.24,4792854,1353097
It is also good practice to close the CURL connection once you are done. I believe that might also play a part in your issues.
If you are still getting issues, check to see if the server the script runs on can access that site.
Update
Upon further investigation, here's what I believe is the root of the problem.
The problem lies with the provider of the CSV file. Perhaps due to some issues on their end, the CSV are generated, but only contain the headers. There were instances where there were indeed data in there.
The data is only avaliable during set hours of the day.
In any case, fetching the empty file = the parser would print #notFound, leading us to assume that there was an issue with CURL.
So my suggestion is to add some further checking to the script to check whether the CSV file actually contains any data at all and is not a file containing just the headings.
Finally, setting a timeout for CURL should fix it as the CSV takes a while to be generated by the provider.
The problem is, I get some parts of the contents but did not get the user's reviews. by Firebug I saw contents but when I checked the source codes NO contents inside HTML tags / no same HTML tags. Here is my code:
<?php
//Headers
include('simple_html_dom.php');
function getPage($page, $redirect = 0, $cookie_file = '')
{
$ch = curl_init();
$headers = array("Content-type: application/json");
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
if($redirect)
{
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
}
curl_setopt($ch, CURLOPT_URL, $page);
if($cookie_file != '') {
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
}
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6');
$return = curl_exec($ch); //Mozilla/4.0 (compatible;)
curl_close($ch);
return $return;
}//EO Fn
//Source
$url = 'http://www.vitals.com/doctor/profile/1982660171/reviews/1982660171';
//Parsing ...
$contents = getPage($url, 1, 'cookies.txt');
$html = str_get_html($contents);
//Output
echo $html->outertext;
?>
Can anyone please help me - what I should do to get the whole page so that I can grab reviews?enter code here
They're just stored as JSON in a <script> block towards the top of the page. Parse it out with RegEx or Simple HTML DOM and run it through json_decode.
var json = {"provider":{"id":"1982660171","display_name":"Stephen R Guy, MD","last_name":"Guy","first_name":"Stephen","middle_name":"Russell","master_name":"Stephen_Guy","degree_types":"MD","familiar_name":"Stephen","years_experience":"27","birth_year":"1956","birth_month":"5","birth_day":"23","gender":"M","is_limited":"false","url_deep":"http:\/\/www.vitals.com\/doctor\/profile\/1982660171\/Stephen_Guy","url_public":"http:\/\/www.vitals.com\/doctors\/Dr_Stephen_Guy.html","status_code":"A","client_ids":"1","quality_indicator_set":[{"type":"quality-indicator\/consumer-feedback","count":"2","suboverall_set":[{"name_short":"Promptness","overall":"3"},{"name_short":"Courteous Staff","overall":"4"},{"name_short":"Bedside Manner","overall":"4"},{"name_short":"Spends Time with Me","overall":"4"},{"name_short":"Follow Up","overall":"4"}],"name":"Consumer Reviews","overall":"4.0","measure_set":[{"feedback_response_id":"1756185","input_source_ids":"{0}","date":"1301544000","value":"4","scale":{"best":"1","worst":"4"},"review":{"type":"review\/consumer","comment":"I will never birth with another dr. Granted that's not saying much as I don't like dr's but I actually find him as valuable as the midwives who I adore. I liked Horlacher but when Kitty left I followed the midwives and then followed again....Dr. Guy is GREAT. I honestly don't know who I'd rather support me at my birth; Margie and Lisa or Dr. Guy. ....I wonder if I can just get all of them.Guy's great. Know what you want. Tell him. Be strong and he'll support you.I give him 10 stars. Oh...my baby's 3 years old now. He's GREAT! ","date":"1301544000"},"sub_measure":[{"name":"Waiting time during a visit","name_short":"Promptness","value":"3","scale":{"best":"4","worst":"1"}},{"name":"Courtesy and professionalism of office staff ","name_short":"Courteous Staff","value":"4","scale":{"best":"4","worst":"1"}},{"name":"Bedside manner (caring)","name_short":"Bedside Manner","value":"4","scale":{"best":"4","worst":"1"}},{"name":"Spending enough time with me","name_short":"Spends Time with Me","value":"4","scale":{"best":"4","worst":"1"}},{"name":"Following up as needed after my visit","name_short":"Follow Up","value":"4","scale":{"best":"4","worst":"1"}}]},{"feedback_response_id":"420734","input_source_ids":"{76}","link":"http:\/\/local.yahoo.com\/info-15826842-guy-stephen-r-md-university-women-s-health-center-dayton","date":"1142398800","value":"4","scale":{"best":"1","worst":"4"},"review":{"type":"review\/consumer","comment":"Excellent Doctor: I really like going to this office. They are truely down to earth people and talk my \"non-medical\" language. I have been using thier office since 1997 and they have seen me through 2 premature pregnancies!","date":"1142398800"}}],"wait_time":"50"}]}};
But again, make sure you have permissions to do this...
I'm having a little trouble updating backgrounds via Twitter's API.
$target_url = "http://www.google.com/logos/11th_birthday.gif";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Expect:'));
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$html = curl_exec($ch);
$content = $to->OAuthRequest('http://twitter.com/account/update_profile_background_image.xml', array('profile_background_image_url' => $html), 'POST');
When I try to pull the raw data via cURL or file_get_contents, I get this...
Expectation Failed The expectation given in the Expect request-header
field could not be met by this server.
The client sent
Expect: 100-continue but we only allow the 100-continue expectation.
OK, you can't direct Twitter to a URL, it won't accept that. Looking around a bit I've found that the best way is to download the image to the local server and then pass that over to Twitter almost like a form upload.
Try the following code, and let me know what you get.
// The URL from an external (or internal) server we want to grab
$url = 'http://www.google.com/logos/11th_birthday.gif';
// We need to grab the file name of this, unless you want to create your own
$filename = basename($url);
// This is where we'll be saving our new file to. Replace LOCALPATH with the path you would like to save the file to, i.e. www/home/content/my_directory/
$newfilename = 'LOCALPATH' . $filename;
// Copy it over, PHP will handle the overheads.
copy($url, $newfilename);
// Now it's OAuth time... fingers crossed!
$content = $to->OAuthRequest('http://twitter.com/account/update_profile_background_image.xml', array('profile_background_image_url' => $newfilename), 'POST');
// Echo something so you know it went through
print "done";
Well, given the error message, it sounds like you should load the URL's contents yourself, and post the data directly. Have you tried that?