I am using CURL to load a web page and return it onto a page on my server. However when the page is returned there are no images showing as they are linked using href="/image.png" etc.... Is there a way using CURL to add the url to any link that starts href="/
function pull_html($url, $device)
{
$ch = curl_init();
if($device == 'iPhone'){
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25');
}elseif($device == 'iPad'){
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25');
}
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch,CURLOPT_BINARYTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,10);
return curl_exec($ch);
curl_close($ch);
}
Use a simple str_replace()
return str_replace('href="/', 'href="'.$url.'/', curl_exec($ch));
http://php.net/manual/en/function.str-replace.php
Related
I once used a PHP Class Library to connect to Google Voice to send SMS Text Messages. The call would work something like this:
$gv = new GoogleVoice("GmailAccount", "GmailPassword");
$gv->sms("PhoneNumber", "TextMsg");
It worked flawlessly until recently, as of 4/20/2015, Google stopped supporting old methods of logging in to Google account. So my script stopped working giving 500 error. Google says you have to use OAuth 2.0 to authenticate however I haven't found any examples online on how to accomplish this with Google Voice. The code is below, I have not written this, please let me know how to adjust the code to use Google's OAuth System.
/*
Version 0.2
License This code is released under the MIT Open Source License. Feel free to do whatever you want with it.
Author lostleon#gmail.com, http://www.lostleon.com/
LastUpdate 05/28/2010
Usage:
*/
class GoogleVoice
{
public $username;
public $password;
public $status;
private $lastURL;
private $login_auth;
private $inboxURL = 'https://www.google.com/voice/m/';
private $loginURL = 'https://www.google.com/accounts/ClientLogin';
private $smsURL = 'https://www.google.com/voice/m/sendsms';
public function __construct($username, $password)
{
$this->username = $username;
$this->password = $password;
}
public function getLoginAuth()
{
$login_param = "accountType=GOOGLE&Email={$this->username}&Passwd={$this->password}&service=grandcentral&source=com.lostleon.GoogleVoiceTool";
$ch = curl_init($this->loginURL);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (iPhone; U; CPU iPhone OS 2_2_1 like Mac OS X; en-us) AppleWebKit/525.18.1 (KHTML, like Gecko) Version/3.1.1 Mobile/5H11 Safari/525.20");
curl_setopt($ch, CURLOPT_REFERER, $this->lastURL);
curl_setopt($ch, CURLOPT_POST, "application/x-www-form-urlencoded");
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $login_param);
$html = curl_exec($ch);
$this->lastURL = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
$this->login_auth = $this->match('/Auth=([A-z0-9_-]+)/', $html, 1);
return $this->login_auth;
}
public function get_rnr_se()
{
$this->getLoginAuth();
$ch = curl_init($this->inboxURL);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$headers = array("Authorization: GoogleLogin auth=".$this->login_auth, 'User-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 2_2_1 like Mac OS X; en-us) AppleWebKit/525.18.1 (KHTML, like Gecko) Version/3.1.1 Mobile/5H11 Safari/525.20');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$html = curl_exec($ch);
$this->lastURL = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
$_rnr_se = $this->match('!<input.*?name="_rnr_se".*?value="(.*?)"!ms', $html, 1);
return $_rnr_se;
}
public function sms($to_phonenumber, $smstxt)
{
$_rnr_se = $this->get_rnr_se();
$sms_param = "id=&c=&number=".urlencode($to_phonenumber)."&smstext=".urlencode($smstxt)."&_rnr_se=".urlencode($_rnr_se);
$ch = curl_init($this->smsURL);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$headers = array("Authorization: GoogleLogin auth=".$this->login_auth, 'User-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 2_2_1 like Mac OS X; en-us) AppleWebKit/525.18.1 (KHTML, like Gecko) Version/3.1.1 Mobile/5H11 Safari/525.20');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_REFERER, $this->lastURL);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $sms_param);
$this->status = curl_exec($ch);
$this->lastURL = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
return $this->status;
}
private function match($regex, $str, $out_ary = 0)
{
return preg_match($regex, $str, $match) == 1 ? $match[$out_ary] : false;
}
}
I will send you to the next answer:
https://stackoverflow.com/a/4131915/2992810
unfortunate google voice had changed their API so you can not use it anymore.
https://github.com/aaronpk/Google-Voice-PHP-API (look at the comments in the head)
Google Voice is not an open API, so they are no maintaining it.
Sorry to tell you but in my own experience today SMS services are so cheap that it will cost you less to actually buy service license than fight with Google and their constant API changes, your website can always go down because of such change.
Thinks about your time as a resource, spending your time will cost you more!
GV4J is a Java library that is able to login to Google Voice, so it might be a good reference for updating your PHP code to be able to authenticate.
I'm writing a php script that deals with page processing via cURL, so I have a function to get and return pages by URL
function get_url($Url){
if (!function_exists('curl_init')){
die('Sorry cURL is not installed!');
}
set_time_limit (20);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $Url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Cookie: age_gate_birthday=19901101"));
curl_setopt($ch, CURLOPT_REFERER, "http://www.facebook.com");
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
return $output;
}
echoing $output in this function always returns a string of HTML, however if I call on this function in another function
function get_vid ($sql, $url) {
$data = get_url($url);
...
the returned value is an empty string, despite the fact that $output had value when get_url() was doing its thing.
Weirdly enough, the error only exists with specific URLs, but works fine with others.
Thank you for trying to help!
UPDATE: It seems CURL returns FALSE randomly on specific links, which seems to be a culprit of this issue, however curl_error is empty, so I'm unable to identify the cause of this.
I think it's because you get a http redirect.
Try to check http code like this :
if (curl_getinfo($ch,CURLINFO_HTTP_CODE) == 302) {
// Manage http redirect here
}
I'm using CURL to process the data readings from the URL. Then I want to get the movie_id of each link and do the next process with the URL that will receive the movie_id so that I get the data in more detail.
My cURL code :
<?php
$url="http://m.21cineplex.com/";
function bacaHTML($url){
$data = curl_init();
curl_setopt($data, CURLOPT_URL, $url);
curl_setopt($data, CURLOPT_HEADER, 1);
curl_setopt($data, CURLOPT_USERAGENT, "Mozilla/5.0 (BlackBerry; U; BlackBerry 9850; en-US) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.0.0.115 Mobile Safari/534.11+");
curl_setopt($data,CURLOPT_COOKIEJAR,'cookie.txt');
curl_setopt($data,CURLOPT_COOKIEFILE,'cookie.txt');
curl_setopt($data, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($data, CURLOPT_RETURNTRANSFER, 1);
$hasil = curl_exec($data);
curl_close($data);
return $hasil;
}
$html= bacaHTML($url);
$html=str_replace(array("\r","\t","\n"),"",$html);
preg_match_all("/<ol id=\"menu_ol\">(.*?)<\/ol>/",$html,$raw);
echo "<p>Now Playing</p>".$raw[0][0]."<p>Coming Soon".$raw[0][1]."Movie News".$raw[0][2]
How do I separate the ID so that I can make the next process?
Ex : Next process with ID
I can access a web page when I type its URL in my browser. However, while using curl to access the details of that web page, I get the message on screen
User information is disabled.
This operation cannot be accepted. User certification is invalid or date expired.
Update page.
I can access the details of my network printer(Canon IR3570) by typing in the IP of that printer in my browser. This opens up the remote UI. However, it doesn't seem to work with curl.
This is my code in PHP curl
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"URL");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20100101 Firefox/13.0.1');
$result= curl_exec($ch);
echo $result;
curl_close ($ch);
What could be the reason for such a message?
Where's UserAgent set? They may filtrate requests.
try adding:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION ,1);
curl_setopt($ch, CURLOPT_USERAGENT, "User-Agent Mozilla/5.0");
curl_setopt($ch, CURLOPT_HEADER ,1); // DO NOT RETURN HTTP HEADERS
This isn't CURL but it works in Visual Basic 2012 - one of the headers solved it for me
Sub Main()
Dim web_client As New System.Net.WebClient
Dim baseDate As DateTime = New DateTime(1970, 1, 1)
Dim diff As TimeSpan = DateTime.Now - baseDate
web_client.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
web_client.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3")
web_client.Headers.Add("Accept-Encoding", "gzip, deflate, sdch")
web_client.Headers.Add("Accept-Language", "en-US,en;q=0.8")
web_client.Headers.Add("Cookie", "iR = 1711753823")
web_client.Headers.Add("Host", "172.23.100.14")
web_client.Headers.Add("Referer", "http://172.23.100.14/jlp.cgi?Flag=Html_Data&LogType=0&Dummy=" & diff.Milliseconds)
web_client.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31")
web_client.DownloadFile("http://172.23.100.14/pprint.csv?Flag=Csv_Data&LogType=0&Dummy=" & diff.Milliseconds, "P:\Transfer\mstavers\printlogs\" & Format(Now, "yyyy-MM-dd-hh-mm-ss") & ".csv")
End Sub
How can I scrape a site using a User-Agent for Ipad?
I have this code below using curl in PHP which outputs the source but can't find the tags still. On Ipad or Safari browser using an Ipad User-Agent, the tags displays when the site is loaded.
Thanks!
<?php
$useragent= "Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10')";
$ch = curl_init ("http://www.cbsnews.com/video/watch/?id=7370279n&tag=mg;mostpopvideo");
curl_setopt ($ch, CURLOPT_USERAGENT, $useragent); // set user agent
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
// curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
echo $output = curl_exec ($ch);
curl_close($ch);
?>
Try using curl from the command line, with a perl script such as this:
my $ua = "Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10";
my $curl = "curl -A '$ua'";
my $server = "http://www.cbsnews.com";
my $startpage = "$server/video/watch/?id=7370279n&tag=mg;mostpopvideo";
my $path = "/path/to/download/to";
open(f, "$curl -L $startpage |") or die "Cannot open website: $!";
while (<f>)
{
if (/<a\s+[^>]*href=\"$server\/([^\"\/])*\"/)
{
my $file = $2;
system("$curl -e $startpage $server/$file > $path/$file");
next;
}
if (/<a\s+[^>]*href=\"$server\/([^\"]+)\/([^\"\/])*\"/)
{
my $folder = $1;
my $file = "$folder/$2";
system("mkdir -p $path/$folder");
system("$curl -e $startpage $server/$file > $path/$file");
next;
}
}
close(f);