I need to fetch data from this web page Sender score
.I try to use cURL but it renders white page.
Here is my code :
$ch = curl_init();
$keyword = "an-example.com";
curl_setopt($ch, CURLOPT_URL, 'https://www.senderscore.org/lookup.php?lookup='.$keyword.'&validLookup=true');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($ch);
print_r($data);
curl_close($ch);
any idea ?
Regards.
You're getting a blank page because of the captcha which is needed to fill in. Perhaps senderscore has an API which you can use? Or maybe there's another website available doing the same thing. I thought this was about scoring email statusus right? Then maybe this site will help you out: http://www.reputationauthority.org/domain_lookup.php?ip=somedomain.nl&Submit.x=0&Submit.y=0&Submit=Search
I can use this site without the need of captcha or any other bot interference.
Related
I am trying to use php curl function to log in to a https webpage "https://portal.opalonline.co.uk/Home/PortalCore/SignIn/SignIn.aspx"
but I have run out of ideas how I can post values to this particular page (username, password) and 'press 'sign in'.
$postfields = array('ctl00_MasterContentContentPane_Signin1_userID_txt'=>'email#address.com',
'ctl00_MasterContentContentPane_Signin1_password_txt'=>'somepassword123');
/* LOG IN TO TalkTalk ACCOUNT */
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://portal.opalonline.co.uk/Home/PortalCore/SignIn/SignIn.aspx?");
curl_setopt($ch, CURLOPT_HEADER, false);
// curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
// curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE_FILE);
// curl_setopt($ch, CURLOPT_COOKIE, COOKIE_FILE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
curl_setopt($ch, CURLOPT_POST, 1);
var_dump($ch);
$string_exec = curl_exec($ch);
var_dump($string_exec);
I can not even display the page with var_dump :( . Ideas / suggestions much appreciated
First, I don't think you can do the 'array' thing like that as that will make PHP/CURL create multipart formpost instead, and this is not such a form. Provide the data in "name=value&name2=value2" style.
Then, make sure you also submit all the hidden fields in the form. There are at least four of them. One of them is set by the HTML to a long value that you need to extract and set, and there is also some javascript magic that sets some of the others. You probably need to use your browser's networking tool to snoop on what exactly your browser sends to be able to mimic that perfectly.
The login page sets cookies and you probably need to pass those cookies on when you submit the login form. So you need to first fetch (GET) the login form page to get the cookies, then file the login POST.
With that fixed, you should be closer. If that isn't all that takes, then continue comparing the browser's request with what your request is sending and make sure they are as similar as possible.
Open the website in google chrome, open the console, to go the network tab.
Login to the website. You should see the request in the network tab. Do a right click on it, select "copy as cURL". It will give you a command line, that will help you understand what you need.
I'm currently using this plugin http://wordpress.org/extend/plugins/repress/ which basically makes my website a proxy so that users can access censored websites like this
www.mywebsite.com/proxy/www.cnn.com
The plugin works well enough but when it comes to absolute links the plugin doesn't parse it properly and the link is still blocked. That plugin development has stopped. So I need to write my own script. I've been searching everywhere and reading up on the tutorials I can find but none specifically helps me in regards to this.
I know how to use php curl to fetch a website and echo it on a blank page. What I don't know is how to set a proxy script to work like the above example where users can type
www.mywebsite.com
followed by
/proxy.php
then their target website
/www.cnn.com
Currently I have this set up:
<?php
$url = 'http://www.cnn.com';
$proxy_port = 80;
$proxy = '92.105.140.115';
$timeout = 0;
$referer = 'http://www.mydomain.com'
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy_port);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYTYPE, 'HTTP');
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);*/
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_REFERER, $referer);
$data = curl_exec($ch);
curl_close($ch);
echo $data;
?>
This pulls the home page but no css or images are retrieved. Likewise all relative links are broken. I have no idea how to apply the proxy_port and proxy variables. I tried
92.105.140.115:80/www.cnn.com
but that doesn't work. I don't quite fully understand this code either since I found it on an example site.
Any answer or links for tutorials is greatly welcome.
Thank You!
To have a completely functioning proxy isn't that simple. There are many such projects already available. Give any a shot:
http://www.surrogafier.info/
https://github.com/Alexxz/Simple-php-proxy-script
http://www.glype.com/
Have fun!
you can not just echo in a page the result of a curl's fetch website because the browsers will interpret the Uris bad, you need that when the user click on a link he goes to your proxy site not to the original site, so you can't just print with echo, you need to edit manual every link in that fetched page before print it to the users, i have a full functional proxy made by php en p.listascuba.com you cant try it.
contact me for more info
I have following URL
http://www.davesinclairstpeters.com/auto2_inventorylist?i=37647&c=12452&npg=1&ns=50&echo=2
I want to retrieve content of this url using curl but everytime I make this request it is showing me error, as it is not passing required parameters
Below is my code
$ch = curl_init(); // start CURL
curl_setopt($ch, CURLOPT_URL, $json_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPGET, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
$response = curl_exec($ch);
That page doesn't give any information stating that the information isn't being passed properly. In fact, it tells you that the information has been recieved - by viewing the source, you can see:
<!--
javax.servlet.forward.request_uri = /auto2_inventorylist
...
javax.servlet.forward.servlet_path = /auto2_inventorylist
...
javax.servlet.forward.query_string = i=37647&c=12452&npg=1&ns=50&echo=2
-->
Which tells you the information has infact been recieved.
Therefore, it's no problem with your code, but with the website itself. You should make sure the URL you are using is valid, or contact that website to get more information.
With regards to your code itself - the curl_setopt($ch, CURLOPT_HTTPGET, true); isn't necessary, as this is already set by default, and you can also pass the URL as an argument of the curl_init function. Doesn't impact performance, but makes for neater code.
$ch = curl_init($json_url); // start CURL
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
$response = curl_exec($ch);
You code is perfectly fine and if there's something wrong returned, simply paste this URL to your web browser and check the result. In this case website simply failed for some reasons. There's nothing you can do about that as problem is NOT on your side.
This URL yields a page of cars with links to more cars. Looks like the URL you're starting with is old, or has some sort of expiration factor that's not obvious.
Not knowing which sort of filtering parameters you're shooting for.. hard to say what else my be wrong, other than your starting URL be bad.
working url:
http://www.davesinclairlincolnstpeters.com/all-inventory/index.htm?listingConfigId=auto-new%2Cauto-used&compositeType=&year=&make=&start=0&sort=&facetbrowse=true&quick=true&preserveSelectsOnBack=true&searchLinkText=SEARCH&showInvTotals=false&showRadius=false&showReset=true&showSubmit=true&facetbrowseGridUnit=BLANK&showSelections=true&dependencies=model%3Amake%2Ccity%3Aprovince%2Ccity%3Astate&suppressAllConditions=false
Alright I am practicing using cURL to login to different webservices. For this pariticular try, I am doing YouTube. This was a pretty big challenge, but I finally got it...almost.
After posting the HUGE amount of post data tags to the login page, you get sent to a checkCookie kind of thing. The checkcookie page verifies that you have the right cookies and then redirects you to youtube.com (logged into your account) This is whats messing me up.
When I have this:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
I get the source of the checkcookie page. It simply says "Document Moved". This isn't what I want, I want to get the source of me being logged in stored into a variable. So I tried something else...
When I use this setup:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
I get sent to the youtube page and I am logged in! It seems to work! Except...I don't want to be redirected off my script. My goal is to get the source of youtube.com with me logged in.
In other words, the cURL is logging in just fine, the problem is I literally get redirected to YouTube. Which I don't want.
Any suggestions? It's like I need to follow the redirects...but not be redirected.
Thanks for any help!
try this:
<?php
function getURL($url) {
$curlHandle = curl_init(); // init curl
curl_setopt($curlHandle, CURLOPT_URL, $url); // set the url to fetch
curl_setopt($curlHandle, CURLOPT_HEADER, 0);
curl_setopt($curlHandle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curlHandle, CURLOPT_TIMEOUT,30);
curl_setopt($curlHandle, CURLOPT_POST, 0);
$content = curl_exec($curlHandle);
curl_close($curlHandle);
return $content;
}
?>
Looks like you are getting redirected because you echo the curl_exec, in which there is a javascript code for redirection. Since you are likely requesting that from your browser, it runs the code and redirects you to YouTube. If that's the case, obvious solution would be to turn off JS or filter what you echo to yourself
Can you help me get this content spinning API working? It was wrote to work with C# but is there a way I can get it to work using PHP? I've been trying to post to the URL stated on that page using cURL, but all I'm getting back is a blank page. Here's the code I'm using:
$url = "http://api.spinnerchief.com/apikey=YourAPIKey&username=YourUsername&password=YourPassword";
// Some content to POST
$post_fields = "SpinnerChief is totally free. Not a 'lite' or 'trial' version, but a 100% full version and totally free. SpinnerChief may be free, but it is also one of the most powerful content creation software tools available. It's huge user-defined thesaurus means that its sysnonyms are words that YOU would normally use in real life, not some stuffy dictionary definition. And there's more...SpinnerChief can only get better, because we listen to our users, We take onboard your ideas and suggestions, and we update SpinnerChief so that it becomes the software YOU want!";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PORT, 9001);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
Can anyone see something wrong I'm doing? Thanks a lot for the help.
The value for CURLOPT_POST should be 1, and the posted data should be set with CURLOPT_POSTFIELDS.