I want to set cookie throw curl request. I used this code but the requested URL return You must enable Javascript and accept cookies. what Im doing wrong here?
cookie.txt file is 0644 permission
$curl = curl_init();
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl, CURLOPT_HEADER, false);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_URL, $base);
curl_setopt ($curl, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($curl, CURLOPT_REFERER, $base);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$str = curl_exec($curl);
curl_close($curl);
The problem is not your code, it's that the page you're trying to access works only by executing javascript. CURL doesn't support that, it just downloads the HTML code of the page, but doesn't execute any javascript.
If you're in the need of retrieving information from a website that needs to execute javascript, you need to rely on solutions that provide headless browsers, like Selenium
Related
I'm trying to visualize a website speed analysis. As a headless browser I use Splash 3.2 ... unfortunately I have problems getting a correct har file.
The first request looks good, after the second request I get only requests that were not cached.
I tried to empty the cache with a post request to the _gc endpoint, unfortunately without success.
My curl requests:
$url = 'http://localhost:8050/render.har?url=' . esc_url( $url ) .'&response_body=1&wait=5&timeout=10';
$cache_url = 'http://localhost:8050/_gc';
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $cache_url);
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS,"cached_args_removed=1");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$server_output = curl_exec($curl);
curl_reset($curl);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl, CURLOPT_HEADER, false);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_REFERER, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$data = curl_exec($curl);
curl_close($curl);
the result of _gc request:
{"cached_args_removed": 0, "pyobjects_collected": 1269, "status": "ok"}
Afterwards I tried to start Splash with --disable-browser-caches to get a correct output, but Splash does not cache anything and therefore makes many requests to the same files, if they occur several times.
Is there another way to flush the browser cache before rendering, or should I prefer to use another headless browser (recommendation)?
#Tobias
You mentioned you're using splash version 3.2
I'm the author of PR 821 which introduced --disable-browser-caches , and according to the changelog , this feature landed in splash version 3.3.
So please upgrade to splash version 3.3 and you should be able to use that feature.
Guys, I am currently working on file hosting premium link generator basically it will be a website from where you can get a premium link of uptobox,rapidgator,uploaded.net and other file hosts sites without purchasing the premium account. Basically, We are purchasing the accounts of this website on behalf of the users and offering this service at a low price. So when I was setting up API of direct download link of rapidgator I was able to get that link but I was getting session is over. I was trying to that API via a software, not via manual coding and I am facing this problem
So I have been getting Rapidgator API reference from Tihs Site:- https://gist.github.com/Chak10/f097b77c32a9ce83d05ef3574a30367d
So I am doing the following Thing With My Debugging Software And I am getting success response but when I just open that URL in my browser it shows Session Id Failed.
So Here Are Steps What I am Doing
Sending a post request on https://rapidgator.net/api/user/login with username and data and I am getting this output
{"response":{"session_id":"g8a13f32hr4cbbbo54qdigrcb3","expire_date":1542688501,"traffic_left":"13178268723435"},"response_status":200,"response_details":null}
Now I am sending a get request (I tried Post Request Too But the Same Thing Happened) on this url with session id and URL embedded in URL https://rapidgator.net/api/file/download?sid=&url=
and I am getting this output
{"response":{"url":"http:\/\/pr56.rapidgator.net\/\/?r=download\/index&session_id=uB9st0rVfhX2bNgPrFUri01a9i5xmxan"},"response_status":200,"response_details":null}
When I try to download the file from the Url through my browser It says Invalid Session and sometimes too many open connections error
Link of the error:- https://i.imgur.com/wcZ2Rh7.png
Success Response:- https://i.imgur.com/MqTsB8Q.png
Rapidgator needs its api to be hit three times with different URLs.
$cookie = $working_dir.rand();
$headers = array("header"=>"Referer: https://rapidgator.net");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://rapidgator.net/api/user/login");
curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');
curl_setopt($ch, CURLOPT_POSTFIELDS, "username=email#domain.ext&password=myplaintextpassword");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$result = curl_exec($ch);
curl_close ($ch);
$rapidgator_json = json_decode($result,true);
return array($rapidgator_json['response']['session_id'],$cookie);
http://rapidgator.net/api/user/login (this is the initial login)
Above link gives you a session id that you need. The response is in JSON
Now we need to request a download link that will allow us to download without having to log in to a human input form. So we will use its api to request a download link using the intial session id we got from the 1st url.
$headers = array("header"=>"Referer: http://rapidgator.net/api/user/login");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://rapidgator.net/api/file/download?sid=$rapidgator_session&url=$rapidgator_file");
curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $working_dir.$rapidgator_cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, $working_dir.$rapidgator_cookie);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$result = curl_exec($ch);
curl_close ($ch);
$rapidgator_json = json_decode($result,true);
return array($rapidgator_json['response']['url']);
Basically, we pass the session id Rapidgator gave us assuming you have properly passed a valid account. Then we include the source url you had obtained (Link to file) http://rapidgator.net/api/file/download?sid=$rapidgator_session&url=$rapidgator_file
After that. Rapidgator will return a JSON response with an url that u can use to obtain the file in question. This allows you to use whatever download method you want
as that link is a session url is valid for a short period of time.
$rapidgator_json['response']['url']
All code above is somewhat incomplete. Some extra checks on the json responces for possible errors/limits are recommended. I used functions on my end but this is enough for you to see what you should be doing. Rapidshare API has other data that can be useful in determining if you have gone over your daily quota. How long the session url is going to last and so on.
I am trying to create a simple PHP script that calls two different REST APIs on two different domains. Both services are HTTPS and require authentication. When I do a curl from the terminal, I get the response in JSON for both domains and everything works beautifully:
curl --user “myuser:mypassword” https://www.example.com/rest/api/2/projects
Notice that it's a GET, not a POST.
The strange thing is that when I try the exact same curl commands from my PHP script neither of them works.
This what happens:
The first domain returns an empty JSON array with no errors. Just this: []
The second domain returns this error in JSON:
{"errors":[
{
"context":null,
"message":"You are not permitted to access this resource",
"exceptionName":"com.atlassian.stash.exception.AuthorisationException"
}
]}
Here's what's NOT happening:
No SSL certificate errors or warnings
No authentication errors.
Even if put in a bad username or password, both services will act exactly the same way.
To me what's suspicious is that both domains don't authenticate my calls which makes me think there's either a problem with my code or in the php curl library.
Here's my code:
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $link3);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$encodedAuth = base64_encode($username.":".$password);
curl_setopt($curl, CURLOPT_HTTPHEADER, array("Authentication : Basic ".$encodedAuth));
curl_setopt($curl, CURLOPT_USERPWD, $username.":".$password);
curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION,true);
curl_setopt($curl, CURLINFO_HEADER_OUT, true);
$status_code = curl_getinfo($curl, CURLINFO_HTTP_CODE); //get status code
I know some of it is redundant, but I wanted to try everything and nothing works. Any ideas?
My environment:
OS X Yosemite (10.10.2)
PHP 5.6.6 (I manually upgraded to the latest version as an attempt to make this work)
The current code mixes various approaches and does it in a conflicting way:
the authentication header is named Authorization: and not Authentication:
the CURLOPT_HTTPAUTH, CURLAUTH_ANY tries to negotiate with the server and it shouldn't (neither does the working cURL command line)
Just use:
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $link3);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_USERPWD, $username.":".$password);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION,true);
# for debugging/non-prod
#curl_setopt($curl, CURLOPT_VERBOSE, true);
#curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
$result = curl_exec($curl);
curl_close($curl);
echo $result;
I have a website which i need to implement a login form to other website , which is based on cakephp.
I don't want to change the current security settings on the Cakephp website.
The login is based on the auth component.
Therefore I've pulled the form from the cakephp site using Curl, to keep the token fields.
When i login, i've got a blackhole message 'The request has been black-holed'.
How can i fix that without lowering the security level?
What are the steps to debug this situation.
Thanks
$url = WEB_APP . '/users/login';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_REFERER, '/');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
$markup = strip_tags($result, '<form><input>');
echo $markup;
I've checked and the problem is that the cookie is missing , for sure.
I tried to use sendcookie on the cookie file i got from curl, and it's saved encoded.
Using setrawcookie returns false.
How can I set the cookie correctly?
my curl function cannot follow the redirection of Facebook external link redirector, l.php and i have no idea what's wrong...
here is the code that i'm working on and i commented the lines that i've tried... and an example link (http://www.facebook.com/l.php?u=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGvhFyNLK66A%26feature%3Dyoutu.be&h=xAQFD_3svAQFKxF5YrtqNQ5cL3lIQxo0uaC9PoB7qAvG7Yw&enc=AZPxNZ8P5q54FREC37UC_MP02pwh2DOmsI5bbFkoQm5VUPUlYeNzQASjarRjhTtcedRkmM3mDjK7J_r_P5pRpYhL)
function connect($u) {
$ch= curl_init();
curl_setopt($ch, CURLOPT_URL, $u);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_HEADER, true);
//curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
//curl_setopt($ch, CURLOPT_REFERER, 'spie');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
//curl_setopt($ch, CURLOPT_AUTOREFERER, true );
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
//curl_setopt($ch, CURLOPT_VERBOSE, true);
//curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
$source=curl_exec($ch);
curl_close($ch);
return $source;
}
thank you..
I first thought this was a redirect issue with cURL (safe mode enabled for instance). But it actually comes from how Facebook redirector works.
There is no Location: header, so curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); won't help you with it.
The Facebook link page actually redirects you using Javascript:
<script type="text/javascript">document.location.replace("http:\/\/www.youtube.com\/watch?v=GvhFyNLK66A&feature=youtu.be");</script>
cURL cannot analyse the content of the page nor execute javascript so this is exepcted behaviour. If you still want to do this, you'll need to parse the content of the page, grab the URL from the javascript, and issue an new cURL request to this URL.
Apparently only HTTP redirects are supported by cURL with the '--location' option.
Reference: https://everything.curl.dev/http/redirects#non-http-redirects