This is a piggy back off this question -- I've discovered some more information such that the question, itself, needed to change.
I'm attempting to pass data from javascript SPA to a php file (dbPatch.php) to another php file (mongoPatch_backend.php). dbPatch.php is effectively acting as a middle-man to get data to appropriate servers.
My javascript fetch looks like this:
const API = PHP_FILE_LOCATION + 'dbPatch.php/';
const query =
"needleKey=" + encodeURIComponent(needleKey) + "&" +
"needle=" + encodeURIComponent(needle) + "&" +
"newData=" + encodeURIComponent(JSON.stringify(newData));
let URI = API;
fetch(URI, {
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
},
body: query
}).then.... blah...blah....
This calls my php file, dbPatch...
<?php
$API = "https://SERVER/php/mongoPatch_backend.php?";
$needleKey = $_REQUEST['needleKey'];
$needle = $_REQUEST['needle'];
$newData = $_REQUEST['newData'];
$postData = "needleKey=".urlencode($needleKey);
$postData .= "&needle=".urlencode($needle);
$postData .= "&newData=".urlencode($newData); //THIS IS THE LINE I TALK ABOUT BELOW
$data = file_get_contents($API.$postData);
echo $data;
?>
which in turn calls my mongoPatch_backend.php file...
<?php
$user = "xxx";
$pwd = 'xxx';
$needleKey = $_REQUEST['needleKey'];
$needle = $_REQUEST['needle'];
$filter = [$needleKey => $needle];
$newData = $_REQUEST['newData'];
$filter = ['x' => ['$gt' => 1]];
$options = [
'projection' => ['_id' => 0],
'sort' => ['x' => -1],
];
$bson = MongoDB\BSON\fromJSON($newData);
$value = MongoDB\BSON\toPHP($bson);
$manager = new MongoDB\Driver\Manager("mongodb://${user}:${pwd}#DBSERVER:27017");
$bulk = new MongoDB\Driver\BulkWrite;
$bulk->update(
[$needleKey => $needle],
['$set' => $value],
['multi' => false, 'upsert' => true]
);
$results = $manager->executeBulkWrite('dbname.users', $bulk);
var_dump($results);
?>
This does not work.
If I call mongoPatch_backend.php directly from the javascript, it DOES work.
This leads me to believe the problem is in the passing of the data located in the dbPatch.php file.
Further, if I call dbPatch with different 'newData' (shorter) it DOES work. This leads me to believe it's something with the data being passed in (but remember, it works if I call directly... so it's right coming out of the javascript).
Spitting out $newData from dbPatch.php via var_dump($_REQUEST['newData']); gives me JSON data which has been stringified but it is not character-escaped. It's about 5,000 characters.
Here is the interesting part.
If I change mongoPatch_backend.php to JUST <?php echo "Hello World"; ?> I STILL do not get anything passed back through dbPatch.php to my SPA. This REALLY makes me think something is wrong in the dbPatch.php file.
So... I comment out the $postData .= "&newData=".urlencode($newData); line from the dbPatch.php ... I DO get the "Hello World" back.
if I just remove .urlencode and instead just have $postData .= "&newData=".$newData; I still get nothing back.
So the problem seems to be with putting $newData in my post. The mongoPatch_backend.php is not even doing anything with the $newData... dbPatch.php (it appears) is simply having trouble sending that data.
Unfortunately... I"m not sure where to go from here... given, I do, indeed, need to send the $newData to the backend.
EDIT: In reponse to suggestions that I use "POST" instead of "GET" I did a search and found this Stack question: POST data to a URL in PHP
From that, I now have this:
dbPatch.php:
$url = 'https://SERVERNAME/php/mongoPatch_backend.php';
$myvars = 'myvar1=' . "TEST" . '&myvar2=' . "ALSOTEST";
$ch = curl_init( $url );
curl_setopt( $ch, CURLOPT_POST, 1);
curl_setopt( $ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt( $ch, CURLOPT_HEADER, 0);
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec( $ch );
echo $response;
and I changed my mongoPatch_backend.php to:
<?php
echo "HELLO WORLD";
?>
... and I get nothing as the response. (that is, I do not get "HELLO WORLD" form the backend).
My PHP log shows no errors.
My curl config from phpinfo() is:
cURL support enabled
cURL Information 7.59.0
Age 4
Features
AsynchDNS Yes
CharConv No
Debug No
GSS-Negotiate No
IDN Yes
IPv6 Yes
krb4 No
Largefile Yes
libz Yes
NTLM Yes
NTLMWB No
SPNEGO Yes
SSL Yes
SSPI Yes
TLS-SRP No
HTTP2 Yes
GSSAPI No
KERBEROS5 Yes
UNIX_SOCKETS No
PSL No
Protocols dict, file, ftp, ftps, gopher, http, https, imap, imaps, ldap, pop3, pop3s, rtsp, scp, sftp, smb, smbs, smtp, smtps, telnet, tftp
Host x86_64-pc-win32
SSL Version OpenSSL/1.1.0h
ZLib Version 1.2.11
libSSH Version libssh2/1.8.0
I'm not entirely sure why, but this question:PHP - CURL is enabled but not working led me to an example of using cURL that worked.
My dbPatch.php now looks like this and appears to work...
<?php
$url = 'https://SERVERNAME/php/mongoPatch_backend.php';
$params = 'needleKey=' . $_REQUEST['needleKey'] . '&needle=' . $_REQUEST['needle'] . '&newData='. $_REQUEST['newData'];
if (! function_exists ( 'curl_version' )) {
exit ( "Enable cURL in PHP" );
}
$ch = curl_init ();
$timeout = 0; // 100; // set to zero for no timeout
$myHITurl = "http://152.61.248.218/php/mongoPatch_backend.php";
curl_setopt ( $ch, CURLOPT_URL, $myHITurl );
curl_setopt ( $ch, CURLOPT_HEADER, 0 );
curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt( $ch, CURLOPT_POSTFIELDS, $params);
curl_setopt ( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );
$file_contents = curl_exec ( $ch );
if (curl_errno ( $ch )) {
echo curl_error ( $ch );
curl_close ( $ch );
exit ();
}
curl_close ( $ch );
echo "$file_contents";
?>
file_get_contents is only to be used to read a file into a string. Think of it like opening a text document in Notepad or Textedit.
For API requests to dynamically rendered PHP files, you'll want to use the PHP library "curl"
http://php.net/manual/en/book.curl.php
http://codular.com/curl-with-php
Related
I am trying to use the curl function in php to login to a specific page. Please check the code below. I connect with my email and password at banggood.com and then i would like to redirect to another private page but it does not work as expected. I get no errors. I am redirected to this page instead ( https://www.banggood.com/index.php?com=account ) using the code below. After i login i want to access a private page where my orders exist. Any help appreciated.
//The username or email address of the account.
define('EMAIL', 'aaa#gmail.com');
//The password of the account.
define('PASSWORD', 'mypassword');
//Set a user agent. This basically tells the server that we are using Chrome ;)
define('USER_AGENT', 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36');
//Where our cookie information will be stored (needed for authentication).
define('COOKIE_FILE', 'cookie.txt');
//URL of the login form.
define('LOGIN_FORM_URL', 'https://www.banggood.com/login.html');
//Login action URL. Sometimes, this is the same URL as the login form.
define('LOGIN_ACTION_URL', 'https://www.banggood.com/login.html');
//An associative array that represents the required form fields.
//You will need to change the keys / index names to match the name of the form
//fields.
$postValues = array(
'email' => EMAIL,
'password' => PASSWORD
);
//Initiate cURL.
$curl = curl_init();
//Set the URL that we want to send our POST request to. In this
//case, it's the action URL of the login form.
curl_setopt($curl, CURLOPT_URL, LOGIN_ACTION_URL);
//Tell cURL that we want to carry out a POST request.
curl_setopt($curl, CURLOPT_POST, true);
//Set our post fields / date (from the array above).
curl_setopt($curl, CURLOPT_POSTFIELDS, http_build_query($postValues));
//We don't want any HTTPS errors.
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
//Where our cookie details are saved. This is typically required
//for authentication, as the session ID is usually saved in the cookie file.
curl_setopt($curl, CURLOPT_COOKIEJAR, COOKIE_FILE);
//Sets the user agent. Some websites will attempt to block bot user agents.
//Hence the reason I gave it a Chrome user agent.
curl_setopt($curl, CURLOPT_USERAGENT, USER_AGENT);
//Tells cURL to return the output once the request has been executed.
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
//Allows us to set the referer header. In this particular case, we are
//fooling the server into thinking that we were referred by the login form.
curl_setopt($curl, CURLOPT_REFERER, LOGIN_FORM_URL);
//Do we want to follow any redirects?
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, false);
//Execute the login request.
curl_exec($curl);
//Check for errors!
if(curl_errno($curl)){
throw new Exception(curl_error($curl));
}
//We should be logged in by now. Let's attempt to access a password protected page
curl_setopt($curl, CURLOPT_URL, 'https://www.banggood.com/index.php?com=account&t=ordersList');
//Use the same cookie file.
curl_setopt($curl, CURLOPT_COOKIEJAR, COOKIE_FILE);
//Use the same user agent, just in case it is used by the server for session validation.
curl_setopt($curl, CURLOPT_USERAGENT, USER_AGENT);
//We don't want any HTTPS / SSL errors.
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
//Execute the GET request and print out the result.
curl_exec($curl);
You're doing several things wrong:
You're trying to login before you have a cookie session, but the site requires you to have a cookie session before sending the login request.
There's an CSRF token tied to your cookie session, here called at, that you need to parse out from the login page html and provide with your login request, which your code doesn't fetch.
Most importantly, there is a captcha image tied to your cookie session that you need to fetch and solve, and who's text you need to append to your login request, which your code is completely ignoring.
Your login request needs the header x-requested-with: XMLHttpRequest - but your code isn't adding that header.
Your login request needs the fields com=account and t=submitLogin fields in the POST data, but your code isn't adding either of them (you try to add them to your URL, but they're not supposed to be in the url, they're supposed to be in the POST data, aka your $postValues array, not the url)
Here's what you need to do:
First do a normal GET request to the login page. This will give you a session cookie id, the CSRF token, and the url to your captcha image.
Store the cookie id and make sure to provide it with all further requests, then parse out the csrf token (it's in the html looking like <input type="hidden" name="at" value="5aabxxx5dcac0" />), and the url for the captcha image (its different for each cookie session, so don't hardcode it).
Then fetch the captcha image, solve it, and add them all to your login request's POST data, along with the username, password, captcha answer, com and t, and add the http header x-requested-with: XMLHttpRequest to the login request too, send it to https://www.banggood.com/login.html, then you should be logged in!
Here's an example implementation using hhb_curl for the web requests (it's a curl_ wrapper taking care of cookies, turning silent curl_ errors into RuntimeExceptions, etc), DOMDocument for parsing out the CSRF token, and deathbycaptcha.com's api for breaking the captcha.
Ps: the example code won't work until you provide a real credited deathbycaptcha.com api username/password on line 6 and 7, also the captcha looks so simple that I think breaking it could be automated if you're sufficiently motivated, I'm not. - edit, seems they improved their captcha since i wrote that, it looks very difficult now. Also, the banggood account is just a temporary testing account, no harm comes of it being compromised, which obviously happens when I post the username/password here)
<?php
declare(strict_types = 1);
require_once ('hhb_.inc.php');
$banggood_username = 'igcpilojhkfhtdz#my10minutemail.com';
$banggood_password = 'igcpilojhkfhtdz#my10minutemail.com';
$deathbycaptcha_username = '?';
$deathbycaptcha_password = '?';
$hc = new hhb_curl ( '', true );
$html = $hc->exec ( 'https://www.banggood.com/login.html' )->getStdOut ();
$domd = #DOMDocument::loadHTML ( $html );
$xp = new DOMXPath ( $domd );
$csrf_token = $xp->query ( '//input[#name="at"]' )->item ( 0 )->getAttribute ( "value" );
$captcha_image_url = 'https://www.banggood.com/' . $domd->getElementById ( "get_login_image" )->getAttribute ( "src" );
$captcha_image = $hc->exec ( $captcha_image_url )->getStdOut ();
$captcha_answer = deathbycaptcha ( $captcha_image, $deathbycaptcha_username, $deathbycaptcha_password );
$html = $hc->setopt_array ( array (
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => http_build_query ( array (
'com' => 'account',
't' => 'submitlogin',
'email' => $banggood_username,
'pwd' => $banggood_password,
'at' => $csrf_token,
'login_image_code' => $captcha_answer
) ),
CURLOPT_HTTPHEADER => array (
'x-requested-with: XMLHttpRequest'
)
) )->exec ()->getStdOut ();
var_dump ( // $hc->getStdErr (),
$html );
function deathbycaptcha(string $imageBinary, string $apiUsername, string $apiPassword): string {
$hc = new hhb_curl ( '', true );
$response = $hc->setopt_array ( array (
CURLOPT_URL => 'http://api.dbcapi.me/api/captcha',
CURLOPT_POST => 1,
CURLOPT_HTTPHEADER => array (
'Accept: application/json'
),
CURLOPT_POSTFIELDS => array (
'username' => $apiUsername,
'password' => $apiPassword,
'captchafile' => 'base64:' . base64_encode ( $imageBinary ) // use base64 because CURLFile requires a file, and i cba with tmpfile() .. but it would save bandwidth.
),
CURLOPT_FOLLOWLOCATION => 0
) )->exec ()->getStdOut ();
$response_code = $hc->getinfo ( CURLINFO_HTTP_CODE );
if ($response_code !== 303) {
// some error
$err = "DeathByCaptcha api retuned \"$response_code\", expected 303, ";
switch ($response_code) {
case 403 :
$err .= " the api username/password was rejected";
break;
case 400 :
$err .= " we sent an invalid request to the api (maybe the API specs has been updated?)";
break;
case 500 :
$err .= " the api had an internal server error";
break;
case 503 :
$err .= " api is temorarily unreachable, try again later";
break;
default :
{
$err .= " unknown error";
break;
}
}
$err .= ' - ' . $response;
throw new \RuntimeException ( $err );
}
$response = json_decode ( $response, true );
if (! empty ( $response ['text'] ) && $response ['text'] !== '?') {
return $response ['text']; // sometimes the answer might be available right away.
}
$id = $response ['captcha'];
$url = 'http://api.dbcapi.me/api/captcha/' . urlencode ( $id );
while ( true ) {
sleep ( 10 ); // check every 10 seconds
$response = $hc->setopt ( CURLOPT_HTTPHEADER, array (
'Accept: application/json'
) )->exec ( $url )->getStdOut ();
$response = json_decode ( $response, true );
if (! empty ( $response ['text'] ) && $response ['text'] !== '?') {
return $response ['text'];
}
}
}
Set CURLOPT_FOLLOWLOCATION to 1 or true, you may also need CURLOPT_AUTOREFERER instead of the static REFERER.
Do you get some cookies into your COOKIEJAR (cookie.txt) ? Remember that the file must already exists and PHP needs write permissions.
If you have PHP executing on localhost then a Network sniffer tool could help debug the problem, try with Wireshark or some equivalent software. Because maybe the request still miss some important HTTP Headers like Host
I am working on a site where realtors can create online presentations for clients and using examples from a website called streeteasy.com, a zillow owned site. I can successfully scrape everything I want with the exception of you have to be logged in in order to see the sold price for closed properties. The login is very low in security as far as I can tell and it sets a cookie for 10 years when you login. I used the chrome developers tools to get the post data and the post URL.
You can see the page comes up just fine but it still says 'Register to see what it closed for about 5 weeks ago' to see the final sale price. When you are logged in, the message is different. I still cannot get this code to work and I have no idea why. I am testing on my WAMP so the cookie.txt file is not an issue with permissions. I even tried to just create my own cookie file with my browser cookies and just access the page but still no luck.
$url = "http://streeteasy.com/sale/1253471";
$login_url = 'https://streeteasy.com/nyc/user/sign_in';
$data = 'utf8=%E2%9C%93&authenticity_token=MYCz6A5PK%2B3I3N%2BgHekaNc4IuQEruBrCPBjSxm1B9dg%3D&do=login&return_to=http%3A%2F%2Fstreeteasy.com%2F&origin=&page_category=&page_type=&boundary=&label=&remember=true&return_to_save_search=&login=john%40telesh.com&password=dman4578';
login($login_url,$data);
echo grab_page ($url);
function login($url,$data){
$fp = fopen("cookie.txt", "w");
fclose($fp);
$login = curl_init();
curl_setopt($login, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($login, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($login, CURLOPT_TIMEOUT, 40000);
curl_setopt($login, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($login, CURLOPT_URL, $url);
curl_setopt($login, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($login, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($login, CURLOPT_POST, TRUE);
curl_setopt($login, CURLOPT_POSTFIELDS, $data);
return curl_exec ($login);
}
function grab_page($site){
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_TIMEOUT, 40);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
return curl_exec ($ch);
}
(i hope that is a dummy account...)
first off, to do a successful login, you need a valid "authenticity_token" which you get on the login page, which is probably different for each session, but your code has a HARDCODED authenticity_token which probably expired long ago, and was only ever valid in your browser. second, your login() function is so bugged that it SHOULD result in a 500 internal server error when calling the script from WAMP, because the output buffer it creates is never ended. third, for some weird reason, seems it REQUIRES the login request to contain a browser-like accept header, for example Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 , but curl, by default, just says Accept: */* - which it doesn't accept.
start a cookie session, grab the sign in page, parse out the authenticity_token from the signin page (actually, just parse out all the "input" tags), and log in with the fresh one, and make sure to send Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 with the login request.
using hhb_curl from https://github.com/divinity76/hhb_.inc.php/blob/master/hhb_.inc.php , here is a working example:
<?php
declare(strict_types = 1);
require_once ('hhb_.inc.php');
$hc = new hhb_curl ();
$hc->_setComfortableOptions ();
$hc->exec ( 'https://streeteasy.com/nyc/user/sign_in' );
$html = $hc->getResponseBody ();
$domd = #DOMDocument::loadHTML ( $html );
$inputs = array ();
foreach ( $domd->getElementsByTagName ( "input" ) as $input ) {
$inputs [$input->getAttribute ( "name" )] = $input->getAttribute ( "value" );
}
assert ( array_key_exists ( 'authenticity_token', $inputs ) );
$inputs ['login'] = 'john#telesh.com';
$inputs ['password'] = 'dman4578';
var_dump ( $inputs );
$hc->setopt_array ( array (
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => http_build_query ( $inputs ),
CURLOPT_URL => 'https://streeteasy.com/nyc/user/sign_in',
CURLOPT_HTTPHEADER => array (
'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
)
) );
$hc->exec ();
$html = $hc->getResponseBody ();
if (false === strpos ( $html, 'You have successfully logged in' )) {
throw new RuntimeException ( 'login failed! (could not find `You have successfully logged in` in the response body!' );
}
hhb_var_dump ( $hc->getStdErr (), $hc->getResponseBody () );
it dumps the logged in html at the end, proving that it has logged in. it even verifies this by checking for the You have successfully logged in string in the response.
-edit: as for parsing out the "sold for" price, you can use DOMDocument for that.. the html is shitty, so its a bit cumbersome, but i found a way to get it:
$hc->exec('http://streeteasy.com/sale/1253471');
$html = $hc->getResponseBody ();
$domd = #DOMDocument::loadHTML ( $html );
$sold_for=NULL;
foreach($domd->getElementsByTagName("div") as $div){
if(false!==strpos($div->getAttribute("class"),'status_sold')){
$sold_for=trim($div->nextSibling->nextSibling->textContent);
break;
}
}
var_dump($sold_for);
output:
string(63) "Sold for $16,550,062
as of about 5 weeks ago"
If the token expires in such a long time, why not just hardcode the cookie session into your cURL requests, simply add the value for the cookie header to be:
$headers = [
'Cookie: _actor=eyJpZCI6IitEZG9VenJLc00wVENZSXFYZWlrVlE9PSJ9--4d8449347e46c32eaae1a8189b83881b7abc6e24; _se_t=f9f7ae31-80cd-4698-bdc8-fbdff739009b; _gcl_au=1.1.73614610.1670886677; pxcts=48f6f381-7a72-11ed-bc8c-4d6a52455343; _pxvid=48f6e270-7a72-11ed-bc8c-4d6a52455343; _ga=GA1.2.1781461336.1670886678; __gads=ID=f12dde3526d46fa2:T=1670886679:S=ALNI_MY5HCF64ez59fkAePUrvcVo2YMjbg; zg_anonymous_id=%2222fd5f18-5b53-4213-8734-1983e805f19e%22; google_one_tap=0; ezab_gold_price_widget=no_widget; ezab_ursa_srp_expanded_map=reduced_map; last_search_tab=rentals; se%3Asearch%3Arentals%3Astate=false%7C%7C%7C%7C; se_login_trigger=8; ezab_orca_1999_show_agent_matches=show_agent_matches_immediately; onboarding_flow=false; se%3Asearch%3Ashared%3Astate=300||||false; anon_searcher_stage=initial; tracked_search=; __gpi=UID=000008d033ec5e0d:T=1670886679:RT=1671549920:S=ALNI_Mbh6kKe3MrPpvhniGrkDAtpQwm-Zg; se%3Abig_banner%3Asearch=%7B%229991594%22%3A1%7D; zjs_anonymous_id=%22f9f7ae31-80cd-4698-bdc8-fbdff739009b%22; _uetvid=485c00c07a7211ed8f4eeb1ea6ad9f20; ezab=%7B%22gold_price_widget%22%3A%22no_widget%22%2C%22orca_1999_show_agent_matches%22%3A%22show_agent_matches_immediately%22%2C%22ursa_srp_expanded_map%22%3A%22reduced_map%22%7D; _gid=GA1.2.608953748.1672024129; _pxff_cc=U2FtZVNpdGU9TGF4Ow==; ki_r=; g_state=; zjs_user_id=%222058917%22; remember_user_token=eyJfcmFpbHMiOnsibWVzc2FnZSI6Ilcxc3lNRFU0T1RFM1hTd2lNbDl6U2paTWIzQlJNMEpZYzNNMlVsbDJjMU1pTENJeE5qY3lNREkwTWpJeExqWTBNRFk0TWpJaVhRPT0iLCJleHAiOiIyMDMyLTEyLTI2VDAzOjEwOjIxWiIsInB1ciI6bnVsbH19--fac8f407a824eba1fb7aeac12d574c6bff565a20; user_auth_token=Q6_wPheB4hx2QseGxyy3; se_lsa=2022-12-25+22%3A10%3A41+-0500; _ses=MVUweDEySUpkZEo1c3pQek96OFpQTmgxWUoyTmIyeElaSWxhR0hCS2I2Z1VESnp4b3NnM2VJQm1Kelo2SzcxeHBIQkxBUmJwWE9zc20valdITU5XM2lRUTJhTjlSa3UzR3p0Yk5yTVJnR1pPQ0RuZ1hsZ1F0SzVCb1dEbkoxYWRCN0hYUzAxZlNZckZ1TVNxZ2diQTNzcXVVL2tIb0lNdk9nb09NNWw1V1p0NDFKNEFGcWp3SkQ2S1N4eElObjJqVWVxRm9IOEVyQXVUMEpORURadGxnUT09LS05S3JuL1Y2TWlNOEdaZFJPYWdlZ3RnPT0%3D--c9565ba32e0b2adae10d9076a0b7442b5edb5d81; ki_t=1670886678145%3B1672024129813%3B1672024241463%3B5%3B102; _px3=d3827445b59b0a26866c26cebaa57f08257be464948e0a69625cdc878a36c4af:bIEndpROKJiy9bn1Wb/Ez8jOc1wjprWLUJjlVz8c37ayGumhnRQ1s9VBQ6XmxIo/gzER64vFdxi2f60X6WLoPA==:1000:Re35eAC4PIw8oV16m1QenX871u0lx4QnQfNVNegIoEFZRiehpPIiiTARGdPIwr6O3A/Aey5zb9hn3BxkW0sD4fU+v/V1zNU/uIlbp9PByj+r4dE8usVlgb3G2grMJw5I+x/yH7N2T6qwRgDiyXcYLTRSCtiwnFQoDvvctIGTuOUOXZmpNUC40YzcPijIrv1UAzXyt0oahTIG815/me9QDA=='
];
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
This will set the header to send the auth cookie string the server checks against; As long as you pass the cookie, and the session is non-expired - you will be able to send all http requests successfully.
I wrote a few scrapers for Streeteasy, know that Distil Networks is the vendor whom they signed up for to protect their server from DDoS/Scrape/Dir Fuzzing - you will likely get banned after a few requests so make sure to use MiT proxy to hit that captcha page to whitelist your server.
Btw if on localhost you're experiencing the same SSL related errors I used to, add the following to your cURL:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPHOST, FALSE);
That out to do the trick; you'll be able to make calls with or without self-signed certs;
I'm trying to get data from external website using cURL in PHP but, somehow it's not working.
I've checked out that CURL enable in phpinfo(). It shows cURL is enabled
But, my code is not working.
<?php
if (! function_exists ( 'curl_version' )) {
exit ( "Enable cURL in PHP" );
}
$ch = curl_init ();
$timeout = 0; // 100; // set to zero for no timeout
$myHITurl = "http://www.google.com";
curl_setopt ( $ch, CURLOPT_URL, $myHITurl );
curl_setopt ( $ch, CURLOPT_HEADER, 0 );
curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt ( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );
$file_contents = curl_exec ( $ch );
if (curl_errno ( $ch )) {
echo curl_error ( $ch );
curl_close ( $ch );
exit ();
}
curl_close ( $ch );
// dump output of api if you want during test
echo "$file_contents";
?>
It goes timeout.
I'm not using WAMP or XAMPP server. The above code runs directly on the server.
I've no idea what's going wrong.
Your code is perfect, I have tested it on my own server (data center in Texas) and it worked fine.
My guess is that your server IP is banned. Try to fetch a different URL, and see if it works for you. If it does then you are banned, if it doesn't then it might be a firewall configuration issue in your server.
disable SELinux if you are on Centos or Fedora or any Redhat Distro
nano /etc/selinux/config
Change
SELINUX=enforcing
to
SELINUX=disabled
I am currently adding the ability to a php back-end system to allow it to print directly and I am trying to get things working with Google's Cloud Print. Imagine the app as an online shopping cart and I want it to print picking notes (completed orders) without the need for someone to login. The server is remote and the destination has Cloud Ready Printers.
So far I have been successful in getting it to print using the interfaces, as long as I am simply passing HTML, plain text or a URL to a PDF. I am able to set the print to color, marginless and the print quality.
However where I have hit a problem is, the PDF which the system creates are not publicly accessible, hence I can't pass a URL to the file, I need to pass the contents of the file.
I have been trying with no success to modify one of the examples I have found on the web HERE. However I don't know the language so am struggling with it.
Another example in python HERE again I have been trying without success!
I'm using PHP and the Zend framework to work with the interface. Here is one sample I have tried, cut down to where I am trying to prepare the file to send, like I say I'm not really sure on translating from python to php, or if the python script even works, but this is what I came up with:
<?php
// Test print a job:
$b64_pathname = PDF_PATH.'ec22c3.pdf'.'.b64';
$fileType = "application/pdf";
// Open the original file and base64 encode it:
$dataHandle = fopen(PDF_PATH.'ec22c3.pdf', "rb");
$dataContent = fread($dataHandle, filesize(PDF_PATH.'ec22ed167763a15e8591a3776f3c65c3.pdf'));
fclose($dataHandle);
$b64data = $fileType.base64_encode($dataContent);
// Store the base64 encoded file:
$ourFileHandle = fopen($b64_pathname, 'w');
fwrite($ourFileHandle, $b64data);
fclose($ourFileHandle);
// Read the contents of the base64 encoded file and delete it:
$fileHandle = fopen($b64_pathname, "rb");
$fileContent = fread($fileHandle, filesize($b64_pathname));
fclose($fileHandle);
unlink($b64_pathname);
// URL encode the file contents:
$file = urlencode($fileContent);
// Add the file and send to the printer:
$client->setParameterPost('content', $file);
$client->setParameterPost('contentType', $fileType);
$client->request(Zend_Http_Client::POST);
?>
Here's a method in php using cUrl (note, I have object level variables called _auth, _username, _password & _printerId).
First, build a function to post with cUrl:
function processRequest($url, $postFields, $referer) {
$ret = "";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_USERAGENT, "");
if(!is_null($postFields)) {
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,
$postFields);
// http_build_query() will properly escape the fields and
// build a query string.
}
if(strlen($this->_auth) > 0) {
$headers = array(
"Authorization: GoogleLogin auth=". $this->_auth,
//"GData-Version: 3.0",
"X-CloudPrint-Proxy", "yourappname"
);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
}
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$ret = curl_exec ($ch);
curl_close ($ch);
return $ret;
}
Then, a function to authorize against Google:
public function authorize() {
$url = "https://www.google.com/accounts/ClientLogin";
$post = array("accountType" => "HOSTED_OR_GOOGLE",
"Email" => $this->_username,
"Passwd" => $this->_password,
"service" => "cloudprint",
"source" => "yourappname");
$resp = $this->processRequest($url, $post, "");
preg_match("/Auth=([a-z0-9_\-]+)/i", $resp, $matches);
$this->_auth = $matches[1];
}
Finally, build a function to submit to the cloud printer:
function printDocument($title, $docBytes)
{
$url = "http://www.google.com/cloudprint/submit?printerid=". $this->_printerId."&output=json";
$post = array(
"printerid" => $this->_printerId,
"capabilities" => "",
"contentType" => "dataUrl",
"title" => $title,
"content" => 'data:application/pdf;base64,'. base64_encode($docBytes)
);
$ret = $this->processRequest($url, $post, "");
echo $ret;
}
In use, call authorize() to get the authentication token. Then just read your file (from wherever) into a variable and pass it to printDocument with the title.
In order to send base64 encoded content you need to send another parameter in submit request:
$client->setParameterPost('contentTransferEncoding', 'base64');
I am working with the Facebook API, and successfully use the following command via terminal to post a message to another users wall.
curl -F 'access_token=XXXXXXXXXX' \
-F 'message=Hello World' \
-F 'to={["id":XXXXXXX]}' \
https://graph.facebook.com/me/feed
This works great. I am trying to do the same via php with this code;
$fields = array(
'access_token' => $t,
'message' => $message,
'to' => '{["id":'.$id.']}'
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
curl_exec($ch);
curl_close($ch);
This code successfuly posts a message, but it does it to my OWN wall (i.e. it is ignoring the 'to' parameter). I'm new to cURL, and I'm sure I am encoding it wrong, or maybe missing a cURL flag, but I've been through several tutorials on POSTing via cURL, including a few SO answers, and I can't see what I'm missing.
Really appreciate any help!
What does this print out?
if ( 'POST' == $_SERVER[ 'REQUEST_METHOD' ]) {
echo 'Posted: ';
print_r( $_POST );
exit;
}
$t = '121';
$message = 'helo Worlds';
$id = 1234;
$fields = array(
'access_token' => $t,
'message' => $message,
'to' => '{["id":'.$id.']}'
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://localhost:8888/testbed/' ); // THIS script
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE );
$out = curl_exec($ch);
curl_close($ch);
echo 'Received[ ' . $out . ' ]';
Prints this on my local box:
Received[ Posted: Array ( [access_token] => 121 [message] => helo Worlds [to] => {[\"id\":1234]} ) ]
UPDATED:
$fields should be a GET like string
para1=val1¶2=val2&...
or an array:
The full data to post in a HTTP "POST"
operation. To post a file, prepend a
filename with # and use the full path.
The filetype can be explicitly
specified by following the filename
with the type in the format
';type=mimetype'. This parameter can
either be passed as a urlencoded
string like
'para1=val1¶2=val2&...' or as an
array with the field name as key and
field data as value. If value is an
array, the Content-Type header will be
set to multipart/form-data. As of PHP
5.2.0, files thats passed to this option with the # prefix must be in
array form to work.