I have a problem logging in to a website with CURL and PHP.
I test with the Firefox add-on HttpRequester and this worked.
Result login:
POST https://www.balatarin.com/sessions
Content-Type: application/x-www-form-urlencoded
session[login]=testeruni&session[password]=123456789&session[remember_me]=1&commit=%D9%88%D8%B1%D9%88%D8%AF&utf8=%E2%9C%93&authenticity_token[![httprequester][1]][1]
-- response --
200 OK
Server: shield
Date: Thu, 19 Jan 2017 13:51:54 GMT
Content-Type: text/html; charset=utf-8
status: 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
x-ua-compatible: IE=Edge,chrome=1
Etag: W/"7418542e936fbdfe20002faf11876845"
Cache-Control: must-revalidate, private, max-age=0
Set-Cookie: _balat_session_new=BAh7C0kiDHVzZXJfaWQGOgZFRmkD964BSSIPc2Vzc2lvbl9pZAY7AEZJIiUzZGUxMzIyN2ZhZDVmMDUzOGE3OGY0YTRhZDkzNmUyMQY7AFRJIhZpbnB1dF9kZXZpY2VfdHlwZQY7AEZJIgpNT1VTRQY7AEZJIhRob3Zlcl9zdXBwb3J0ZWQGOwBGVEkiCmZsYXNoBjsARm86JUFjdGlvbkRpc3BhdGNoOjpGbGFzaDo6Rmxhc2hIYXNoCToKQHVzZWRvOghTZXQGOgpAaGFzaHsGOgtub3RpY2VUOgxAY2xvc2VkRjoNQGZsYXNoZXN7BjsKSSI22YbYrtiz2Kog2YXYtNiq2LHaqSDahtmG2K8g2KjYp9mE2KfahtmHINi02YjbjNivLgY7AFQ6CUBub3cwSSIQX2NzcmZfdG9rZW4GOwBGSSIxT3krNk5nM1NTM2IreXc4SUtxbW9yN2NmMXQrdUNLWWdubFRRYmpidmtNTT0GOwBG--2c2a72f8ec27564250ba084d97998aefba4af11a; path=/; secure; HttpOnly geo=0
X-Request-Id: 521288561d7cfff0ef8fe8d72080760c
X-Runtime: 0.188862
X-Rack-Cache: miss
Content-Encoding: gzip
Via: 1.1 google
Alt-Svc: clear
Expires: Thu, 19 Jan 2017 13:51:54 GMT
X-Firefox-Spdy: h2
but it does not login with curl in PHP. I tested all headers in my CURL but it does not login, only works with HttpRequester.
public function actionLoggin()
{
$url = 'https://www.balatarin.com/sessions';
$headers[] = 'Content-Type: application/x-www-form-urlencoded';
$headers[] = 'Host: www.balatarin.com';
$headers[] = 'User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0';
$headers[] = 'Referer: https://www.balatarin.com/login';
$params = array(
'session[login]' => 'testeruni',
'session[password]' => '123456789',
'session[remember_me]' => '0',
'commit' => 'ورود',
'utf8' => '✓',
'authenticity_token' => '',
);
//open connection
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($params));
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'bala_cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'bala_cookie.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
}
Here is my cookie file:
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
www.balatarin.com FALSE / FALSE 0 logged_in 1
#HttpOnly_www.balatarin.com FALSE / TRUE 0 _balat_session_new BAh7CToOcmV0dXJuX3RvMDoMdXNlcl9pZGkDj60BOhJsb2dpbl9yZXRyaWVzMEkiD3Nlc3Npb25faWQGOgZFRkkiJTgwN2ZmMDRjMGUzMzkyMDIyZWY5YzBmZTQxN2FmZWMzBjsIVA%3D%3D--d47dd61bc9900449cca69ebd727041c3946a13ba
www.balatarin.com FALSE / FALSE 0 geo 0
www.balatarin.com FALSE / FALSE 1516368886 corr b8ed93fa279a469a637b
Related
I'm trying to embed Facebook's posts (e.g. video) using oEmbed format. According to Facebook documentation, oEmbed is now supported. I'm trying this PHP code:
$json_post = #file_get_contents('https://www.facebook.com/plugins/video/oembed.json/?url={MY VIDEO URL HERE}');
$oembed = json_decode($json_post);
var_dump($oembed);
I already used the same code for Instagram with success, now I'm getting a NULL result. oEmbed works good if i directly write the URL on the browser. Am i missing something?
Thanks.
Update
I tried with Curl:
$url='https://www.facebook.com/plugins/video/oembed.json/?url=https%3A%2F%2Fwww.facebook.com%2Ffacebook%2Fvideos%2F10153231379946729%2F';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
//curl_setopt($ch, CURLOPT_NOBODY, TRUE); // remove body
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$page = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
print_r($page);
curl_close($ch);
Now i get:
HTTP/1.1 302 Found
Location: https://www.facebook.com/unsupportedbrowser
access-control-allow-method: OPTIONS
Access-Control-Expose-Headers: X-FB-Debug, X-Loader-Length
Access-Control-Allow-Origin: https://www.facebook.com
Vary: Origin
Access-Control-Allow-Credentials: true
Content-Type: text/html
X-FB-Debug: gGcZzyllZadlcn/6jz2HqqouIcDnhTzxzR+etWXhZEnOcditfsaIUw0WjgO3nELHzveRCYa1UM86D3LA/nLnNw==
Date: Wed, 11 Jan 2017 10:18:47 GMT
Connection: keep-alive
Content-Length: 0
HTTP/1.1 200 OK
X-XSS-Protection: 0
public-key-pins-report-only: max-age=500; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; pin-sha256="q4PO2G2cbkZhZ82+JgmRUyGMoAeozA+BSXVXQWB8XWQ="; report-uri="http://reports.fb.com/hpkp/"
Pragma: no-cache
Cache-Control: private, no-cache, no-store, must-revalidate
Expires: Sat, 01 Jan 2000 00:00:00 GMT
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=15552000; preload
X-Frame-Options: DENY
Vary: Accept-Encoding
Content-Type: text/html
X-FB-Debug: zwArox8KyM3BtwLymhiARCTltrrcE/pDqSWdqbHgstXVBEbIXG57Od2MfDnqgqSX5Tj43qoe8uYhphzwoZcXeg==
Date: Wed, 11 Jan 2017 10:18:48 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Still waiting for a reply.
Thank You.
Set the user agent with the curl and try,
$browser = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.16 (KHTML, like Gecko) \Chrome/24.0.1304.0 Safari/537.16';
curl_setopt($ch, CURLOPT_USERAGENT, $browser);
Here is the answer with file_get_content,
$options = array(
'http'=>array(
'method'=>"GET",
'header'=>"Accept-language: en\r\n" .
"Cookie: foo=bar\r\n" . // check function.stream-context-create on php.net
"User-Agent: Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.102011-10-16 20:23:10\r\n" // i.e. An iPad
)
);
$context = stream_context_create($options);
$json_post = #file_get_contents('https://www.facebook.com/plugins/video/oembed.json/?url=https%3A%2F%2Fwww.facebook.com%2Ffacebook%2Fvideos%2F10153231379946729%2F', false, $context);
$oembed = json_decode($json_post);
var_dump($oembed)
I need to parse with simpl_html_dom german site.I have a problem with german Umlauts, because utf-8 don't support Umlauts. I know ,if convert text from UTF-8 to UTF-16 or ISO-8859-1 problem solved.I use CURL for get content page. This page have charset ISO-8859-1.I try to set CURLOPT_ENCODING ISO-8859-1 ,but Curl always return the utf-8 text.I don't know what do.
Code of this method.
public function testsec()
{
require_once DIR_SYSTEM.'library'.DIRECTORY_SEPARATOR.'simpleHtml'.DIRECTORY_SEPARATOR.'simple_html_dom.php';
$regexpSecond = "~Möglicherweise.*? Vielen Dank~su";
$headers = array(
"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0",
"Accept: text/plain",
"Connection: keep-alive",
);
$fp = fopen(DIR_ADMIN.'logCurl.txt','w+');
$head = fopen(DIR_ADMIN.'headers.txt','w+');
$curl = curl_init("http://test.site.com/bla-bla-bla");
curl_setopt($curl, CURLOPT_RETURNTRANSFER,true);
curl_setopt($curl, CURLOPT_ENCODING , "UTF-16");
curl_setopt($curl, CURLOPT_VERBOSE, 1);
curl_setopt($curl, CURLOPT_STDERR, $fp);
curl_setopt($curl, CURLOPT_HEADER ,$headers);
curl_setopt($curl, CURLOPT_WRITEHEADER, $head);
$result = curl_exec($curl);
curl_close($curl);
fclose($fp);
fclose($head);
$html = str_get_html($result);
echo mb_detect_encoding($result); //utf-8
}
Headers response
HTTP/1.1 200 OK
Date: Sun, 03 Jul 2016 05:22:34 GMT
Server: Apache
Set-Cookie: JTLSHOP=c1qv3vafghmf3ih43g5m96epi4; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: max-age=1, private, must-revalidate
Pragma: no-cache
Vary: Accept-Encoding
X-Powered-By: PleskLin
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1
UTF-8 support Umlauts.
http://www.periodni.com/unicode_utf-8_encoding.html#german_special_characters
If you want to convert charset, Use ICONV function.
I'm trying to upload a file via cURL but something is missing. I forces this request to be HTTP 1.0 because cURL adds the Expect: 100 header if I use HTTP 1.1 so thats why the extra header. Here is a simple test code:
<?php
if(isset($_POST["id"])) {
$data = array("id" => $_POST["id"]);
$data["file"] = "#".realpath($_FILES["file"]["tmp_name"]);
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Authorization: Bearer 0e39ffba-66cd-4933-9e94-fcdf600c2453',
'Connection: keep-alive'
));
curl_setopt($ch, CURLOPT_URL, "http://localhost:8080/test-api/upload");
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_VERBOSE, false);
curl_setopt($ch, CURLOPT_HTTP_VERSION, 1);
$response = curl_exec($ch);
var_dump($response);
exit;
}
?>
My Jersey based server picks it up, and I can see these headers:
INFO: 25 * Server has received a request on thread http-nio-8080-exec-1
25 > POST http://localhost:8080/test-api/upload
25 > authorization: Bearer 0e39ffba-66cd-4933-9e94-fcdf600c2453
25 > connection: keep-alive
25 > content-length: 261
25 > content-type: multipart/form-data; boundary=------------------------53f7ba34739b4d9e
25 > host: localhost:8080
See the content-length? It's way too short. When I send the same file and the same request via my Postman REST client, I get these headers:
INFO: 26 * Server has received a request on thread http-nio-8080-exec-3
26 > POST http://localhost:8080/test-api/upload
26 > accept-encoding: gzip, deflate
26 > accept-language: hu-HU,hu;q=0.8,en-US;q=0.6,en;q=0.4
26 > authorization: Bearer 0e39ffba-66cd-4933-9e94-fcdf600c2453
26 > cache-control: no-cache, no-cache
26 > connection: keep-alive
26 > content-length: 144954
26 > content-type: multipart/form-data; boundary=----WebKitFormBoundarye5Tg0kEqi10nEBwv
26 > cookie: ff_uvid=126143952; _ga=GA1.1.459454356.1439469592; CAKEPHP=9mffidqo8203ugktan4roc0u82
26 > host: localhost:8080
26 > origin: chrome-extension://fdmmgilgnpjigdojojpjoooidkmcomcm
26 > user-agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36
The content-length now is set property. What could be wrong here?
It sounds like you're using PHP 5.6.0 or later. As of this release, the # prefix for file uploads is disabled by default. You can enable it with
curl_setopt($ch, CURLOPT_SAFE_UPLOAD, false);
This option was added in 5.5, but the default was false for backward compatibility; 5.6 changed the default incompatibly.
The preferred way to perform file uploads starting with 5.5 is with the CurlFile class.
$data["file"] = new CurlFile(realpath($_FILES["file"]["tmp_name"]));
You have to actually insert the filecontent, this differs from the cli-version of curl.
try:
$data["file"] = file_get_contents($_FILES["file"]["tmp_name"]);
In my first cURL request i upload a file and i set a $_SESSION variable with the name, extension etc. In my second cURL request i want to move the uploaded file from tmp folder to user folder but baddly the $_SESSION variable is empty. why?
first request code code looks like this:
$upload = curl_init();
curl_setopt($upload, CURLOPT_URL, "http://localhost/upload/" );
curl_setopt($upload, CURLOPT_POST, true );
curl_setopt($upload, CURLOPT_RETURNTRANSFER, true );
curl_setopt($upload, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"] );
curl_setopt($upload, CURLOPT_HTTPHEADER, array("Set-Cookie: data=" . urldecode($cookie) ));
curl_custom_postfields($upload, $fields, $files);
$res = curl_exec($upload);
curl_close($upload);
and the second request code: following the first request:
$submit = curl_init();
curl_setopt($submit, CURLOPT_URL, "http://localhost/" );
curl_setopt($submit, CURLOPT_RETURNTRANSFER, true );
curl_setopt($submit, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"] );
curl_setopt($submit, CURLOPT_POST, count($fields));
curl_setopt($submit, CURLOPT_POSTFIELDS, $fields_string );
curl_setopt($submit, CURLOPT_HEADER, true);
curl_setopt($submit, CURLOPT_HTTPHEADER, array("Cookie: data=" . urldecode($cookie) ));
$res = curl_exec($submit);
curl_close($submit);
is there any option to keep session alive? is the same problem i meet on AJAX requests when i start using javascript with AJAX i think.
my response header:
HTTP/1.1 200 OK
Date: Wed, 04 Mar 2015 02:24:22 GMT
Server: Apache/2.2.12 (Win32) DAV/2 mod_ssl/2.2.12 OpenSSL/0.9.8k mod_autoindex_color PHP/5.3.0 mod_perl/2.0.4 Perl/v5.10.0
X-Powered-By: PHP/5.3.0
Set-Cookie: PHPSESSID=g7hc328ij8lr63mps6ub44gat2; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Length: 22
Content-Type: text/html
I think you need a cookie jar to keep track of your session:
curl_setopt($ch, CURLOPT_COOKIEJAR, "/tmp/cookies");
I'm trying to implement a functionnality like facebook, when you paste a link it's grabbing some information (h1, desc, images, ...) from the page and display them.
I already face several issues that I managed to fix (gzip, cookies, user agent, ...) but on this one I'm not sure what is blocking my request.
The link in question is http://www.mixcloud.com
Here is my PHP script:
protected function getContent()
{
$ch = curl_init();
$headers = array(
'Accept: */*',
// 'Accept-Encoding: gzip,deflate,sdch',
// 'Accept-Language: en-US,en;q=0.8,es;q=0.6,fr;q=0.4,pt;q=0.2',
// 'Cache-Control: no-cache',
// 'Connection: keep-alive'
);
$debug = TRUE;
// Set the request type
curl_setopt($ch, CURLOPT_VERBOSE, $debug);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_NOBODY, FALSE);
curl_setopt($ch, CURLOPT_URL, $this->url);
curl_setopt($ch, CURLOPT_USERAGENT, $this->userAgent);
curl_setopt($ch, CURLOPT_REFERER, $this->referrer);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, $debug);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_ENCODING , 'gzip');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5');
curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, '/tmp/cookies.txt');
$data = curl_exec($ch);
var_dump($data);die;
return curl_exec($ch);
}
Here is the verbose response:
* Adding handle: conn: 0x7f937504e400
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f937504e400) send_pipe: 1, recv_pipe: 0
* About to connect() to www.mixcloud.com port 80 (#0)
* Trying 46.23.65.210...
* Connected to www.mixcloud.com (46.23.65.210) port 80 (#0)
> GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5
Host: www.mixcloud.com
Accept-Encoding: gzip
Referer: https://www.google.com.au
Accept: */*
< HTTP/1.1 403 Forbidden
* Server nginx/1.5.8 is not blacklisted
< Server: nginx/1.5.8
< Date: Tue, 18 Feb 2014 06:39:45 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< Content-Encoding: gzip
<
* Connection #0 to host www.mixcloud.com left intact
string(376) "HTTP/1.1 403 Forbidden\r\nServer: nginx/1.5.8\r\nDate: Tue, 18 Feb 2014 06:39:45 GMT\r\nContent-Type: text/html\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nVary: Accept-Encoding\r\nContent-Encoding: gzip\r\n\r\n<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body bgcolor="white">\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx/1.5.8</center>\r\n</body>\r\n</html>\r\n"
Now if I try to execute the curl command in the shell it's working fine:
$ curl -i 'http://www.mixcloud.com' -v
* Adding handle: conn: 0x7fe28b004000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fe28b004000) send_pipe: 1, recv_pipe: 0
* About to connect() to www.mixcloud.com port 80 (#0)
* Trying 46.23.65.210...
* Connected to www.mixcloud.com (46.23.65.210) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: www.mixcloud.com
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Tue, 18 Feb 2014 06:41:30 GMT
Date: Tue, 18 Feb 2014 06:41:30 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Content-Length: 194847
Content-Length: 194847
< Connection: keep-alive
Connection: keep-alive
< Vary: Accept-Encoding
Vary: Accept-Encoding
* Server gunicorn/0.17.4 is not blacklisted
< Server: gunicorn/0.17.4
Server: gunicorn/0.17.4
< Vary: Cookie, User-Agent, X-Requested-With, X-Ignore-Block
Vary: Cookie, User-Agent, X-Requested-With, X-Ignore-Block
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
x-content-type-options: nosniff
< Set-Cookie: csrftoken=ciOosbUNp5EL8t5tiQQzkoeaJIDJ3VfO; Domain=.mixcloud.com; expires=Tue, 17-Feb-2015 06:41:30 GMT; Max-Age=31449600; Path=/
Set-Cookie: csrftoken=ciOosbUNp5EL8t5tiQQzkoeaJIDJ3VfO; Domain=.mixcloud.com; expires=Tue, 17-Feb-2015 06:41:30 GMT; Max-Age=31449600; Path=/
< Set-Cookie: eventstream=; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/
Set-Cookie: eventstream=; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/
<
<!DOCTYPE html> ...
I know that the cURL for PHP and cURL are different, but I can't see what I am missing.
Anyone?
Cheers,
Maxime
Ok I've found what was the issue. It was the user-agent.
It's really weird. I was using this user-agent:
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5
With this user agent I was getting a 403. I've updated it using the following one:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36
And it's now working well. I can't believe that people are still rejecting request for specific user agent...