I'm trying to make a POST sending a page with the following code:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.olx.es/posting_success.php");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
curl_setopt($ch, CURLINFO_HEADER_OUT, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTREDIR, 2);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, array("itemid" => $json['id'], "sh" => $json['sh']));
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-type: application/x-www-form-urlencoded"));
$response = curl_exec($ch);
curl_close($ch);
And in doing so I returned a server error saying "Invalid Request".
I've noticed that if I do the request manually by telnet, the first time I returned a 302, despite the fact that I've put to follow him, not work, and maybe it's this.
With wireshark obtained what the customer actually sent, what am I doing wrong?
**Hypertext Transfer Protocol:**
POST /posting_success.php HTTP/1.1
Host: www.olx.es
Connection: keep-alive
Content-Length: 52
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: http://www.olx.es
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
Content-Type: application/x-www-form-urlencoded
Referer: http://www.olx.es/posting.php?categ_id=322
Accept-Encoding: gzip,deflate,sdch
Accept-Language: es-ES,es;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
**Line-based text data: application/x-www-form-urlencoded:**
itemid=501347053&sh=b9302ed20769ae3717f896a33a369aa2
Sorry for my english..
It Seems You have Problem in POST request. please Try
$fields=array('itemid' => $json['id'], 'sh' =>urlencode ($json['sh']));
Format Your Fields
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; }
rtrim($fields_string,'&');
Then Use in Curl OPt
curl_setopt($ch1,CURLOPT_POST,count($fields));
curl_setopt($ch1,CURLOPT_POSTFIELDS,$fields_string);
There is no need to make custom request, sent such headers or use CURLINFO_HEADER_OUT.
Try this:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.olx.es/posting_success.php");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTREDIR, 2);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query( array("itemid" => $json['id'], "sh" => $json['sh']) ));
$response = curl_exec($ch);
curl_close($ch);
the problem is, this form is not multipart/form-data, and then you will to create a query string to post. See the manpage http://php.net/manual/en/function.curl-setopt.php. you give then an array. The the header will be set a multipart/form-data. And this is the invalid request.
THE SOLUTION IS
The string is simple to build "&itemid=".json['id']."&sh=".json['sh'];
put this in the CURLOPT_POSTFIELDS.
saludos (bye) and sorry for my english...
Related
Can anybody please tell what i am doing wrong in below code ?
I am working on php script to login and scrape data from https://beams.us.yazaki.com/BEAMSLogin/Login.aspx?ReturnUrl=%2FBeams%2FDefault.aspx
I have tried with Below curl code.
<?php
define('USERNAME', 'XXX');
define('PASSWORD', 'XXX');
define('DOMAIN', 'XXX');
define('USER_AGENT', 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36');
define('COOKIE_FILE', 'cookie.txt');
define('LOGIN_FORM_URL', 'https://beams.us.yazaki.com/BEAMSLogin/Login.aspx?ReturnUrl=%2fBeams%2fDefault.aspx');
define('LOGIN_ACTION_URL', 'https://beams.us.yazaki.com/BEAMSLogin/Login.aspx?ReturnUrl=%2fBeams%2fDefault.aspx');
$postValues = array(
'txtUserName' => USERNAME,
'txtPassword' => PASSWORD,
'lstDomain' => DOMAIN
);
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, LOGIN_ACTION_URL);
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_POSTFIELDS, http_build_query($postValues));
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_COOKIEJAR, COOKIE_FILE);
curl_setopt($curl, CURLOPT_USERAGENT, USER_AGENT);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_REFERER, LOGIN_FORM_URL);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, false);
curl_exec($curl);
if(curl_errno($curl)){
throw new Exception(curl_error($curl));
}
curl_setopt($curl, CURLOPT_URL, 'https://beams.us.yazaki.com/BEAMS/SearchAndResults.aspx?topic=component');
curl_setopt($curl, CURLOPT_COOKIEJAR, COOKIE_FILE);
curl_setopt($curl, CURLOPT_USERAGENT, USER_AGENT);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
echo curl_exec($curl);
?>
I am getting the output like below page.
Object not found!
The requested URL was not found on this server. The link on the referring page seems to be wrong or outdated. Please inform the author of that page about the error.
If you think this is a server error, please contact the webmaster.
Error 404
localhost
Apache/2.4.43 (Win64) OpenSSL/1.1.1g PHP/7.4.8 *
And the browser is getting redirected to http://localhost/BeamsLogin/AuthenticationSelect.aspx?ReturnUrl=%2fBEAMS%2fSearchAndResults.aspx%3ftopic%3dcomponent&topic=component
I am earnestly waiting for response.
Thanks.
EDIT 1
#Shibon
I think this website doesn't use any csrf token. I am giving HTTP header live content. Please check once.
https://beams.us.yazaki.com/BEAMSLogin/Login.aspx?ReturnURL=%2fBeams%2fDefault.aspx
Host: beams.us.yazaki.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://beams.us.yazaki.com/
Connection: keep-alive
Upgrade-Insecure-Requests: 1
GET: HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Tue, 15 Sep 2020 06:19:19 GMT
Content-Length: 3376
Try the bellow code
$ch= curl_init();
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_COOKIEJAR, "-"); //to set session and cookie for the current $ch object
curl_setopt($ch, CURLOPT_URL, LOGIN_ACTION_URL);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($postValues));
$resp = curl_exec($ch);
if(!$resp) {
print_r(curl_error($ch));
exit;
}
curl_setopt($curl, CURLOPT_URL, 'https://beams.us.yazaki.com/BEAMS/SearchAndResults.aspx?topic=component');
curl_setopt($ch, CURLOPT_POST, true);
echo curl_exec($ch);
In this have set cookie initially and the used that in all request
I try curl a website for show it in mine but always I stuck in a white page. I think because it have a redirection to a login form but I dont sure if is the reason because I failed. You can access the url that I use without login.
Here the code
$url = "http://www.faf.es/pnfg/NPcd/NFG_CmpJornada?cod_primaria=1000120&CodCompeticion=16867461&CodGrupo=17910021&CodTemporada=15&CodJornada=26&Sch_Codigo_Delegacion=1&Sch_Tipo_Juego=2";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_TIMEOUT, 99999);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$output = curl_exec($ch);
curl_close($ch);
htmlentities($output);
The html result:
<html>
<head>
<title></title>
</head>
<body>
<!--
<h1>No se ha aceptado el cookie</h1>
-->
</body>
</html>
Your request url only validate requests which have JSESSIONID cookie. So before you must grab valid cookie:
<?php
$url = "http://www.faf.es/pnfg/NPcd/NFG_CmpJornada?cod_primaria=1000120&CodCompeticion=16867461&CodGrupo=17910021&CodTemporada=15&CodJornada=26&Sch_Codigo_Delegacion=1&Sch_Tipo_Juego=2";
$ch = curl_init();
$useragent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com/');
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_TIMEOUT, 99999);
$output = curl_exec($ch);
curl_close($ch);
echo $output;
This gives you three headers for requests:
HTTP/1.1 302 Movido temporalmente
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=648967C5FC907B1225EC61E9A65443E5; Path=/pnfg
Location: http://www.faf.es/pnfg/NLogin
Content-Length: 0 Date: Mon, 22 Jun 2020 19:33:30 GMT
Connection: close
HTTP/1.1 302 Movido temporalmente
Server: Apache-Coyote/1.1
Cache-Control: no-cache
Pragma: no-cache
Set-Cookie: JSESSIONID=9973308FA3AF81BE66C2F8C124671870; Path=/pnfg
Location: http://www.faf.es/pnfg/NLogin?NSess=1
Content-Length: 0 Date: Mon, 22 Jun 2020 19:33:30 GMT
Connection: close
HTTP/1.1 200 OK Server: Apache-Coyote/1.1
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html;charset=ISO-8859-15
Content-Length: 117
Date: Mon, 22 Jun 2020 19:33:30 GMT
Connection: close
We have three header result because we said, request has CURLOPT_FOLLOWLOCATION option. So, we made three request in curl.
I identified, second JSESSIONID cookie value is valid. So, for getting web site information we must use cookie as below:
<?php
$url = "http://www.faf.es/pnfg/NPcd/NFG_CmpJornada?cod_primaria=1000120&CodCompeticion=16867461&CodGrupo=17910021&CodTemporada=15&CodJornada=26&Sch_Codigo_Delegacion=1&Sch_Tipo_Juego=2";
$ch = curl_init();
$useragent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36';
curl_setopt($ch, CURLOPT_URL, $url);
// Secondary request cookie.
curl_setopt($ch, CURLOPT_COOKIE, "JSESSIONID=9973308FA3AF81BE66C2F8C124671870");
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com/');
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_TIMEOUT, 99999);
$output = curl_exec($ch);
curl_close($ch);
echo $output;
When web site content comes, do whatever you want.
You should revisit the htmlentities() documentation again; it doesn't do what it appears you believe it does.
In short, htmlentities() returns a string. You aren't capturing or otherwise outputting this string result anywhere. Use echo (or similar) to make sure the result of htmlentities($output) is output to where you expect it.
$url = "http://www.faf.es/pnfg/NPcd/NFG_CmpJornada?cod_primaria=1000120&CodCompeticion=16867461&CodGrupo=17910021&CodTemporada=15&CodJornada=26&Sch_Codigo_Delegacion=1&Sch_Tipo_Juego=2";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_TIMEOUT, 99999);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$output = curl_exec($ch);
curl_close($ch);
echo htmlentities($output);
We have an assignment to filter authentic curl requests from robots. I am sending a curl request to the site, but it's returning to me an invalid image file(i know because when i view it with my browser it works). It somehow knows my request is not authentic. Is there a field I'm overlooking here, I'm trying to mimic a browser request exactly.
$header_arr = array(
'0' =>'Host: www.myittest.com',
'1' =>'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0',
'2' =>'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*\/*;q=0.8',
'3' =>'Accept-Language: en-US,en;q=0.5',
'4' =>'Accept-Encoding: gzip, deflate',
'5' =>'Connection: keep-alive',
);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header_arr);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_MAXREDIRS, 6);
$raw=curl_exec($ch);
You have requested gzip/deflate encoding but haven't made curl aware of it so it doesn't decode the image. Adding this should fix it:
curl_setopt($ch, CURLOPT_ENCODING, '');
I am trying to figure out why pasing a custom header is resulting in a 400 BAD REQUEST from the server.
$headers = array(
'API KEY: asdf',
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0" );
curl_setopt($ch, CURLOPT_URL, 'http://url');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1 );
curl_setopt($ch, CURLOPT_POSTFIELDS, 'stuff');
curl_setopt($ch, CURLOPT_COOKIEFILE, './tmp/cookie.txt');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_PROXYPORT, '0000');
curl_setopt($ch, CURLOPT_PROXYTYPE, 'HTTP');
curl_setopt($ch, CURLOPT_PROXY, '0.0.0.0');
$result = curl_exec($ch);
curl_close($ch);
I thought that using CURLOPT_HTTPHEADER would add a custom header to the request, but I'm now wondering whether it's simply overriding everything else I set?
There are more reasons a server will give a 400 response than just a header value. Without more information about the endpoint it's difficult to say what's causing the 400 response. With the exception of the extra "," in the headers array in the example the code looks okay. cURL Options
I am trying to login into Tesco Mobile, using PHP curl, I have the following code:
$postData='user=XXX&password=XXXXX';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://paymonthly.tescomobile.com/orderentry/LoginSubmit.do');
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, TRUE);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_REFERER, 'https://paymonthly.tescomobile.com/orderentry/LoginSubmit.do');
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$result = curl_exec ($ch);
curl_close($ch);
But have been unsuccessful, please can anyone help how I can achieve this. Please find the output from Live HTTP below:
Thanks in advance.
https://paymonthly.tescomobile.com/orderentry/LoginSubmit.do
POST /orderentry/LoginSubmit.do HTTP/1.1 Host:
paymonthly.tescomobile.com User-Agent: Mozilla/5.0 (Macintosh; Intel
Mac OS X 10.7; rv:18.0) Gecko/20100101 Firefox/18.0 Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate
Referer:
https://paymonthly.tescomobile.com/orderentry/LoginSubmit.do;jsessionid=60af415cd448d5728a9fa98cdf99
Cookie: JSESSIONID=60af415cd448d5728a9fa98cdf99;
reevoomark_marker=349049566;
reevoo_utma=1.324660794.1360800582.1360800582.1360800582.1;
reevoo_utmb=1.5.6.1360800582; reevoo_utmc=1;
reevoo_utmz=1.1360800582.1.1.utmgclid=CK6Vp5nBtLUCFeXKtAodVGIAVA|utmccn=(not%20set)|utmcmd=(not%20set)|utmctr=tescomobile;
reevoomark_bh_session=0af0a16fc00f5fbc8b0d28c571721a72;
CUSTOMER_UUID=04fdc3e6-59e1-4038-81b7-ec7c5d63c9d4;
__utma=152930744.490151867.1360800710.1360800710.1360800710.1; __utmb=152930744; __utmc=152930744; __utmz=152930744.1360800710.1.1.utmccn=(referral)|utmcsr=phone-shop.tesco.com|utmcct=/tesco-mobile/my-tesco-mobile/|utmcmd=referral
Connection: keep-alive Content-Type: application/x-www-form-urlencoded
Content-Length: 32 user=dinoc83&password=c079378643 HTTP/1.1 200 OK
X-Powered-By: JSP/2.1 Server: Sun GlassFish Enterprise Server v2.1.1
Content-Type: text/html;charset=UTF-8 Transfer-Encoding: chunked Date:
Thu, 14 Feb 2013 00:15:27 GMT
Try to change :
$postData='user=XXX&password=XXXXX';
by $postData=array('user=XXX&password=XXXXX');
There are some code samples in : http://php.net/manual/en/function.curl-setopt.php
Regards,
Just checked form
Form attribute action have parameter jsessionid In your code you don't use it. I think you should in first load page with form, then parse response code and get correct URL for post. And this need for initialize cookies.
Try to this in this way it's working for us posting data in another server
$postData='user=XXX&password=XXXXX'
$url ="https://paymonthly.tescomobile.com/orderentry/LoginSubmit.do";
$ch = curl_init() or die(curl_error());
curl_setopt($ch, CURLOPT_POST,1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$postData);
curl_setopt($ch, CURLOPT_URL,$url);// here the request is sent to payment gateway
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result=curl_exec($ch) or die(curl_error());
curl_close($ch);
You can also set $postData value this way:
$postData = array( 'user'=>'XXX', 'password'=>'XXXX' );
And note what:
Passing an array to CURLOPT_POSTFIELDS will encode the data as multipart/form-data, while passing a URL-encoded string will encode the data as application/x-www-form-urlencoded.