I'm using cURL with PHP to connect to my University admin website to provide a mobile-user-friendly interface in order to access my informations, such as grades. So basically I just have a form in which I put the login informations and my code uses these credentials to connect to the admin website and returns the wanted informations that I show in a mobile way.
The problem is that the connection doesn't work and I get a blank page excepted when I log on my account from any other device before attempting to connect with my program. That is, if for example this morning I accessed my account normally (and log out since), my code will work, but tomorrow it won't unless I access first normally my account.
I've been studying the connection process with Chrome's development tool again and again and I don't know where I'm wrong. The only suspicion I have is that the first page loaded with the credentials sent returns a 302 FOUND and the redirection is not applied by cURL, but the first page returns a connexion cookie, which I assumed was the only thing needed to be log in correctly. I think maybe the insertion of the created cookie into the server's database should be done on the second page in order for this one to be accepted next...
Here is my code:
$lien = 'https://isa.epfl.ch/imoniteur_ISAP/!logins.tryToConnect';
$login = $_POST['login'];
$password = $_POST['passwd'];
$postfields = array(
'ww_x_username' => $login,
'ww_x_password' => $password,
'ww_x_urlAppelant' => ''
);
$path_cookie = 'cookie.txt';
if (!file_exists(realpath($path_cookie))) touch($path_cookie);
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $lien);
curl_setopt($curl, CURLOPT_COOKIESESSION, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_POSTFIELDS, $postfields);
curl_setopt($curl, CURLOPT_COOKIEJAR, realpath($path_cookie));
curl_setopt($curl, CURLOPT_AUTOREFERER, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
$host = array(
0 => 'Host: '. parse_url("http://isa.epfl.ch", PHP_URL_HOST),
1 => 'Referer: https://isa.epfl.ch/imoniteur_ISAP/!logins.htm',
2 => 'Origin: https://isa.epfl.ch'
);
curl_setopt($curl, CURLOPT_HTTPHEADER, $host);
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13');
$return = curl_exec($curl);
curl_close($curl);
unset($curl);
I hope someone could help me! I'll be happy to give more precisions if necessary!
Thanks
Romain
I've solved the issue! It was pretty simple, maybe so simple that I didn't see the solution earlier...
If someone wants more explanation, I'll be happy to give it.
Related
I am trying to login into to a remote site using curl. ( before doing some data scraping)
Using the following code I am producing a cookies.txt file that has the following:
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
#HttpOnly_www.xxx.com FALSE / TRUE 0 xxxv5 h_r4hXtn-gNAilZwhvHjYdE3Vr4HewhxtGrxja57LbW03-M9MLNqZSeiW7lQ2wRT9lZypNsAiX0gS0Ev1PrvNkGLmwL3B8ZmyOUMLYbTYbSW0y_aPGrIFlEp4skDzh0GJGIGtFHisCmQjEMlu0CJr0UEw2rCT9jbjzg0IyOnFYxNffaMPo229NZWV7HDfCK5M1_y6MPNvW_Kt-h4qTy8YmqGbfBwKxB-bulV78MSXU9ZWz_DVvdu6jXfPiHwCBDMV8FFBLaXm5rqYgNzvbsq8JLe1xkTPn1PNJhyizUa-hlwB6ev8HNwIwBpzs7406l6mL3VgyrDJpay6bHNoMtjh4fLwI7KapFANhFHfn57mg4
#HttpOnly_www.xxx.com FALSE / TRUE 0 ASP.NET_SessionId txakhdi15oeqxyfq53f44dts
When I manually log into the web site the cookie names are correct. So I think I am creating the login ( otherwise the cookies would not be created) but when I output
echo 'HELLO html1 = '.$html1;
I see the page telling me I have entered the wrong username and password.
Code as follows:
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
$username = 'xxx';
$password = 'xxx';
// echo 'STARTING';
//login form action url
$url="https://www.xxxx.com/Login";
$postinfo = "username=".$username."&password=".$password;
$cookie_file_path = "cookie.txt";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_NOBODY, false);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
//set the cookie the site has for certain features, this is optional
curl_setopt($ch, CURLOPT_COOKIE, "cookiename=0");
curl_setopt($ch, CURLOPT_USERAGENT,
"Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, $_SERVER['REQUEST_URI']);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_MAXREDIRS,5); // return into a variable
// curl_setopt($ch, CURLOPT_UPLOAD, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST" );
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postinfo);
// set content length
$headers[] = 'Content-length: 0';
$headers[] = 'Transfer-Encoding: chunked';
curl_setopt($ch, CURLOPT_HTTPHEADER , $headers);
$html1 = curl_exec($ch);
echo 'HELLO html1 = '.$html1;
I cannot show the site for security reasons. ( which may be a killer)
Can anyone point me in the right direction?
first off, this won't work: ini_set('display_startup_errors', 1);
- the startup phase is already finished before the userland php code starts to run,
so this setting is set too late. it must be set in the php.ini config file. (not strictly true, but close enough, like on windows you can do crazy registry hacks to enable it, and you can set it with .user.ini files, etc, more info here http://php.net/manual/en/configuration.php )
second, obvious error here is that you don't urlencode $username and $password in $postinfo = "username=".$username."&password=".$password; -
if the username OR password contains any characters with special meanings in urlencoded format, you'll send the wrong credentials and won't get logged in (this includes &,=,#, spaces, and many other characters). fixed version would look like $postinfo = "username=".urlencode($username)."&password=".urlencode($password);
third, don't use CURLOPT_CUSTOMREQUEST for POST requests,
just use CURLOPT_POST.
fourth, your Content-length header is outright lying. the
correct length is actually 'Content-length: '.strlen($postinfo) - which with your code, is definitely not 0 -
but you shouldn't set this header at all, curl will do it for you
if you don't, and unlike you, curl won't mess up the code calculating
the size, so get rid of the entire line.
fifth, this code is also wrong:
$headers[] = 'Transfer-Encoding: chunked';
your curl code here is NOT using chuncked transfers,
and if it were, curl would send that header automatically,
so get rid of it.
sixth, don't just call curl_setopt, if there's an
error setting any of your options, curl_setopt will return
bool(false), and you should watch out for such errors,
use curl_error to extract the error message, and throw an exception,
if such an error occur. - instead of what your code is doing right now,
silently ignoring any curl_setopt errors. use something like
function ecurl_setopt($ch,int $option, $value){if(!curl_setopt($ch,$option,$value)){throw new \RuntimeException('curl_setopt failed!: '.curl_error($ch));}}
if fixing all of these problems is not enough to log in, you're not giving us enough information to help you any further. what does the browsers http login request look like? or what is the login url?
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
$username = 'xxx';
$password = 'xxx';
//login form action url
$url="https://www.xxxx.com/Login";
$postinfo = array("username"=>$username,"password"=>$password);
$cookie_file_path = "cookie.txt";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_SSL_VERIFYHOST,false);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER,false);
curl_setopt($ch,CURLOPT_COOKIEFILE,$cookie_file_path);
curl_setopt($ch,CURLOPT_COOKIEJAR,$cookie_file_path);
curl_setopt($ch, CURLOPT_USERAGENT,
"Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postinfo);
$html = curl_exec($ch);
echo $html;
Above code must works fine.
If there is still an issue, you must check cookie.txt file permissions.
Also if there is an invisible data needs to be sent including post, you can check it using firefox Live Http Headers plugin.
It is not as simple as reading the HTML page using curl. You need to supply a POST value for the submit button. If there is any javascript that executes prior to the activation of ACTION script, then that has to be looked at as well.
Usually you get better results if you use Selenium. See http://www.seleniumhq.org/
EDIT1:
If the server is rejecting your post string try: curl_setopt($handle, CURLOPT_POSTFIELDS, http_build_query($data));
I apologize in advance for my English. I have small problem.
I want to get Final Effective URL from page
streamuj.tv/video/00e276bf5841bf77c8de?streamuj=original&authorize=ac13bb77d3d863ca362315b9b4dcdf3e
When you put a link into the browser gives me to .flv file
But when I put it through PHP gives me s3.streamuj.tv/unauthorized.flv
When I try it through this: getlinkinfo.com/info?link=http%3A%2F%2Fwww.streamuj.tv%2Fvideo%2F00e276bf5841bf77c8de%3Fstreamuj%3Doriginal%26authorize%3Dac13bb77d3d863ca362315b9b4dcdf3e&x=49&y=11
So everything is fine indicates that
s4.streamuj.tv:8080/vid/d0fe77e1020b6414a16aa5316c759add/58aaf1dd/00e276bf5841bf77c8de_hd.flv?start=0
My PHP CODE:
<?php
session_start();
include "simple_html_dom.php";
$proxy = array("189.3.93.114:8080");
$proxyNum = 0;
$proxy = explode(':', $proxy[$proxyNum]);
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, 'http://www.streamuj.tv/video/00e276bf5841bf77c8de?streamuj=original&authorize=ac13bb77d3d863ca362315b9b4dcdf3e');
curl_setopt($curl, CURLOPT_FILETIME, true);
curl_setopt($curl, CURLOPT_NOBODY, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($curl, CURLOPT_PROXY, $proxy[0]);
curl_setopt($curl, CURLOPT_PROXYPORT, $proxy[1]);
$header = curl_exec($curl);
$info = curl_getinfo($curl);
curl_close($curl);
$u1 = $info['url'];
echo "u1: $u1</br>";
$u2 = str_replace("flv?start=0","flv",$u1);
echo $u2;
?>
Where is the problem? Why it makes unauthorized.flv?
Solution
Server was checking client legitimacy via user-agent HTTP header parameter.
Using custom user-agent solved the problem.
curl_setopt($curl, CURLOPT_HTTPHEADER, array( 'user-agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2950.0 Iron Safari/537.36' ));
Original post:
Most likely the generated flv URL is not pointing to static place. It
probably uses sessionID + cookie / verifies IP (one of these, or
both).
Without knowing what header you have to request with via CURL, you
probably won't get a relevant response.
I have a script that gathers a session id, puts it together with a URL and then redirects to the URL. This works perfectly in the browser and mx player for Android. But on kodi, there seems to be an error. Kodi seems to use my server as the host of the file. So instead of using: streamsite.com/index.m3u8, it uses MYSERVER.com/index.m3u8. This is driving me crazy since I do not even know how to code. This is my script:
<?php
$url = link.tojson
$cURL = curl_init();
curl_setopt($cURL,CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($cURL, CURLOPT_URL, $url); curl_setopt($cURL, CURLOPT_HTTPGET, true);
curl_setopt($cURL, CURLOPT_RETURNTRANSFER, true);
curl_setopt($cURL, CURLOPT_HTTPHEADER, array( 'Content-Type: application/json', 'Accept: application/json' ));
$result = curl_exec($cURL);
curl_close($cURL);
$json=json_decode($result,true);
$pre=$json[0]['id'];
$stream='streamsite.com/index.m3u8?&sessionId='.$pre. '';
ini_set('user_agent', 'Mozilla/5.0 (Linux; Android 6.0; en-US; Nexus 5 Build/Veneno ROM) MXPlayer/1.8.3
');
header("Location:$stream");
die();
?>
Try using a proper URL:
$stream="http://streamsite.com/index.m3u8?sessionId=$pre";
header("Location:$stream");
Also I don't know what you think that call to ini_set() will accomplish, but it won't.
I'm attempting to autologin to https://www.myicomfort.com/ to retrieve data. Tried some of the examples posted but does not seem to work. Maybe I'm not using the correct field names when passing the username and password. Can someone pls help? Still learning PHP. Thanks!
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"https://www.myicomfort.com/");
curl_setopt($ch, CURLOPT_COOKIEFILE,1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 120);
curl_setopt($ch, CURLOPT_TIMEOUT, 120);
$post_array = array(
'ctl00$RightContent$txtUserName'=>'xxxname',
'ctl00$RightContent$txtPwd'=>'xxxpassword',
);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_array);
$output = curl_exec($ch);
curl_close($ch);
`
You are missing the other fields. You can check which fields are being sent by pressing Ctrl+Shift+I in Chrome, then try to login. Click on the page and you will see something like this:
__LASTFOCUS:
__EVENTTARGET:
__EVENTARGUMENT:
__VIEWSTATE:/wEPDwUKMTczNjcxMDc0Mg9kFgJmD2QWAgIDD2QWBAIDD2QWBAI...
__EVENTVALIDATION:/wEWCAKI/qyXDAKSptf/CwKJn6v3AQKdg7/fBwKNoqOVDAK...
ctl00$RightContent$hdnPwd:
ctl00$RightContent$txtUserName:asfasdfa
ctl00$RightContent$txtPwd:dfasdf
ctl00$RightContent$chkRemember:on
ctl00$RightContent$btnLogin:Log in
Try to submit those data too. Once you've done this, you should be able to login.
Make sure you are using all the input fields required, doing a net sniff on information being transmitted from the form results in this
'__EVENTARGUMENT'
'__EVENTTARGET'
'__EVENTVALIDATION' /wEWCAKI/qyXDAKSptf/CwKJng .. etc
'__LASTFOCUS'
'__VIEWSTATE' /wEPDwUKMTczNjcxMDc0Mg9kFgJmD2QWAg .. etc
'ctl00$RightContent$btnLog...' Log in
'ctl00$RightContent$chkRem...' on
'ctl00$RightContent$hdnPwd'
'ctl00$RightContent$txtPwd' vyvuy
'ctl00$RightContent$txtUse...' iuhgiu
See also this question on SO where I believe they suggested to at a missing login button
U can look into the source code of that page, there are many hidden input fields. Just provide your username and password is not enough.
Solutions:
get all these fields and send them out together using PHP
use some library which can simulate browser, then u can only provide username and password when logging in. This would be a little slow, but NOT the bottleneck of your code.
I'm trying to access a file from a distant server with
$fp = #fsockopen($ip,$port,$errno,$errstr,1);
fputs($fp, "GET /randomfiletoget.html HTTP/1.0\r\nUser-Agent: Mozilla\r\n\r\n");
But the server needs a HTML authentification.
How should I pass the login/password to access the file?
Thanks!
To potential downvoters, I wrote authentification on purpose. Change Host to you domain name. It should return HTML output, with valid username and password.
function authentification($url, $username, $password){
$headers = array(
"Host=example.com",
"User-Agent=Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.11) Gecko/20101012 Firefox/3.6.11",
"Accept=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language=en-us,en;q=0.5",
"Accept-Encoding=gzip,deflate",
"Accept-Charset=ISO-8859-1,utf-8;q=0.7,*;q=0.7",
"Date: ".date(DATE_RFC822)
);
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($curl, CURLOPT_USERPWD, "$username:$password");
return curl_exec($curl);
}
As webarto mentioned in his comment, you should look into cURL, it supports basic HTTP auth among many other handy features.
For one, you should use Host header too. Todays hosting environments need it in 99%.
Basic http autentication is done with headers and Base64 encoding. This wikipedia article will tell you everÃthing. Or use PHP's cURL functions.