I have some code similar to Jason Lattimer's below that used to work however now fails. I can find the "Identifier" in the wsdl anymore. Can anyone provide assistance on this?
function GetADFS($url) {
$ch = curl_init ();
curl_setopt ( $ch, CURLOPT_URL, $url . "XrmServices/2011/Organization.svc?wsdl=wsdl0" );
curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt ( $ch, CURLOPT_TIMEOUT, 60 );
curl_setopt ( $ch, CURLOPT_SSL_VERIFYPEER, false );
$response = curl_exec ( $ch );
curl_close ( $ch );
$responsedom = new DomDocument ();
$responsedom->loadXML ( $response );
// **************************************************
// **************************************************
// Can no longer find this Identifier
// **************************************************
// **************************************************
$identifiers = $responsedom->getElementsbyTagName ( "Identifier" );
$identifier = $identifiers->item ( 0 )->textContent;
return str_replace ( "http://", "https://", $identifier );
}
It's in namespace 'ms-xrm' so replace
$identifiers = $responsedom->getElementsbyTagName ( "Identifier" );
with
$identifiers = $responsedom->getElementsbyTagNameNS ( "ms-xrm","Identifier" );
and check. Next thing is to check in actual WSDL if you see this node if not then there may be a problem with configuration of your CRM because that node is only available when you are using ADFS.
Related
Hello I would like to get some data from a url. I already tried to make a request through the console, postman, browset - all of them worked correctly. But if I make a request using php (guzzle, symfony http client) it fails on SSL. Does anybody know how to get response from this url by curl? Thanks!
Please try the following code ( That's worked for me ):
<?php
function curl( $url, $data = array(), $headers = array(), $ssl_required = false ) {
$handle = curl_init( $url );
curl_setopt( $handle, CURLOPT_RETURNTRANSFER, true );
// Set post data if exist
if ( !empty( $data ) ) {
curl_setopt( $handle, CURLOPT_POST, true );
curl_setopt( $handle, CURLOPT_POSTFIELDS, $data );
}
// Set custom headers if exist
if ( count( $headers ) )
curl_setopt( $handle, CURLOPT_HTTPHEADER, $headers );
// If url was ssl, need to true
if ( $ssl_required )
curl_setopt($handle, CURLOPT_SSL_VERIFYPEER, false);
$output = curl_exec( $handle );
curl_close( $handle );
return $output;
}
echo curl("https://www.skroutz.gr/c/900/fakoi-epafhs.json");
I do not receive a successful login to $loginUrl (i.e. no cookie.txt file is generated in the same directory as the file) based on the code below and therefore am unable to load HTML data from $url (i.e. echoes did not load). When I view the curl_exec for loginUrl, it looks like it doesn't submit the username and password to the form, although I have $store = curl_exec($ch), as the form is displayed instead of a successful login.
function parseDOM($data)
{
global $projectID, $sRedirect, $database;
libxml_use_internal_errors(true);
$dom = new DOMDocument();
if(!$dom->loadHTML($data))
{
echo "did not load";
}
}
$ch = #curl_init();
if($ch)
{
$username = 'username';
$password = 'password';
//$url = 'https://global-factiva-com.libproxy.lib.unc.edu/ha/default.aspx#./!?&_suid=14977301633480007720669669887936';
//trying different URL
$url = 'https://global.factiva.com.libproxy.lib.unc.edu/redir/default.aspx?P=sa&NS=16&AID=9UNI011500&f=g&an=j000000020010807dw8b00lc2&cat=a';
//loginUrl is the same as the URL for the form post action
$loginUrl = 'https://sso.unc.edu/idp/profile/SAML2/POST/SSO;jsessionid=A2C0B6480084BED37E1104E903B07AA9?execution=e1s1';
//Set the URL to work with
curl_setopt($ch, CURLOPT_URL, $loginUrl);
// ENABLE HTTP POST
curl_setopt($ch, CURLOPT_POST, 1);
//Set the post parameters
curl_setopt($ch, CURLOPT_POSTFIELDS, 'j_username='.$username.'&j_password='.$password);
//Handle cookies for the login
$cookie=dirname(__FILE__)."\\cookie.txt";
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
//execute the request (the login)
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$store = curl_exec($ch);
//now access the URL that requires login
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$content=curl_exec($ch);
$headers = curl_getinfo($ch);
curl_close($ch);
parseDOM($content);
}
Here is the approach I would use. First, use Google Chrome and open the network inspector. If you then manually log in then you'll be able to see all the request headers sent, the form fields, etc.
Armed with this information you can construct a curl request and specify all the custom headers. I have worked with systems before that reject requests without a legitimate referer or user-agent.
So, for example ..
<?php
$username = 'hello';
$password = 'letmein';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"https://sso.unc.edu/idp/profile/SAML2/POST/SSO?execution=e1s1");
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS,'j_username:='.$username.'&j_password:='.$password.'&_eventId_proceed:=');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$headers = [
'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding:gzip, deflate, br',
'Accept-Language:en-US,en;q=0.8,es;q=0.6',
'Cache-Control:max-age=0',
'Connection:keep-alive',
'Content-Length:57',
'Content-Type:application/x-www-form-urlencoded',
'Host:sso.unc.edu',
'Origin:https://sso.unc.edu',
'Referer:https://sso.unc.edu/idp/profile/SAML2/POST/SSO?execution=e1s1',
'Upgrade-Insecure-Requests:1',
'User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
];
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$output = curl_exec ($ch);
curl_close ($ch);
echo $output;
?>
Once you run this then hopefully you are logged in and the cookie is set. You can then make a second request using a new curl_init() to the second URL and include the CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR parameters.
Hopefully this gives you something to work from. Good luck.
you haven't told us where you want to login, but in a comment, you posted this link https://auth.lib.unc.edu/ezproxy_auth.php?url=https://global.factiva.com/ha/default.aspx , which itself links to 4 different login pages. however, the CURLOPT_VERBOSE log you posted in the comment at Cookies File Not Generated PHP and cURL Login from Remote Site suggests you are trying to login to the site called Onyen. after doing some research, turns out they have a very odd login system, which starts at https://auth.lib.unc.edu/ezproxy_auth.php?url=https://global.factiva.com/ha/default.aspx - make a GET request to that url, this will create a cookie session that you'll need for all subsequent requests, and give you needed information in the HTML. parse out the HTML <form-element that contains the <input-element that contains Onyen in the value-property (the input form to look for, looks like <input name="submit" value="Onyen Sign In" accesskey="o" type="submit">), this form element gives you 3 input elements who's value you need to add to your next GET request, whos url you get from the form's action attribute. i suspect all the values are constant, except the 1 called auth, which is probably unique to each cookie session, or ip address, or something like that. the URL as generated from my browser (and later, by php) turned out to be https://auth.lib.unc.edu/authentication.php?url=https://global.factiva.com/ha/default.aspx&auth=shibboleth&submit=Onyen+Sign+In - now if you did everything correctly, generating the correct url and sending it with the cookies recieved in the previous request, it should respond with a 302 Found http-redirect, which you must follow. following this redirect, you get another page with a html with a single <form tag, whose url you must extract, and <input elements who's name and value you must parse out, and add to your neext POST request, which goes to https://sso.unc.edu/idp/profile/SAML2/POST/SSO - now this POST gives a http 302 Found redirect, which you must also follow. now follwing that redirect, you finally get to https://sso.unc.edu/idp/profile/SAML2/POST/SSO?execution=e2s1, where there's a html with a single <form tag, with <input elements who's name and value you must parse out, and fill out the j_username and j_password inputs, and add to your next POST request, which goes to https://sso.unc.edu/idp/profile/SAML2/POST/SSO?execution=e2s1 - now sending that POST request with a valid username/password and cookie session, will probably get you logged in. here's an implementation, using DOMDocument/DOMXpath for HTML parsing, and hhb_curl from https://github.com/divinity76/hhb_.inc.php/blob/master/hhb_.inc.php for http/cookies stuff (its a libcurl wrapper), just replace username_here with a real username, and password_here with a real password, on line 72 & 73.
<?php
declare(strict_types = 1);
require_once ('hhb_.inc.php');
function getFormUrl(\hhb_curl $hc, \DOMNode $form): string {
$url = $form->getAttribute ( "action" );
if (empty ( $url )) {
$url = '';
}
if (! parse_url ( $url, PHP_URL_HOST )) {
$url = 'https://' . rtrim ( parse_url ( $hc->getinfo ( CURLINFO_EFFECTIVE_URL ), PHP_URL_HOST ), '/' ) . '/' . ltrim ( $url, '/' );
}
if (false === strpos ( $url, '?' )) {
$url .= '?';
}
return $url;
}
$hc = new hhb_curl ( 'https://auth.lib.unc.edu/ezproxy_auth.php?url=https://global.factiva.com/ha/default.aspx', true );
$hc->exec ();
// hhb_var_dump ( $hc->getStdErr (), $hc->getStdOut () );
$domd = #DOMDocument::loadHTML ( $hc->getResponseBody () );
$form = (new DOMXPath ( $domd ))->query ( '//input[contains(#value,\'Onyen Sign In\')]/parent::form' )->item ( 0 );
$url = getFormUrl ( $hc, $form );
// probably looks like $url = 'https://auth.lib.unc.edu/authentication.php?';
$queryparms = array ();
foreach ( $form->getElementsByTagName ( "input" ) as $input ) {
$url .= urlencode ( $input->getAttribute ( "name" ) ) . '=' . urlencode ( $input->getAttribute ( "value" ) ) . '&';
}
$url = substr ( $url, 0, - 1 );
// hhb_var_dump ( $url );
$hc->exec ( $url );
// hhb_var_dump ( $hc->getStdErr (), $hc->getStdOut () );
$domd = #DOMDocument::loadHTML ( $hc->getResponseBody () );
$form = $domd->getElementsByTagName ( "form" )->item ( 0 );
$url = getFormUrl ( $hc, $form );
$posts = array ();
foreach ( $form->getElementsByTagName ( "input" ) as $input ) {
$name = $input->getAttribute ( "name" );
if (empty ( $name )) {
continue;
}
$posts [$name] = $input->getAttribute ( "value" );
}
// hhb_var_dump ( $posts );
$hc->setopt_array ( array (
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => http_build_query ( $posts ),
CURLOPT_URL => $url
) );
$hc->exec ();
// hhb_var_dump ( $hc->getStdErr (), $hc->getStdOut () );
$domd = #DOMDocument::loadHTML ( $hc->getResponseBody () );
$form = $domd->getElementsByTagName ( "form" )->item ( 0 );
$url = getFormUrl ( $hc, $form );
$posts = array ();
foreach ( $form->getElementsByTagName ( "input" ) as $input ) {
$name = $input->getAttribute ( "name" );
if (empty ( $name )) {
continue;
}
$posts [$name] = $input->getAttribute ( "value" );
}
foreach ( $form->getElementsByTagName ( "button" ) as $button ) {
$name = $button->getAttribute ( "name" );
if (empty ( $name )) {
continue;
}
$posts [$name] = $button->getAttribute ( "value" );
}
assert ( isset ( $posts ['j_username'] ), 'failed to find the username input!' );
assert ( isset ( $posts ['j_password'] ), 'failed to find the password input!' );
$posts ['j_username'] = 'username_here';
$posts ['j_password'] = 'password_here';
hhb_var_dump ( $posts );
$hc->setopt_array ( array (
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => http_build_query ( $posts ),
CURLOPT_URL => $url
) );
$hc->exec ();
hhb_var_dump ( $hc->getStdErr (), $hc->getStdOut () );
edit: fixed a bug where a <button> element's name didnt get appended to the POST data.
Say we have this array
$args = array('responseType' => 'Xml',
'serverName' => 'vl18278.dinaserver.com',
'command' => 'Vps_GetUsedSpace',
) ;
This array composes an URL to send through cURL. I need to replace vl18278.dinaserver.com with a variable $vps, but when I replace it, the URL show a %5B0%5D just before the = sign of the attribute serverName:
responseType=Xml&serverName%5B0%5D=vl18278.dinaserver.com&command=Vps_GetUsedSpace
If I dont replace the vl18278.dinaserver.com, the URL is correct.
What is wrong with my code? Why are those %5B0%5D getting into my URL? :(
Thanks in advance.
Complete code:
<?php
$listavps = simplexml_load_file('servers.xml');
foreach ($listavps->servers->server as $vps) {
$urlApi = 'url.php';
$username = 'user';
$password = 'pass';
$args = array('responseType' => 'Xml',
'serverName' => 'vl18278.dinaserver.com',
'command' => 'Vps_GetUsedSpace',
) ;
$args = ( is_array ( $args ) ? http_build_query ( $args, '', '&' ) : $args );
$headers = array();
$handle = curl_init($urlApi);
if( $handle === false ) // error starting curl
{
$error = '0 - Couldn\'t start curl';
}
else
{
curl_setopt ( $handle, CURLOPT_FOLLOWLOCATION, true );
curl_setopt ( $handle, CURLOPT_RETURNTRANSFER, true );
curl_setopt ( $handle, CURLOPT_URL, $urlApi );
curl_setopt( $handle, CURLOPT_USERPWD, $username.':'.$password );
curl_setopt( $handle, CURLOPT_HTTPAUTH, CURLAUTH_BASIC );
curl_setopt( $handle, CURLOPT_TIMEOUT, 60 );
curl_setopt( $handle, CURLOPT_CONNECTTIMEOUT, 4); // set higher if you get a "28 - SSL connection timeout" error
curl_setopt ( $handle, CURLOPT_HEADER, true );
curl_setopt ( $handle, CURLOPT_HTTPHEADER, $headers );
$curlversion = curl_version();
curl_setopt ( $handle, CURLOPT_USERAGENT, 'PHP '.phpversion().' + Curl '.$curlversion['version'] );
curl_setopt ( $handle, CURLOPT_REFERER, null );
curl_setopt ( $handle, CURLOPT_SSL_VERIFYPEER, false ); // set false if you get a "60 - SSL certificate problem" error
curl_setopt ( $handle, CURLOPT_POSTFIELDS, $args );
curl_setopt ( $handle, CURLOPT_POST, true );
$response = curl_exec ( $handle );
echo $args;
if ($response)
{
$response = substr( $response, strpos( $response, "\r\n\r\n" ) + 4 ); // remove http headers
// parse response
$responseSimpleXml = simplexml_load_string($response);
if( $responseSimpleXml === false )
{
// invalid xml response
}
else
{
// parse response
$errorCode = $responseSimpleXml->response->responseCode ;
echo $errorCode;
if( $errorCode == 1000 ) // success
{
$usado = $responseSimpleXml->response->data->total_space;
$capacidad = $responseSimpleXml->response->data->space_limit;
echo 'Usado: '.$usado.'</br>Total: '.$capacidad.'.';
}
else // normal errors
{
$errors = $responseSimpleXml->response->errors;
foreach( $errors->error as $error )
{
// process error
}
}
}
}
else // http response code != 200
{
$error = curl_errno ( $handle ) . ' - ' . curl_error ( $handle );
}
curl_close($handle);
}
}
?>
Your variable $server must be an array, because, once decoded, %5B0%5D is [0].
My guess is to use $server[0] instead of $server wherever you replace the value. Without the replacement code, it is hard to determine.
I solved this using rawurlencode in the $listavps variable before using it.
<?php
$listavps = simplexml_load_file('servers.xml');
foreach ($listavps->servers->server as $key => $tag) {
$vps = rawurlencode ($tag);
$urlApi = 'url.php';
$username = 'user';
$password = 'pass';
$args = array('responseType' => 'Xml',
'serverName' => $vps,
'command' => 'Vps_GetUsedSpace',
) ;
I need to make multiple API calls to the uClassify Sentiment classifier to get the sentiment for a number of tweets. Since I have a lot of tweets to index, simply using cURL is not enough (it takes nearly 2 minutes to fully index around 228 tweets).
Without sentiment analysis, indexing is almost instantaneous so the problem is definitely due to the high number of API calls.
I have instead considered to use the curl_multi_init. Whenever an API call is made, curl_init() is called and rather than processing the call, the handle is added to curl_multi. Once all the handles are added, I use the curl_multi_exec() function to process all the handles.
Here is a simplified version of my application to only show the sentiment part:
$mh = curl_multi_init ();
foreach ($tweets as $tweet){
getSentiment ( $tweet, $mh );
}
executeHandles($mh);
function getSentiment($tweet, $mh) {
$tweet = str_replace ( ' ', '+', $tweet );
$prefix = 'http://uclassify.com/browse/uClassify/Sentiment/ClassifyText?';
$key = 'readkey=' . CLASSIFY_KEY . '&';
$text = 'text=' . $tweet . '&';
$version = 'version=1.01';
$url = $prefix . $key . $text . $version;
// $xml = getXML($url, $mh);
addHandle ( $url, $mh );
// $xml = file_get_contents($url, false, $context); ---- TOO SLOWh
// $mood = parseSentiment($xml);
// return $mood;
}
function addHandle($url, $mh) {
$ch = curl_init ();
$timeout = 5;
curl_setopt ( $ch, CURLOPT_URL, $url );
curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt ( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );
curl_multi_add_handle ( $mh, $ch );
// $xml = curl_exec($ch);
curl_close ( $ch );
// return $xml;
}
function executeHandles($mh) {
if (! empty ( $mh )) {
$active = null;
// execute the handles
do {
$mrc = curl_multi_exec ( $mh, $active );
} while ( $mrc == CURLM_CALL_MULTI_PERFORM );
while ( $active && $mrc == CURLM_OK ) {
if (curl_multi_select ( $mh ) == - 1) {
usleep ( 100 );
}
do {
$mrc = curl_multi_exec ( $mh, $active );
} while ( $mrc == CURLM_CALL_MULTI_PERFORM );
}
}
}
This is returning
curl_multi_exec(): 12 is not a valid cURL handle resource in C:\xampp\htdocs\Twitter\twitteroauth-master\index.php on line 299
This is referring to this line of code:
$mrc = curl_multi_exec ( $mh, $active );
Now this is just my first time using cURL so I am not sure if I am missing some important detail. I cannot understand why this error is happening, I do not have any curl statements that are happening after curl_close() etc.
Any help would be greatly appreciated, thank you!
so if you need those handles, why did you close them?
function addHandle($url, $mh) {
$ch = curl_init ();
$timeout = 5;
curl_setopt ( $ch, CURLOPT_URL, $url );
curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt ( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );
curl_multi_add_handle ( $mh, $ch );
}
I have an AWS setup with Apache/PHP server on Port 80 and a REST Tomcat server on 8080.
If I try to access REST Services using IP Address A.B.C.D:8080/restapp from outside it works.
However if I try invoking from PHP code on the same box, it throws an internal error. Need your expert help in debugging this:
Checklist:
Security Profile:
8080 and 80 opened for 0.0.0.0/0
URL to be invoked: http://ec2-A-B-C-D.us-west-1.compute.amazonaws.com/myapp/ba-simple-proxy1.php?url=http://ec2-A-B-C-D.us-west-1.compute.amazonaws.com:8080/restapp/rest/user
ERROR RESPONSE:
"NetworkError: 500 Internal Server Error - http://ec2-A-B-C-D.us-west-1.compute.amazonaws.com/myapp/ba-simple-proxy1.php?url=http://ec2-A-B-C-D.us-west-1.compute.amazonaws.com:8080/restapp/rest/user"
Code Snippet from PHP - ba-simple-proxy1.php:
//print $url;
if ( !$url ) {
// Passed url not specified.
$contents = 'ERROR: url not specified';
$status = array( 'http_code' => 'ERROR' );
} else if ( !preg_match( $valid_url_regex, $url ) ) {
// Passed url doesn't match $valid_url_regex.
$contents = 'ERROR: invalid url';
$status = array( 'http_code' => 'ERROR' );
} else {
$ch = curl_init( $url );
if ( strtolower($_SERVER['REQUEST_METHOD']) == 'post' ) {
curl_setopt( $ch, CURLOPT_POST, true );
curl_setopt( $ch, CURLOPT_POSTFIELDS, $_POST );
}
if ( $_GET['send_cookies'] ) {
$cookie = array();
foreach ( $_COOKIE as $key => $value ) {
$cookie = array();
foreach ( $_COOKIE as $key => $value ) {
$cookie[] = $key . '=' . $value;
}
if ( $_GET['send_session'] ) {
$cookie[] = SID;
}
$cookie = implode( '; ', $cookie );
curl_setopt( $ch, CURLOPT_COOKIE, $cookie );
}
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
curl_setopt( $ch, CURLOPT_HEADER, true );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt( $ch, CURLOPT_USERAGENT, $_GET['user_agent'] ? $_GET['user_agent'] : $_SERVER['HTTP_USER_AGENT'] );
list( $header, $contents ) = preg_split( '/([\r\n][\r\n])\\1/', curl_exec( $ch ), 2 );
//print $ch;
$status = curl_getinfo( $ch );
curl_close( $ch );
}
Turns out the php_curl lib was not part of PHP5 installation. I installed it and everything works fine now.