I am trying to use a Http client to store the HTML from a web page. The following code snippet shows how I have configured the Http client, it uses php-http/guzzle6-adapter.
I know from my tests that the client works properly when pointed at other webpages.
<?php
require_once(__DIR__.'/vendor/autoload.php');
use Http\Adapter\Guzzle6\Client as GuzzleAdapter;
use GuzzleHttp\Psr7\Request;
$config = [
'verify' => false,
'timeout' => 2
];
$adapter = GuzzleAdapter::createWithConfig($config);
$request = new Request('GET', 'https://workingwithchildren.wa.gov.au/');
// Returns a Psr\Http\Message\ResponseInterface
$response = $adapter->sendRequest($request);
echo $response->getBody();
?>
However page I am trying to resolve https://workingwithchildren.wa.gov.au/ returns the following error, no matter what I do.
The requested URL was rejected. Please consult with your administrator.
Your support ID is: 9283834035315018727
I pointed my browser at the website and used Chrome Developer Tools to examine the Request/Response data being exchanged. The screenshot below shows the Response data my browser received.
I noticed that the site is setting cookies that seem to relate to security and CPFS and I would imagine these cookies are what are stopping my client from resolving the web page successfully. But I don't know how to fix this. I'd imagine this is a problem others have faced before. Any help would be much appreciated.
For anyone experiencing a similar problem the solution I found was to, as the commenter Scuzzy suggested, add User-Agent data to my guzzle config.
Related
I am running an IIS 8 / PHP web server and am attempting to write a so-called 'proxy script' as a means of fetching HTTP content and loading it onto an HTTPS page.
Although the script does run successfully (outputting whatever the HTTP page sends) in some cases - for example, Google.com, Amazon.com, etc. - it does not work in fetching my own website and a few others.
Here is the code of proxy.php:
<?php
$url = $_GET['url'];
echo "FETCHING URL<br/>"; // displays this no matter what URL I enter
$ctx_array = array('http' =>
array(
'method' => 'GET',
'timeout' => 10,
)
);
$ctx = stream_context_create($ctx_array);
$output = file_get_contents($url, false, $output); // times out for certain requests
echo $output;
When I set $_GET['url'] to http://www.ucomc.net, the script fails. With most other URLs, it works fine.
I have checked other answers on here and other places but none of them describe my issue, nor do the solutions offered solve it.
I've seen some suggestions to similar problems that involve changing the user agent, but when I do this it not only does not solve the existing problem but prevents other sites from loading as well. I do not want to rely on third-party proxies (don't trust the free ones/want to deal with their query limit and don't want to pay for the expensive ones)
Turns out that it was just a problem with the firewall. Testing it on a PHP sandbox worked fine, so I just had to modify the outgoing connections settings in the server firewall to allow the request through.
I’m trying to invoke a WCF service (.NET) from PHP. It’s a little more complicated than just using a SoapClient since the service uses a WS2007FederationHttpBinding to authenticate.
Here’s the code I’m using at the moment. I haven’t even added credentials as I’m not sure how, but regardless, I’m not even at the point where I’m getting access denied errors.
$wsdl = "https://slc.centershift.com/sandbox40/StoreService.svc?wsdl";
$client = new SoapClient($wsdl,array(
//'soap_version'=>SOAP_1_2 // default 1.1, but this gives 'Uncaught SoapFault exception: [HTTP] Error Fetching http headers'
));
$params = array();
$params['SiteID'] = 123;
$params['GetPromoData'] = false;
$ret = $client->GetSiteUnitData(array('GetSiteUnitData_Request'=>$params));
print_r($ret);
Which WSDL should I be pointing to?
https://slc.centershift.com/Sandbox40/StoreService.svc?wsdl
Seems to be very short, but includes a reference to (note the wsdl0) https://slc.centershift.com/Sandbox40/StoreService.svc?wsdl=wsdl0
https://slc.centershift.com/Sandbox40/StoreService.svc?singleWsdl
Seems to have everything in it.
Do I need to specify SOAP 1.2? When I do, I get a connection timeout ([HTTP] Error Fetching http headers). When I don’t, the default of SOAP 1.1 is used and I get a [HTTP] Cannot process the message because the content type 'text/xml; charset=utf-8' was not the expected type 'application/soap+xml; charset=utf-8'. Is this because I’m not authenticated yet, or because I’m using the wrong SOAP version?
How to authenticate in PHP? Here’s the corresponding .NET/C# code. Do I need to somehow put these as SOAP headers? Or am I thinking about it all wrong, and I need to do some kind of authentication before I even call the method (from what I read, I’m supposed to get a token back and then use it for all future method calls – I think I see an example of this in an answer here on Stack Overflow.
If I call $client->__getFunctions(), using either WSDL and either SOAP version, I’m getting a valid list of all functions, so I assume either of these is fine and my real issue is the authentication.
Other programmers I’ve talked to had spent time trying to get this to work, but gave up and instead implemented a proxy in .NET. They pass their parameters from PHP to their own unsecured .NET service, which in turn calls this secure service. It works, but seems crazily inefficient to me, and counter-productive, as the purpose of WCF is to support all types of clients (even non-HTTP ones!).
I’ve read How to: Create a WSFederationHttpBinding on MSDN, but it didn’t help.
You can use this URL for WSDL https://slc.centershift.com/Sandbox40/StoreService.svc?singleWsdl. This WSDL has all definitions.
You have to use 1.2 because this webservice works with SOAP 1.2 version. I tried it with 1.1 and 1.2 and both of them gived error. 1.1 is version error, 1.2 is timeout error. I think there is an error at this test server. I used it with svcutil to generate code but it gived error too. Normaly it should get information and generate the code example to call service.
Normally you can add authenticate parameters with SoapHeader or directly add to options in SoapClient consruct (if service authentication is basic authentication). I write below code according to your screenshot. But it gives timeout after long wait.
$wsdl = "https://slc.centershift.com/sandbox40/StoreService.svc?wsdl";
$client = new SoapClient($wsdl,array('trace' => 1,'soap_version' => SOAP_1_2));
$security = array(
'UserName' => array(
'UserName'=>'TestUser',
'Password'=>'TestPassword',
'SupportInteractive'=>false
)
);
$header = new SoapHeader('ChannelFactory','Credentials',$security, false);
$client->__setSoapHeaders($header);
$params = array();
$params['SiteID'] = 100000000;
$params['Channel'] = 999;
try {
$ret = $client->GetSiteUnitData($params);
print_r($ret);
}catch(Exception $e){
echo $e->getMessage();
}
__getFunctions works, because it prints functions defined in WSDL. There is no problem with getting WSDL information at first call. But real problem is communication. PHP gets WSDL, generates required SOAP request then sends to server, but server is not responding correctly. SOAP server always gives a response even if parameters or request body are not correct.
You should communicate with service provider, I think they can give clear answer to your questions.
Having worked with consuming .NET WS from PHP before I believe you would need to create objects from classes in PHP that matches the names that .NET is expecting. The WSDL should tell you the types it is expecting. I hope this assist with your path forward!
If the SOAP call works from a C# application, you could use Wireshark (with the filter ip.dst == 204.246.130.80) to view the actual request being made and then construct a similar request from php.
Check this answer to see how you can do a custom SOAP call.
There's also the option of doing raw curl requests, since it might be easier to build your xml body, but then you would have to parse the response yourself with simplexml.
I have an interesting situation when calling the Shopify API. I use the standard procedure for calling the url and get the data, like this:
define('SHOPIFY_SHOP', 'myteststore.myshopify.com');
define('SHOPIFY_APP_API_KEY', 'xxxx');
define('SHOPIFY_APP_PASSWORD', 'yyy');
$shop_url = 'https://'.SHOPIFY_APP_API_KEY.':'.SHOPIFY_APP_PASSWORD.'#'.SHOPIFY_SHOP;
$response = Requests::get($shop_url.'/admin/products.json');
And I correctly get the response, parse the data and all works great. Now, when I put it to the actual server (Ubuntu 12.04), I noticed a weird message from the Spotify API:
[API] Invalid API key or access token (unrecognized login or wrong password)
I tried creating a new app, but still its the same. So the same file and the same set works on my machine, but not on the server. (only difference in the file is the path to requests library, require_once './Requests/library/Requests.php'; for Linux and require_once '..\Requests\library\Requests.php'; for Windows) As stated, I use the requests library and I assume there has to be some trick where the library (or something else) rewrites the URl and it doesn't get to Shopify correctly.
I tried using CURL with the URL directly, and it works that way as well. Can anyone point me what might be causing this?
Update: I moved to another library which solved the issue, but would like to know what was causing this since I had great experience with Requests up to this point.
I'm starting to use the same lib, and I stumbled upon something relevant right after finding this question:
https://github.com/rmccue/Requests/issues/142#issuecomment-147276906
Quoting relevant part:
This is an intentional part of the API design; in a typical use case,
you won't necessarily need data sent along with a request. Building
the URL for you is just a convenience.
Requests::get is a helper function designed to make GET requests
lightweight in the code, which is why there's no $data parameter
there. If you need to send data, use Requests::request instead
$response = Requests::request( 'http://httpbin.org/get', $headers, $data, Requests::GET, $options );
// GET is the default for type, and $options can be blank, so this can be shortened:
$response = Requests::request( 'http://httpbin.org/get', $headers, $data );
I couldn't figure why is this happening, it appears the Requests library is stripping the parameters from GET requests, so I moved to unirest library and this solved the issue.
I'm trying to get a JSON string from a page in my Laravel Project. Using this:
$json = file_get_contents($url);
$data = json_decode($json, TRUE);
return View::make('adventuretime.marceline')
->with('json', $json)
->with('title', 'ICE KING')
->with('description', 'I am the Ice King')
->with('content', 'ice king');
But since I'm only using a localhost, I think this doesn't work that's why it doesn't output anything. I want to know what is the proper way for it to be flexible and be able to get the JSON string with any $url value using php?
Looking at the comments above, it is possible that the $url you are using is not valid, check it out by pointing your browser there and see what happens.
If you are sure that the $url is fine, but you still get the 404 Not Found error - verify that you have proper Laravel routing defined for that address. If the routes are fine, maybe you forgot to do
composer dump-autoload
after making modifications in your routes.php. If so, try the above and refresh the browser to see if it helps.
Furthermore, bear in mind that using your current function, you can submit only GET requests. What is more, this function might not be available for fetching remote urls, on some hosting servers due to security reasons. If you still want to use it, it'd be good to check
if($json !== FALSE)
before you process the $json response. If the file_get_contents fails it will return false.
Reffering to the part of your question
what is the proper way for it to be flexible and be able to get the JSON string with any $url
I'd suggest using cURL, as a standard and convenient way to fetch remote content. Using cURL you have better control over the process of sending the http request and receiving the "answer" it returns. Personaly, in my Laravel 4 apps I often use this package jyggen/curl. You can read the docs for it here: jyggen docs
If you are not satisfied with cURL and you want greater control try Guzzle As the authors state, Guzzle is a PHP HTTP client & framework for building RESTful web service clients.
I'm trying to use file_get_contents() to get the response from a server and this error was encountered. Could someone tell me what is the reason and how to fix it? The portion of the code is:
$api = "http://smpp5.routesms.com:8080/bulksms/sendsms?username=$username&password=$password&source=$source&destination=$destin&dlr=$dlr&type=$type&message=$message";
$resp = file_get_contents($api);
The server responded correctly while I pasted the url in the browser.
I learned that this is caused by the server rejecting the client's HTTP version, but I have no idea why that is happening in my case.
Any help is much appreciated. Thanks in advance
I found the problem, and it was a simple coding error -- missing url encoding.
The reason I didn't notice it at first was because the code was ok before I did some editing, and I'd missed out the urlencode() function before calling the server, which caused a space in the url.
This does seem to be the reason this error occurs for most people. So if you encounter this, use urlencode() on all variables which may contain white space in it's value used as URL parameters. So in the case in my question the fixed code will look like:
$api = "http://smpp5.routesms.com:8080/bulksms/sendsms?username=$username&password=$password&source=$source&destination=$destin&dlr=$dlr&type=$type&message=" . urlencode($message);
$resp = file_get_contents($api);
Also, thanks for all of your time and responses, those were informational.
You could create a stream context with the HTTP version set to 1.0 and use that context with file_get_contents:
$options = array(
'http' => array(
'protocol_version' => '1.0',
'method' => 'GET'
)
);
$context = stream_context_create($options);
$api = "http://smpp5.routesms.com:8080/bulksms/sendsms?username=$username&password=$password&source=$source&destination=$destin&dlr=$dlr&type=$type&message=$message";
$resp = file_get_contents($api, false, $context);
By the way: Don’t forget to escape your URI argument values properly with urlencode.
I ran into the same issue and in my case the culprit was an errant newline/CRLF character at the end of the request URL, which does not get caught by urlencode() (or maybe it does encode it but it still causes the server to produce the error). Once I found the problem the requests began to work again, even without the stream context options.
Hopefully this will help others.
Can you sniff what's happening on the wire? Seeing the format of the HTTP request as it goes out on the wire would help a lot.
Without seeing that, my best guess would be that the server isn't well-implemented, and is rejecting a HTTP/1.1 request. Try setting --http1.0 on Curl and seeing what happens...
Some time we still get error with
file_get_contents($api);
in that case, try this:
fopen($api,"r");
I was also facing this same issue..
later i found that while retrieving the results from mysql, Limit $count ,
$count was -ve. fixing that the url worked fine.
There is some problem in url only, and its not a file_get_contents or http version issue..