$_POST/$_GET Submitted URL too large - php

I have php scripts to migrate data from one database do another. In one script I call one another to Get the data from the original database, and then it sends it to another. Something like this:
migrateData.php <---> getData.php
and then migrateData.php <---> putData.php
In putData.php im using $_GET, but when its too many Data I get the error saying "The length of the requested URL exceeds the capacity limit of this server".
I've seen people saying to use $_POST, but i'm not really sure about it.
My migrateData.php:
<?php
echo "Migration";
$url = "http://localhost/mig/getData.php";
$client = curl_init($url);
curl_setopt($client,CURLOPT_RETURNTRANSFER,true);
$response = curl_exec($client);
echo "<br>";
echo "<br>";
echo $response;
$url = "http://localhost/mig/putData.php?data=".$response;
$url = str_replace ( ' ', '%20', $url);
$client = curl_init($url);
curl_setopt($client,CURLOPT_RETURNTRANSFER,true);
$response = curl_exec($client);
echo $response;
?>
Any help would be appreciated

Yes when you send a get request to the server (apache, nginx), it may limit the allowed size of a client's HTTP request-line (e.g. ) and the HTTP request header field size.
For example if you have access to the server and its an Apache server you can increase the Apache limits for limit request line and limit request field size.
In case of nginx, you can increase the large_client_header_buffers parameter in nginx.conf.
Using POST method will send the fields outside of the client's HTTP request-line, effectively bypassing the server constraint.
Actually POST request are also limited but the default size is bigger.
Using POST or PUT method is recommended instead of changing the server config.
<?php
//Your code to get the response here
// ...
$data = ['data' => $response];
// Prepare new cURL resource
$ch = curl_init('http://localhost/mig/putData.php');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
// Submit the POST request
$result = curl_exec($ch);
// Close cURL session handle
curl_close($ch);
//Output the result
echo $result;
?>

Apache's max $_GET request length is 8190 bytes (https://stackoverflow.com/a/1289611/575047) so trying to send heaps of data in a GET request is a bad idea because it might get cut off.
POST is ideal. By default, apache sets post_max_size to 8mb I believe.
To POST, you just need to add these lines I believe to your putData curl
curl_setopt($client, CURLOPT_POST, 1);
curl_setopt($client, CURLOPT_POSTFIELDS, $response);

Related

PHP cURL request takes up the entire Timeout

I currently have a Laravel application, which is doing a CURL request from one route to another route within the same route. My CURL looks like this:
//LOGGING THAT A CURL CALL IS ABOUT TO BE MADE
$url = env('APP_URL') . '/tests/add/results';
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //return server error
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLINFO_HEADER_OUT, FALSE);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $test_post_data);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$response = curl_exec($ch);
In the route, that the POST is being sent to, the first thing i log, is the request data:
//LOGING THAT I RECEIVED THE CURL CALL in receiving function
I'm noticing, that the logs for the request data get logged exactly the same amount of time as the timeout, meaning the request is actually being sent 10 seconds after the initial call.
In my logs i'll see something like:
10:10:10 - LOGGING CURL CALL
10:10:20 - Recieving CURL call
If i change the timeout to 30, then the log shows 30 seconds later that i received the CURL call.
Does anyone have any idea why this may be happening?
The response from the CURL just comes back as false always.
I did the following to make a post request work:
Instead of calling the route via CURL i did a post directly to the route function
$testController = new TestsController;
$test_data_request = new \Illuminate\Http\Request();
$test_data_request->setMethod('POST');
$test_data_request->request->add( $test_post_data );
$testId = $testController->addTestResults($test_data_request);
You've not provided enough information, but i think, the problem will be one or more of the following:
The WebServer http://127.0.0.1:8000 is not running
The script located on http://127.0.0.1:8000/tests/add/results is running too long and the request timeouts before it is completed
The requested path is returning redirect headers and creates an infinite loop
The response is too big to finish the data transfer in thirty seconds (very wierd if on localhost)
Try some more debugging and provide more information on this, so we may help you.
PS: Firstly i would try to catch the headers (curl_setopt($ch, CURLINFO_HEADER_OUT, true);) and print out the response (var_dump($response);) - or save it to some file :)

File get contents returns false

So I'm trying to have users verify that they own the domain. So I generated a file and have them upload it to their site. So I then have to verify it, so what I do is
file_get_contents($url.'/'.$token.'.html');
All this returns is
bool(false)
Here's more of the code
$url = $_POST['url'];
//Get site info
$gin = $con->prepare("SELECT * FROM verify WHERE url = :url");
$gin->bindValue(':url', $url);
$gin->execute();
//Get token
$t = $gin->fetch(PDO::FETCH_ASSOC);
$token = $t['token'];
$url = $t['url'];
//Get content
var_dump(file_get_contents($url.'/'.$token.'.html'));
I have 3 columns in the table token, which is the string in the file. url which is the url obviously, its in example.com format. And a verified column which is either 1 or 0. Any ideas?
Based on my experience with fetching third-party content from more than 1 million domain names, I would not recommend you to use file_get_contents() because this PHP function cannot handle page redirects, site that requires a valid user-agent etc. The issue you are experiencing might be specific to a certain domain names only. A better approach to your problem is to use curl.
function download_content($url) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Firefox 32.0");
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
Usage:
$returned_content = download_content('http://stackoverflow.com');
Try to add http:// before $url and put a valid url. It will work
var_dump(file_get_contents('http://'.$url.'/'.$token.'.html')); // With a valid URL
If you enable error_reporting(E_ALL), then you will probably see that the use of HTTP URLs are disallowed due to an ini setting.
Warning: you are possibly opening a hole by allowing arbitrary prefixes in file_get_contents. Try to use parse_url to validate that you actually have a HTTP URL. Then you should probably consider using cURL and disable external redirects (otherwise one could pass a URL such as http://bit.ly/something# and still pass your tests).

file_get_contetnts and soap client don't work with long query

Sorry if this is duplicate,I would think it would be but couldn't find anything
I have an application using soap client web-service to connect to other server but recently I found out the requests with long text do not send to web-service.
I thought it related to some PHP configs and set post_max_size to 200M and set max_input_vars to 10000 but the problem did not solve ...
today I found out file_get_contents don't work as well as soap client in my server and I am sure one of php configuration made this problem.
here is the code that dont work and when I run this just wait for a while and display nothing ( white page)
$url="http://somedomain.com/webservice.php?Username=user&Password=pass&Det=13218%5E04134414401%5E7015373%5E3%5E51%5E14%252F2141-9305-310209-424926%25D8%25A2%25D9%2585%25D9%2588%25D8%25B2%25D8%25B4%2B%25D9%2583%25D8%25A7%25D8%25B1%25D8%25A8%25D8%25B1%25D8%25AF%25D9%258A%2BHTML%2B5.0%2B%2526%2BCSS%2B3.0%2B%25D9%2587%25D9%2585%25D8%25B1%25D8%25A7%25D9%2587%2B%25D8%25A8%25D8%25A7%2BResponsive%2BWeb%2BDesign%25D9%2585%25D8%25B1%25D8%25AC%25D8%25B9%2B%25D8%25A2%25D9%2585%25D9%2588%25D8%25B2%25D8%25B4%25D9%258A%2B%25D8%25A8%25D8%25B1%25D9%2586%25D8%25A7%25D9%2585%25D9%2587%25E2%2580%258C%25D9%2586%25D9%2588%25D9%258A%25D8%25B3%25D9%258A%2BASP.NET%2B4.5%2B%25D8%25AF%25D8%25B1%2BC%2523%2B%2528%25D8%25A7%25D8%25B2%2B%25D9%2585%25D8%25A8%25D8%25AA%25D8%25AF%25D9%258A%2B%25D8%25AA%25D8%25A7%2B%25D9%25BE%25D9%258A%25D8%25B4%25D8%25B1%25D9%2581%25D8%25AA%25D9%2587%2529%25D9%2585%25D8%25B1%25D8%25AC%25D8%25B9%2B%25D9%2583%25D8%25A7%25D9%2585%25D9%2584%2BASP.Net%2BMVC%2B4%5E2825%5E819700%5E1%5E0%5E%25D8%25A7%25D9%2585%25DB%258C%25D8%25B1%25D8%25B1%25D8%25B6%25D8%25A7%2B%25D8%25A7%25D8%25B3%25D8%25AF%25D9%25BE%25D9%2588%25D8%25B1%5E%25D8%25AA%25D8%25A8%25D8%25B1%25DB%258C%25D8%25B2%2B-%2B%25DA%2586%25D9%2587%25D8%25A7%25D8%25B1%25D8%25B1%25D8%25A7%25D9%2587%2B%25D9%2584%25D8%25A7%25D9%2584%25D9%2587%2B%25D9%2585%25D8%25AC%25D8%25AA%25D9%2585%25D8%25B9%2B%25D8%25AA%25D8%25AC%25D8%25A7%25D8%25B1%25DB%258C%2B%25D9%2584%25D8%25A7%25D9%2584%25D9%2587%2B%25D8%25B7%25D8%25A8%25D9%2582%25D9%2587%2B%25D8%25A7%25D9%2588%25D9%2584%2B%25D9%2588%25D8%25A7%25D8%25AD%25D8%25AF%2B5%5E5178735466%5Einfo%2540pendarepars.com%5E154000%5E12320&Ordertip=1";
echo file_get_contents($url);
You can avoid this issue by sending the request as a post not Get
You should check what the server is responding with using cUrl. This snippet should help you on your way.
// Prep cUrl
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$return = curl_exec($ch);
// Output info
echo "<pre>";
if(curl_error($ch))
{
echo "Error: ".curl_error($ch)."\n";
}
else
{
print_r(curl_getinfo($ch));
echo htmlentities($return);
}
Look for http_code, if it's 414 then the server can't handle the length of the URI.

https connection passing another server

I want to do the following scenario but i dont really know if this is possible at all or what techniques i need to use:
So im using an external api to store information of my web application. Now i want to send the api request to my server which sends it to the api server maybe by using curl and wait for the response.. when i receive the response i update something at my server like insert row in mysql and send the response back to the requester
Now the api url is using https and my server is currently using http.. will this be a problem to test or do i need https also?
And how should i implement this using php? Maybe some kind of redirect? Or do i have to rebuild the whole request?
Thanks in advance
There are a lot of unknowns about your question that make it hard to answer, but here goes.
You certainly don't have to offer your own application over HTTPS and you can query an HTTPS API and then deliver your own response via HTTP. I should warn you that you are inherently making the external service less secure by doing this. If you are acting as a proxy for the external service, I would suggest that you either get an SSL certificate and do it the proper way.
The lack of details in your question make it hard to answer a specific question, but here is a small code block that should get you started:
<?php
$url = "https://service/endpoint";
$params = array(
'api_token' => 'mytoken',
'param1' => 'myparam',
);
$url .= http_build_query($params);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
if (curl_errno($ch)) {
echo 'received error: ' . curl_error($ch) . ' (code: ' . curl_errno($ch);
} else {
echo 'received response: ' . $result;
}
What nate has said is roughly right, but if you want to post from HTTP to HTTPS with cURL, set this option as well:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
Which stops cURL from verifying the peer’s certificate (I'm guessing you don't care about using SSL between the second server and the API).
Code on the app server:
$data = http_build_query(
// Your parameters go here
);
file_get_contents('http://middleserverurl.com/?' . $data);
Code on the middle server:
$apiurl = "https://apiurl.com/?" . http_build_query($_GET);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$result = curl_exec($ch);
// Decode the result data here
// Insert into your database here
// Send something back to your app server e.g.:
$data = http_build_query(
// What you want to send back form the result
);
file_get_contents('http://appserverurl.com/?' . $data);

PHP Redirect a file from another server to end user

I want to be able to allow user to enter in variable URL which file they would like to download from remote server URL e.g /download.php?url=fvr_anim_foxintro_V4_01.jpg
<?php
$url = $_GET['url'];
header("Location: http://fvr.homestead.com/files/animation/" . $url);
?>
The above is purely an example I grabbed from google images. The problem is I do not want the end user to be allowed to see where the file is originally coming from so it would need to get the file download to the server and the server passes it along to the end user. Is there a method of doing this?
I find many examples for files hosted on the server but no examples for serving files hosted on a remote server. In other words I would be passing them along. The files would be quite large (up to 100MB)
Thanks in advance!
You can use cURL for this:
<?php
$url = "http://share.meebo.com/content/katy_perry/wallpapers/3.jpg";
$ch = curl_init();
$timeout = 0;
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
// Getting binary data
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
$image = curl_exec($ch);
curl_close($ch);
// output to browser
header("Content-type: image/jpeg");
echo $image;
?>
Source: http://forums.phpfreaks.com/topic/120308-solved-curl-get-image/
Of course, this example is just for an image (as you've suggested) but you can use cURL for all kinds of remote data retrieval via HTTP GET, POST, PUT, DELETE, etc. Search around the web for "php curl" and you'll find an endless supply of information.
The ideal solution would be to use PHP's cURL Library, but if you're using shared hosting keep in mind this library may be disabled.
Assuming you can use cURL, you simply echo the Content-type header with the appropriate MIME Type and echo the results from curl_exec().
To get a basic idea of how to use the cURL library, look at the example under the curl_init() function.

Categories