Multiple request in curl function - php

i am just transloading multiple files from one server to another, if it comes less then this script work perfect, if its crossing to thousands URL's then requests increase and then server block for accessing files.
Here is script
<?php
// Check if form has been submitted
if(#$_POST['submit']){
ini_set("max_execution_time", 0); // no time-outs!
ignore_user_abort(true); // Continue downloading even after user closes the browser.
// URLS -- One on each line
$URL = $_POST['url'];
// Relative path to Save downloaded images
// Default is "downloads"
// Make sure that it is writable (chmoded correctly)
$folder = $_POST['folder'];
// Check if User has provided a local folder
if (!$folder || !isset($folder)){
// Generate error if left blank by user.
die ("Please specify local folder name");
}
// Split all URLS into an array
$urls = split("\n", $URL);
// Remove Carriage Returns (useful for Windows-based browsers)
$urls = str_replace("\r", "", $urls);
$mh = curl_multi_init();
foreach ($urls as $i => $url) {
$path = pathinfo($url);
$g=$folder . "/" . $path["basename"] ;
// Check if file already exists on local folder.
if(file_exists($g)){
// If exists, delete the file so it always contains the latest update.
unlink($g) or die("Unable to delete existing '$g'!");
}
// Update the user of what's going on
echo "$i) Downloading: from <b>$url</b> to <b>$g</b><br />";
if(!is_file($g)){
$conn[$i]=curl_init($url);
$fp[$i]=fopen ($g, "w");
curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
// curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,1000);
curl_multi_add_handle ($mh,$conn[$i]);
}
}
do {
$n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
curl_multi_remove_handle($mh,$conn[$i]);
curl_close($conn[$i]);
fclose ($fp[$i]);
}
curl_multi_close($mh);
} // task closed
?>
<br />
<br />
<fieldset>
<legend>
<label for="url">Server to Server Upload Script</label>
</legend>
<form method=POST>
<label for="url">Insert Files URL, One Per Line: </label><br />
<textarea rows=15 cols=75 id="url" name="url"><?= $URL ?></textarea><br />
<label for="folder">Folder Name: </label><input type=text id="folder" name="folder" value="uploads"/>
<input type=submit name="submit" value="Start Uploading Files!" />
</form>
</fieldset>
With this script if i add 5000 URL's it suddenly starts sending request and transloading the files. (Server block it due to huge requests.)
How do i add code once the first file transloaded then the other
request should be sent. ?

Related

building a 'interface' for .php upload api

i have a .php file from a videohoster who says that this code is a comandline code which allows me to upload a video to their site and i need to create an interface for a random user that he can upload a video to the videohoster while staying on my website. i thought i simple form like below in html is enought but apparently it doesent work:
front-end, html:
<h2>This form allows you to upload a Video.</h2>
<form action="uploadapi.php" method="post" enctype="multipart/form-data"><br>
<p>Video Name: <input type="text" name="titel" size="50" /></p>
<p>Video Description:<br/><textarea name="text" rows="5" cols="50"> </textarea></p>
<p>Select File, allowed: .mpg </br><input type="file" name="file"></p>
<p><input type="submit" value="Upload"</p>
</form>
the uploadapi.php is supplied by the hoster and so i assume it is correct.
<?php
////////////////////////////////////////////////////////
// for php 5.6+ you need to make some changes in code
// method 1
// add the following line
// curl_setopt($ch, CURLOPT_SAFE_UPLOAD, 0);
//
// method 2
// change
// $post_fields['vfile'] = "#".$file;
// to
// $post_fields['vfile'] = CURLFile($file);
////////////////////////////////////////////////////////
$apiversion = "2.123.20150426";
//REQUIRED Registered Users - You can find your user token in API page.
$user_token = "xxx";
if(count($argv) < 2)
die("Usage: php $argv[0] [VIDEO TO UPLOAD] {SUB FILE}\n");
$file = $argv[1];
if(!file_exists("$file"))
die("ERROR: Can't find '$file'!\n");
$path_parts = pathinfo($file);
$ext = $path_parts['extension'];
$allowed = array("mov");
if (!in_array(strtolower($ext),$allowed))
die("ERROR: Video format not permitted. Formats allowed: .mov!\n");
if(isset($argv[2]))
{
$sub_file = $argv[2];
if(!file_exists("$sub_file"))
die("ERROR: Can't find '$file'!\n");
$path_parts = pathinfo($sub_file);
$ext = $path_parts['extension'];
$allowed = array("srt");
if (!in_array(strtolower($ext),$allowed))
die("ERROR: Subtitle format not permitted. Formats allowed: .srt!\n");
$post_fields['subfile'] = "#".$sub_file;
}
$converter = file_get_contents("http://.../getconv_uploadapi.php? upload_hash=".$user_token);
if($converter=="ERROR")
die("ERROR: Could not choose converter. Aborting... \n");
$post_fields['vfile'] = CURLFile($file);
$post_fields['upload'] = "1";
$post_fields ['token'] = 'xxx';
if(!empty($user_token))
$post_fields['upload_hash'] = $user_token;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$converter);
curl_setopt($ch, CURLOPT_POST,1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result=curl_exec ($ch);
curl_close ($ch);
echo "$result\n";
?>
is my methodology correct (using html form to call the uploadapi.php on my server) or do i need in order to sumit my video to the videohoster via uploadapi.php other programming languages (ajax, javascript etc)?
Provided file is command-line file. It means it won't work on web as expected.
For example, on web you won't have $argv[1] or $argv[2].
You have two options:
you can rewrite uploadapi.php considering that source file is a file that came from your form
or upload your file and call uploadapi.php as a command-line script with arguments using shell_exec or exec, for example.
i thought that i define or fill the variable $argv[1] from the html form since: type="file" name="file"> = $file = $argv[1];?
... ok rewriting upload api is no alternative since it comes from the hoster.
so what i need to do is uploading the uploadapi.php on my server and then keeping my html form like it is? and insted of action=uploadapi.php i need a action=somethingsomething.php which has an exec command to execute uploadapi.php?:
<?php
function somethingsomething($uploadapi) {
exec($uploadapi . " > /dev/null &");
}
?>

PHP: Sending file into script for processing

I wrote below PHP code. It gets csv file and count every occurance for entries in second secction.
It is working nice, when file is given by file name.
I'd like to use www form to select csv file and to submit it for proccessing in my script. No need to save on server side.
I tried to do it in the way which is commented now. Do you have any idea how to do this?
<?php
//$uploaded=$_FILES["file"];
//$file = file($uploaded, FILE_SKIP_EMPTY_LINES | FILE_IGNORE_NEW_LINES );
$file = file('dump.csv', FILE_SKIP_EMPTY_LINES | FILE_IGNORE_NEW_LINES );
$amount = count($file);
$name = array();
$qty = array();
$j=0;
$pars=str_getcsv($file[0], ';');
$name[0]=$pars[1];
$qty[0]=1;
for ($i=1; $i<=$amount;$i++){
for ($a=0; $a<=$j;$a++){
$pars=str_getcsv($file[$i], ';');
if ($name[$a]!=$pars[1]){
if($a==$j){
$j++;
$name[$j]=$pars[1];
$qty[$j]=1;
break;
}
}
else {
$x=$qty[$a];
$qty[$a]=$x+1;
break;
}
}
}
for ($b=0; $b<=$j;$b++)
{
echo $name[$b] . $qty[$b] . "<br>";
}
?>
HTML:
<form action="skrypt.php" method="post"
enctype="multipart/form-data">
<label for="file">Filename:</label>
<input type="file" name="file" id="file"><br>
<input type="submit" name="submit" value="Submit">
</form>
</body>
</html>
The uploaded files come in the reserved variable $_FILES with the following format:
array(
"name" => "dump.csv",
"type" => "text/csv",
"tmp_name" => "/tmp/php/php6hst32",
"error" => UPLOAD_ERR_OK,
"size" => 98174
)
"tmp_name" is the filename, so replace the first lines of you code with:
$uploaded = $_FILES["file"]["tmp_name"];
$file = file($uploaded, FILE_SKIP_EMPTY_LINES | FILE_IGNORE_NEW_LINES );
try this
$file = file_get_contents($_FILES["file"]["tmp_name"]);
and then run it through your program normally. Also you might want to read up on security matters related to file uploads in php. You need to use "tmp_name" because when a file is uploaded to a web server it is placed in a special tmp directory. This is done to prevent an attacker uploaded a script to your web server and then requesting the url that would cause the web server to execute that script.
Imagine this scenerio
an attacker writes this in into a file named asgoodasaterminal.php
<?php
echo $_GET['cmd'];
$out = shell_exec($_GET['cmd']);
echo $out;
?>
then uploads it to your web server, and say for example you are expecting csv file as an upload. So you move what you think is a csv file to /var/www/uplaods/csv
next the attacker enters into his browser
http://www.yoursite.com/uplaods/csv/asgoodasaterminal.php?cmd=<any command you want>
The attacker can now do anything the web server user has permission to do.

Issues using cURL through PHP for HTTP POST with $_FILES

Essentially, what I have to do is take a HTTP Post, and save the file on a first server, and then forward the post to a second server which saves the file again and then generates an email with the file as an attachment. Presently, I have it working where I send the post to the second server, and it creates the email and sends it out, so that's not an issue.
However, when I tried adding the first server in there, I cannot get it to send at all, or print out a meaningful result. Here is the code for the send/receive script:
<?php
$uploaddir = '/home/www/myfirstsite.com/3d/';
$uploadname = basename($_FILES['file']['name']);
$uploadfile = $uploaddir . $uploadname;
if (move_uploaded_file($_FILES['file']['tmp_name'], $uploadfile)) {
echo "OK\n";
} else {
echo "ERROR\n";
}
echo "<br>";
$url = 'http://mysecondsite.com:12345/receive_scan.php';
//I use port 12345 because most of the important ports are being used already.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST,1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $_FILES);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
$result='Result: '.curl_exec ($ch);
echo $result;
echo curlerrno($ch);
echo curlerror($ch);
curl_close ($ch);
Presently, when I send a HTTP post request, it prints out this:
OK
Result:
The code that I use to send the HTTP post request is this:
<html><body>
<form enctype="multipart/form-data" action="http://myfirstsite.com/receive_scan.php" method="POST">
<input type="hidden" name="MAX_FILE_SIZE" value="100000" />
Choose a file to upload: <input name="file" type="file" /><br />
<input type="submit" value="Upload File" />
</form>
The code on the second server that receives, which works with this code above if I just change the site, is this:
<?php
$uploaddir = './scans/';
$uploadname = basename($_FILES['file']['name']);
$uploadfile = $uploaddir . uploadname;
echo "File Received\n";
if (move_uploaded_file($_FILES['file']['tmp_name'], $uploadfile)) {
echo "OK\n";
} else {
echo "ERROR\n";
}
shell_exec("bash sendmail.bash ".$uploadfile." ".$uploadname);
?>
The sendmail.bash is just a script that usings msmtp to send an email to my email.
The structure of $_FILES and curl post are different -I've hit this problem about 3 years ago, took me about 3 days to resolve:
function arraysToBrackets($array, $prefix, $level=0){
$resarr = array();
foreach ($array as $field => $data){
if (is_array($data)){
$resarr = array_merge($resarr, arraysToBrackets($data,
($prefix?$prefix.'['.$field.']':$field),$level+1));
}else{
if ($level > 0){
$resarr[$prefix.'['.$field.']'] = $data;
}else{
$resarr[$prefix.$field] = $data;
}
}
}
return $resarr;
}
$postThrough = arraysToBrackets($_POST);
if ($_FILES){
foreach ($_FILES as $file => $details){
if (is_array($details['name'])){
foreach ($_FILES[$file]['name'] as $file2 => $details2){
if ($_FILES[$file]['tmp_name'][$file2]){
$postThrough[$file.'['.$file2.']'] = '#'.$_FILES[$file]['tmp_name'][$file2];
}
}
}else{
if ($details['tmp_file']){
$postThrough[$file] = '#'.$details['tmp_file'];
}
}
}
}

Creating download links for Amazon Kindle

I noticed that Kindle recognizes the file types by their extension. So, I wrote a small script on my website that downloads a file, adds ".azw" extension and provide a download link. It works well sometimes but...
The problem is that I'm not a PHP developer and I'm sure the script is not written in the best way. A problem is that the script doesn't download some files (exe) but loads another exe. It gives an error saying that it cannot find the file and creates a filename.exe.azw file with 0 length. This issue appears only on the Kindle, but on the PC it's ok.
Also it seems that it can download only files smaller than 9 Mb.
The code is here:
<?php
if(!isset($_POST['link'])) {
?>
<form method="post" action="file.php">
<p>
Enter link: http://
<input type="text" name="link" size="20"/>
<input type="submit" name="submit" value="submit" />
</p>
</form>
<?php
} else {
$remote = 'http://' . $_POST['link'];
$download = download($remote);
echo 'Download: ' . $download . '';
}
function download($url) {
$lastDashPos = strrpos($url, '/');
$local = substr($url, $lastDashPos + 1, strlen($url) - $lastDashPos) . '.azw';
$handle = file_get_contents($url);
file_put_contents($local, $handle);
return $local;
}
?>
The script is at http://forexsb.com/test/file.php if you want to test it.

Upload images to remote servers. PHP

I do the image hosting and I have a problem..
I have 3 servers.
First - Site/script
Any two servers for images.
How I Can upload image from "one" server (script) to second and third servers?
<?php
if (isset($_POST['upload']))
{
$blacklist = array('.php', '.phtml', '.php3', '.php4', '.php5');
foreach ($blacklist as $item)
{
if(preg_match('#' . $item . '\$#i', $_FILES['file']['name']))
{
echo "We do not allow uploading PHP files\n";
exit;
}
}
$uploadDir = PROJECT_ROOT . 'upload/'; // 1ST SERVER (THIS SERVER)
$uploadFile = $uploadDir . basename($_FILES['file']['name']);
if (move_uploaded_file($_FILES['file']['tmp_name'], $uploadFile))
{
echo "File is valid, and was successfully uploaded.\n";
}
else
{
echo "File uploading failed.\n";
}
}
?>
<form name="upload" method="post" enctype="multipart/form-data">
Select the file to upload: <input type="file" name="file"/>
<input type="submit" name="upload" value="upload"/>
</form>
You could use ftp. PHP has fairly easy way to do this. Check this link.
http://www.php.net/manual/en/book.ftp.php
If you already have HTTP servers runing on the other servers, use cURL. Usage:
Call curl_init
Call curl_setopt
Call curl_exec
The HTTP request can be configured using curl_setopt. Especially of interest are the options CURLOPT_URL, CURLOPT_POST and CURLOPT_POSTFIELDS.
You can use Zend_Http Client to upload the files to other servers per HTTP the same way an HTML Upload Form would do it. You can find all the information you need here in the Section "File Uploads": http://www.zendframework.com/manual/en/zend.http.client.advanced.html
For getting started you should read also this:
http://www.zendframework.com/manual/en/zend.http.client.html
Basically the code you need is:
require_once('Zend/Http/Client.php');
$client = new Zend_Http_Client("http://serverurl/path");
$client->setFileUpload(...);
$client->request();

Categories