setting extra variables to Curl connection string - php

I am using php/curl to make a connection to an external api. It is working fine but right now my connection string only references one variable from a form.
How can I make it references more than one?
This is how it looks like now
$curl_connection =curl_init('https://api.ssllabs.com/api/v2/analyze?host='.$_REQUEST['host']);
When making a call the above will look like this:
https://api.ssllabs.com/api/v2/analyze?host=www.yourdomain.com
My form has been updated and contains more input values which I want to reference as part of the curl connection and needs to look like this:
https://api.ssllabs.com/api/v2/analyze?host=www.yourdomain.com&s=69.54.183.66&ignoreMismatch=on
Each reference is seperated by the & symbol.
On my html form the input names are as follows:
host
s
ignoreMismatch
There are others aswell.. but the above is a sample
I hope that's clear enough for your to know what I am trying to do.
Here is a portion of my jquery / Ajax
var hostName = $("input#host").val();
var ignoreMismatch = $("#ignoreMismatch").val();
var fromCache = $("#fromCache").val();
dataString = "host=" + hostName + "&fromCache=" + fromCache + "&ignoreMismatch=" + ignoreMismatch;
enter code here
$.ajax({
type: "GET",
url: "api2.php",
data: dataString,
dataType: "json",
I can see this part is working. From my web console I can see the URL is built and posted to the api2.php file.
For example https://172.21.121.37/labs/api2.php?host=ocsp.verisign.com&fromCache=on&ignoreMismatch=on
Here is my api2.php file which is currently only setup to receive host= string.
<?php
header( "Content-Type: application/json" );
//create cURL connection
$curl_connection =curl_init('https://api.ssllabs.com/api/v2/analyze?host='.$_REQUEST['host']);
///set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
//perform our request
$result = curl_exec($curl_connection);
echo $result;
//close the connection
curl_close($curl_connection);
?>
This is working code by the way but I just wanted to give you guys more context as I tried some of the suggestions provided and they didn't work. If someone can help update my api2.php file I will test the updates.

This should help you
foreach($_GET as $name => $value) {
$encoded .= urlencode($name).'='.urlencode($value).'&';
}
$encoded = substr($encoded, 0, strlen($encoded)-1);
$ch = curl_init("https://api.ssllabs.com/api/v2/analyze?" . $encoded);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
echo $output;

Related

Scrape product Image url from a website where content is uploading dynamiclly

I am not able to scrape the product images. I am using ajax. My ajax file is test.html and here is my code :-
$( "#click_me" ).click(function () {
$.ajax({
url: "test.php",
asyn:false,
success: function(result){
console.log(result);
}});
});
Test.php file code :-
$url="http://www.kohls.com/catalog/bedroom-mattresses-accessories-furniture.jsp?CN=Room:Bedroom+Category:Mattresses%20%26%20Accessories+Department:Furniture&cc=bed_bath-TN3.0-S-mattresses";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT,"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0 ");
$out = curl_exec($ch);
curl_close($ch);
$out = str_replace("\n", '', $out);
echo $out;
Note: please check the $url. The images are populating dynamically and we are not able to scrape them . Please I need quick guidance , I have used pythonjs as well to scrape them but that didn't work !!!
Thanks !!!
you need to parse out the images from the HTML. DOMDocument is a good choice for this.
example code (UNTESTED but should work in theory)
$url="http://www.kohls.com/catalog/bedroom-mattresses-accessories-furniture.jsp?CN=Room:Bedroom+Category:Mattresses%20%26%20Accessories+Department:Furniture&cc=bed_bath-TN3.0-S-mattresses";
$html=file_get_contents($url);
$domd=#DOMDocument::loadHTML($html);
foreach($domd->getElementsByTagName("img") as $img){
$src=$img->getAttribute("src");
if(empty($src)){continue;}
$src='http://www.kohls.com'.$src;
$filename=basename($src);
echo "downloading ".$filename.PHP_EOL;
file_put_contents($filename,file_get_contents($src));
}
just replace file_get_contents with your curl functions if you want curl
(also this is rather memory hungry, as the entire image will be downloaded to ram no matter how big it is. with curl, you could optimize it with CURLOPT_FILE to write to a file directly. could save a lot of RAM if you want to download images from NASA or the like)

Download an Excel file with PHP and Curl

I have a repetitive task that I do daily. Log in to a web portal, click a link that pops open a new window, and then click a button to download an Excel spreadsheet. It's a time consuming task that I would like to automate.
I've been doing some research with PHP and cUrl, and while it seems like it should be possible, I haven't found any good examples. Has anyone ever done something like this, or do you know of any tools that are better suited for it?
Are you familiar with the basics of HTTP requests? Like, do you know the difference between a POST and a GET request? If what you're doing amounts to nothing more than GET requests, then it's actually super simple and you don't need to use cURL at all. But if "clicking a button" means submitting a POST form, then you will need cURL.
One way to check this is by using a tool such as Live HTTP Headers and watching what requests happen when you click on your links/buttons. It's up to you to figure out which variables need to get passed along with each request and which URLs you need to use.
But assuming that there is at least one POST request, here's a basic script that will post data and get back whatever HTML is returned.
<?php
if ( $ch = curl_init() ) {
$data = 'field1=' . urlencode('somevalue');
$data .= '&field2[]=' . urlencode('someothervalue');
$url = 'http://www.website.com/path/to/post.asp';
$userAgent = 'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)';
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
$html = curl_exec($ch);
curl_close($ch);
} else {
$html = false;
}
// write code here to look through $html for
// the link to download your excel file
?>
try this >>>
$ch = curl_init();
$csrf_token = $this->getCSRFToken($ch);// this function to get csrf token from website if you need it
$ch = $this->signIn($ch, $csrf_token);//signin function you must do it and return channel
curl_setopt($ch, CURLOPT_HTTPGET, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 300);// if file large
curl_setopt($ch, CURLOPT_URL, "https://your-URL/anything");
$return=curl_exec($ch);
// the important part
$destination ="files.xlsx";
if (file_exists( $destination)) {
unlink( $destination);
}
$file=fopen($destination,"w+");
fputs($file,$return);
if(fclose($file))
{
echo "downloaded";
}
curl_close($ch);

How to pass a nested array using php cURL

I am collecting form posts via jQuery via the.val() method, validating those posts and passing errors back to the form inputs, or passing true to the .$post method which invokes a PHP cURL script.
A typical var will look like this:
var industs_servedVal = $('#industs_served').val();
In this case it is a select multiple form field. I understand that the .val() method in jQuery passes an array, so that seems reasonable, and am I right in saying that the var will also collect the array.
I then pass var industs_servedVal to the $.post method like this ( then slide up a thank you note):
$.post('../inc/form_sendsf_modified.php', {
othervars: othervarsVal;
industs_served: industs_servedVal,
}, function(data) {
$('#sendEmail').slideUp('slow', 'swing', function() {
$('#sendEmail').replaceWith('<h3>Thank you!</h3><p>Your message was sent to us. We\'ll get back to you as soon as we can.</p>');
});
});
}
return false;
});
The file "form_sendSF_modified.php" handles those posts and sends to the Sales Force Cloud using cURL. This works; however, the problem is that Sales Force shows "array" as the values received for the multiple field array, not the array values themselves. Is there a problem in the way I am collecting the array and passing it to sales force. Is the foreach loop capable of sending the multiple field array values as well as the other values as an array as shown in the code.
$post_data['00N70000002U2fA'] = $_POST['industs_served']; //Array
//$otherpost data
//cURL CODE for post
//traverse array and prepare data for posting (key1=value1)
foreach ( $post_data as $key => $value) {
$post_items[] = $key . '=' . $value;
}
//create the final string to be posted using implode()
$post_string = implode ('&', $post_items);
//create cURL connection to SFDC
$curl_connection = curl_init('https://www.salesforce.com/servlet/servlet.WebToLead?encoding=UTF-8');
//set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
//set data to be posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
//perform our request
$result = curl_exec($curl_connection);
//show information regarding the request
//print_r(curl_getinfo($curl_connection));
//echo curl_errno($curl_connection) . '-' .
curl_error($curl_connection);
//close the connection
curl_close($curl_connection);
//End cURL
You can use array itself (but it will change Content-Type header to multipart/form-data).
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_data);
Or you can use build string with http_build_query function, look example #3.
use this
$post = "ac=on&p=1&pr[]=0&pr[]=1&a[]=3&a[]=4&pl=on&sp[]=3&ct[]=3&s=1&o=0&pp=3&sortBy=date";
parse_str($post,$fields);
$url = 'http://example.com/';
//open connection
$ch = curl_init();
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_POST, true);
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields);
curl_setopt($ch,CURLOPT_RETURNTRANSFER, true);
//execute post
$result = curl_exec($ch);
//close connection
curl_close($ch);

email current url on if else condition with no send button

)
i want the smallest possible script to send current URL via email without a send button
but as a result of an if { } else {} condition..
ideas anyone?
ok.. i have gotten so far.. got a script on one end reciving and mailing
the desired data to the desired email.. then i need a way to send the current
page URL without a send button but as a result of an if (!notsomething) {} else{ send!!}
so i have this so far AND it works!! but i am getting partial URL
instead of "http://www.example.com/subfile.php" i am getting just "/subfile.php"
$websiteurl = $_SERVER['REQUEST_URI'];
$curl_connection = curl_init('http://www.sagive.co.il/downloads/wordpress/linkRemovedNotification.php');
//set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
//set data to be posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $websiteurl);
//perform our request
$result = curl_exec($curl_connection);
//show information regarding the request
print_r(curl_getinfo($curl_connection));
echo curl_errno($curl_connection) . '-' . curl_error($curl_connection);
//close the connection
curl_close($curl_connection);
For some weird reason it wont send when i use:
$websiteurl = $_SERVER["SERVER_NAME"].$_SERVER["REQUEST_URI"];
.
SO.. thats the last problem.. ideas anyone?
Look into jQuery for this probably
<script rel="javascript" src="jquery.js"></script>
<script>
$.post('ajax/test.html?url='+window.location.pathname)
</script>
Ok! solved it :) - thanks for your help and your time.
here is the code i used and it works (Collected from sites across the web)
//extract data from the post
$currentw = $_SERVER["SERVER_NAME"].$_SERVER["REQUEST_URI"];
$currentth = SpaceJunkie;
//set POST variables
$url = 'http://www.example.com';
$fields = array(
'siteurl'=>urlencode($currentwebsite),
'themename'=>urlencode($currentheme),
);
//url-ify the data for the POST
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; }
rtrim($fields_string,'&');
//open connection
$ch = curl_init();
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string);
//execute post
$result = curl_exec($ch);
//close connection
curl_close($ch);

cURL post data to asp.net page

I am trying to call the __doPostback javascript function in a asp.net page from php using curl.
I learnt that this can be done by making a post request to the asp.net page with the appropriate parameters.
So in curl,
I make a get request / just use file_get_contents to retrieve the initial page.
From this, I extract the values for __VIEWSTATE and __EVENTVALIDATION.
So far everything seems ok.
Now, I understand that we need to make a post request using cURL with __VIEWSTATE and other parameters required. ( values for the fields present in the asp.net form )
I am unable to construct the CURLOPT_POSTFIELDS correctly.
For instance, I am trying this out,
$postoptions1='__EVENTTARGET='.('ctl00$ContentPlaceHolder1$gRef').'&__EVENTARGUMENT='.('$2');
$postoptions2 = '&__VIEWSTATE='.urlencode($viewState) ;
$otherparams = '&ctl00$ContentPlaceHolder1$ddlName=Abc';
And before using setopt for CURLOPT_POSTFIELDS, I am doing,
urlencode ($postoptions1.$postoptions2.$otherparams)
This does not work. The submit results are not shown, which means, the required parameter __VIEWSTATE was not found in my post request.
If I change the order of the parameters and place __VIEWSTATE as the first parameter, the results page is shown but the other parameter values are not honoured.
I think there is some problem with the way I am encoding the parameters.
Please tell me how to construct the parameters for the post request to a asp.net page.
Thanks.
--Edited--
Here is the complete code:
$resultsPerPage='10';
$url = "www.example.com"; // url changed
$curl_connection = curl_init($url);
function sendCurl($curl_connection,$url,$params,$isPost=false) {
//$post_string = $params;
$post_string = http_build_query($params);
//$post_string = build_query_string($params);
//$post_string = urlencode($params);
echo 'After Encode'.$post_string;
$cookie="/cookie.txt";
//set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 300);
curl_setopt($curl_connection, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_HEADER, 0); // don't return headers
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl_connection,CURLOPT_REFERER, $url);
if($isPost) {
curl_setopt ($curl_connection, CURLOPT_POST, true);
//set data to be posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
curl_setopt($curl_connection,CURLOPT_COOKIEJAR,$cookie);
}
else {
curl_setopt($curl_connection,CURLOPT_COOKIEFILE,$cookie);
}
$response1 = curl_exec($curl_connection);
if($response1 === false)
{
echo 'Curl error: ' . curl_error($curl_connection);
}
else
{
echo 'Operation completed without any errors';
}
return $response1;
} **// First time, get request to asp.net page
$response1 = sendCurl($curl_connection,$url,'',false);
$viewState=getVStateContent($response1);
$eventValidation =getEventValidationContent($response1);
$simpleParams = '&__VIEWSTATE='.$viewState.'&ctl00$ContentPlaceHolder1$ddlManuf=&ctl00$ContentPlaceHolder1$ddlCrossType=&ctl00$ContentPlaceHolder1$ddlPageSize='.$resultsPerPage.'&ctl00$ContentPlaceHolder1$btnSearch=Search&ctl00_ToolkitScriptManager1_HiddenField=&__EVENTTARGET=&__EVENTARGUMENT=';
// Second post - for submitting the search form
$response2= sendCurl($curl_connection,$url,$simpleParams,true);
----**
What you want is http_build_query, which will format an array as proper HTTP parameters.
Edit: To clarify what this should probably look like:
$params = array(
'__EVENTTARGET' => 'ctl00$ContentPlaceHolder1$gRef',
'__EVENTARGUMENT' => '$2',
'__VIEWSTATE' => $viewState,
'ctl00$ContentPlaceHolder1$ddlName' => 'Abc'
);
curl_setopt($curlHandler, CURLOPT_POSTFIELDS, http_build_query($params));
Also, what's ctl00$ContentPlaceHolder1$ddlName supposed to be?
Don't urlencode() the ampersands (&) linking the parameters together, just the keys & values (the stuff on either side of the ampersands).

Categories