Curl Drops Parameters - php

SO, I have been fighting with a piece of code that I want to use to get a remote page's source code using curl.
The code executes successfully, both in the browser and on command line. However, I get the of the main file only. When parameters are added, they are not considered whatsoever in the output.
The Code:
STACK : Ubuntu, Nginx, PHP-FPM 7.2
$urlcontent = 'https://XXX.YYY.COM/file/?var1=value1' ;
// Create a new cURL resource
$curl = curl_init();
if (!$curl) {
die("Couldn't initialize a cURL handle");
}
// Set the file URL to fetch through cURL
curl_setopt($curl, CURLOPT_URL, $urlcontent);
// Set a different user agent string (Googlebot)
curl_setopt($curl, CURLOPT_USERAGENT, 'CodiBot/2.1');
// Follow redirects, if any
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
// Fail the cURL request if response code = 400 (like 404 errors)
curl_setopt($curl, CURLOPT_FAILONERROR, true);
// Return the actual result of the curl result instead of success code
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
// Wait for 10 seconds to connect, set 0 to wait indefinitely
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
// Execute the cURL request for a maximum of 50 seconds
curl_setopt($curl, CURLOPT_TIMEOUT, 50);
// Do not check the SSL certificates
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
// Fetch the URL and save the content in $html variable
$html = curl_exec($curl);
// Check if any error has occurred
if (curl_errno($curl))
{
echo 'cURL error: ' . curl_error($curl);
}
else
{
// cURL executed successfully
print_r(curl_getinfo($curl));
print_r($html);
}
curl_close($curl);
PROBLEM
I get the content for https://XXX.YYY.COM/file but not the corresponding ?var1=value1 part. IN other words, as I feed info to be retrieved to DB I get only the html of the main file.
I tried :
curl_setopt($ch, CURLOPT_POSTFIELDS, 'foo=1&bar=2&baz=3');
I know the remote server may have CORS enabled, but I tried the same url using a remote curl retriever and it succeeded. SO, it may not be the remote server

Related

PHP & CURL scraping

I have a problem when I run this script in Google Chrome I got a blank page. When I use another link of a web site, it works successfully. I do not what is happening.
$curl = curl_init();
$url = "https://www.danmurphys.com.au/dm/home";
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($curl);
echo $output;
There are some conditions which make your result blank. Such as:
Curl error.
Redirection without response body and the curl doesn't follow the redirection.
The target host doesn't give any response body.
So here you have to find out the problem.
For the first possibility, use curl_error and curl_errno to confirm that the curl wasn't errored when its runtime.
For the second, use CURLOPT_FOLLOWLOCATION option to make sure the curl follows the redirection.
For the third possibility, we can use curl_getinfo. It returns an array which contains "size_download". The size_download shows you the length of the response body. If it is zero that is why you see a blank page when printing it.
One more, try to use var_dump to see the output (debug purpose only). There is a possibility where the curl_exec returns bool false or null. If you print the bool false or null it will show a blank.
Here is the example to use all of them.
<?php
$curl = curl_init();
$url = "https://www.danmurphys.com.au/dm/home";
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
$output = curl_exec($curl);
$info = curl_getinfo($curl);
$err = curl_error($curl);
$ern = curl_errno($curl);
if ($ern) {
printf("An error occurred: (%d) %s\n", $ern, $err);
exit(1);
}
curl_close($curl);
printf("Response body size: %d\n", $info["size_download"]);
// Debug only.
// var_dump($output);
echo $output;
Hope this can help you.
Update:
You can use CURLOPT_VERBOSE to see the request and response information in details.
Just add this
curl_setopt($curl, CURLOPT_VERBOSE, true);
It doesn't need to be printed, the curl will print it for you during runtime.

Icecast metadata update by php

I need to get Icecast metadata auto update by PHP let say every 15 min which will be done by cPanel cronjob.
i had the below code but it does't work(it works if I use header location to redirect however cronjob won't be able to do that)
<?PHP
$url="http://tgftp.nws.noaa.gov/data/observations/metar/stations/KJFK.TXT";
$info=file_get_contents($url);
$url_info = "http://username:password#icecast:8000/admin/metadata?mount=/mymount&mode=updinfo&song=" . urlencode($info);
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url_info);
curl_setopt($ch, CURLOPT_HEADER, 0);
// grab URL and pass it to the browser
curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
?>
Try checking for errors after you execute the call using curl_error:
<?php
$url="http://tgftp.nws.noaa.gov/data/observations/metar/stations/KJFK.TXT";
$info=file_get_contents($url);
$url_info = "http://username:password#icecast:8000/admin/metadata?mount=/mymount&mode=updinfo&song=" . urlencode($info);
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url_info);
curl_setopt($ch, CURLOPT_HEADER, 0);
// grab URL and pass it to the browser, check for errors
if (curl_exec($ch) === FALSE)
{
print 'Curl-Error occurred: ' . curl_error($ch).', error code: '.curl_errno($ch);
}
// close cURL resource, and free up system resources
curl_close($ch);

PHP: cURL or header - faster and better way to send cross domain data and redirect

I create one small API what collect contact form informations from one website and store in database on another website in managment application what I also build.
On website where is contact form is this code:
// collect all fields
$save = array(
'oAuth'=>'{KEY}',
'secret'=>'{SECRET-KEY}',
'category'=>'Category',
'name'=>'Jon Doe',
'email'=>'jon#doe.com',
'phone'=>'123 456 7890',
// ... etc. other fields
);
// made GET request
$fields=array();
foreach($save as $key=>$val){
$fields[]=$key."=".rawurlencode($val);
}
// Set cross domain URL
$url='http://api.mydomain.com/';
// Send informations in database
$cURL = curl_init();
curl_setopt($cURL,CURLOPT_URL, $url.'?'.join("&",$fields));
curl_setopt($cURL, CURLOPT_RETURNTRANSFER, true);
curl_setopt($cURL, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($cURL, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($cURL, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($cURL, CURLOPT_TIMEOUT, 2);
curl_setopt($cURL, CURLOPT_HTTPHEADER, array('Accept: application/json'));
$output=curl_exec($cURL);
curl_close($cURL);
usleep(1000);
// redirect
header("Location: http://somemysite.com/thank-you-page");
session_destroy();
exit;
My question is do to use cURL like now for this or to use header() function to send GET?
I ask because I not use output here and sometimes redirection start before cURL finish request.
What is faster way to send GET info and not lost data?
The redirection will not start before cURL finishes. What may happen is that cURL fails to load you remote page, and you don't handle that error. You should add something like this:
if($output === false || curl_getinfo($cURL, CURLINFO_HTTP_CODE) != 200) {
// Here you handle your error
}
If you are looking for an alternative to load remote webpages, you can set allow_url_fopen to true in your php.ini and then use this simple function instead of cURL:
$output = file_get_contents($url.'?'.join("&",$fields));

How can I get a live data from API with cURL save or stream that data with another script?

I have a hardware which records data in every seconds. I can connect to the hardware with a browser and use the hardware’s interface and get the live data. For example: I can call the realtime data with command like this:
http://192.168.100.120:2345/realtime
That’s what I can see in the browser:
DM_NumLogChans=5
DM_NumDataModes=1
DM_LogicalChan=1
DM_ChanType=SEQUENTIAL
DM_NumDims=2
DM_DataMode=1
DM_DataModeType=TIMHIS
DM_AxisLabel.Dim1=Time
DM_AxisLabel.Dim2=Value
DM_AxisUnits.Dim1=secs
DM_AxisUnits.Dim2=microstrain
DM_SampleRate=1.000000
DM_TimeBase=0.0
DM_ChanName=bridge_1
DM_UserMin=-583.220764
DM_UserMax=940.916199
DM_Start=
-439.779 -391.875 -680.114 1001.37 0
-442.068 -396.62 -680.945 1001.37 0
-443.571 -399.705 -680.639 1001.37 0
-445.598 -404.848 -684.662 1001.37 0
A new row appear in each seconds. I would like to get this data and save it to a file or display it in real time in my php program. How can I catch the data? I tried with cURL. I think that is the solution but I am really new to this. I would appreciate any help or advice you could give me.
Try this code with your url
function curl_download($Url){
// is cURL installed yet?
if (!function_exists('curl_init')){
die('Sorry cURL is not installed!');
}
// OK cool - then let's create a new cURL resource handle
$ch = curl_init();
// Now set some options (most are optional)
// Set URL to download
curl_setopt($ch, CURLOPT_URL, $Url);
// Set a referer
curl_setopt($ch, CURLOPT_REFERER, "http://www.example.org/yay.htm");
// User agent
curl_setopt($ch, CURLOPT_USERAGENT, "MozillaXYZ/1.0");
// Include header in result? (0 = yes, 1 = no)
curl_setopt($ch, CURLOPT_HEADER, 0);
// Should cURL return or print out the data? (true = return, false = print)
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Timeout in seconds
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
// Download the given URL, and return output
$output = curl_exec($ch);
// Close the cURL resource, and free system resources
curl_close($ch);
return $output;
}
echo curl_download("192.168.100.120:2345/realtime");

cURL operation and PHP

I have a frontend code
$ch = curl_init();
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
//curl_setopt($ch, CURLOPT_HTTPHEADER, array('Accept: application/json'));
curl_setopt($ch, CURLOPT_URL, $url);
//make the request
$responseJSON = curl_exec($ch);
$response_status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
if ($response_status == 200) { // success
// remove any "problematic" characters from the json string and then decode
if (debug) {
echo "----finish API of getAPI inside basic_function with status==200---";
echo "<br>";
echo "-------the json response is-------" ; //.$responseJSON;
var_dump($responseJSON);
//print_r($responseJSON);
echo "<br>";
}
return json_decode( preg_replace( '/[\x00-\x1F\x80-\xFF]/', '', $responseJSON ) );
}
and I have a backend code which executed when cURL fired its operation with its URL. The backend code would therefore activated. So, I know cURL is operating.
$output=array (
'status'=>'OK',
'data'=>'12345'
)
$output=json_encode($output)
echo $output;
and $output shown on browser as {"status":"OK","data":"12345"}
However, I gone back to the frontend code and did echo $responseJSON, I got nothing. I thought the output of {"status":"OK","data":"12345"} would gone to the $responseJSON. any idea?
Here's output on Browser, something is very odd! the response_status got 200 which is success even before the parsing of API by the backend code. I expect status =200 and json response after the {"status":"OK","data":"12345"}
=========================================================================================
inside the get API of the basic functions
-------url of cURL is -----http://localhost/test/api/session/login/?device_duid=website&UserName=joe&Password=1234&Submit=Submit
----finish API of getAPI inside basic_function with status==200---
-------the json response is-------string(1153)
"************inside Backend API.php******************
---command of api is--/session/login/
---first element of api is--UserName=joe
--second element of api is---Password=1234
---third element of api is----Submit=Submit
----fourth element of api is---
-------inside session login of api-------------
{"status":"OK","data":"12345"}
Have you tried with curl_setopt($ch, CURLOPT_TIMEOUT, 10); commented?
See what happends if you comment that line.
Also try with the a basic code, if that works, smthing you added later is wrong:
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http://www.example.com/");
curl_setopt($ch, CURLOPT_HEADER, false);
// grab URL and pass it to the browser
curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
Try var_dump($responseJSON)
If it returns false try
curl_error ( $ch )
Returns a clear text error message for the last cURL operation.
Are you sure your $url is correct?

Categories