I'm still a bit new to using curl to pull data and I've recently started using Fiddler to help find what options need to be set.
I'm trying to see if I can pull an image from a site. I first hit a search page - I set the search parameters, then start hitting links in the results. When I attempt to go a link in one of the results for an image, I get an empty string returned from curl_exec().
The weird thing is - at one point, it worked - I got the data back and successfully saved the image locally. But then it stopped, and I have no idea what I was doing to have it working. Naturally, everything works OK in the browser. :(
I'm using Simple HTML DOM to parse through results and cUrl for the actual page requests. curl_error() does not show an error, curl_getinfo() thinks everything is OK too. It's probably something trivial, but I'm not sure how to troubleshoot it beyond where I am.
<?php
include 'includes/simple_html_dom.php';
$url = "http://nwweb.co.bell.tx.us/NewWorld.Aegis.WebPortal/Corrections/InmateInquiry.aspx";
// Get Cookie - ASP.NET_SessionId
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
$r = curl_exec($ch);
preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $r, $matches);
$cookies = array();
foreach($matches[1] as $item)
{
parse_str($item, $cookie);
$cookies = array_merge($cookies, $cookie);
}
$sessionCookie = "ASP_NET_SessionId=".$cookies['ASP_NET_SessionId'];
// now load up page into Simple HTML DOM and get all inputs - ignore buttons and populate our dates
$startDate = "02%2F01%2F2000";
$endDate = "02%2F07%2F2016";
$getInputs = str_get_html($r);
$inputs = $getInputs->find('input');
$inputs_array = array();
$buttons_array = array();
for ($i=0; $i<count($inputs); $i++)
{
if ($inputs[$i]->type != "submit")
{
$inputs_array[$inputs[$i]->id] = $inputs[$i]->value;
if (stripos($inputs[$i]->id, "FromDate") > 0)
$inputs_array[$inputs[$i]->id] = $startDate;
if (stripos($inputs[$i]->id, "ToDate") > 0)
$inputs_array[$inputs[$i]->id] = $endDate;
}
}
// build up our curl data - includes hidden inputs, our to & from dates, plus the Search button
$curl_data = http_build_query($inputs_array)."&ctl00%24DefaultContent%24uxSearch=Search";
// POST the data, include session cookie
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $curl_data);
curl_setopt($ch, CURLOPT_COOKIE, $sessionCookie);
$response = curl_exec($ch);
// this shows that we can get data
// find the links from the HTML
$htmlDom = str_get_html($response); // load up Simple HTML DOM
// get the table of results
$divTable = $htmlDom->find('div#ctl00_DefaultContent_uxResultsWrapper',0)->find('table',0);
$rows = $divTable->find('tr');
for ($i=1; $i<count($rows);$i++)
{
if ($i>3) break; // limit the length of script for debugging
$link = $rows[$i]->find('td',1)->find('a',0)->href;
// build up query to get inmate details from the link above
$url = "http://nwweb.co.bell.tx.us/NewWorld.Aegis.WebPortal/Corrections/".$link;
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_COOKIE, $sessionCookie);
$page = curl_exec($ch);
$pageData = str_get_html($page);
// Now find the Photo, there's a thumb in div.BookingPhotos
// It is linked to a full size image, the link is of the form http://nwweb.co.bell.tx.us/NewWorld.Aegis.WebPortal/GetImage.aspx?ImageKey=17C030IS, but in the href, it has ../GetImage.aspx?ImageKey=xxxx
$photoLink = $pageData->find('div.BookingPhotos',0)->find('a',0)->href;
// get rid of .. and put the base URL on the front
$imgLink = str_replace("..", "http://nwweb.co.bell.tx.us/NewWorld.Aegis.WebPortal", $photoLink);
// now attempt to pull the image
$ch = curl_init($imgLink);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_COOKIE, $sessionCookie);
// here is the PROBLEM - NO DATA RETURNED
$imgData = curl_exec($ch); // I get a header back, but NO data
}
?>
Related
I have this code:
$status_url = $site_properties["status_url"];
//$listeners_url = $site_properties["listeners_url"];
//$messages_url = $site_properties["messages_url"];
//$html = file_get_html($status_url);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $status_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$res = curl_exec($ch);
curl_close($ch);
$dom = new DomDocument();
# $dom->loadHTML($res);
$radio_listeners = $dom->getElementById('listeners_cont');
echo $radio_listeners->textContent;
I was wondering how can i write this script to wait a few seconds (10 for example), so the setInterval ajax in $status_url page, will be started and all the fields will be updated correctly.
Some screen shot to explain:
curl and PHP doesn't execute any javascript at all. You need something like PhantomJS - or just get the AJAX call in your browser's network tab and implement this call only.
How can I remove firebase specific data? I use the
php Kreait\Firebase library.
$fg = $database->getReference('raw_check_out')->orderByChild('reciptno')->equalTo($recipt)->getSnapshot();
$reb = $fg->getValue();
$fg->remove();
but this is not working.
Based on your code example:
$fg = $database
->getReference('raw_check_out')
->orderByChild('reciptno')
->equalTo($recipt)
->getSnapshot();
Here $fg does not hold the reference, but the snapshot.
If you want to remove the reference after you have retrieved the data you need, you need the reference itself:
$fg = $database->getReference('raw_check_out');
$query = $fg->orderByChild('reciptno')->equalTo($recipt);
$reb = $query->getSnapshot()->getValue();
$fg->remove();
This function is for unsubscribing users if you want to remove "user-id-8776" and your structure is like:
public function unsubscribe($uid)
{
$ref = $this->database->getReference()->getChild('users')->getChild($uid)->orderByChild($uid)->getReference();
$path = $ref->getUri()->getPath();
$ref->remove();
}
For any other data is more or less the same, you just have to find the reference of data you want to delete it and then you can remove it.
Check Image
You have another option to resolve this problem. This solution given below:
$url = "https://<projectID>.firebaseio.com/chatList/".<jsonFileName>."json";
$ch = curl_init();
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "DELETE");
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/plain'));
$result = curl_exec($ch);
curl_close($ch);
[Edited for better explanation and code included]
Hi! I have a php script on my web server that logs in to my heat pump web interface nibeuplink.com and gets all my temperature readings and so forth and returns them in a json-format.
freeboard.io is a free service for visualizing data, so I'm making a freeboard.io for my heat pump values. in freeboard.io I can add any json data as a data source, so I have added the link to my php-script. It fetches the data once but it seems there is some kind of cached values that it uses after that so they are not updated with new values from the script. freeboard.io uses a get-function to get the url. If i use a normal web browser to run the php script and refresh it, the values are updated - and also immediately updated in freeboard.io. Freeboard.io has a setting to automatically update the data source every 5 seconds.
It seems that there is something that triggers the script correctly when it is fetched from my web browser, but not when it is fetched from freeboard.io that uses a get function every 5 seconds to get new data.
in freeboard I can add headers to the get request, is there some header that would help me here to discard any cached data?
I hope that explains my problem better.
Is there anything i can add to my code in the beginning to always force an override of any cached data?
<?php
/*
* read nibe heatpump values from nibeuplink status web page and return them in json format.
* based on: https://www.symcon.de/forum/threads/25663-Heizung-Nibe-F750-Nibe-Uplink-auslesen-auswerten
* to get the code which is required as parameter, log into nibe uplink, open status page of your heatpump, and check url:
* https://www.nibeuplink.com/System/<code>/Status/Overview
*
* usage: nibe.php?email=<email>&password=<password>&code=<code>
*/
// to add additional debug output to the resulting page:
$debug = false;
date_default_timezone_set('Europe/Helsinki');
$date = time();
// Create temp file to store cookies
$ckfile = tempnam ("/tmp", "CURLCOOKIE");
// URL to login page
$url = "https://www.nibeuplink.com/LogIn";
// Get Login page and its cookies and save cookies in the temp file
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // Accepts all CAs
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_COOKIEJAR, $ckfile); // Stores cookies in the temp file
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
// Now you have the cookie, you can POST login values
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 2);
curl_setopt($ch, CURLOPT_POSTFIELDS, "Email=".$_GET['email']."&Password=".$_GET['password']);
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile); // Uses cookies from the temp file
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // Tells cURL to follow redirects
$output = curl_exec($ch);
curl_setopt($ch, CURLOPT_URL, "https://www.nibeuplink.com/System/".$_GET['code']."/Status/ServiceInfo");
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_POST, 0);
$result = curl_exec($ch);
$pattern = '/<h3>(.*?)<\/h3>\s*<table[^>]*>.+?<tbody>(.+?)<\/tbody>\s*<\/table>/s';
if ($debug) echo "pattern: <xmp>".$pattern."</xmp><br>";
$pattern2 = '/<tr>\s*<td>(.+?)<span[^>]*>[^<]*<\/span>\s*<\/td>\s*<td>\s*<span[^>]*>([^<]*)<\/span>\s*<\/td>\s*<\/tr>/s';
if ($debug) echo "pattern2: <xmp>".$pattern2."</xmp><br>";
preg_match_all($pattern, $result, $matches);
// build json format from matches
echo '{';
$first = true;
foreach ($matches[1] as $i => $title) {
echo ($first ? '"' : ',"').trim($title).'":{';
$content = $matches[2][$i];
preg_match_all($pattern2, $content, $values);
$nestedFirst = true;
foreach ($values[1] as $j => $field) {
echo ($nestedFirst ? '"' : ',"').trim($field).'":"'.$values[2][$j].'"';
$nestedFirst = false;
}
echo "}";
$first = false;
}
echo ",\"time\":{\"Last fetch\":\"$date\"}";
echo "}";
if ($debug) {
echo "<pre><xmp>";
echo print_r($matches);
echo "<br><br>";
echo $result;
echo "</xmp></pre>";
}
?>
You can make an ajax call to php script to refresh the part of webpage. I don't understand what do you mean by io i.e. are you talking about fetching the data from database and if any changes occurred in database then only newly added records must be fetched. If you mean it in that sense then you can use cookie to track any new records added into database and only if it finds new records it can make ajax call to php script to run your algorithm on fetched total dataset.
I wants to create a php script for alerts from my work website when new notice is published, so following the page url
http://www.mahapwd.com/nit/ueviewnotice.asp?noticeid=1767
from this page i want a variable for Date & Time of Meeting (Date and time seperately two variables)
Place of Meeting and Published On
please help me to create a perfect php script.
I tried to create following script but it gives to many errors
<?php
$url1 = "http://www.mahapwd.com/nit/ueIndex.asp?district=12";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($ch);
preg_match("/href=(.*)\", $data, $urldata);
$url2 = "http://www.mahapwd.com/nit/$urldata[1];
curl_setopt($ch, CURLOPT_URL, $url2);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data2 = curl_exec($ch);
preg_match("/Published On:</b>(.*)<\/font>", $data, $pubDt);
$PubDate = $pubDt[1];
preg_match("/Time of Meeting:</b>(.*) ", $data, $MtDt);
$MeetDate = $MtDt[1];
preg_match("/Time of Meeting:</b>$MtDt[1] (.*)</font>", $data, $MtTime);
$MeetTime = $MtTime[1];
preg_match("/Place of Meeting:</b>(.*)<\/font>", $data, $pubDt);
$PubDate = $pubDt[1];
?>
Hello i have done simple code for you. You can download simple_html_dom.php from http://simplehtmldom.sourceforge.net/
require_once "simple_html_dom.php";
$url='http://www.mahapwd.com/nit/ueviewnotice.asp?noticeid=1767';
//parse url
for ($i=0;$i<1;$i++) {
$html1 = file_get_html($url);
if(!$html1){ echo "no content"; }
else {
//here is parsed html
$string1 = $html1;
//now you need to find table
$element1=$html1->find('table');
//here is a table you need
$input=$element1[2];
//now you can select row from here
foreach($input->find('td') as $element) {
//in here you can find name than save it to database than check it
}
}
}
I am trying to get the parent element, I think of an XML tag. Basically I need to go through multiple <HotelRoomResponse> results and find this parent tag that contains a child tag with this exact number value: <roomTypeCode>17918</roomTypeCode> I am not sure how to do this or what would be the best way. Because I then need to get ALL the information in that specific <HotelRoomResponse>. Here is an example XML response:
<HotelRoomResponse>
<cancellationPolicy> </cancellationPolicy>
<rateCode>200482409</rateCode>
<roomTypeCode>17918</roomTypeCode>
<rateDescription>
Deluxe Sunset View - All Inclusive-Up to $300Resort Credit
</rateDescription>
<roomTypeDescription>
Deluxe Sunset View - All Inclusive-Up to $300Resort Credit
</roomTypeDescription>
<supplierType>E</supplierType>
</HotelRoomResponse>
So there are various of these result types and I need to loop through it and find this specific one.
Here is how I am connecting to the XML:
$ch = curl_init();
$fp = fopen('room_request.xml','w');
curl_setopt($ch, CURLOPT_URL, "http://api.ean.com/ean-services/rs/hotel/v3/avail?cid=55505&minorRev=13&apiKey=4sr8d8bsn75tpcuja6ypx5g3&locale=en_US¤cyCode=USD&customerIpAddress=10.184.2.9&customerUserAgent=Mozilla/5.0+(Windows+NT+6.1)+AppleWebKit/535.11+(KHTML,+like+Gecko)+Chrome/17.0.963.79+Safari/535.11&customerSessionId=&xml=<HotelRoomAvailabilityRequest><hotelId>".$hid."</hotelId><arrivalDate>05/14/2012</arrivalDate><departureDate>05/18/2012</departureDate><RoomGroup><Room><numberOfAdults>3</numberOfAdults><numberOfChildren>0</numberOfChildren><childAges>0</childAges></Room></RoomGroup><includeDetails>true</includeDetails></HotelRoomAvailabilityRequest>");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Accept: application/xml'));
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FILE, $fp);
$val = curl_exec($ch);
curl_close($ch);//Close curl session
fclose($fp); //Close file overwrite
$avail = simplexml_load_file('room_request.xml');
Any ideas are welcome.
To find all HotelRoomResponse nodes which has a roomTypeCode child node with the value '17918', use the following:
$match = $avail->xpath("/HotelRoomResponse[child::roomTypeCode[text() = '17918']]");
EDIT: $match will be an array holding all matches.
Ok figured it out!!!! Here is what I used to get a node with text = value. Then I got all sibling elements.
// load as file
$contents = new SimpleXMLElement($source,null,true);
$result = $contents->xpath('HotelRoomResponse[roomTypeCode="17918"]');
foreach($result as $key=>$node)
{
$cancelPolicy = $node->cancellationPolicy;
}
$xml = new SimpleXMLElement('room_request.xml');
/* Search for <HotelRoomResponse><roomTypeCode> */
$result = $xml->xpath('/HotelRoomResponse/roomTypeCode');
Result will give you a list of nodes you can then check and get the parent node if appropriate
see here http://www.php.net/manual/en/simplexmlelement.xpath.php
Edit 2.
Key was the namespace
<?php
$ch = curl_init();
$fp = fopen('room_request.xml','w');
curl_setopt($ch, CURLOPT_URL, "http://travellinginmexico.com/test/room_request.xml");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Accept: application/xml'));
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FILE, $fp);
$val = curl_exec($ch);
curl_close($ch);//Close curl session
fclose($fp); //Close file overwrite
$xml = new SimpleXMLElement(file_get_contents('room_request.xml'));
/* Search for <HotelRoomResponse><roomTypeCode> */
$xml->registerXPathNamespace('ns2', 'http://v3.hotel.wsapi.ean.com/');
$result = $xml->xpath("HotelRoomResponse[child::roomTypeCode[text() = '153725']]");
foreach($result as $obj=>$node)
{
var_dump($node->roomTypeCode);
}
With the example you sent will get specific information