Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
How can I automatically upload to a database data from tables like the one in this page? I can use their function "export" and then manually download the .csv file, and upload it, but if I want everyday the data from each game, it is a pain... do you think it is possible to automatize it? The only solution would be by scraping their website?
Thanks
You could use the cURL lib of PHP.
Here there's an example:
<?php
$ch = curl_init();
$timeout = 0; // set to zero for no timeout
$url = 'http://www.basketball-reference.com/boxscores/201506160CLE.html'; // set the page url
curl_setopt ($ch, CURLOPT_URL, $url);//enter your url here
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$file_contents = curl_exec($ch); //get the page contents
include "simple_html_dom.php";
$table = $file_contents;
$html = str_get_html($table);
$printing = false;
//header('Content-type: application/ms-excel');
$fp = fopen("php://output", "w");
foreach($html->find('tr') as $element){
$arr = array();
foreach ($element->find('tr') as $element2) {
if(!$printing)
$printing = strpos($element2,'Scoring') !== false;
if($printing){
//echo $element2 -> plaintext . "<br>";
$arr[] = $element2 -> plaintext; //comment here
}
}
fputcsv($fp, $arr);
}
fclose($fp);
?>
You have to download a file from here.
You get a text/csv file in your page, if you prefere a plain it's sufficient switch comments (comment line where is requested and decomment the others)
You can parse your new csv as you want
Related
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 2 years ago.
Improve this question
User can paste image url on form, then my app download this on local server. I would like validate this image, but i have problem.
Input:
$image = "http://example.com/image.png";
Rules, but not working
'image' => [
'required',
'url',
'mimes:jpeg,png',
'max:5120',
'dimensions:min_width=400,min_height=400',
new FileExtensionRule(['png', 'jpg', 'jpeg'])
]
How can I validate image from external source.
First you have to make custom rule Custom Validation Rules
php artisan make:rule RemoteImage
and then put your logic inside this function in the RemoteImage class
public function passes($attribute, $value)
{
return strtoupper($value) === $value;
}
to get file type
$image = "http://example.com/image.png";
$ext = pathinfo($image, PATHINFO_EXTENSION);
to get the file size to validate
using curl
$ch = curl_init('http://example.com/image.png');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_NOBODY, TRUE);
$data = curl_exec($ch);
$size = curl_getinfo($ch, CURLINFO_CONTENT_LENGTH_DOWNLOAD);
curl_close($ch);
echo $size;
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself.
Closed 8 years ago.
Improve this question
I have this code,
<?php
// XML File
$feed_url = "http://api.worldweatheronline.com/free/v1/weather.ashx?q=dubai&format=xml&extra=localObsTime&num_of_days=5&key=42esgfa4pfqp6zwgr4egjbph";
// INITIATE CURL.
$curl = curl_init();
// CURL SETTINGS.
curl_setopt($curl, CURLOPT_URL,"$feed_url");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 0);
// GRAB THE XML FILE.
$xxml = curl_exec($curl);
curl_close($curl);
$xml = new SimpleXMLElement($xxml);
$flag=0;
// Loop through ... only pick the first one
foreach ($xml->current_condition as $item) {
if($flag==0){
echo"
Dubai Time And Date: {$item->localObsDateTime}
<br />
";
}
$flag=1;
}
?>
The above code will give me local time and date of the city of Dubai.
I need to take the city value from a text input box and replace "dubai" with that.
This will do the text box part.
I tried to incorporate, but getting some weird time.
<?php
if(isSet($_POST['city']) && $_POST['city'])
{
// XML File
$feed_url = "http://api.worldweatheronline.com/free/v1/weather.ashx?q=" . urlencode(trim($_POST['city'])) . "&format=xml&extra=localObsTime&num_of_days=5&key=xxxxxxxxx";
// INITIATE CURL.
$curl = curl_init();
// CURL SETTINGS.
curl_setopt($curl, CURLOPT_URL,"$feed_url");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 0);
// GRAB THE XML FILE.
$xxml = curl_exec($curl);
curl_close($curl);
$xml = new SimpleXMLElement($xxml);
$flag=0;
// Loop through ... only pick the first one
foreach ($xml->current_condition as $item) {
if($flag==0){
echo
ucfirst($_POST['city']) . " Time And Date: {$item->localObsDateTime}
<br />
";
}
$flag=1;
}
}
?>
<form method="post" action="">
<input type="text" name="city" value="<?php echo (isSet($_POST['city']) && $_POST['city'] ? $_POST['city'] : ''); ?>">
<input type="submit" value="Search City">
</form>
i see http://xooplate.com/templates/download/13693 return a file.
i have using
$ch = curl_init("http://xooplate.com/templates/download/13693");
$fp = fopen("example_homepage.zip", "w");
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
function get_include_contents($filename) {
if (is_file($filename)) {
ob_start();
include $filename;
$contents = ob_get_contents();
ob_end_clean();
return $contents;
}
return false;
}
$string = get_include_contents('http://xooplate.com/templates/download/13693');
but not working, i expected one a help, thank for all
Updated Answer :
Hey buddy, i checked that site. they put onClick jQuery method on the download button (widget.js line 27) with the post domain security. I tried to trigger the onClick event but due to domain check security functiion server only allows http://www.stumbleupon.com and http://xooplate.com to post the data using AJAX. So it wouldn't be possible using curl or PHPDOM. Sorry. :) Great Security.
Below Solution is not the answer !
Hi I found a solution using PHP Simple HTML DOM Library. Just Download it and use the following code
<?php
include("simple_html_dom.php");
$html = file_get_html('http://xooplate.com/templates/details/13693-creative-circular-skills-ui-infographic-psd');
foreach($html->find('a[class=btn_green nturl]') as $element)
echo $element->href . '<br>';
?>
First of all have a look at here,
www.zedge.net/txts/4519/
this page has so many text messages , I want my script to open each of the message and download it,
but i am having some problem,
This is my simple script to open the page,
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.zedge.net/txts/4519");
$contents = curl_exec ($ch);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_close ($ch);
?>
The page download fine but how would i open every text message page inside this page one by one and save its content in a text file,
I know how to save the content of a webpage in a text file using curl but in this case there are so many different pages inside the page i've downloaded how to open them one by one seperately ?
I've this idea but don't know if it will work,
Downlaod this page,
www.zedge.net/txts/4519
look for the all the links of text messages page inside the page and save each link into one text file (one in each line), then run another curl session , open the text file read each link one by one , open it copy the content from the particular DIV and then save it in a new file.
The algorithm is pretty straight forward:
download www.zedge.net/txts/4519 with curl
parse it with DOM (or alternative) for links
either store them all into text file/database or process them on the fly with "subrequest"
// Load main page
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, "http://www.zedge.net/txts/4519");
$contents = curl_exec ($ch);
$dom = new DOMDocument();
$dom->loadHTML( $contents);
// Filter all the links
$xPath = new DOMXPath( $dom);
$items = $xPath->query( '//a[class=myLink]');
foreach( $items as $link){
$url = $link->getAttribute('href');
if( strncmp( $url, 'http', 4) != 0){
// Prepend http:// or something
}
// Open sub request
curl_setopt($ch, CURLOPT_URL, "http://www.zedge.net/txts/4519");
$subContent = curl_exec( $ch);
}
See documentation and examples for xPath::query, note that DOMNodeList implements Traversable and therefor you can use foreach.
Tips:
Use curl opt COOKIE_JAR_FILE
Use sleep(...) not to flood server
Set php time and memory limit
I used DOM for my code part. I called my desire page and filtered data using getElementsByTagName('td')
Here i want the status of my relays from the device page. every time i want updated status of relays. for that i used below code.
$keywords = array();
$domain = array('http://USERNAME:PASSWORD#URL/index.htm');
$doc = new DOMDocument;
$doc->preserveWhiteSpace = FALSE;
foreach ($domain as $key => $value) {
#$doc->loadHTMLFile($value);
//$anchor_tags = $doc->getElementsByTagName('table');
//$anchor_tags = $doc->getElementsByTagName('tr');
$anchor_tags = $doc->getElementsByTagName('td');
foreach ($anchor_tags as $tag) {
$keywords[] = strtolower($tag->nodeValue);
//echo $keywords[0];
}
}
Then i get my desired relay name and status in $keywords[] array.
Here i am sharing of Output.
If you want to read all messages in the main page. then first you have to collect all link for separate messages. Then you can use it for further same process.
I'm starting to help a friend who runs a website with small bits of coding work, and all the code required will be PHP. I am a C# developer, so this will be a new direction.
My first stand-alone task is as follows:
The website is informed of a new species of fish. The scientific name is entered into, say, two input controls, one for the genus (X) and another for the species (Y). These names will need to be sent to a website in the format:
http://www.fishbase.org/Summary/speciesSummary.php?genusname=X&speciesname=Y&lang=English
Once on the resulting page, there are further links for common names and synonyms.
What I would like to be able to do is to find these links, and call the URL (as this will contain all the necessary parameters to get the particular data) and store some of it.
I want to save data from both calls and, once completed, convert it all into xml which can then be uploaded to the website's database.
All I'd like to know is (a) can this be done, and (b) how difficult is it?
Thanks in advance
Martin
If I understand you correctly you want your script to download a page and process the downloaded data. If so, the answers are:
a) yes
b) not difficult
:)
Oke... here some more information: I would use the CURL extension, see:
http://php.net/manual/en/book.curl.php
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "example.com");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
?>
I used a thing called snoopy (http://sourceforge.net/projects/snoopy/) 4 years a go.
I took about 500 customers profiles from a website that published them in a few hours.
a) Yes
b) Not difficult when have experience.
Google for CURL first, or allow_url_fopen.
file_get_contents() will do the job:
$data = file_get_contents('http://www.fishbase.org/Summary/speciesSummary.php?genusname=X&speciesname=Y&lang=English');
// Отправить URL-адрес
function send_url($url, $type = false, $debug = false) { // $type = 'json' or 'xml'
$result = '';
if (function_exists('curl_init')) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($ch);
curl_close($ch);
} else {
if (($content = #file_get_contents($url)) !== false) $result = $content;
}
if ($type == 'json') {
$result = json_decode($result, true);
} elseif ($type == 'xml') {
if (($xml = #simplexml_load_file($result)) !== false) $result = $xml;
}
if ($debug) echo '<pre>' . print_r($result, true) . '</pre>';
return $result;
}
$data = send_url('http://ip-api.com/json/212.76.17.140', 'json', true);