I have a txt file with 5000 lines with URL's. WHat i'm trying to do is to open every url to extract every url (that first url have).
My problem is, the first line the script opens the URL and tell me how many links i have with no problem. But for the rest of the URL's in the file isnt showing anything...the array show something like this:
Array
(
)
Array
(
)
My code:
$homepage = file_get_contents('***mytxt file****');
$pathComponents = explode(",", trim($homepage)); //line breaker
//echo "<pre>";print_r($pathComponents);echo "</pre>";
$count_nlines = count($pathComponents);
for ($i=0;$i<3;$i++) {
$request_url = $pathComponents[$i];
//echo $request_url . "<br>";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $request_url); // The url to get links from
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // We want to get the respone
$result = curl_exec($ch);
$regex='|<a.*?href="(.*?)"|';
preg_match_all($regex,$result,$parts);
$links=$parts[1];
echo "<pre>";print_r($links);echo "</pre>";
curl_close($ch);
}
Any ideas?!
It looks like you're looping through the wrong thing. Try changing this:
for ($i=0;$i<3;$i++) {
To this:
for ($i = 0; $i <= count($pathComponents); $i++)
Related
I am currently working on Roblox API. I am stuck on one question. Reason :
I have this link https://avatar.roblox.com/v1/users/2/currently-wearing.
This shows what specified users have equipped on them. this link right here shows this:
{"assetIds":[382537569,607702162,607785314]}
My goal is to get the assetIds to string.
I tried this:
<?php
$id = 2;
$ch = file_get_contents("https://avatar.roblox.com/v1/users/$id/currently-wearing");
$ch = curl_init($ch);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($ch);
$data = json_decode($data);
$id = "$data->assetIds";
echo $id;
But it Shows Array. I need some help.
The cURL is not necessary as the file_get_contents() get you the data you appear to require
$id = 2;
$ch = file_get_contents("https://avatar.roblox.com/v1/users/2/currently-wearing");
$data = json_decode($ch);
foreach ($data->assetIds as $id){
echo $id . PHP_EOL;
}
RESULT
382537569
607702162
607785314
As mentioned earlier , i am working on the local server as of now in xampp. I have created 2 files index.php and test.php. What i want to achieve is that , index.php will send json data to test.php , with the json data received , the test.php is able to use that json data to turn the statistics into graph.
I am working with the first step , but however , nothing seems to display on my test.php when i tried to do a var_dump($data) , and what i get is NULL. Tried alot of solutions online but none to seems to fix it. I am relatively new to this , so really thanks and appreciate of your help. First php is index.php , second php is test.php.
Do i require a live server as of now in order to see the results in test.php or local server unable to display the result?
<?php
$array = array();
$product = array();
$product[0]['id_product'] = 'A01';
$product[0]['name_product'] = 'Sandal';
$product[0]['price_product'] = '500';
$product[1]['id_product'] = 'A02';
$product[1]['name_product'] = 'Shoes';
$product[1]['price_product'] = '2500';
$array['id'] = '123';
$array['note'] = 'this is my short example';
$array['data'] = $product;
$data = json_encode($array);
$ch = curl_init('http://localhost:8080/practice3/test.php');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, count($data));
$result = curl_exec($ch);
var_dump($result);
?>
```````````````````````````````````
<?php
$fp = fopen('php://input', 'r');
$raw = stream_get_contents($fp);
$data = json_decode($raw,true);
echo "hello";
echo $data['id'];
echo $data['note'];
foreach ($data['data'] as $key) {
echo 'id_product : '.$key['id_product'].'<br/>';
echo 'name_product : '.$key['name_product'].'<br/>';
echo 'price_product : '.$key['price_product'].'<br/>';
}
?>
````````````````````````````````````
I'm trying to get the redirect url from a stream using php.
Here's the code I have right now:
<?php
$stream = 'https://api.soundcloud.com/tracks/178525956/stream?client_id=XXXXXX';
$ch = curl_init($stream);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$url = curl_exec($ch);
echo $url;
curl_close($ch);
?>
Which outputs as a string:
{"status":"302 - Found","location":"THE_URL_I_WANT"}
So how ould I go about getting the url I want as a variable?
Thanks
It's simple use json_decode
$data = json_decode($url);
$your_url = $data->location;
How about
$data = json_decode($url);
$location = $data["location"];
I finally got my scraper working (somewhat), but now I'd like to know how I can automatically go to the next page and scrape the same info from there. I'm using cURL to copy the entire page (otherwise I get a 500 error). Here's my code:
<?
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "http://example.com/results.asp?&j=t&page_no=1");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$html = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
// print $html . "\n";
require 'simple_html_dom.php';
$dom = new simple_html_dom();
$dom->load($html);
foreach($dom->find("div[#id='schoolsearch'] tr") as $data){
$tds = $data->find("td");
if(count($tds)==3){
$record = array(
'school' => $tds[1]->plaintext,
'city' => $tds[2]->plaintext
);
print json_encode($record) . "\n";
file_put_contents('schools.csv', json_encode($record) . "\n", FILE_APPEND);
}
}
?>
It's not perfect, but it's what works right now! Anyone know how I can move to the next pages?
Wrap it in a loop:
$maxPages = 10;
for ($i = 1; $i <= $maxPages; $i++) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://example.com/results.asp?&j=t&page_no=$i");
etc...
}
You'll need to tidy up a bit more,to avoiding including that file on every page, but you get the idea.
I want to search number of links or URL on http://public-domain-content.com
and store them in an array and then just randomly select any one from array and just display or echo
How can i do that in php
If I understood what you're asking, you can achieve this using file_get_contents();
After using file_get_contents($url), which gives you a string, you can loop through the result string searching for spaces to tell the words apart. Count the number of words, and store the words in an array accordingly. Then just choose a random element from the array using array_rand()
However, sometimes there are security problems with file_get_contents().
You can override this using the following function:
function get_url_contents($url)
{
$crl = curl_init();
$timeout = 5;
curl_setopt ($crl, CURLOPT_URL,$url);
curl_setopt ($crl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($crl, CURLOPT_CONNECTTIMEOUT, $timeout);
$ret = curl_exec($crl);
curl_close($crl);
return $ret;
}
http://php.net/manual/en/function.curl-setopt.php <--- Explanation about curl
Example code:
$url = "http://www.xxxxx.xxx"; //Set the website you want to get content from
$str = file_get_contents($url); //Get the contents of the website
$built_str = ""; //This string will hold the valid URLs
$strarr = explode(" ", $str); //Explode string into array(every space a new element)
for ($i = 0; $i < count($strarr); $i++) //Start looping through the array
{
$current = #parse_url($strarr[$i]) //Attempt to parse the current element of the array
if ($current) //If parse_url() returned true(URL is valid)
{
$built_str .= $current . " "; //Add the valid URL to the new string with " "
}
else
{
//URL invalid. Do something here
}
}
$built_arr = explode(" ", $built_str) //Same as we did with $str_arr. This is why we added a space to $built_str every time the URL was valid. So we could use it now to split the string into an array
echo $built_arr[array_rand($built_arr)]; // Display a random element from our built array
There is also a more extended version to checking URLs, which you can explore here:
http://forums.digitalpoint.com/showthread.php?t=326016
Good luck.