Nested loop with recursive function? - php

I need to do a recursive loop on every result suggested by google up to a user-defined depth and save results in a multidimensional array, explored later on.
I want to get this result.
google
google app
google app store
google app store games
google app store games free
google maps
google maps directions
google maps directions driving
google maps directions driving canada
...
Currently, my recursive function returns replicated results from the second nesting.
google
google app
google app
google app store
google app store
google app
google app store
google app store
google app store
...
I think the problem comes from the array (parent results) that I pass as an argument to my function recursive_function() to each nested loops.
$child = recursive_function($parent[0][1], $depth, $inc+1);
Recursive function
// keywords at line or spaced
$keywords = explode("\n", trim("facebook"));
$result = recursive_function($keywords, 2);
function recursive_function($query, $depth, $inc = 1)
{
$urls = preg_filter('/^/', 'http://suggestqueries.google.com/complete/search?client=firefox&q=', array_map('urlencode', $query));
$parent = curl_multi_function($urls);
array_multisort($parent[0][1]);
if (count($parent[0][1]) === 0 || $inc >= $depth)
{
$out[] = $parent[0][1];
}
else
{
$child = recursive_function($parent[0][1], $depth, $inc+1);
$out[] = $child;
}
return $out;
}
Function curl
function curl_multi_function($data, $options = array())
{
// array of curl handles
$curly = array();
// data to be returned
$result = array();
// multi handle
$mh = curl_multi_init();
// loop through $data and create curl handles
// then add them to the multi-handle
foreach ($data as $id => $d)
{
$curly[$id] = curl_init();
$url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
curl_setopt($curly[$id], CURLOPT_URL, $url);
curl_setopt($curly[$id], CURLOPT_HEADER, 0);
curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curly[$id], CURLOPT_SSL_VERIFYPEER, 0);
// post?
if (is_array($d))
{
if (!empty($d['post']))
{
curl_setopt($curly[$id], CURLOPT_POST, 1);
curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']);
}
}
// extra options?
if (!empty($options)) {
curl_setopt_array($curly[$id], $options);
}
curl_multi_add_handle($mh, $curly[$id]);
}
// execute the handles
$running = null;
do
{
curl_multi_exec($mh, $running);
}
while($running > 0);
// get content and remove handles
foreach($curly as $id => $c)
{
$result[$id] = curl_multi_getcontent($c);
// decode json result
$result[$id] = json_decode(utf8_encode($result[$id]));
curl_multi_remove_handle($mh, $c);
}
// all done
curl_multi_close($mh);
return $result;
}
Thank's

I've changed your recursive_function a little bit:
function recursive_function($query, $depth, $inc = 1)
{
$urls = preg_filter('/^/', 'http://suggestqueries.google.com/complete/search?client=firefox&q=', array_map('urlencode', $query));
$parent = curl_multi_function($urls);
foreach ($parent as $key => $value) {
array_multisort($value[1]);
$words = explode(' ', $value[0]);
$lastWord = end($words);
if (count($value[1]) === 0 || $inc >= $depth) {
$out[$lastWord] = [];
} else {
unset($value[1][0]);
$child = recursive_function($value[1], $depth, $inc+1);
$out[$lastWord] = $child;
}
}
return $out;
}
It generates an array like this:
[
google =>
[
app =>
[
store =>
[
games =>
[
free => []
]
]
]
...
]
]
Is that what you want?

Related

curl not downloading all pdf files PHP

im trying to download multiple pdfs with php. i get an array of urls and each url redirects to a website that contains a pdf file if something is wrong with that url it just redirects to a html page, so i've been googling and found this to download all pdfs to the server:
public function download ($data, $simultaneous = 1, $save_to)
{
$loops = array_chunk($data, $simultaneous, true);
foreach ($loops as $key => $value)
{
foreach ($value as $urlkey => $urlvalue)
{
$ch[$urlkey] = curl_init($urlvalue["url"]);
curl_setopt($ch[$urlkey], CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch[$urlkey], CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch[$urlkey], CURLOPT_SSL_VERIFYHOST, false);
}
$mh = curl_multi_init();
foreach ($value as $urlkey => $urlvalue)
{
curl_multi_add_handle($mh, $ch[$urlkey]);
}
$running = null;
do {
curl_multi_exec($mh, $running);
} while ($running);
foreach ($value as $urlkey => $urlvalue)
{
$response = curl_multi_getcontent($ch[$urlkey]);
file_put_contents($save_to.$urlvalue["saveas"], $response);
curl_multi_remove_handle($mh,$ch[$urlkey]);
curl_close($ch[$urlkey]);
}
}
}
for some reason this downloads only some of the files
anyone has any idea why this is not working?
any help would be appreciated

telegram bot split keyboard rows[columns]

I am writing my telegram bot codes in PHP. I would like to split my inline keyboard into 2 or 3 columns. Here is my code:
foreach ($categories as $cat) {
$key[] = array(
array('text'=>$cat['name'],'callback_data'=>'sub-'.$cat['id'])
);
if ($k % 2 == 0) {
$keyoptions[] = $key;
$key = array();
}
$k++;
}
$telegram->SendMessage($userid, $keyoptions);
but my code doesn't work. Where is the problem and how can I solve my issue?
EDIT :
i just used this code
$keyoptions = array_chunk($keyoptions,3);
but still can't find the problem;
Telegram API: inline_keyboard: Array of Array of InlineKeyboardButton
Example:
keyboard: [
["uno :+1:"],["uno \ud83d\udc4d", "due"],["uno", "due","tre"]
]
I don't know what library you are using and what are those fields in your code, but this a working with native telegram API:
function inlineKeyboard($text, $chatID, $btnNames, $callBackDatas)
{
$inlineRow = array(); // this is array for each row of buttons
$i = 0;
foreach ($btnNames as $name) {
array_push($inlineRow, array("text" => $name, "callback_data" => $callBackDatas[$i]));
$i++;
}
/* if you need multiple rows then just create other inlineRow arrays
and push to this array below */
$inlineKeyboard = array($inlineRow);
$keyboard = array(
"inline_keyboard" => $inlineKeyboard
);
$postfields = array
(
'chat_id' => "$chatID",
'text' => $text,
'reply_markup' => json_encode($keyboard)
);
send('sendMessage', $postfields);
}
define('BaseURL', 'https://api.telegram.org/bot<TOKEN>');
function send($method, $datas)
{
$url = BaseURL . "/" . $method;
if (!$curld = curl_init()) {
exit;
}
curl_setopt($curld, CURLOPT_POST, true);
curl_setopt($curld, CURLOPT_POSTFIELDS, $datas);
curl_setopt($curld, CURLOPT_URL, $url);
curl_setopt($curld, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($curld);
curl_close($curld);
return $output;
}

Can't execute HTTP request to Youtube Data API from PHP

I'm writing a mass youtube link finder, which imports a list of titles from an array, generates an API url and the executes them with curl_multi.
However, curl returns blank data for each link. Links are fine as I can access to them correctly via Chrome.
file_get_contents() tried in another script with a url amongst those returns an ERR_EMPTY_RESPONSE in Chrome.
Any help would be much appreciated,
EDIT: Code:
function getYTUrl($urls){
$curls = array();
$result = array();
$arrjson = array();
$mh = curl_multi_init();
foreach ($urls as $key => $value) {
echo $value;
$curls[$key] = curl_init();
curl_setopt($curls[$key], CURLOPT_URL, $value);
curl_setopt($curls[$key], CURLOPT_HEADER, 0);
curl_setopt($curls[$key], CURLOPT_RETURNTRANSFER, true);
curl_setopt($curls[$key],CURLOPT_SSL_VERIFYPEER,false);
curl_multi_add_handle($mh, $curls[$key]);
}
$active = null;
do{
$mrc = curl_multi_exec($mh, $active);
}
while ($active);
foreach ($urls as $key => $value) {
$result[$key] = curl_multi_getcontent($curls[$value]);
curl_multi_remove_handle($mh, $value);
}
curl_multi_close($mh);
}

Multithreading PHP Function

Currently when I execute this function with say 60 URL's I get a HTTP 504 error. Is there anyway to multithread this so that I no longer get a 504 error and iterate throughout the entire list of URL's?
<?php
namespace App\Http\Controllers;
use Request;
use App\Http\Controllers\Controller;
class MainController extends Controller
{
public function parse()
{
$input = Request::all();
$csv = $input['laraCsv'];
$new_csv = trim(preg_replace('/\s\s+/', ',', $csv));
$headerInfo = [];
//$titles = [];
$csvArray = str_getcsv($new_csv, ",");
$csvLength = count($csvArray);
$i = 0;
while ($i < $csvLength) {
if(strpos($csvArray[$i], '.pdf') !== false) {
print_r($csvArray[$i]);
}
else{
array_push($headerInfo, get_headers($csvArray[$i], 1));
}
//sleep(3);
//echo file_get_contents($csvArray[$i]);
$i++;
}
return view('csvViewer')->with('data', $headerInfo)->with('urls', $csvArray);
}
}
I've used digitalocean in the past before but I'm not sure what error codes they give if you run out of time, (also set_time_limit(0); should already be in your code).
See if this works:
<?php
function getHeaders($data) {
$curly = array();
$result = array();
$mh = curl_multi_init();
foreach ($data as $id => $url) {
$curly[$id] = curl_init();
curl_setopt($curly[$id], CURLOPT_URL, $url);
curl_setopt($curly[$id], CURLOPT_HEADER, true);
curl_setopt($curly[$id], CURLOPT_NOBODY, true);
curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, true);
curl_multi_add_handle($mh, $curly[$id]);
}
$running = null;
do {
curl_multi_exec($mh, $running);
} while ($running > 0);
foreach($curly as $id => $c) {
$result[$id] = array_filter(explode("\n", curl_multi_getcontent($c)));
curl_multi_remove_handle($mh, $c);
}
curl_multi_close($mh);
return $result;
}
$urls = array(
'http://google.com',
'http://yahoo.com',
'http://doesnotexistwillitplease.com'
);
$r = getHeaders($urls);
echo '<pre>';
print_r($r);
So once you've gotten all your URLs into an array, run it like getHeaders($urls);.
If it doesn't work try it only with 3 or 4 urls first. Also set_time_limit(0); at the top as mentioned before.
Are you sure it is because of your code ? it could also be the server configuration.
about HTTP 504
This problem is entirely due to slow IP communication between back-end
computers, possibly including the Web server. Only the people who set
up the network at the site which hosts the Web server can fix this
problem.

Simultaneous HTTP requests in PHP with cURL

I'm trying to take a rather large list of domains query the rank of each using the compete.com API as seen here -> https://www.compete.com/developer/documentation
The script I wrote takes a database of domains I populated and initiates a cURL request to compete for the rank of the website. I quickly realized that this was very slow because each request was being sent one at a time. I did some searching and came across this post-> http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/ which explains how to perform simultaneous HTTP requests in PHP with cURL.
Unfortunately that script will take an array of 25,000 domains and try to process them all at once. I found that batches of 1,000 work quite well.
Any idea how to send 1,000 queries to compete.com then wait for completion and send the next 1,000 until the array is empty? Here's what I'm workin with thus far:
<?php
//includes
include('includes/mysql.php');
include('includes/config.php');
//get domains
$result = mysql_query("SELECT * FROM $tableName");
while($row = mysql_fetch_array($result)) {
$competeRequests[] = "http://apps.compete.com/sites/" . $row['Domain'] . "/trended/rank/?apikey=xxx&start_date=201207&end_date=201208&jsonp=";
}
//first batch
$curlRequest = multiRequest($competeRequests);
$j = 0;
foreach ($curlRequest as $json){
$j++;
$json_output = json_decode($json, TRUE);
$rank = $json_output[data][trends][rank][0][value];
if($rank) {
//Create mysql query
$query = "Update $tableName SET Rank = '$rank' WHERE ID = '$j'";
//Execute the query
mysql_query($query);
echo $query . "<br/>";
}
}
function multiRequest($data) {
// array of curl handles
$curly = array();
// data to be returned
$result = array();
// multi handle
$mh = curl_multi_init();
// loop through $data and create curl handles
// then add them to the multi-handle
foreach ($data as $id => $d) {
$curly[$id] = curl_init();
$url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
curl_setopt($curly[$id], CURLOPT_URL, $url);
curl_setopt($curly[$id], CURLOPT_HEADER, 0);
curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);
// post?
if (is_array($d)) {
if (!empty($d['post'])) {
curl_setopt($curly[$id], CURLOPT_POST, 1);
curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']);
}
}
curl_multi_add_handle($mh, $curly[$id]);
}
// execute the handles
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);
// get content and remove handles
foreach($curly as $id => $c) {
$result[$id] = curl_multi_getcontent($c);
curl_multi_remove_handle($mh, $c);
}
// all done
curl_multi_close($mh);
return $result;
}
?>
Instead of
//first batch
$curlRequest = multiRequest($competeRequests);
$j = 0;
foreach ($curlRequest as $json){
You can do:
$curlRequest = array();
foreach (array_chunk($competeRequests, 1000) as $requests) {
$results = multiRequest($requests);
$curlRequest = array_merge($curlRequest, $results);
}
$j = 0;
foreach ($curlRequest as $json){
$j++;
// ...
This will split the large array into chunks of 1,000 and pass those 1,000 values to your multiRequest function which uses cURL to execute those requets.
https://github.com/webdevelopers-eu/ShadowHostCloak
This does exactly what you want. Just pass empty argument to new Proxy() to bypass proxy and make direct requests.
You can stuff 1000 requests in it and call $proxy->execWait() and it will process all requests simultaneously and exit that method when everything is done... Then you can repeat.

Categories