I have a free script and I would like to ask if it's possible to replace or automate the search function. For example every hour. Right now I have to press the search button to find new proxies but I want to search automatically and update them in my database, maybe using a cron job.
if(isset($_POST['search'])) { // hit search button
$script_start = $pb->microtime_float();
ob_flush();
flush();
$proxylisttype = $pb->returnProxyList($_REQUEST['listtype']); // make sure request vars are clean
$sitestoscour = $pb->returnSitesScour($_REQUEST); // make sure request vars are clean
$finallist = $pb->returnFinalList($sitestoscour);
$finallist = $pb->arrayUnique($finallist); // eliminate the dupes before moving on
if(AUTO_BAN == 1) { // remove banned proxies
$finallist = $pb->autoBan($finallist);
}
$script_end = $pb->microtime_float(); // stop the timer
}
You can either do it with curl from a php script or command line (or wget). That way you can set the $_POST:
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL, "http://yoururl.com'");
curl_setopt($ch,CURLOPT_POST, true);
curl_setopt($ch,CURLOPT_POSTFIELDS, "search=your_query");
$result = curl_exec($ch);
curl_close($ch);
Then make that script run every hour by setting up a cron job.
You could also do it with wget:
wget --post-date="search=query" http://yoururl.com
Related
So i made this little code here:
<?php
$steamData = file_get_contents("http://steamcommunity.com/profiles/76561198258195397/inventory/json/730/2");
$data = json_decode($steamData, true);
$items = $data['rgDescriptions'];
foreach($items as $key => $item){
$market_hash_name = $item["market_hash_name"];
$market_hash_name = htmlentities($market_hash_name, ENT_QUOTES, 'UTF-8');
$sql = "SELECT price FROM itemprice WHERE market_hash_name='".$market_hash_name."'";
$result = $conn->query($sql);
$itemprice = "-";
if ($result->num_rows > 0)
{
while($row = $result->fetch_assoc())
{
if($row['price']>0.02){
$itemprice = $row['price'];
$itemprice = floatval($itemprice);
$itemprice = $itemprice*1.05;
}else{
$itemprice = 0;
}
}
}
echo '<div class="items"><center><img src="http://steamcommunity-a.akamaihd.net/economy/image/' . $item["icon_url"] . '" height="192" width="256"></center><div><center><h5 style="color:#'.$item["name_color"].'">'.$item["market_name"].' - <strong><b><a style="color:green">$'.$itemprice.'</a></b></strong></h5></center></div></div>';
}
?>
How can i make this script run with ajax until it gives some data back?Because its like 1/10 times working.
How can i make this script run with ajax until it gives some data
back?
I think you're asking the wrong question here. When you make an ajax request to your server and your server fails to honor it correctly it should be assumed that your server made the mistake and that you caused the problem*. However when Steam fails to provide information, your server should attempt to hold Steam's server accountable rather than immediately propagate the problem to your client's code.
Ultimately, the problem lies here:
$steamData = file_get_contents("http://steamcommunity.com/profiles/76561198258195397/inventory/json/730/2");
You don't make any checks to ensure that the data you've gotten from Steam is the correct data - or even that this network call succeeded at all! For this you need a non-trival HTTP request in order to check the status, which means you have to get rid of file_get_contents() and use the curl functions instead to make sure that the request succeeded as you expect:
<?php
do {
//If this isn't the first time in the loop, pause for a second to prevent spamming their servers
if(isset($statusCode)){
sleep(1);
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://steamcommunity.com/profiles/76561198258195397/inventory/json/730/2");
//The curl request should return the body of the text
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
//Apparently the URL provided initially returns with a 3XX response, this option follows the redirection.
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$steamData = curl_exec($ch);
$statusCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
//Repeat this command until we get a successful request
//Note that this is slightly naive - Any 2XX should be deemed successful.
} while ($statusCode !== 200);
$data = json_decode($steamData, true);
//...
Doing this should greatly improve those 1/10 odds of success, but it doesn't guarantee success as relying on network calls is inherently out of your control. If it were to happen that Steam shut down, repeatedly sending ajax requests to your server wouldn't accomplish anything and is actually harmful for the client*. In these situations your server should respond with an HTTP 500 error (that is, indicating the the server messed up) and display an error message to your user that the request can't be completed and not attempt to rerequest the data.
*I should also mention that sending multiple ajax requests means that the client is utilizing their bandwidth cap to correct for an issue that occurred on the server, which is unideal.
I am looking to collect the titles of all of the posts on a subreddit, and I wanted to know what would be the best way of going about this?
I've looked around and found some stuff talking about Python and bots. I've also had a brief look at the API and am unsure in which direction to go.
As I do not want to commit to find out 90% of the way through it won't work, I ask if someone could point me in the right direction of language and extras like any software needed for example pip for Python.
My own experience is in web languages such as PHP so I initially thought of a web app would do the trick but am unsure if this would be the best way and how to go about it.
So as my question stands
What would be the best way to collect the titles (in bulk) of a
subreddit?
Or if that is too subjective
How do I retrieve and store all the post titles of a subreddit?
Preferably needs to :
do more than 1 page of (25) results
save to a .txt file
Thanks in advance.
PHP; in 25 lines:
$subreddit = 'pokemon';
$max_pages = 10;
// Set variables with default data
$page = 0;
$after = '';
$titles = '';
do {
$url = 'http://www.reddit.com/r/' . $subreddit . '/new.json?limit=25&after=' . $after;
// Set URL you want to fetch
$ch = curl_init($url);
// Set curl option of of header to false (don't need them)
curl_setopt($ch, CURLOPT_HEADER, 0);
// Set curl option of nobody to false as we need the body
curl_setopt($ch, CURLOPT_NOBODY, 0);
// Set curl timeout of 5 seconds
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
// Set curl to return output as string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// Execute curl
$output = curl_exec($ch);
// Get HTTP code of request
$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
// Close curl
curl_close($ch);
// If http code is 200 (success)
if ($status == 200) {
// Decode JSON into PHP object
$json = json_decode($output);
// Set after for next curl iteration (reddit's pagination)
$after = $json->data->after;
// Loop though each post and output title
foreach ($json->data->children as $k => $v) {
$titles .= $v->data->title . "\n";
}
}
// Increment page number
$page++;
// Loop though whilst current page number is less than maximum pages
} while ($page < $max_pages);
// Save titles to text file
file_put_contents(dirname(__FILE__) . '/' . $subreddit . '.txt', $titles);
I have a website, that uses WP Super Cache plugin. I need to recycle cache once a day and then I need to call 5 posts (URL adresses) so WP Super Cache put these posts into cache again (caching is quite time consuming so I'd like to have it precached before users come so they dont have to wait).
On my hosting I can use a CRON but only for 1 call/hour. And I need to call 5 different URL's at once.
Is it possible to do that? Maybe create one HTML page with these 5 posts in iframe? Will something like that work?
Edit: Shell is not available, so I have to use PHP scripting.
The easiest way to do it in PHP is to use file_get_contents() (fopen() also works), if the HTTP stream wrapper is enabled on your server:
<?php
$postUrls = array(
'http://my.site.here/post1',
'http://my.site.here/post2',
'http://my.site.here/post3',
'http://my.site.here/post4',
'http://my.site.here/post5',
);
foreach ($postUrls as $url) {
// Get the post as an user will do it
$text = file_get_contents();
// Here you can check if the request was successful
// For example, use strpos() or regex to find a piece of text you expect
// to find in the post
// Replace 'copyright bla, bla, bla' with a piece of text you display
// in the footer of your site
if (strpos($text, 'copyright bla, bla, bla') === FALSE) {
echo('Retrieval of '.$url." failed.\n");
}
}
If file_get_contents() fails to open the URLs on your server (some ISP restrict this behaviour) you can try to use curl:
function curl_get_contents($url)
{
$ch = curl_init($url);
curl_setopt_array($ch, array(
CURLOPT_CONNECTTIMEOUT => 30, // timeout in seconds
CURLOPT_RETURNTRANSFER => TRUE, // tell curl to return the page content instead of just TRUE/FALSE
));
$text = curl_exec($ch);
curl_close($ch);
return $text;
}
Then use the function curl_get_contents() listed above instead of file_get_contents().
An example using PHP without building a cURL request.
Using PHP's shell exec, you can have an extremely light function like so :
$siteList = array("http://url1", "http://url2", "http://url3", "http://url4", "http://url5");
foreach ($siteList as &$site) {
$request = shell_exec('wget '.$site);
}
Now of course this is not the most concise answer and not always a good solution also, if you actually want anything from the response you will have to work with it a different way to cURLbut its a low impact option.
Thanks to Arkascha tip I created a PHP page that I call from CRON. This page contains simple function using cURL:
function cache_it($Url){
if (!function_exists('curl_init')){
die('No cURL, sorry!');
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $Url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 50); //higher timeout needed for cache to load
curl_exec($ch); //dont need it as output, otherwise $output = curl_exec($ch);
curl_close($ch);
}
cache_it('http://www.mywebsite.com/url1');
cache_it('http://www.mywebsite.com/url2');
cache_it('http://www.mywebsite.com/url3');
cache_it('http://www.mywebsite.com/url4');
I want to process a script every minute using a cron on my server but I need to pass a variable in the URL or some other way. I have researched this and I've seen solutions using arguments in the cron but I don't think that works with what I'm doing.
Here is what I am trying to do:
script.php (runs every minute)
<?php
$marker = $_GET['marker'];
$accountObj = new etAccounts($consumer);
$request_params = new TransactionHistoryRequest();
$request_params->__set('count', 50); //how many will be shown
if($marker_get != ''){
$request_params->__set('marker', $marker_get); //starting point ex. 14293200140265
}
$json = $accountObj->GetTransactionHistory($account, $request_obj, $request_params );
echo $json; //shows most recent 50 transactions starting from marker value
//process json data here...
//included in json is a marker variable that will be used to return the next 50 json results
//after data is processed reload the page with marker in URL
header('Location: script.php?marker=14293200140265');
?>
I understand that cron is CLI on the server side and that it can't process redirections or header locations but how is this possible. I saw someone mention using CURL, how might this work? Example?
Simple example to send post variables to an url:
$fields = array(
'id' => $id,
'mail' => $mail,
);
$url = "yourdomain.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
Based on the comments, you don't need to redirect or use query variables.
You could use a loop that runs while your marker variable is not empty:
$your_marker_variable = '';
do {
$accountObj = new etAccounts($consumer);
$request_params = new TransactionHistoryRequest();
$request_params->__set('count', 50); //how many will be shown
if($marker_get != '') {
$request_params->__set('marker', $marker_get); //starting point ex. 14293200140265
}
$json = $accountObj->GetTransactionHistory($account, $request_obj, $request_params );
echo $json; //shows most recent 50 transactions starting from marker value
//process json data here...
//included in json is a marker variable that will be used to return the next 50 json results
// set the new value of $your_marker_variable
$your_marker_variable = ...
} while (!empty($your_marker_variable));
Note that the undefined and unused variables in the script you posted, make it hard to see what variable is used for what, so you would need to adapt this a bit.
I have an array containing the contents of a MySQL table. I need to put each of these contents into curl_multi_handles so that I can execute them all simultaneously
Here is the code for the array, in case it helps:
$SQL = mysql_query("SELECT url FROM urls") or die(mysql_error());
while($resultSet = mysql_fetch_array($SQL)){
$urls[]=$resultSet
}
So I need to put be able to send data to each url at the same time. I don't need to get any data back, and in fact I'll be having them time out after two seconds. It only needs to send the data and then close.
My code prior to this, was executing them one at a time. here is that code:
$SQL = mysql_query("SELECT url FROM shells") or die(mysql_error()); while($resultSet = mysql_fetch_array($SQL)){
$ch = curl_init($resultSet['url'] . $fullcurl); //load the urls and send GET data
curl_setopt($ch, CURLOPT_TIMEOUT, 2); //Only load it for two seconds (Long enough to send the data)
curl_exec($ch);
curl_close($ch);
So my question is: How can I load the contents of the array into curl_multi_handle, execute it, and then remove each handle and close the curl_multi_handle?
You still call curl_init and curl_setopt. Then you load it into a multi_handle, and keep calling execute until it's done. This is based on the documentation at curl_multi_init. Since you're timing out in two seconds, and not processing responses, I think you can just sleep for two seconds at a time. curl_multi_select might be better if you actually need to process the responses.
$SQL = mysql_query("SELECT url FROM shells") ;
$mh = curl_multi_init();
$handles = array();
while($resultSet = mysql_fetch_array($SQL)){
//load the urls and send GET data
$ch = curl_init($resultSet['url'] . $fullcurl);
//Only load it for two seconds (Long enough to send the data)
curl_setopt($ch, CURLOPT_TIMEOUT, 2);
curl_multi_add_handle($mh, $ch);
$handles[] = $ch;
}
// Create a status variable so we know when exec is done.
$running = null;
//execute the handles
do {
// Call exec. This call is non-blocking, meaning it works in the background.
curl_multi_exec($mh,$running);
// Sleep while it's executing. You could do other work here, if you have any.
sleep(2);
// Keep going until it's done.
} while ($running > 0);
// For loop to remove (close) the regular handles.
foreach($handles as $ch)
{
// Remove the current array handle.
curl_multi_remove_handle($mh, $ch);
}
// Close the multi handle
curl_multi_close($mh);
If i were you, i would write class mysql and a class curl.
Its very good at all.
First i would create a method witch would return all urls from a passed mysql result.
Something like
public function getUrls($mysql_fetch_array)
{
foreach($mysql_fetch_array as $result)
{
$urls[] = $result["url"];
}
}
then you could write a method like curlSend($url,$param)
//remember you have to edit i dont know your full code so its just
// a way you could do it
public function curlSend($url,$param="")
{
$ch = curl_init($resultSet['url'] . $fullcurl); //load the urls and send GET data
curl_setopt($ch, CURLOPT_TIMEOUT, 2); //Only load it for two seconds (Long enough to send the data)
curl_exec($ch);
curl_close($ch);
}
public function send()
{
$urls = getUrls($this->mysql->result($sql));
foreach($urls as $url)
{
$this->curlSend($url);
}
}
Now this is how you could do it.