My website is pausing for a second when running my PHP code. What can I do to speed it up?
I can't put the PHP code at the bottom of the page and use CSS "position: absolute" to move it back to the top because of responsive web design issues with iPhones.
I don't really want to remove the 2 timeouts because if the 3rd party website (blockchain.info) goes offline the page won't load.
The code is for an advertisement to displayed after reading a bitcoin wallet balance. The code you see below is copied 7 times for each of the 7 ads on the page (ad1, ad2, ad3 etc.).
I know HTML and CSS but don't really know much about PHP or javascript/jQuery etc. (but I can copy and paste).
$ch = curl_init('http://whateverlink.com' . $address);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1) ;
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 1);
$ad1 = trim(curl_exec($ch)/100000000);
if ($ad1 > 0.001) {echo
'Display Ad1';
} elseif ($ad1 > -0.0001) {echo
'No ad yet';
}
Multi-curl can speed up your script.
You should have a look at the answer to this question: PHP Parallel curl requests
and the following PHP class providing an easy interface for running multiple concurrent CURL requests:
https://github.com/petewarden/ParallelCurl
Related
I have a pastebin scraper script, which is designed to find leaked emails and passwords, to make a website like HaveIBeenPwned.
Here is what my script is doing:
- Scraping Pastebin links from https://psbdmp.ws/dumps
- Getting a random proxy using this Random Proxy API (because Pastebin bans your IP if you hammer too many requests): https://api.getproxylist.com/proxy
- Doing a CURL request to the Pastebin links, then doing a preg_match_all to find all the email addresses and passwords in the format email:password.
The actual script seems to be working alright, but it isn't optimized enough, and is giving me a 524 timeout error after some time, which I suspect is because of all those CURL requests.Here is my code:
api.php
function comboScrape_CURL($url) {
// Get random proxy
$proxies->json = file_get_contents("https://api.getproxylist.com/proxy");
$proxies->decoded = json_decode($proxies->json);
$proxy = $proxies->decoded->ip.':'.$proxies->decoded->port;
list($ip,$port) = explode(':', $proxy);
// Crawl with proxy
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
comboScrape('email:pass',$curl_scraped_page);
}
index.php
require('api.php');
$expression = "/(?:https\:\/\/pastebin\.com\/\w+)/";
$extension = ['','1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16','17','18','19','20'];
foreach($extension as $pge_number) {
$dumps = file_get_contents("https://psbdmp.ws/dumps/".$pge_number);
preg_match_all($expression,$dumps,$urls);
$codes = str_replace('https://pastebin.com/','',$urls[0]);
foreach ($codes as $code) {
comboScrape_CURL("https://pastebin.com/raw/".$code);
}
}
524 timeout error - err, seems you're running php behind a web server (apache? nginx? lighthttpd? IIS?) don't do that, run your code from php-cli instead, php-cli can run indefinitely and never timeout.
because Pastebin bans your IP if you hammer too many requests - buy a pastebin.com pro account instead ( https://pastebin.com/pro ), it costs about $50 (or $20 around Christmas & Black Friday), and is a lifetime account with a 1-time payment, and gives you access to the scraping api ( https://pastebin.com/doc_scraping_api ), with the scraping api you can fetch about 1 paste per second, or 86400 pastes per day, without getting ip banned.
and because of pastebin.com's rate limits, there is no need to do this asynchronously with multiple connections (it's possible, but not worth the hassle. if you actually needed to do that however, you'd have to use the curl_multi API)
I would like to show what CoreTemp is monitoring on my personal website. The CoreTemp reports in what I can tell JSON format. However, it appears that when I do a cron or a GET requests, nothing ever shows from the CoreTemp page. I think this may have to do with it must return 5 results before displaying it on the local webpage at http://localhost:5200/.
Example output from CoreTemp Monitoring page:
{"CpuInfo":
{"uiLoad":[2,3],
"uiTjMax":[100],
"uiCoreCnt":2,
"uiCPUCnt":1,
"fTemp":[49,48],
"fVID":1.065918,
"fCPUSpeed":3292.04028,
"fFSBSpeed":99.7588,
"fMultiplier":33,
"CPUName":"Intel Core i5 4310M (Haswell) ",
"ucFahrenheit":0,
"ucDeltaToTjMax":0,
"ucTdpSupported":1,
"ucPowerSupported":1,
"uiStructVersion":2,
"uiTdp":[37],
"fPower":[5.889584],
"fMultipliers":[33,33]},
"MemoryInfo":{
"TotalPhys":16282,
"FreePhys":8473,
"TotalPage":17304,
"FreePage":8473,
"TotalVirtual":8388608,
"FreeVirtual":8388003,
"FreeExtendedVirtual":1,
"MemoryLoad":47}}
Again, I'm getting stuck even to have any data show up from a simple GET or curl request from php.
Simple php code:
<?php
$exe = curl_init();
curl_setopt($exe, CURLOPT_URL, "http://localhost:5200/");
curl_setopt($exe, CURLOPT_HEADER, 0);
curl_setopt($exe, CURLOPT_RETURNTRANSFER, 1);
$raw = curl_exec($exe);
curl_close($exe);
echo $raw;
What am I missing or is this a problem with the Monitoring plug in itself?
guys.
I'm with serious trouble trying to solve this.
The scenario:
Here at work we use the Vulnerability Management tool QualysGuard.
Skipping all technical details, this tool basically detects vulnerabilities in all servers and for each vulnerability in each server it creates a Ticket Number.
From the UI I can access all these tickets and download a CSV file with all of them.
The other way of doing it is by using the API.
The API uses some cURL calls to access the database and retrieve the info that I specify in the parameters.
The method:
I'm using a script like this to get the data:
<?php
$username="myUserName";
$password="myPassword";
$proxy= "myProxy";
$proxyauth = 'myProxyUser:myProxyPassword';
$url="https://qualysapi.qualys.com/msp/ticket_list.php?"; //This is the official script, provided by Qualys, for doing this task.
$postdata = "show_vuln_details=0&SINCE_TICKET_NUMBER=1&CURRENT_STATE=Open&ASSET_GROUPS=All";
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyauth);
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_USERPWD, $username . ":" . $password);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata);
curl_setopt ($ch, CURLOPT_POST, 1);
$result = curl_exec ($ch);
$xml = simplexml_load_string($result);
?>
The script above works fine. It connects to the API, pass some parameters to it and the ticket_list.php file generates an XML file with all I need.
The Problems:
1-) This script only allows a limit of 1000 results in the XML file it returns.
If my request has generated more than 1000 results, the script creates a TAG like this, at the end of the XML:
<TRUNCATION last="5066">Truncated after 1000 records</TRUNCATION>
In this case, I would need to execute anoter cURL call, with the parameters bellow:
$postdata = "show_vuln_details=0&SINCE_TICKET_NUMBER=5066&CURRENT_STATE=Open&ASSET_GROUPS=All";
2-) There are approximately 300,000 tickets in Qualys' database (cloud), and I need to download all of them and insert in MY database, which is used by an application that I'm creating. This application has some forms, which are filled by the user and a bunch of queries are run against the database.
The doubt:
What would be the best way for me to do the task above?
I've got some ideas, but I'm at a complete loss.
I thought:
**1-)**Create a function that does the call above, parses the xml and if the tag
TRUNCATION exists, it gets its value and call itself again, doing it recursively until a result without the tag TRUNCATIONcomes.
The problem with this one is that I weren't able to merge the XML results of each call, and I'm not sure if it would cause memory issues, since it would be needed nearly 300 cURL calls. This script would be executed automatically by using the server's cronTab in a non-business period.
2-) Instead of retrieving all the data, I make the forms that I've mentioned post the data to the script and make the cURL calls with the parameters that the user POSTed. But again I'm not sure if that would be good, since I would still need to do multiple calls, depending on the parameters that the user sends.
3-) This is a crazy one: Use some sort of Macro software to record me while I log in the UI, go to the page where the tickets are located, click the download button, check the CSV option and click to download again. Then, export this script to some language like python or java, create a task in the cronTab and create a script that parses the CSV downloaded and inserts the data to the database. (Crazy or not? =P )
Any help is very welcome, maybe the answer is right before my eyes and I haven't gotten yet.
Thanks in advance!
I believe the proper way would involve a queue worker, however, If I were you I'd make your script grab 5 of these XML files in one execution- grab 1, insert rows, remove from memory, repeat. Then, I'd test it by running it a few times manually to see what sort of execution time and memory it requires. Once you've got a good idea of the execution time and you can see memory will not be a problem, schedule a cron for a little under double that time. If all goes well it should be about a minute between runs and you can have it all in your DB within an hour.
I'm trying to get information from a site by parsing/scraping it via PHP & Curl. But sometimes the current page doesn't finish loading, so the script runs without anything happening. It's a simple script like this...
...
curl_setopt($curl, CURLOPT_URL, $url);
$page = curl_exec($curl);
...
Is there a way to simply retry the loading of the same page if the page doesn't finish loading after (for example) 60 sec, without interrupting the complete script?
It would be great if someone could help me out with a way to realize this task.
You can use CURLOPT_TIMEOUT which is the maximum number of seconds to allow cURL functions to execute.
curl_setopt($ch, CURLOPT_TIMEOUT, timeout_in_seconds);
Something simple would be..
$bol=true;
while($bol)
{
$page = curl_exec($curl);
if($page=="")//Or whatever curl_exec returns on timeout
$bol=true;
}
I wish to include JSPs include files which contain java code in a PHP template. The two includes in question are a header file, and a footer file. Anyone any experience of doing this? We are considering just doing a HTTP request to grab the resulting HTML from the JSP files independantly, but aren't sure if there will be slight performance issues with doing so.
Is there any better solution using some of the tools within Apache to perform this?
echo file_get_contents('http://full/link/to/jsp/page');
If you JSP page echos a header, body structure, you'll need to strip it out. You can do that from the JSP side or PHP.
That's disabled on some systems so you might need to use cURL (it also allows you to post back which you might need to do if you're playing with forms).
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL, 'http://full/link/to/jsp/page');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
echo curl_exec($ch);
You can't include a JSP page into a PHP page.
You can do what you are thinking of though: doing a HTTP request to get HTML content from JSP and embed that into the PHP result. Not pretty, but will work.
There is the Java / PHP Integration extension, but it doesn't allow to compile Java code. I don't think there is a way to compile Java from PHP, if not executing command line commands.
Depending on your requirements, if you don't want to impact page loading, you could also perform an AJAX request to grab the content once the HTML page is loaded, and inject it in the page : this would move the problem to the client.
Does this JSP page change frequently, or depend on PHP page's parameters (some kind of advertisement) ?
You could also cache the output of your JSP (even by parameters) for a pair of hours or a whole day, to avoid calling the page on every request.