I am converting a Java program into PHP for a webpage, the only problem is that I am using a free web hosting service (000Webhost) and it does not let me make time limit changes. The program compiles data from (currently, 66, however, more are possible) different webpages, and puts the data into a table. I can provide code, though, I am by no means a PHP scripter, so that may be the source of my problems. Can someone help me figure out my issues?
<?php
$html = file_get_contents("http://www.someURL.com");
$newString = explode("data = [", $html);
$sOne = explode("]", $newString[1]);
$names = explode(", ", $sOne[0]);
$urls = array();
foreach($names as $name) {
$chars = str_split($name);
foreach ($chars as $ch) {
if (!ctype_alnum($ch) && $ch != '-' && $ch != '_' && $ch != ' ') {
$name = trim(str_replace($ch, ' ', $name));
}
}
$urls[count($urls)] = 'http://www.someURL.com/' .str_replace(' ', '_', $name);
}
echo count($urls);
for ($i = 0; $i < count($urls); $i++) {
$url = $urls[$i];
$html = #file($url);
if (strpos($html, "404 - Page not found") === false) {
echo $i;
flush(); #ob_flush();
}
}
echo '<br>';
echo '<br>';
?>
The echos in there are a debug, telling me how many pages I need to process, and how many I actually have processed.
Thanks.
you can run you php processing in a local host, WAMP, LAMP , XAMP etc, and then change your php time limit to 0 (unlimited) , fill the local DB in all the data you just saved and then move the local db to the production db
Related
I'm using PHP simple_html_dom and file_get_html in a foreach loop which loops about 4,000 times.
Here's what I'm doing. I'm extracting content from multiple urls on same host.
There are 4,000+ urls all on the same host. example.com
Example url: example.com/product.php?prod=00283
On the next pass-thru of the loop example.com/product.php?prod=03418
next pass-thru of loop example.com/product.php?prod=08192
etc, etc until all 4000+ urls are looped over.
Tested 100 urls took over 1 and a half minutes to parse and output content.
Is it possible to setup a multithreaded approach in the foreach loop to bring back the contents of all 4000 urls simultaneously?
In other words, do I use sockets or some other kind of multithreaded approach to iterate over all the urls in parallel?
Here is my sample code below so you can see what I'm doing here.
include 'simple_html_dom.php';
$partlist_file = 'parts.txt';
$partlist = file('partnumberlist.txt', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$stock = '';
$isFirst = true;
$firstline = 'sku,qty,visibility,is_in_stock' . "\n";
$output_first = '';
foreach($partlist as $parts => $part) {
$html = file_get_html('example.com/product.php?prod=' . $part);
$ret = $html->find('span#cph_lb_stock_buy');
foreach($ret as $element) {
$stock = $element->plaintext;
$stock = preg_replace(array('/\\n/','/\\r/'),'',$stock);
$stock = trim($stock);
if($stock == 'Not in stock.') {
$stock = '0,1,0';
} elseif($stock == 'In Stock & Ready to Ship!') {
$stock = '4,4,4';
}
if ($isFirst) {
$output = $firstline;
$isFirst = false;
}
$output .= 'K' . $part . ',' . $stock . "\n";
}
}
//print_r (trim($output,"\n"));
file_put_contents($partlist_file, trim($output,"\n"));
I'm running a php script that reads from some pipe-delimited txt files, those files contain around 22,000 records. I'm using PHP file() function to read those files, BTW the files have some inter-relation too, I'm pasting the code here for better understanding
public function getGames()
{
$resource = self::DATAFILES.'Product.txt';
$games = array_slice($this->readFile($resource), 1);
$data = array();
$count = 1;
foreach($games as &$records)
{
$game = new Game();
$attributes = explode($this->delimiter,$records);
$game->api = (int) $attributes[0];
$game->console_id = (string) $attributes[1];
$game->title = (string) $this->getTitle($attributes[2]);
$game->barcode = (string) $attributes[4];
$game->image = $this->getCoverImage($attributes[0]);
$game->releateDate = strtotime($attributes[8]);
$data[] = $game;
//if($count == 100): break; else: $count++; endif;
}
print '<pre>'; print_r($data);
}
public function getTitle($titleID)
{
$resource = self::DATAFILES.'Title.txt';
$titles = array_slice($this->readFile($resource), 1);
foreach($titles as $records)
{
$attributes = explode($this->delimiter,$records);
$pattern = '/^' . preg_quote($attributes[0], '/') . '$/';
if (preg_match($pattern, $titleID))
{
return $attributes[2];
break;
}
}
}
public function getCoverImage($gameID)
{
$resource1 = self::DATAFILES.'ProductImage.txt';
$coverImages = array_slice($this->readFile($resource1), 1);
foreach($coverImages as $img_records)
{
$image_attributes = explode($this->delimiter,$img_records);
$pattern1 = '/^' . preg_quote($image_attributes[0], '/') . '$/';
$pattern2 = '/^' . preg_quote($image_attributes[3], '/') . '$/';
if (preg_match($pattern1, $gameID) && preg_match($pattern2, 'Cover'))
{
return $image_attributes[2];
break;
}
}
So the problem I'm facing here is that - when I run that script it will just go on and on nothing happens i.e never throws any error neither print the returned array, however when I limit's the loop iteration up to 100 it works which means nothing wrong with the code so I thought perhaps, php script execution time making trouble so I changed it to max_execution_time = 0 - but still not getting the results. Any help or suggestion, would love hear that.. :)
BTW I'm using Xampp Apache for my local dev !
I would like to have a quick script listing all active hosts in a LAN, and I am a bit lost. From other posts I figured that this can be done most effectively by polling the DHCP server (in my case a Lancom router) using SNMP.
However, I am not familiar with the SNMP commands in PHP. Is snmpwalk() the correct function? Can I get snmpwalk() or any other php function to return an array that contains a list of all live hosts?
Finding live hosts
The best way to be sure you get all live hosts is with a ping sweep of the subnet, using a tool like nmap. Since Windows hosts don't respond to pings by default, it also includes a brief TCP port scan as well. The syntax from the linux CLI is nmap -sP 192.0.2.0/24 (substitute your subnet instead of 192.0.2.0/24).
SNMP query
I don't think SNMP will really solve your problem, but I will include what I can to assist... When you use the PHP SNMP Extension, you first need to know the OID for the table with the correct information. The LANCOM-1711-MIB is one possibility, but it's hard to know for sure; you should contact LANCOM support if you don't know which SNMP OID to poll.
Let's just go on the assumption that staDhcpLanIpadd (OID: 1.3.6.1.4.1.2356.500.2.1712.1.32.21.1.2) is the OID you need. At this point, you would snmpwalk the router using SNMPv2c and the SNMP community you configured on it. Presumably, this OID gives you the list of DHCP addresses issued; however, that doesn't mean they are live at the time you poll the router (someone could have unplugged the cable, or turned them off).
So I wrote a script that probes the Lancom router and pulls out the DHCP/BOOTP table. It may be used for all those who need to monitor such routers, and therefore I am sharing it. It also outputs a nice HTML table; the function BetterTable() can be used on any 2D array.
You will need to set IP, userid, and pwd (first three variables) in order to use the script on your router.
<?php
$router_ip = '';
$username = '';
$password = '';
$port = 23;
$timeout = 10;
$connection = fsockopen($router_ip, $port, $errno, $errstr, $timeout);
if(!$connection){
echo "Connection failed\n";
exit();
} else {
fputs($connection, "$username\r\n");
fputs($connection, "$password\r\n");
fputs($connection, "cd setup/dhcp/dhcp-table \r\n");
fputs($connection, "dir \r\n");
fputs($connection, " ");
$j = 0;
while ($j < 16) {
fgets($connection);
$j++;
}
stream_set_timeout($connection, 2);
$timeoutCount = 0;
$content ='';
$DhcpArray = '';
(int) $index =0;
$DhcpFile = "C:\IP-Symcon\webfront\user\images\LancomDhcp.txt";
$fh = fopen($DhcpFile, 'w') or die("can't open file");
//$DhcpArray[0] = array ('IP-Address', 'MAC-Address', 'Timeout', 'Hostname', 'Type', 'LAN-Ifc', 'Ethernet-Port', 'VLAN-ID', 'Network-Name');
while (!feof($connection)){
$content = fgets($connection);
$content = str_replace("\r", '', $content);
$content = str_replace("\n", "", $content);
$lineArray = explode(' ', $content);
if (isValidIp($lineArray [0]))
{
$DhcpArray[$index]['IP-Address'] = substr ($content, 0,17);
$DhcpArray[$index]['MAC-Address'] = substr ($content, 17,32-18);
$DhcpArray[$index]['Timeout'] = substr ($content, 31,41-32);
$DhcpArray[$index]['Hostname'] = substr ($content, 40,108-41);
$DhcpArray[$index]['Type'] = substr ($content, 107,125-108);
$DhcpArray[$index]['LAN-Ifc'] = substr ($content, 124,137-125);
$DhcpArray[$index]['Ethernet-Port'] = substr ($content, 136,152-137);
$DhcpArray[$index]['VLAN-ID'] = substr ($content, 151,161-152);
$DhcpArray[$index]['Network-Name'] = substr ($content, 160);
fwrite($fh, $content);
$index +=1;
}
# If the router say "press space for more", send space char:
if (preg_match('/MORE/', $content) ){ // IF current line contain --More-- expression,
fputs ($connection, " "); // sending space char for next part of output.
} # The "more" controlling part complated.
$info = stream_get_meta_data($connection);
if ($info['timed_out']) { // If timeout of connection info has got a value, the router not returning a output.
$timeoutCount++; // We want to count, how many times repeating.
}
if ($timeoutCount >2){ // If repeating more than 2 times,
break; // the connection terminating..
}
}
$content = substr($content,410);
BetterTable($DhcpArray);
fclose($fh);
}
echo "End.\r\n";
//--------------------------------------------------------------------
function isValidIp($ip)
{/* PCRE Pattern written by Junaid Atari */
return !preg_match ( '/^([1-9]\d|1\d{0,2}|2[0-5]{2})\.('.
'(0|1?\d{0,2}|2[0-5]{2})\.){2}(0|1?'.
'\d{0,2}|2[0-5]{2})(\:\d{2,4})?$/',
(string) $ip )
? false
: true;
}
//--------------------------------------------------------------
function BetterTable($twoDimArray)
{
$i = 0;
echo "<table>
<table class='BetterTable' border='1'>";
echo "<tr>";
echo '<td>Line #
</td>';
foreach ($twoDimArray[0] as $fieldName => $fieldValue)
{
echo '<td>'.$fieldName. '</td>';
}echo '</tr>';
$i = 0;
foreach ($twoDimArray as $rowName => $rowValue)
{
if ($i%2 == 0)
Echo "<tr bgcolor=\"#d0d0d0\" >";
else
Echo "<tr bgcolor=\"#eeeeee\">";
$fields = count($twoDimArray[$i]);
$y = 0;
echo '<td>'.$i. '</td>';
foreach ($rowValue as $fieldName => $fieldValue)
{
echo '<td>'.$fieldValue. '</td>';
$y = $y + 1;
}
echo '</tr>';
$i = $i + 1;
}
echo '</table>';
}
?>
I'm trying to parse the resources contained in a resources.arsc file as discussed in this question. I know the androidmanifest.xml file identifies resources located in the .arsc file. I have successfully managed to parse the header of the .arsc file, I can't figure out how to parse the resources themselves.
Can somebody please help me figure out how to parse the resources contained in an .arsc file?
My parsing code so far:
<?php
$doc = fopen('resources.arsc', 'r+');
for($i=1;$i<10;$i++){
$res[$i] = _unpack('V', fread($doc, 4));
}
for ($i = 0, $j = $res[6]; $i <= $j; $i++) {
$word = fread($doc, 4);
$stroffs[] = _unpack('V', $word);
}
$strings = array();
$curroffs = 0;
foreach($stroffs as $offs){
//read length
$len = _unpack('v', fread($doc, 2));
//read string
if($len>0){
$str = fread($doc, $len*2);
}else{
$str = '';
}
//null
$wd = fread($doc, 2);
//utf-16le
$strings[] = mb_convert_encoding($str, 'gbk', 'UTF-16LE');
//curr offset
$curroffs += ($len+1)*2 + 2;
}
$tpos = ftell($doc);
read_doc_past_sentinel($doc);
//fseek($doc, $tpos + $tpos % 4);
$i = 0;
$xmls = $strings;
print_r($xmls);
//------------------------------------
//and then...somebody konw format or continue parse?
//------------------------------------
function read_doc_past_sentinel(&$doc){
$pos = ftell($doc);
$count= 0;
while($word = fread($doc, 4)){
if(_unpack('V', $word)==-1)break;
}
$n = 1;
if ($count < $n){
while($word = peek_doc($doc, 4)){
if(_unpack('V', $word) != -1)break;
fread($doc, 4);
$n++;
if(isset($count) && $count >= $n)break;
}
echo 'skip '.$n.' chars<br />';
}
}
function peek_doc(&$doc, $size){
$data = fread($doc, $size);
fseek($doc, ftell($doc)-$size);
return $data;
}
function _unpack($m, $b){
//if(!$b)return '';
$res = unpack($m, $b);
return $res[1];
}
?>
This is a fairly complicated binary file. You will need way more code than that to parse it. :)
My suggestion would be to use the same code that the platform does -- that is the ResTable and related classes found here:
frameworks/base/include/utils/ResourceTypes.h
frameworks/base/libs/utils/ResourceTypes.cpp
Note that ResourceTypes.h also has definitions for the complete structure of the resource table (which the classes there use to parse it).
You may also just be able to use the aapt tool. This has a number of options for parsing resource-related data out of an .apk:
aapt d[ump] [--values] WHAT file.{apk} [asset [asset ...]]
badging Print the label and icon for the app declared in APK.
permissions Print the permissions from the APK.
resources Print the resource table from the APK.
configurations Print the configurations in the APK.
xmltree Print the compiled xmls in the given assets.
xmlstrings Print the strings of the given compiled xml assets.
If there is some other data you want not available with those commands, consider modifying the tool code in frameworks/base/tools/aapt to add stuff to parse out what you want. This tool is using ResTable to parse the resources.
EDIT3: It seems that the problem occurs on my localhost XAMPP PHP 5.3 setup, not on any of the remote servers running php 5.2 that I've tested. Still unclear if its php or xampp (or maybe the combination) that causes the error /EDIT3
I have an xml with about 12000 names to add to an array.
The xml structure looks like this:
<smdusersync datetime="2010-12-13 13:51:16">
<userstoadd>
<User fnamn="Adam" enamn="Svensson" email="adam#darkeye.se" password="3906" />
<User fnamn="Brooke" enamn="Jarnbjer" email="brooke#gmail.com" password="2729" />
<User fnamn="Caesar" enamn="Carlsson" email="caesar#comhem.se" password="1668" />
<!-- about 12000 other users -->
</userstoadd>
<userstoremove>
</userstoremove>
</smdusersync>
EDIT2: I've tried with other xml examples, including programmatically generated with no attbutes etc, and that doesn't matter - still the same problem described below... /EDIT2
When running a simple foreach loop on the xml userstoadd child, strange things start to
happen when I push objects to an array.
(Please note that the example below has the code that causes the error - it's not a useful example in any way...)
Running a simple foreach loop (here just pushing a testcounter to the array) works as expected:
$user_xml = simplexml_load_file('users.xml');
$xml_count = $user_xml->userstoadd->children()->count();
$users_arr = array();
$test_count = 0;
foreach ($user_xml->userstoadd->children() as $user) {
array_push($users_arr, $test_count); // << Works as expected!
$test_count++;
}
echo $xml_count, '/', $test_count;
The $xml_count and $test_count always have the same value.
When I do the same, except for that I push a simple object to the array, everything works fine if the number of xml users <= 9940:
$user_xml = simplexml_load_file('users.xml');
$xml_count = $user_xml->userstoadd->children()->count();
$users_arr = array();
$test_count = 0;
foreach ($user_xml->userstoadd->children() as $user) {
$dummy_object = new StdClass(); // << Empty dummy object
array_push($users_arr, $dummy_object); // << CAUSES ERROR if more than 9940 items...!
$test_count++;
}
echo $xml_count, '/', $test_count; // << SHOULD BE THE SAME!
When having 9940 xml user items, the output is as expected 9940/9940.
BUT when having 9941 xml user items, the output is 9941/19881!
And when having 9942 xml user items, the output is 9942/19882!
Suddenly almost 10000 difference! And the content of the user items doesn't seem to matter... I've copied and moved xml user items with the same result...
EDIT: When more than 9940 items, it suddeny doubles so that 9941 items give (9940 x 2) + 1 = 19881.
No difference when using one single xml user item 12000 times. /EDIT
Any idea what's going on here?
The following insanity solves the problem for me:
You might want to comment out the test(50000); line.
<?php
test(9996); // works (echoes 9996/9996/9996)
test(9997); // BREAKS on my XAMPP php 5.3.1! (echoes 9997/19993/19993)
test(50000); // test with larger number
function test($items_to_create) {
$xml_str = '<root>' . chr(13);
for($i=0; $i<$items_to_create; $i++) {
$xml_str .= '<item attr="t' . $i . '"/>' . chr(13);
}
$xml_str .= '</root>';
$xml = new SimpleXMLElement($xml_str);
$xml_children = $xml->children();
$xml_count = $xml_children->count();
$arr = array();
$test_count = 0;
/*
foreach ($xml->children() as $user) {
$dummy_object = new StdClass();
array_push($arr, $dummy_object);
$test_count++;
}
*/
for ($i=0; $i<$xml_count; $i++) {
$dummy_object = new StdClass();
$dummy_object->foo = $xml_children[$i]->attributes()->attr;
array_push($arr, $dummy_object);
$test_count++;
}
$arr_count = count($arr);
echo '<br />CTRL+F for me:', $xml_count, '/', $test_count , '/', $arr_count, '<hr />';
echo '<pre>', print_r($arr, true), '</pre>';
}
?>
Does it fix it for you?
As you describe it I can't see any relation to XML handling. Please run this code to verify:
$xml_count = 9950;
$users_arr = array();
$test_count = 0;
foreach (range(0, $xml_count) as $user) {
$dummy_object = new StdClass();
array_push($users_arr, $dummy_object);
$test_count++;
}
echo $xml_count, '/', $test_count, '/', count($users_arr);
It should print the number 9950 three times. If that's not the case on your machine, your PHP is broken.
Ok, I've isolated the as somhow related to my XAMPP php 5.3 setup.
Test on two different remote servers with php 5.2 and there it works as expected.
The following code breaks on my XAMPP with php 5.3.1
<?php
test(9996); // works (echoes 9996/9996/9996)
test(9997); // BREAKS on my XAMPP php 5.3.1! (echoes 9997/19993/19993)
function test($items_to_create) {
$xml_str = '<root>' . chr(13);
for($i=0; $i<$items_to_create; $i++) {
$xml_str .= '<item />' . chr(13);
}
$xml_str .= '</root>';
$xml = new SimpleXMLElement($xml_str);
$xml_count = $xml->children()->count();
$arr = array();
$test_count = 0;
foreach ($xml->children() as $user) {
$dummy_object = new StdClass();
array_push($arr, $dummy_object);
$test_count++;
}
$arr_count = count($arr);
echo "<br/>", $xml_count, '/', $test_count , '/', $arr_count;
}