So checked via a phpinfo() and Safe Mode on my server is off, Curl is activated and there are no reasons for it not to work.
I also made sure Sharrre.php is in my root directory. Even included the Curlurl to the php file. Tried both absolute and relative linking. The google button with the counter shows as soon it is uploaded but not as expected because the counter shows 0 the entire time.
The culprit seems to be: $json = array('url'=>'','count'=>0);
After a few lines of other code we got this:
if(filter_var($_GET['url'], FILTER_VALIDATE_URL)){
if($type == 'googlePlus'){ //source http://www.helmutgranda.com/2011/11/01/get-a-url-google-count-via-php/
$contents = parse('https://clients6.google.com/rpc?key=AIzaSyCKSbrvQasunBoV16zDH9R33D88CeLr9gQurl=' . $url . '&count=true');
preg_match( '/window\.__SSR = {c: ([\d]+)/', $contents, $matches );
if(isset($matches[0])){
$json['count'] = (int)str_replace('window.__SSR = {c: ', '', $matches[0]);
}
}
So either the google url code is not valid anymore or... well maybe there is something wrong with the suspected culprit because:
when changed to a value higher than 0 $json = array('url'=>'','count'=>15);
It shows 15 counts as you can see. I want it to be dynamic though and get the counts I already have and update those per click.
What can be done to solve this?
In my particular case the problem was in the asignement of the URL to the Curl object.
The original script sharrre.php sets the URL by asigning it to an array element of the curl object, but this is not working and causes Google counter not retrieve any amount.
Instead, the URL must be asigned by the curl_setopt() function.
This resolved this problem in my case:
sharrre.php:
//...
$ch = curl_init();
//$options[CURLOPT_URL] = $encUrl; // <<<--- not working! comment this line.
curl_setopt_array($ch, $options);
curl_setopt($ch, CURLOPT_URL, $encUrl ); // <<<--- Yeeaa, working! Add this line.
//...
Hope this help.
Related
I've a URL which becomes a query string with the http_build_query function.
But I have a parametertimestamp which I cannot edit. And × becomes a multiplication sign x.
Is there a workaround for this?
This is my array which gets passed to the http_build_query function.
$parameters = array(
"transaction_id"=>uniqid("FF-"),
"timestamp"=> time(),
"order_total"=>$_SESSION['total_price'],
"order_total_with_vat"=>$_SESSION['total_price'] * 1.21,
"order_vat"=>"21",
"payment_method"=>"ideal",
"payment_status"=>"1",
"customer_name"=>$_SESSION['customer_data']['naam'],
"customer_address"=>$_SESSION['customer_data']['address'],
"customer_city"=>$_SESSION['customer_data']['city'],
"customer_zipcode"=>$_SESSION['customer_data']['zipcode'],
"customer_country"=>$_SESSION['customer_data']['country'],
"customer_email"=>$_SESSION['customer_data']['email'],
"customer_telephone"=>$_SESSION['customer_data']['telephone'],
);
Output of url:
http://somedomain/subdir/someapi/order?transaction_id=FF-58e2451c5aea9×tamp=1491223836&order_total=156695&order_total_with_vat=189600.95&order_vat=21&payment_method=ideal&payment_status=1&customer_name=t&customer_address=t&customer_city=t&customer_zipcode=t&customer_country=t&customer_email=t%40t&customer_telephone=t&product%5B0%5D=5&product%5B1%5D=5&product%5B2%5D=5&product%5B3%5D=5&product%5B4%5D=5&product%5B5%5D=5&product%5B6%5D=5
Preferred output:
http://somedomain/subdir/someapi/order?transaction_id=FF-58e2451c5aea9×tamp=1491223836&order_total=156695&order_total_with_vat=189600.95&order_vat=21&payment_method=ideal&payment_status=1&customer_name=t&customer_address=t&customer_city=t&customer_zipcode=t&customer_country=t&customer_email=t%40t&customer_telephone=t&product%5B0%5D=5&product%5B1%5D=5&product%5B2%5D=5&product%5B3%5D=5&product%5B4%5D=5&product%5B5%5D=5&product%5B6%5D=5
http_build_query function:
case 'POST':
curl_setopt( $curlHandler, CURLOPT_POST, true );
$url .= '?' . http_build_query( $parameters );
break;
Currently it is working correctly. The issue is that when you echo your URL the sequence × will make your browser replace it with the multiplication symbol x. To echo it and show the correct way in the browser try this:
echo htmlspecialchars($url);
This will display the desired URL.
#Novice was right, he commented on my question:
It's just a display issue in browser but your curl data will go just like what you have shown in your preferred output , no need to add anything extra just hit and go! – Novice
When I checked the apache access.log I could see the preferred query string posted.
Currently, I am trying to use the Twitch API to grab Twitch stats such as current viewers, Title, and more I am running into an issue with #File_Get_Contents when using this my request seem to be delayed or not getting them as quick as I refresh, i.e. I think the results may be cache'd.
For example here is my old code
$twitch = json_decode(curl_get_file_contents('https://api.twitch.tv/kraken/channels/'.$twitch_channel), true);
$display_name = $twitch['display_name'];
$game = $twitch['game'];
$status = $twitch['status'];
$url = $twitch['url'];
$avatar = $twitch['logo'];
$views = $twitch['views'];
$followers = $twitch['followers'];
The main issue with this is it didnt seem like it updated every time I refreshed, so I looked into using cURL for better results + I heard it's much quicker with load time!
Here is my current curl code
$requesturl='https://api.twitch.tv/kraken/channels/' . $twitch_username;
$ch=curl_init($requesturl);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$cexecute=curl_exec($ch);
curl_close($ch);
$twitch = json_decode($cexecute,true);
$display_name = $twitch['display_name'];
$game = $twitch['game'];
$status = $twitch['status'];
$url = $twitch['url'];
$avatar = $twitch['logo'];
$views = $twitch['views'];
$followers = $twitch['followers'];
Your log is showing PHP Notice, You don't have any error. I change a lil bit your code to test it and It's working. So you're probably just no printing your vars.
Check your code with a print_r online
So ultimnately I figured out what the issue was, it's XAMP Curl not working for whatever reason. It's setup properly in the PHP.ini, I have the Two .DLL in my Sys32. So I dont know why this isn't working
I want to fill a database table with certain items from the Steam Marketplace, specifically at the moment, guns from CSGO. I can't seem to find any database or list already of all the gun names, skin names and skin qualities, which is what I want.
One way I thought of to do it is to get to the list of items I want, EG "Shotguns", and save each item on the page into the database, and go through each page of that search. EG:
http://steamcommunity.com/market/search?appid=730&q=shotgun#p1_default_desc
http://steamcommunity.com/market/search?appid=730&q=shotgun#p2_default_desc
Ect..
Firstly, I'm not exactly sure how I would do that, and secondly, I wanted to know if there would be an easier way.
I plan on using the names of items to later get the prices by substituting the names into this: http://steamcommunity.com/market/priceoverview/?currency=3&appid=730&market_hash_name=StatTrak%E2%84%A2%20P250%20%7C%20Steel%20Disruption%20%28Factory%20New%29
And updating the prices every hour or so by running that check for every item. (probably at least a few thousand..)
The general gist of what you need to do boils down to:
Identify the urls you need to parse. In your case you'll notice that the results are loaded via ajax. Right-click the page, click 'inspect element' and go to the network tab. You'll see that the actual url is: http://steamcommunity.com/market/search/render/?query=&start=<STARTVALUE>&count=<NUMBEROFRESULTS>&search_descriptions=0&sort_column=quantity&sort_dir=desc&appid=730&category_730_ItemSet%5B%5D=any&category_730_TournamentTeam%5B%5D=any&category_730_Weapon%5B%5D=any&category_730_Type%5B%5D=tag_CSGO_Type_Pistol&category_730_Type%5B%5D=tag_CSGO_Type_SMG&category_730_Type%5B%5D=tag_CSGO_Type_Rifle&category_730_Type%5B%5D=tag_CSGO_Type_SniperRifle&category_730_Type%5B%5D=tag_CSGO_Type_Shotgun&category_730_Type%5B%5D=tag_CSGO_Type_Machinegun&category_730_Type%5B%5D=tag_CSGO_Type_Knife
Identify what the response type is. In this case it is json, and the data we want is inside a html-snippet
Find the framework required to parse it. You can use json_decode(...) to decode the json string. This question will give more information how to parse html.
You can now feed these urls to a function that loads the page. You can use file_get_contents(...) or the curl library.
Enter the values you parse from the response into your database. Make sure that the script does not get killed when it runs for too long. This question will give you more information about that.
You can use the following as a framework. You'll have to figure the structure of the html yourself, and lookup a tutorial of the html parser and mysql library you want to use.
<?php
//Prevent this script from being killed. Please note that if this script never
//ends, you'll have to kill it manually
set_time_limit( 0 );
//The api does not allow for more than 100 results at a time
$start = 0;
$count = 100;
$maxresults = PHP_INT_MAX;
$baseurl = "http://steamcommunity.com/market/search/render/?query=&start=$1&count=$2&search_descriptions=0&sort_column=quantity&sort_dir=desc&appid=730&category_730_ItemSet%5B%5D=any&category_730_TournamentTeam%5B%5D=any&category_730_Weapon%5B%5D=any&category_730_Type%5B%5D=tag_CSGO_Type_Pistol&category_730_Type%5B%5D=tag_CSGO_Type_SMG&category_730_Type%5B%5D=tag_CSGO_Type_Rifle&category_730_Type%5B%5D=tag_CSGO_Type_SniperRifle&category_730_Type%5B%5D=tag_CSGO_Type_Shotgun&category_730_Type%5B%5D=tag_CSGO_Type_Machinegun&category_730_Type%5B%5D=tag_CSGO_Type_Knife";
while( $start < $maxresults ) {
//Constructing the next url
$url = str_replace( "$1", $start, $baseurl );
$url = str_replace( "$2", $count, $url );
//Doing the request
$ch = curl_init();
curl_setopt( $ch, CURLOPT_URL, $url );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1 );
$result = json_decode( curl_exec( $ch ), TRUE );
curl_close( $ch );
//Doing things with the result
//
//First let's see if everything went according to plan
if( $result == NULL || $result["success"] !== TRUE ) {
echo "Something went horribly wrong. Please edit the script to take this error into account and rerun it.";
exit( -1 );
}
//Bookkeeping for the next url we have to fetch
$count = $result["pagesize"];
$start += $count;
$maxresults = $result["total_count"];
//This is the html we have to parse
$html = $result["results_html"];
//Look up an example how to parse html, and how to get data from it
//Look up how to make a database connection and how to insert data into
//your database
}
echo "And we are done!";
i need to know if is possible to load an html page and submit the form inside this page using php. so something like:
<?php
$html = fopen("http://www.mysite.com","r");
//get position of form...
...
//submit it
?>
is possible? can someone help me? thanks!!!
EDIT:
i have to submit this form
https://annunci.ebay.it/pubblica-annuncio
my problem is that in this page there is an image upload and i don't know how to do that using php( scraping it )
You can also use curl to POST to any URL, for instance the form's action url.
$ch = curl_init('http://example.com/form_action.php');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, array('your' => 'data', 'goes' => 'here');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
This will call the URL http://example.com/form_action.php as if it was called from a form with 'your' set to value 'data' and 'goes' set to value 'here'.
To find out the URL you need to POST, you can inspect source code. When doing that, check the "name" atribute on the <input> tags you want to send.
EDIT: If the POST url and the fields can change, you should check #Adan's answer.
Basically this is what you need to do
1- Get the content of the HTML page using file_get_contents() (bearing in mind the security risks)
2- Parse the HTML using DOMDocument
3- Get the form's attributes, most importantly (ACTION, METHOD) using DOMDocument
4- Get the form's fields' names using DOMDocument
5- Then send to the ACTION url using the method METHOD a request with the data you want replacing the fields using cURL
you can use curl for getting page in php. as mentioned in answer #Lumbendil. For parsing the HTML you can use libraries like
http://simplehtmldom.sourceforge.net/
Or you can use
http://code.google.com/p/phpquery/
As another option, which would be more clean, you could use the eBay API. It provides methods to add new items, and it probably has already built libraries for php, such as the PHP Accelerator toolkit for eBay.
I am providing a code that I got from net to get the contents of a page. After that you can use jquery(maybe) to force the submit function.
$url = "URL OF YOUR PAGE"; // I have tested page from same server
$lines = file( $url );
foreach( $lines as $line_num => $line ) {
$line = htmlspecialchars( $line );
$line = str_replace( "<", '<span><', $line );
$line = str_replace( ">", '></span>', $line );
$line = str_replace( "<!–", '<em><!–', $line );
$line = str_replace( "–>", '–></em>', $line );
echo "<span class=\"linenumber\">Line <strong>$line_num </strong></span> : " . $line . "<br/>\n";
}
The above code gave me contents from another page on same server. Now you have to find a way around to check if a form exist and then ; force submit that form.
Parsing HTML / JS codes to get info using PHP.
www.asos.com/Asos/Little-Asos-Union-Jack-T-Shirt/Prod/pgeproduct.aspx?iid=1273626
Take a look at this page, it's a clothes shop for kids. This is one of their items and I want to point out the size section. What we need to do here is to get all the sizes for this item and check whether the sizes are available or not. Right now all the sizes for this items are:
3-4 years
4-5 years
5-6 years
7-8 years
How can you say if the sizes are available or not?
Now take a look at this page first and check the sizes again:
www.asos.com/Ralph-Lauren/Ralph-Lauren-Long-Sleeve-Big-Horse-Stripe-Rugby-Top/Prod/pgeproduct.aspx?iid=1111751
This item has the following sizes:
12 months
18 months - Not Available
24 months
As you can see 18 months size is not available, it is indicated by the "Not Available" text next to the size.
What we need to do is go the page of an item, get the sizes and check the availability of each sizes. How can I do this in PHP?
EDIT:
Added a working code and a new problem to tackle.
Working code but it needs more work:
<?php
function getProductVariations($url) {
//Use CURL to get the raw HTML for the page
$ch = curl_init();
curl_setopt_array($ch,
array(
CURLOPT_RETURNTRANSFER=>true,
CURLOPT_HEADER => false,
CURLOPT_URL => $url
)
);
$raw_html = curl_exec($ch);
//If we get an invalid response back from the server fail
if ($raw_html===false) {
throw new Exception(curl_error($ch));
}
curl_close($ch);
//Find the variation JS declarations and extract them
$raw_variations = preg_match_all("/arrSzeCol_ctl00_ContentMainPage_ctlSeparateProduct\[[0-9]+\].*Array\((.*)\);/",$raw_html,$raw_matches);
//We are done with the Raw HTML now
unset($raw_html);
//Check that we got some results back
if (is_array($raw_matches) && isset($raw_matches[1]) && sizeof($raw_matches[1])==$raw_variations && $raw_variations>0) {
//This is where the matches will go
$matches = array();
//Go through the results of the bracketed expression and convert them to a PHP assoc array
foreach($raw_matches[1] as $match) {
//As they are declared in javascript we can use json_decode to process them nicely, they just need wrapping
$proc=json_decode("[$match]");
//Label the fields as best we can
$proc2=array(
"variation_id"=>$proc[0],
"size_desc"=>$proc[1],
"colour_desc"=>$proc[2],
"available"=>(trim(strtolower($proc[3]))=="true"),
"unknown_col1"=>$proc[4],
"price"=>$proc[5],
"unknown_col2"=>$proc[6], /*Always seems to be zero*/
"currency"=>$proc[7],
"unknown_col3"=>$proc[8],
"unknown_col4"=>$proc[9], /*Negative price*/
"unknown_col5"=>$proc[10], /*Always seems to be zero*/
"unknown_col6"=>$proc[11] /*Always seems to be zero*/
);
//Push the processed variation onto the results array
$matches[$proc[0]]=$proc2;
//We are done with our proc2 array now (proc will be unset by the foreach loop)
unset($proc2);
}
//Return the matches we have found
return $matches;
} else {
throw new Exception("Unable to find any product variations");
}
}
//EXAMPLE USAGE
try {
$variations = getProductVariations("http://www.asos.com/Asos/Prod/pgeproduct.aspx?iid=803846");
//Do something more useful here
print_r($variations);
} catch(Exception $e) {
echo "Error: " . $e->getMessage();
}
?>
The above code works, but there's a problem when the product needs you to select a colour first before the sizes are displayed.
Like this one:
http://www.asos.com/Little-Joules/Little-Joules-Stewart-Venus-Fly-Trap-T-Shirt/Prod/pgeproduct.aspx?iid=1171006
Any idea how to go about this?
SOLUTION:
function curl($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
return curl_exec($ch);
curl_close ($ch);
}
$html = curl('http://www.asos.com/pgeproduct.aspx?iid=1111751');
preg_match_all('/arrSzeCol_ctl00_ContentMainPage_ctlSeparateProduct\[(.*?)\] \= new Array\((.*?),\"(.*?)\",\"(.*?)\",\"(.*?)\"/is',$html,$bingo);
echo print_r($bingo);
Link: http://debconf11.com/stackoverflow.php
You are on your own now :)
EDIT2:
Ok, we are close to solution...
<script type="text/javascript">var arrSzeCol_ctl00_ContentMainPage_ctlSeparateProduct = new Array;
arrSzeCol_ctl00_ContentMainPage_ctlSeparateProduct[0] = new Array(1164,"12 months","SailingOrange","True","","59.00","0.00","£","","-59.00","0.00","0");
arrSzeCol_ctl00_ContentMainPage_ctlSeparateProduct[1] = new Array(1165,"18 months","SailingOrange","False","","59.00","0.00","£","","-59.00","0.00","0");
arrSzeCol_ctl00_ContentMainPage_ctlSeparateProduct[2] = new Array(1167,"24 months","SailingOrange","True","","59.00","0.00","£","","-59.00","0.00","0");
</script>
It is not loaded via ajax, instead array is in javascript variable. You can parse this with PHP, you can clearly see that 18 months is a False, which means it is not available.
EDIT:
This sizes are loaded via javascript, therefore you cannot parse them since they are not there.
I can extract only this...
<select name="drpdwnSize" id="ctl00_ContentMainPage_ctlSeparateProduct_drpdwnSize" onchange="drpdwnSizeChange(this, 'ctl00_ContentMainPage_ctlSeparateProduct', arrSzeCol_ctl00_ContentMainPage_ctlSeparateProduct);">
<option value="-1">Select Size</option>
</select>
You can sniff JS to check if you can load sizes based on product id.
First you need: http://simplehtmldom.sourceforge.net/
Forget file_get_contents() it is ~5 slower than cURL.
You then parse this piece of code (html with id ctl00_ContentMainPage_ctlSeparateProduct_drpdwnSize)
<select id="ctl00_ContentMainPage_ctlSeparateProduct_drpdwnSize" name="ctl00$ContentMainPage$ctlSeparateProduct$drpdwnSize" onchange="drpdwnSizeChange(this, 'ctl00_ContentMainPage_ctlSeparateProduct', arrSzeCol_ctl00_ContentMainPage_ctlSeparateProduct);">
<option value="-1">Select Size</option><option value="1164">12 months</option><option value="1165">18 months - Not Available</option><option value="1167">24 months</option></select>
You can then use preg_match(),explode(),str_replace() and others to filter out values you want. I can write it but I don't have time right now :)
The most simple way to fetch the content of a URL is to rely on fopen wrappers and just use file_get_contents with the URL. You can use the tidy extension to parse the HTML and extract content. http://php.net/tidy
You can download the file using fopen() or file_get_contents(), as Raoul Duke said, but if you have experience with the JavaScript DOM model, the DOM extension might be a bit easier to use than Tidy.
I know for a fact that the DOM extension is enabled by default in PHP, but I am a bit unsure if Tidy is (the manual page only says it's "bundeled", so I suspect that it might not be enabled).