We are currently performing searches on a Database and returning results in JSON format to use on a Google Maps. The file that we call is named getvenues.php and works great on the server. It accepts a number of parameters and returns the results based on the query.
We then have a separate file that checks to see if there's a JSON file on the server which contains the results, matches its age against a setting, and then returns the data either from the cache file, or builds a new cache file if it's too old.
Since there are several thousand possible search options, we only cache single searches (either on a County, Region or Type). The JavaScript always calls our search_cache_builder.php file. If there is more than one search parameter, the file simply gets the contents returned by getvenues.php and serves it up without any caching.
Everything works great except for one particular combination. If a search is run for venue_type=Castle / Fort and venue_name=Leeds Castle, the search_cache_builder.php returns an empty array, even though accessing getvenues.php directly returns the required data.
Here's a sample of the getvenues.php working with this data > http://dev.weddingvenues.com/lincweb/getvenues.php?type=Castle%20/%20Fort&venue_name=Leeds%20Castle
And here's what the search_cache_builder.php script returns with an identical search (the address we are sending to is correct) > http://www.weddingvenues.com/search_cache_builder.php?type=castle%20/%20fort&venue_name=Leeds%20Castle
Here's the code for the search_cache_builder.php file, which relates to this particular query:
$get_string = '?';
foreach($_GET as $key => $value)
{
if($get_string === '?')
{
$get_string .= $key . '=' . $value;
}
else
{
$get_string .= '&' . $key . '=' . $value;
}
}
//$get_string = str_replace(' ', '', $get_string);
// Otherwise, we need to serve up the page as is:
$file_url = GET_VEN_URL . $get_string;
echo file_get_contents($file_url);
Can anyone offer an explanation as to why the search_cache_builder.php file is returning an empty array?
You should urlencode() your parameter values.
In fact, while your getvenues.php receives parameters directly from a browser it behaves OK, 'cause they are correctly urlencoded.
I tried what follows in my computer towards your service and it works:
define ("GET_VEN_URL", "http://www.weddingvenues.com/getvenues.php");
$get_string = '?';
foreach($_GET as $key => $value)
{
if($get_string === '?')
{
$get_string .= $key . '=' . urlencode($value);
}
else
{
$get_string .= '&' . $key . '=' . urlencode($value);
}
}
$file_url = GET_VEN_URL . $get_string;
echo "<pre>";
echo $file_url;
echo "<pre>";
echo file_get_contents($file_url);
Because $get_string === '?' is ALWAYS false, change it to ==, ? is not a boolean.
I would not recommend using GET parameters, &,? as part of the file name.
Beside this, it will quickly hit into 5k (or 4k, can not recall) limit for a file name.
You can do a sort by $GET, md5 (or hash) the array to a random string.
As long you ensure the hashing mechanism is consistent, you can easily retrieve the cache file.
First try to urldecode values, then use http_build_query to generate your get_string, your problem must be the special chars in values (like /).
Edit:
If you change the order it works: http://www.weddingvenues.com/search_cache_builder.php?venue_name=Leeds%20Castle&type=castle%20/%20fort
Related
I have made a small script which uses the Twitch API. The API only allows a maximum of 100 results per query. I would like to have this query carry on until there are no more results.
My theory behind this, is to run a foreach or while loop and increment the offset by 1 each time.
My problem however, is that I cannot change the foreach parameters within itself.
Is there anyway of executing this efficiently without causing an infinite loop?
Here is my current code:
<?php
$newcurrentFollower = 0;
$offset=0;
$i = 100;
$json = json_decode(file_get_contents("https://api.twitch.tv/kraken/channels/greatbritishbg/follows?limit=25&offset=".$offset));
foreach ($json->follows as $follow)
{
echo $follow->user->name . ' (' . $newcurrentFollower . ')' . "<br>";
$newcurrentFollower++;
$offset++;
$json = json_decode(file_get_contents("https://api.twitch.tv/kraken/channels/greatbritishbg/follows?limit=25&offset=".$offset));
}
?>
Using a While loop:
while($i < $total)
{
$json = json_decode(file_get_contents("https://api.twitch.tv/kraken/channels/greatbritishbg/follows?limit=25&offset=".$offset));
echo $json->follows->user->name . ' (' . $newcurrentFollower . ')' . "<br>";
$newcurrentFollower++;
$offset++;
$i++;
}
Ends up echoing this (No names are successfully being grabbed):
Here is the API part for $json->follows:
https://github.com/justintv/Twitch-API/blob/master/v2_resources/channels.md#get-channelschannelfollows
You can use this:
$offset = 0;
$count = 1;
do {
$response = json_decode(file_get_contents(
'https://api.twitch.tv/kraken/channels/greatbritishbg/follows?limit=100&offset=' . $offset
));
foreach($response->follows as $follow) {
echo $follow->user->name . ' (' . ($count++) . ')' . "</br>";
}
$offset+=25;
} while (!empty($response->follows));
You want to use a while loop here, not just a foreach. Basically:
while (the HTTP request returns results)
{
foreach ($json->follows as $follow)
{
do stuff
}
increment offset so the next request returns the next one not already processed
}
The trickiest part is going to be getting the while condition right so that it returns false when the request gets no more results, and will depend on what the API actually returns if there are no more results.
Also important, the cleanest way would be to have the HTTP request occur as part of the while condition, but if you need to do some complicated computation of the JSON return to check the condition, you can put an initial HTTP request before the loop, and then do another request at the end of each while loop iteration.
The problem is you're only capturing the key not the value. Place it into a datastructure to access the information.
Honestly I find a recursive function much more effective than a iterative/loop approach then just update a datatable or list before the next call. It's simple, uses cursors, lightweight and does the job. Reusable if you use generics on it too.
This code will be in c#, however I know with minor changes you'll be able to get it working in php with ease.
query = //follower object get request//
private void doProcessFollowers(string query)
{
HTTPParse followerData = new HTTPParse(); //custom json wrapper. using the basic is fine. Careful with your cons though
var newRoot = followerData.createFollowersRoot(query); // generates a class populated by json
if (newRoot[0]._cursor != null)
{
populateUserDataTable(newRoot); //update dataset
doProcessFollowers(newRoot[0]._links.next); //recurse
}
}
Anyway - This just allows you to roll through the cursors without needing to worry about indexes - unless you specifically want them for whatever reason. If you're working with generics you can just reuse this code without issue. Find a generic example below. All you need to do to make it reuseable is pass the correct class within the <> of the method call. Can work for any custom class that you use to parse json data with. Which is basically what the 'createfollowerroot()' is in the above code, except that's hard typed.
Also I know it's in c# and the topic is php, with a few minor changes to syntax you'll get it working easily.
Anyway Hope this helped somebody
Generic example:
public static List<T> rootSerialize<T>(JsonTextReader reader)
{
List<T> outputData = new List<T>();
while (reader.Read())
{
JsonSerializer serializer = new JsonSerializer();
var tempData = serializer.Deserialize<T>(reader);
outputData.Add(tempData);
}
return outputData;
}
I have a page that lists out items according to numerous parameters ie variables with values.
listitems.php?color=green&size=small&cat=pants&pagenum=1 etc.
To enable editing of the list, I have a parameter edit=1 which is appended to the above querystring to give:
listitems.php?color=green&size=small&cat=pants&pagenum=1&edit=1
So far so good.
WHen the user is done editing, I have a link that exits edit mode. I want this link to specify the whole querystring--whatever it may be as this is subject to user choices--except remove the edit=1.
When I had only a few variables, I just listed them out manually in the link but now that there are more, I would like to be able programmatically to just remove the edit=1.
Should I do some sort of a search for edit=1 and then just replace it with nothing?
$qs = str_replace("&edit=1, "", $_SERVER['QUERY_STRING']);
<a href='{$_SERVER['PHP_SELF']}?{$qs}'>return</a>;
Or what would be the cleanest most error-free way to do this.
Note: I have a similar situation when going from page to page where I'd like to take out the pagenum and replace it with a different one. There, since the pagenum varies, I cannot just search for pagenum=1 but would have to search for pagenum =$pagenum if that makes any difference.
Thanks.
I'd use http_build_query, which nicely accepts an array of parameters and formats it correctly. You'd be able to unset the edit parameter from $_GET and push the rest of it into this function.
Note that your code has a missing call to htmlspecialchars(). A URL can contain characters that are active in HTML. So when outputting it into a link: Escape!
Some example:
unset($_GET['edit']); // delete edit parameter;
$_GET['pagenum'] = 5; // change page number
$qs = http_build_query($_GET);
... output link here.
Here's my shot:
/**
* Receives a URL string and a query string to remove. Returns URL without the query string
*/
function remove_url_query($url, $key) {
$url = preg_replace('/(?:&|(\?))' . $key . '=[^&]*(?(1)&|)?/i', "$1", $url);
$url = rtrim($url, '?');
$url = rtrim($url, '&');
return $url;
}
Returns:
remove_url_query('http://example.com?a', 'a') => http://example.com
remove_url_query('http://example.com?a=1', 'a') => http:/example.com
remove_url_query('http://example.com?a=1&b=2', 'a') => http://example.com?b=2
Kudos to David Walsh.
Another solution, to avoid & problems too!!!
remove_query('http://example.com/?a=valueWith**&**inside&b=value');
Code:
function remove_query($url, $which_argument=false){
return preg_replace('/'. ($which_argument ? '(\&|)'.$which_argument.'(\=(.*?)((?=&(?!amp\;))|$)|(.*?)\b)' : '(\?.*)').'/i' , '', $url);
}
It wouldn't work if edit=1 is the first variable:
listitems.php?edit=1&color=green&...
You can use the $_GET variable to create the query string yourself. Something like:
$qs = '';
foreach ($_GET as $key => $value){
if ($key == 'pagenum'){
// Replace page number
$qs .= $key . '=' . $new_page_num . '&';
}elseif ($key != 'edit'){
// Keep all key/values, except 'edit'
$qs .= $key . '=' . urlencode($value) . '&';
}
}
I want to add a query string to a URL, however, the URL format is unpredictable. The URL can be
http://example.com/page/ -> http://example.com/page/?myquery=string
http://example.com/page -> http://example.com/page?myquery=string
http://example.com?p=page -> http://example.com?p=page&myquery=string
These are the URLs I'm thinking of, but it's possible that there are other formats that I'm not aware of.
I'm wondering if there is a standard, library or a common way to do this. I'm using PHP.
Edit: I'm using Cbroe explanation and Passerby code. There is another function by Hamza but I guess it'd be better to use PHP functions and also have cleaner/shorter code.
function addQuery($url,array $query)
{
$cache=parse_url($url,PHP_URL_QUERY);
if(empty($cache)) return $url."?".http_build_query($query);
else return $url."&".http_build_query($query);
}
// test
$test=array("http://example.com/page/","http://example.com/page","http://example.com/?p=page");
print_r(array_map(function($v){
return addQuery($v,array("myquery"=>"string"));
},$test));
Live demo
I'm wondering if there is a standard, library or a common way to do this. I'm using PHP.
Depends on how failsafe – and thereby more complex – you want it to be.
The simplest way would be to look for whether there’s a ? in the URL – if so, append &myquery=string, else append ?myquery=string. This should cover most cases of standards-compliant URLs just fine.
If you want it more complex, you could take the URL apart using parse_url and then parse_str, then add the key myquery with value string to the array the second function returns – and then put it all back together again, using http_build_query for the new query string part.
Some spaghetti Code:
echo addToUrl('http://example.com/page/','myquery', 'string').'<br>';
echo addToUrl('http://example.com/page','myquery', 'string').'<br>';
echo addToUrl('http://example.com/page/wut/?aaa=2','myquery', 'string').'<br>';
echo addToUrl('http://example.com?p=page','myquery', 'string');
function addToUrl($url, $var, $val){
$array = parse_url($url);
if(isset($array['query'])){
parse_str($array['query'], $values);
}
$values[$var] = $val;
unset($array['query']);
$options = '';$c = count($values) - 1;
$i=0;
foreach($values as $k => $v){
if($i == $c){
$options .= $k.'='.$v;
}else{
$options .= $k.'='.$v.'&';
}
$i++;
}
$return = $array['scheme'].'://'.$array['host'].(isset($array['path']) ? $array['path']: '') . '?' . $options;
return $return;
}
Results:
http://example.com/page/?myquery=string
http://example.com/page?myquery=string
http://example.com/page/wut/?aaa=2&myquery=string
http://example.com?p=page&myquery=string
You should try the http_build_query() function, I think that's what you're looking for, and maybe a bit of parse_str(), too.
I have a button on a page with an onClick I dynamically set via Jquery.
That button starts the download of a file.
I'm using mod_rewrite to turn http://somesite.com/blah1/blah2/blah3/blah4 into index.php&q=$1
Then I access q, split on / and I get any user requested path as an array.
Works great except I can't send an encoded path as one of my variables:
<?php
$html = '<button onClick="window.location = \'' . SITE_URL . 'downloadLog/' . urlencode($some_path) . ">Click me</button>';
Looked into AllowEncodedSlashes On for Apache.
Tried it.. Always end up with either %2Fvar%2Fadm%2Fmessages even after urldecode() or rawurldecode(), or apache attempting to go somewhere that doesn't exist even with my mod_rewrite on / converting these to q=.
So.. the new idea is I'll just create a form onClick and submit that. POST will have no problem with my urlencoded path as a parameter.
Since this is destined to go in an onClick, and my onClick= uses double quotes (this is part of another huge piece of code I can't easily change right now) I'm having trouble getting the escaping correct I think. Broke down and didn't use quotes around my id/name as a last resort, still no dice.
So here is what I came up with. This does not error in firebug, but I don't see a POST happening either.
function get_inline_post($a) {
$output .= 'this_form = $(\'<form submit=' . SITE_URL . $a['submit'] . ' ></form>\').html(\'\')';
unset($a['submit']);
foreach ( $a as $key => $value ) {
$output .= '.append(\'<input id=' . $key . ' name=' . $key . ' type=hidden>\')';
}
$output .= ';';
foreach ($a as $key => $value ) {
$output .= '$(\'#' . $key . '\').val(\'' . rawurlencode($value) . '\');';
}
$output .= 'this_form.submit();';
return $output;
}
Which given the right variables ends up building this to go in my onClick:
<button class="ui-corner-all"
onClick="
this_form = $('<form submit=/downloadHostLog/messages></form>').html('').append('<input id=link_hcmdb_id name=link_hcmdb_id type=hidden >').append('<input id=added_path name=added_path type=hidden >');
$('#link_hcmdb_id').val('046345D4771C4D3FBD2EF33CBE038028');$('#added_path').val('var%2Fadm');this_form.submit();
"
title="Kill me now" type="button">
<span class="ui-icon ui-icon-copy"></span>
</button>
Remember,I'm trying to initiate a download here so answers with .post() are useless. Thank you.
Edit:
I'm open to using the original idea of just urlencoding it in the window.location script, but I've wasted all day on it already. Actually made a table and tried every variation of:
single urlencoding, double urlencoding encoding,Different levels of encoding going in versus coming out.
rawurlencode vs urlencode,
AllowEncodedSlashes On/Off
At this point I'm guessing that my combination of using mod_rewrite, splitting on / to generate the original query path as an array, and AllowEncodedSlashes just may not work together.
I'm also eliminating the fact that the buttons are sent to the user via ajax, and this button actually ends up in a jqgrid.
This is part of the reason I want to simply inline the onClick on generation instead of doing
$().click(function(){blah});
for every single result row as every row has 5 different parameters other than what I show here that are unique to every row for security purposes... so I end up calling the bind 50-500 times anyway which was painfully slow.
What I was doing previously was just generating a uid for each row, storing that in a session variable, and dereferencing it back when the user visited /downloadLog/UID. That worked as well, but I like to keep my session information under 4k for performance reasons and doing this took it to 16k easily.
Simply replacing / with | before encoding and changing it back after worked but I refused to resort to building in a bug like that.
EDIT: #dqhendricks My current mod_rewrite statement:
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)? index.php?q=$1 [L,QSA]
The corresponding php that processes that:
foreach ( preg_split('/\//',$_GET['q']) as $key => $value)
{
$g[$key] = urldecode($value);
}
Given the above I tried AllowEncodedSlashes On and Off. Neither one allowed /blah/blah/%2F%var%2Fadmin/blah to be even directed into index.php. Got 404.
Don't quote me on this, but I believe frameworks like Zend just redirect everything to index.php (no GET request variables), then gets the request data from something like $_SERVER['ORIG_PATH_INFO']*. This gives you the request data prior to decoding the url. You can then split the request string, and decode each variable yourself.
*Unsure exactly, but something like this. Would have to look into Zend Framework's routing class to be sure.
EDIT
Dug up the code that gets the request uri from zend framework. Part of the Zend_Controller_Request_Http class, but you get the idea.
public function setRequestUri($requestUri = null)
{
if ($requestUri === null) {
if (isset($_SERVER['HTTP_X_REWRITE_URL'])) { // check this first so IIS will catch
$requestUri = $_SERVER['HTTP_X_REWRITE_URL'];
} elseif (
// IIS7 with URL Rewrite: make sure we get the unencoded url (double slash problem)
isset($_SERVER['IIS_WasUrlRewritten'])
&& $_SERVER['IIS_WasUrlRewritten'] == '1'
&& isset($_SERVER['UNENCODED_URL'])
&& $_SERVER['UNENCODED_URL'] != ''
) {
$requestUri = $_SERVER['UNENCODED_URL'];
} elseif (isset($_SERVER['REQUEST_URI'])) {
$requestUri = $_SERVER['REQUEST_URI'];
// Http proxy reqs setup request uri with scheme and host [and port] + the url path, only use url path
$schemeAndHttpHost = $this->getScheme() . '://' . $this->getHttpHost();
if (strpos($requestUri, $schemeAndHttpHost) === 0) {
$requestUri = substr($requestUri, strlen($schemeAndHttpHost));
}
} elseif (isset($_SERVER['ORIG_PATH_INFO'])) { // IIS 5.0, PHP as CGI
$requestUri = $_SERVER['ORIG_PATH_INFO'];
if (!empty($_SERVER['QUERY_STRING'])) {
$requestUri .= '?' . $_SERVER['QUERY_STRING'];
}
} else {
return $this;
}
} elseif (!is_string($requestUri)) {
return $this;
} else {
// Set GET items, if available
if (false !== ($pos = strpos($requestUri, '?'))) {
// Get key => value pairs and set $_GET
$query = substr($requestUri, $pos + 1);
parse_str($query, $vars);
$this->setQuery($vars);
}
}
$this->_requestUri = $requestUri;
return $this;
}
Let's say I have some code like this
if(isset($_GET['foo']))
//do something
if(isset($_GET['bar']))
//do something else
If a user is at example.com/?foo=abc and clicks on a link to set bar=xyz, I want to easily take them to example.com/?foo=abc&bar=xyz, rather than example.com/?bar=xyz.
I can think of a few very messy ways to do this, but I'm sure there's something cleaner that I don't know about and haven't been able to track down via Google.
Here's one way....
//get passed params
//(you might do some sanitizing at this point)
$params=$_GET;
//morph the params with new values
$params['bar']='xyz';
//build new query string
$query='';
$sep='?';
foreach($params as $name=>$value)
{
$query.=$sep.$name.'='.urlencode($value);
$sep='&';
}
If you are updating the query string you need ot make sure you don't do something like
$qs="a=1&b=2";
$href="$qs&b=4";
$href contains "a=1&b=2&b=4"
What you really want to do is overwrite the current key if you need to .
You can use a function like this. (disclaimer: Off the top of my head, maybe slightly bugged)
function getUpdateQS($key,$value)
{
foreach ($_GET as $k => $v)
{
if ($k != $key)
{
$qs .= "$k=".urlencode($v)."&"
}
else
{
$qs .= "$key=".urlencode($value)."&";
}
}
return $qs
}
View report
Just set the link that changes bar to xyz to also have foo=abc if foo is already set.
$link = ($_GET['foo'] == 'abc') ? 'foo=abc&bar=xyz' : 'bar=xyz';
?>
Click Me
You would have to render out the links with the proper URL querystring to make that happen. This is a design decision that you would need to make on your end depending on how your system is setup.
I have some sites that have this issue, and what I do is setup a querystring global variable that sets the current page data the top of the page request.
Then when I am rendering the page, if I need to make use of the current query string I do something like:
echo '<a href="myurl.php' . querystring . '&bar=foo';
It's not the cleanest, but it all depends on how your system works.
Save some code and use the built-in http_build_query. I use this wrapper in one of my projects:
function to_query_string($array) {
if(is_scalar($array)) $query = trim($array, '? \t\n\r\0\x0B'); // I could split on "&" and "=" do some urlencode-ing here
else $query = http_build_query($array);
return '?'.$query;
}
Also, though it isn't often used, you can have $_GET on the left-hand side of an assignment:
$_GET['overriden_or_new'] = 'new_value';
echo 'Yeah!';
Other than that, just do what Paul Dixon said.