I'm trying to figure out a way to pull all tweets of a specific search term via PHP and the Twitter search api.
So functionality would include
1. Include a search term
2. Pull terms from each page.
3. Only pull new terms from last search
4. Export to a db or a flat file.
I'm pretty clear on all of these except for traversing across multiple pages
The twitter API takes a page number parameter. In the atom results, there are link elements, with rel attributes for next and previous. This will be your best indicator as to whether you should go looking for a 2nd page and so on. The href attribute of that tag will even tell you the URL you should request.
The query you create also takes a since_id parameter. You'll want to store the largest id number you see in your responses and use it in subsequent requests so that you don't have to filter duplicates.
As for data storage, your selection is probably best guided by what you plan to do with the results... if you're going to be doing any querying, you should probably file it away in a database, i.e. MySQL. If you're just logging, flat file should do you fine.
The search API has a page parameter:
page: Optional. The page number (starting at 1) to return, up to a max of roughly 1500 results (based on rpp * page. Note: there are pagination limits.
Example: http://search.twitter.com/search.atom?q=devo&rpp=15&page=2
Does that help?
The twitter search api takes a page parameter:
# page: Optional. The page number (starting at 1) to return, up to a max
of roughly 1500 results (based on rpp
* page. Note: there are pagination limits.
* Example: http://search.twitter.com/search.atom?q=devo&rpp=15&page=2
Related
On my WP site I have a custom post type, and have modified the loop so that the results display as markers on a Google Map. I would like these results searchable.
To do this I need a second search page template, (so that I can display the search results on the map). Getting a custom post type search is easy - but I can't find any hint of whether I can create a second site search template. Is there a way to, say, create a second search page that uses "/x?=" instead of "/s?="...
I'm simply trying to scope out whether I'm wasting my time looking for a solution - does anyone have an idea whether this is even possible?!
There is a similar question here Best way to implement <next>, <prev> element links from search list
But I'm not satisfied with that answer, and my case is slightly different, and I want to know the best way more commonly.
I have a search page, let's say, searching poems. (currently the data is from mysql, but I'm planning to index them into apache solr)
Each result on the search result page (let's say, /search?keyword=flower&author=john) is a link, which you can click and will be redirected to the poem detail page.
(let's say, /poem/1234/This-is-the-best-poem)
Now I want "prev" and "next" links on the detail page if the page is clicked from a search result page. and it uses the same order of the search results.
What I can think is to add full search parameters to the detail page, so the "detail page with pagination" actually is a search results but perpage = 1.
But I have to do the full query everytime showing a poem detail page.
I wonder is there a better way ?
Thanks and apologize for my bad English.
As for the Solr part of this, you can use DeepPaging, take a look here:
http://heliosearch.org/solr/paging-and-deep-paging/
You get a 'cursorMark' from Solr when you use DeepPaging which works
like sort of a book mark for your next query results.
You will have to requery, but you will stay in the same result set using that flag,
so when you do this only for the next document you will have to get only one row,
remember your current position where to start,
save the cursor mark and repeat all of that for the next.
By remembering previous cursorMarks you can also go back.
Regards,
Markus
You have to get the poem from the db for the detail. But the rest of the parameters can be passed on from the list page, including a list (array) of poem IDs retrieved in the paged list. You could use this until you reach the end of list, to retrieve the next page list.
You might also want to consider caching options available on the server. APC extension is common on PHP.
I'm trying to get all the favorites of a certain user.
I used /users/{id}/favorites, but the returned list always has the last 50 favorited tracks, and I couldn't find a "next_href" or anything to get the rest.
Is there any way to get the whole list? Or is it just me who made something wrong and it's working perfect for everybody?
you need to make batch page requests with offset=0,50,100,etc parameter in the url. and join the results when all pages (tracks) are loaded.
I'm building a CRUD application and need to implement two forms of search:
The basic search function accepts a keyword and searches every column in the database table for that particular keyword. The search keyword is sent as an URL-parameter in the form www.website.com/category?q=keyword. Nothing too special here...
In the advanced search form users can specify up to 5 keywords and for each of those they can select a column from the database table which needs to be searched for this keyword. In short: the advanced search form takes up to 5 keyword-column pairs. This form gets submitted via POST (because i want to avoid hitting limitations on maximum number of URL-characters).
The results need to be paginated, showing 10 records per page. I've searched for pagination classes online, but they all fetch the total amount of records in the database table, and then return the selection that needs to be displayed for that page. Because I'm working with a large set of data here, I can't afford to do it this way.
This lead me to create my own pagination class, which takes the requested page number (and if applied, the $_GET and $_POST parameters from the search forms). Based on that it calculates the total number of results, calculates the total number of pages, decides whether there's a 'previous' and 'next' page... It also returns an SQL statement (with LIMIT and OFFSET values) to be executed by the particular Model (using my own MVC architecture (aka no framework)). This allowed me to only fetch the results for the requested page.
Problem: when the user navigates through the pages (aka clicking 'previous' or 'next' buttons), the $_POST data is lost. However, these values are needed in order to get the results for another page.
This particular problem made me think about the search forms on forums. I submitted a query on one of them and noticed a search ID getting appended to the URL:
example.com/forum/search.php?searchid=5672532
This means that the form data is being kept somewhere, somehow... and I'm thinking what the best option might be, from most to least plausible:
- Database table where each search input gets saved for e.g. 30 minutes
- Sessions
- Files
(hidden fields are obviously not an option, because they need a form submit button to be pressed in order to get sent with the request)
I'd like to here your opinions on this, some of you must have encountered this problem already...
Saving the search parameters in JSON or URL encoded in a database for a set amount of time (have an expiry column) seems the best solution.
Note that in that way, you'll need to query the database over and over, with different LIMIT clauses.
I'm working for a client who asks me to prepare a module for its website (written using Yii) with the following features:
It shows 3 elements from a mysql_result, starting from offset=0.
The user can click on any of them to mark them as "read". That makes the desired item to disappear and to appear what would be the following item. That is, if you are showing items 3 4 5 and you click on 4, that item would disappear and appear the number 6, meaning the result would be 3 5 6.
The elements being shown are partial views and the button that "deletes" the item is a widget within each partial view.
The user can move through the list of items using some << and >> arrows to go back and forth the result.
One of the options the client gave to me was to show a list of pages ( 1 | 2 | 3 | 4... and so), but deleting items means the number of pages will decrease at some time in the future, plus if the amount is pretty big, I would need something more flexible such as only showing the current and other 4 pages at most. Another option is to keep the << and >> arrows.
I've tried to convince the client that pagination and a "live list" is a pretty bad idea, but he rejected the idea to limit the visualization to only the first 3 items (keeping in mind that eventually you will delete them and thus will be able to see the following items).
I'm developing it using Yii, MySQL and jQuery, and I'm not able to use CPagination because of this living list. I'm not asking the code, just some guidelines because I got lost the third time I tried doing it.
Some basics about the system:
I got a controller which loads the first 3 items of this module.
I got some actions in this controller that fetches the next item within a page (which may not be the same of the current object, though. One of my problems resides here), and the full page.
Each item is able to mark itself as "read", which will make the item not appearing the next time you fetch some results.
Every 1 second I check for items that have been marked as read, remove from DOM and append some new items using the action I defined in the second bullet.
Every time the user hits the << or >>, I reload the previous/next page (That would not be a problem apparently. If you're in the last page and there aren't more items to add, you just remain there. However if you empty the page, I don't have any method to detect that and scroll one page back).
As you may see, this headache would be easier without the pagination buttons, but the client obligues me to put them. What would you do guys? Thanks in advance
EDIT: The client decided to get the results in a random flavor. There is no more pagination, so the problem has disappeared. The #thaddeusmt answer may not have helped me really much, but I'll give it as valid, as it might be plenty useful for other people with similar problems than mine. Cheers
It seems to me like the CGridView or CListView should basically do this automatically. They support AJAX updating/paging out-of-the-box.
I assume that you have an AJAX action that like "actionMarkRead()" which you are calling when the user clicks. I assume that this sets some database field somewhere saying that the user has "Read" that item. To make this work with the CListView, just make sure that the CDataProvider has a condition which checks that "read" field (might have to JOIN a table, I don't know what your DB looks like). Then, when the list reloads via AJAX, it will have the correct # of pages to represent the smaller number of pages the CDataProvier query is returning.
I just tested this and it works!
The way I tested it is I set up a CGridView with 'ajaxUpdate'=>true,. Then in my CDataProvider I set 'pagination'=>1 to make it easy to test. Then I used the default AJAX actionDelete in my Controller to delete the items. Every time I deleted an item via that AJAX action link in the CGridView, the grid refreshed via AJAX and page count shrunk by 1. Seems to work like a charm!
Cheers and good luck!