I've created a rather large Business Intelligence System over the years. It already offers vCard export of customers and contacts. But this of course is a tedious process, where each employee has to manually import the vCard file on their phones.
So I was reading about CardDAV and thought this might be a better solution. However, all I really need is to be able to provide a read-only source for contacts. It should not be possible for anyone to make changes to the contacts (well, except temporarily in their own phonebook - until next synchronization happens). And all other functionality isn't interesting either. I only need the "synchronize contacts from BI to phones" part.
I was hoping it would be simple. Something along the lines of just using the url to the vCard generated file (or PHP file that generates it). But I can see this question has been asked a few times before, and no one has given any answers, so I guess it's not as simple as that.
Can anyone share some light on this? Is it possible to just provide a simple read-only url that is compatible with the CardDAV protocol?
And if not - are there then some other protocol that supports something like that?
It isn't possible with a single endpoint URL, but it isn't super complicated either. To make it read only, you'll reject PUTs w/ a "403 Forbidden" and optionally also add the relevant WebDAV permission properties (though many clients might ignore those).
You'll need:
One endpoint for the CardDAV principal record, this represents the user accessing your system and points the client to the "CardDAV home". It is a simple XML document in response to a PROPFIND.
One endpoint for the CardDAV "home", this is a WebDAV collection that contains the "contact folders" you expose, quite likely just one. This is also a simple XML, again hit with a PROPFIND.
One endpoint representing the CardDAV contacts folder, this is the thing pointing to the actual vCard resources. An XML that lists the URLs for the contained vCards, again hit with a PROPFIND.
Well, and one endpoint for each vCard, this is queried w/ a GET. (and if you want to allow creation/modification/deletion, w/ a PUT or DELETE)
This is a great resource on the protocol: https://sabre.io/dav/carddav/
Another option might be LDAP, but that's a little more complicated than CardDAV. (You could use openldap to serve the protocol, and fill it using LDIF files).
Related
I have created a site, that uses 1 code base, but multiple domains access that codebase.
Now the content served-up shows different CSS and imagery.
My question i'm running into, is, how do I generate a Sitemap file for each domain.
I have looked at using, http://www.xml-sitemaps.com/ and using their Script, but that will only work for 1 domain.
Other than creating my own code to do the site scraping, I don't see any other route. Do you know of another solution instead of having to start from scratch? Ideally would love to hit the ground running.
Note: The script needs to crawl the site. Thoughts?
Creating Mutliple sitemaps for single codebase something challanging job but not the impossible one. I am supposing that you are using some kind of framework for working of website.
Theer are many problem which comes during creating such thing are:
How to identify which request is coming from which website. So, the problem is to create the sitemap for specific site for the request is recieved.
Somehow, if you identify the which request is coming from which website then your website is dynamic. How to record these paramenters.
Where to store the such a huge database. Somehow if you resolve these things the mutlitple sites requests/parameters which database would be large enough to store such large requests.
If you somehow manage the huge database then next problem about submitting such a huge xml to search engine.
Sitemap would start growing daily the time for creation would certainly go up and so the request of crawling from website would also grow daily.
If your sitemap grows huge and same pages are submitted for different website then content would be marked as spam along with website.
There are some problems which can be unseen or predicted and thus it will be risky thing. Now to do it.
solutions
For problem 1st and 2nd we have to use the PHP $_SERVER - which provides the information about Server and execution environment information, such as parameters, hostname, requested host and many other things.
Now for problem 3 to 6, we have use the text files to store the requests one file for one domain and provide request details. The file must be flushed after particular time i.e. daily, weekly etc.
While creating sitemap we have to read the file and get the unqiue parameters, so that the sitemap doesn't include multiple same urls.
warning: It is highly recommended that don't do such thing as it will trigger the spamming and would be identified soon and marked as spammers websites.
Assumptions built into this answer:
The URI elements trailing the domain name are the same on each domain, for all pages.
i.e. http://site-one.com/page/1 is the same as http://site-two.com/page/1
You are able to manipulate the file provided by xml-sitemaps. This is a concern if you need to generate this on a continuous basis, meaning you would need to create a script to do the following per href.
If you don't mind using the service you mentioned at http://www.xml-sitemaps.com and , then by far the easiest way to do this would be to use that service and then change all absolute URLs to relative URLs. You can write any link that looks like
http://www.example.com/category/page
as a relative link at
/category/page
In short, that starting slash is the key, indicating to the browser to 'use the current domain'. You can do a find and replace on all all instances of http://www.example.com/, converting to / + the remaining string of URI elements.
I have a client who has a feed of leads which have Name, IP, Address, OptIn Time/Date, and I want them to be able to post the data to my hosted SQL database. If you are familiar with lead generation you will get what Im trying to do.
Id also like to know if its possible to write a script and place it on my server so that when someone posts a CSV file to it I can have the script automatically post the data in the CSV to the SQL server.
Is this possible? And are there any tutorials our reference manuals, sources, etc. I can use to accomplish this?
The answer to your question is Yes.
You can go about this two ways:
Write an API for your database which is consumed by those wishing to search/write/query your database. To do this, you can use any language that you are comfortable with. PHP, XML and Python are not interchangeable. XML is a format specification, it describes what the data should look like when its being transported between two systems. So you can use any programming language that provides XML libraries to write your code. In addition to XML, JSON has emerged as the more popular transport format especially for mobile and web applications.
The second option is to use a service like apigee, google cloud endpoints and mashery which will automate a lot of this process for you. Each requires its own amount of effort (with google cloud endpoints perhaps requiring the most effort). For example apigee will automatically create an API for you as long as you can provide it access to your data source.
I am a newbie onto GTFS and found some information on my research that said one has to provide the GTFS feed in txt formats to obtain the routes/transit information, etc.
Now my question is,
1) If we create our own txt formats, and upload onto the GTFS feed provider, would it result in showing up on the google maps as well?
2) I would like to have my own GTFS server code which will take data from my DB and process it and provide the best transit routes. Is it possible? Assume I have the ability to run Python as well as PHP scripts.
Any help would be best appreciated!
Thanks in advance
No, since you would need to enter into an agreement with Google for them to use your data, and they're unlikely to take you seriously unless you're affiliated with an actual transit agency. But if you're curious you can read about the steps involved.
Yes, it's possible, and there are open-source routing engines available for you to use, like OpenTripPlanner and Graphserver. This is pretty heavy-duty stuff, however. If what you have is a basic Web-hosting account and you just want to do "something interesting" with transit data, setting up an online trip planner is probably not the place to start.
I think the most straightforward solution would be for you to run OpenTripPlanner on a server of your own. This would provide your users with a familiar-looking website they can use to generate trip plans from your data while leaving you complete control over the data itself.
Note that running OpenTripPlanner would require a fairly powerful server plus map data from OpenStreetMap (which I'm assuming is available for your area) in addition to your own transit data. On the project's website you'll find setup instructions for Ubuntu to give you an idea of what's involved.
I'm assuming you're already able to generate a GTFS bundle; that is, to produce a ZIP file containing comma-separated data files as specified in the GTFS Reference. With an OpenTripPlanner server set up, your workflow would be as simple as
Making changes to your transit data.
Generating a new GTFS bundle.
Uploading the bundle to a specific folder on your OpenTripPlanner server.
Restarting OpenTripPlanner.
Optionally, notifying your users of the changes.
Every step except the first could be automated with a script.
In response to your first question, Google needs to be informed of the transit feed. Here's the latest link from Google to get you started [https://support.google.com/transitpartners/answer/1106422]. They also require confirmation from an authorised representative of the transit agency that this is an authorised GTFS feed. I should note that txt formats is not strictly correct. The file you need to create is a GTFS file (General Transit Feed Specification). In essence this is a zipped file of mandatory and optional txt files, in CSV format. For you to create the GTFS file, you'll need to create the multiple files based on a detailed understanding of GTFS or use a GTFS Editor/GTFS API like what can be found at AddTransit.
2) You can install routing software on your own servers. However, if Google is using your GTFS data, then another alternative is to create a simple form on your website for customers to enter their from & to locations. You can then use Google's Transit options on Maps to return the proposed route. Here's a simple example that you could extend to meet your needs: https://addtransit.com/blog/2016/01/add-google-maps-public-transport-directions/
I am writing a set of HTML based resources, stored in a mysql db on our server. The resources contain images references, which are stored as a relative paths.
I can login to an admin panel and create and edit resources very easily.
This bit is all done, and working well.
However, we are going to want to provide this resource in two ways: packaged and hosted. The one that is providing real problems is the hosted solution:
We want to host the database and all image resources, however we want to give access to the resources via a set of templates for other users to host on their own site. This is so they can brand the templates accordingly, and have the resource available at their own URL e.g.
http://www.example.com/discojoe
There are two questions I have on this process, which are causing me real headaches:
• I realise the obvious security implication of providing even read-only access to our mysql server. The only alternative I think of is some server side code running on our server, which when requested, squirts the requested data back to the user. This removes the need for them to have any mysql connection.
Are there any examples of the above type of scenario online I can read up on, or can you give some pointers on how I would go about addressing this solution?
• For users whom we stream the data to (if the above solution is sensible), how could I go about updating the image SRC value on the fly within the HTML content. Whilst I want it stored as a relative URL in the database, when I retrieve the data from the database, I want to convert all image srcs from relative to absolute, with the absolute URL that I specify.
I realise the obvious security
implication of providing even
read-only access to our mysql server.
The only alternative I think of is
some server side code running on our
server, which when requested, squirts
the requested data back to the user.
This removes the need for them to have
any mysql connection.
You could create an REST API(I would return JSON) using predefined queries with PDO prepared statements(safe against SQL-injections). With a little bit of care you could make it pretty safe. Ofcourse if the resources should be protected, you must also add authentication to your system using simple API keys for example. I think you could generate these key easily the same way you prevent CSRF($token = md5(uniqid(rand(), TRUE));). Maybe you should add a little bit more entropy, but I think this is going to be sufficient. But if you want to really do it correctly you should use oauth instead.
with a little bit of mod_rewriting you could write pretty URLs.
For users whom we stream the data to
(if the above solution is sensible),
how could I go about updating the
image SRC value on the fly within the
HTML content. Whilst I want it stored
as a relative URL in the database,
when I retrieve the data from the
database, I want to convert all image
srcs from relative to absolute, with
the absolute URL that I specify.
I think you could use any of the many available template languages to achieve this. Even jquery has one built-in
Create a REST-style web service. That is, set up an HTTP server that responds to data requests by using some server code to load up your templates, alter the URLs and other things (relative to absolute), and sends it to the client as fragments of HTML (or even CSS).
Your user, running on another web server, can use an HTTP client package to consume your web service, incorporate the resulting code fragments into her page, and send it out.
Alternatively you could build your code fragments so they function in iframe objects. In that case your user would build her code to deliver iframe objects to her end-users with references to your server in them.
Finally, your web service could deliver XML or JSON, and be consumed by AJAX-style javacscript in the end-user's browsers.
You are absolutely right to prevent direct access to your mySQL table server from random clients.
For example I need to grab from http://gmail.com/ the number of free storage:
Over <span id=quota>2757.272164</span> megabytes (and counting) of free storage.
And then store those numbers in a MySql database.
The number, as you can see, is dynamically changing.
Is there a way i can setup a server side script that will be grabbing that number, every time it changes, and saving it to database?
Thanks.
Since Gmail doesn't provide any API to get this information, it sounds like you want to do some web scraping.
Web scraping (also called Web
harvesting or Web data extraction) is
a computer software technique of
extracting information from websites
There are numerous ways of doing this, as mentioned in the wikipedia article linked before:
Human copy-and-paste: Sometimes even
the best Web-scraping technology can
not replace human’s manual examination
and copy-and-paste, and sometimes this
may be the only workable solution when
the websites for scraping explicitly
setup barriers to prevent machine
automation.
Text grepping and regular expression
matching: A simple yet powerful
approach to extract information from
Web pages can be based on the UNIX
grep command or regular expression
matching facilities of programming
languages (for instance Perl or
Python).
HTTP programming: Static and dynamic
Web pages can be retrieved by posting
HTTP requests to the remote Web server
using socket programming.
DOM parsing: By embedding a
full-fledged Web browser, such as the
Internet Explorer or the Mozilla Web
browser control, programs can retrieve
the dynamic contents generated by
client side scripts. These Web browser
controls also parse Web pages into a
DOM tree, based on which programs can
retrieve parts of the Web pages.
HTML parsers: Some semi-structured
data query languages, such as the XML
query language (XQL) and the
hyper-text query language (HTQL), can
be used to parse HTML pages and to
retrieve and transform Web content.
Web-scraping software: There are many
Web-scraping software available that
can be used to customize Web-scraping
solutions. These software may provide
a Web recording interface that removes
the necessity to manually write
Web-scraping codes, or some scripting
functions that can be used to extract
and transform Web content, and
database interfaces that can store the
scraped data in local databases.
Semantic annotation recognizing: The
Web pages may embrace metadata or
semantic markups/annotations which can
be made use of to locate specific data
snippets. If the annotations are
embedded in the pages, as Microformat
does, this technique can be viewed as
a special case of DOM parsing. In
another case, the annotations,
organized into a semantic layer2,
are stored and managed separated to
the Web pages, so the Web scrapers can
retrieve data schema and instructions
from this layer before scraping the
pages.
And before I continue, please keep in mind the legal implications of all this. I don't know if it's compliant with gmail's terms and I would recommend checking them before moving forward. You might also end up being blacklisted or encounter other issues like this.
All that being said, I'd say that in your case you need some kind of spider and DOM parser to log into gmail and find the data you want. The choice of this tool will depend on your technology stack.
As a ruby dev, I like using Mechanize and nokogiri. Using PHP you could take a look at solutions like Sphider.
Initially I thought it was not possible thinking that the number was initialized by javascript.
But if you switch off javascript the number is there in the span tag and probably a javascript function increases it at a regular interval.
So, you can use curl, fopen, etc. to read the contents from the url and then you can parse the contents looking for this value to store it on the datanase. And set this up a cron job to do it on a regular basis.
There are many references on how to do this. Including SO. If you get stuck then just open another question.
Warning: Google have ways of finding out if their apps are being scraped and they will block your IP for a certain period of time. Read the google small print. It's happened to me.
One way I can see you doing this (which may not be the most efficient way) is to use PHP and YQL (From Yahoo!). With YQL, you can specify the webpage (www.gmail.com) and the XPATH to get you the value inside the span tag. It's essentially web-scraping but YQL provides you with a nice way to do it using maybe 4-5 lines of code.
You can wrap this whole thing inside a function that gets called every x seconds, or whatever time period you are looking for.
Leaving aside the legality issues in this particular case, I would suggest the following:
Trying to attack something impossible, stop and think where the impossibility comes from, and whether you chose the correct way.
Do you really think that someone in his mind would issue a new http connection or even worse hold an open comet connection to look if the common storage has grown? For an anonimous user? Just look and find a function that computes a value based on some init value and the current time.