Task: linking the new PDF's with the websites database (phpmyadmin)
link: http://thepeninsulaqatar.com/pdf/the-peninsula-newspaper?page=57
Explanation: I am uploading all the old PDFs to the website's Upload folder (in the httpdocs directory)
Obstacle: How to link all the uploaded PDFs with the phpmyadmin database.
I can always supply any additional information, since I can't really locate the problem from the tables.
What's meant by linking: So I mean here that after uploading the PDFs on their designated folder that are already names on the database, the database automatically auto-detects and when you press the button it prompts you to download the PDFs as in page one from the website I sent. What I mean by phpmyadmin: MySQL database I guess. It's embedded at most web hosting servers.
There is no easy and short answer to this. Three general approaches come into mind. None is better than the other, it depends on your situation which you prefer:
auto detection:
you need some form of script that runs periodically and scans for new files. This is typically done by a cron job, but that again is typically not offered for typical low budget web space. An alternative would be what is called a "poor mans cron" approach. Works, but annoying and complex.
registering during the upload:
a file upload typically is implemented in a script accepting and storing the file. The straight forward approach would be to enhance that script to also register the uploaded file inside the database. This is easy to implement, but you need to modify the existing script.
live scanning the file system:
instead of registering the uploaded files in some database you could also simply live scan the file system each time you want to offer a selection of a file. This obviously is slower than a database query, but it saves you from having to register the files at all, so far easier to implement. And it is very robust, since you have a single point of truth about what files exist under what name and url.
None of the approaches is trivial, but this certainly is not voodoo magic. It's impossible to give final script snippets here, the current state of the question does not offer enough information for this. Especially the existing situation, so existing scripts are missing.
Related
I have seen many questions concerning the storage of user uploaded image files onto a web application, but most of these are dealing with the following:
Indexing of the images, so as to retrieve them later
How to store them (on the server itself as a file or in the database)
I have a question in regards to this subject, but the question is:
In what directory do I put the uploaded image file? (or other file type, for that matter)
I have a small group I am running php apps for. There is very little files that get uploaded, but nontheless, they get uploaded.
I currently have them in my public html document root under /var/www/images/* , however I am told that it is not smart to store your user uploaded content straight to the /var/www/* directory and that it should be stored elsewhere.
However I cannot find a straightforward statement of where "elsewhere" is.
Keep in mind I do not have a server farm where I can establish certain servers for specific purposes (such as uploaded user files).
Therefore, on a single webserver that hosts usual scripting files, etc. what is the best storage practice for such content?
Thank you.
I don't think there's necessarily a 'best practice' per se; anywhere on your server will be fine, so long as you're able to retrieve the images later on. Typically they'd go inside a folder under /var/www/images/.
Personally I'd recommend creating an individual folder to store these user-uploaded images in (as something like /var/www/images/user_uploads), so that they don't get confused with other images you might have uploaded directly to /var/www/images/ (such as backgrounds or core imagery).
I'm facing a dilemma on how to implement file upload and download in a PHP website.
I have these criteria:
Performance - does not give performance issues to the website
File size - around 2GB and up.
Authorization - I want to be able to change who can access the files in PHP. Allow multiple users to gain access to a single file.
User friendly - no additional tools to use.
So here are the methods I'm currently looking at and how I assess them based on my criteria:
Database BLOB
Writing the file data into the output stream will take time and blocks other requests (is this correct?)
I read somewhere that there's a size limit for BLOB.
OK - I can easily control who can download the files here.
OK - No additional tools, just the website.
FTP
OK - since it is designed to store files.
OK - file system is the limit.
I need to create another credentials for each user aside from the username and password for the website. I assume I have to move the file from one location to another to update authorization, but how if multiple users can access one file? Shared directory? It looks messy.
Need another tools/program for accesing their files, need to remember another username and password.
My questions:
Based on my assumptions, do you thimk I understand the methods correctly?
If my assumptions are wrong, is there a way I can do this functionality while meeting my criteria?
PS Please excuse my English.
Why not just use the file system to store the files and store the path to the given file (+ permissions, if needed) in addition in a database.
The upload folder isn't accessible from the public and an wrapper script serves the content to the user.
Performance shouldn't be a problem as you just move/copy the uploaded file to a dedicated data directory.
File size isn't a problem (as long, as you have enough disk space)
The wrapper script handles permissions and serves files to the users
It's as friendly, as you design your ui
for me the usage of BLOB is not the best. I thought about BLOB to upload pictures in my own website, but the best way to upload file is to put them directly on ur server locally.
I'm building a PHP based upload service for some of our clients. I am using SWFUpload so that I can view the progress of a file as it uploads. I've got it pretty much built, but am running into one last issue before we can release it to the public.
Many (almost all) of our clients are Mac-based and are uploading sets of files that include InDesign Files, Fonts, Illustrator Files, etc. Most of the times the images files are OK, but occasionally (and always with Type 1 Fonts) the file will become corrupted because it is losing the resource fork.
I understand why this is happening (moving from a multi-fork system to a single-fork system), but I can not find any elegant solution. In my research the best answer I've found so far is "have the user compress it". I know that works, but it's unreasonable - in our client's opinion - for us to require them to compress every set of files they are going to send.
Are there any better solutions for keeping those resource forks alive? Of course, I would prefer a solution that is straight javascript/php, but would settle for something that is flash based or (least preferably) java based.
My only requirements for the new solution would be:
View upload progress
User doesn't have to manually compress files
Here's some information about my system
Ubuntu 10.10 Server running a standard LAMP install
PHP5
SWFUpload (wtv the most recent version is)
Uploads handle files. If the browser and the underlying OS is not able to deal with forks in this procedure (map anything file onto the file model for uploads), then you're bound to what you get by the systems architecture.
Resource fork: The resource fork is a construct of the Mac OS operating system used to store structured data in a file, alongside unstructured data stored within the data fork. A resource fork stores information in a specific form, such as icons, the shapes of windows, definitions of menus and their contents, and application code (machine code).
If that's a blocker to you you might have chosen the wrong field to work in. Just saying, if you run into systematic borders, there is not much you can do about. Even if you work for graphic designers and mac users.
The swfupload would need a feature to deal with forks. For that, flash would need a feature to deal with forks. For that the browser would eventually need a feature to deal with forks. And so on.
Next to this chain, another question remains: How to deal with forks? As the upload only maps one file to a chunk of binary data, how to map the fork as well? Append it? Add an additional file?
So on the technical level this does not sound like easily solveable. All components and systems in the file input chain must support a feature that is commonly not supported at all.
So as you can not offer something to the user that does not exist, the only thing you can do is make your application more usable or user-friendly. E.g. by providing the right notes at the right time (e.g. when a user selects a Type 1 file for uploading, to remind him/her to select the fork as well). Communicating with the user can help, but keep in mind that a user needs to be spoken with in a language he/she understands.
So if you know that certain file types have forks, address the issue to someone who can solve it: The user. You can't.
You don't have to use swfupload to monitor progress.
Here are some file that demonstrate this: https://github.com/senica/Booger/tree/master/assets/js/jquery-upload
It is not documented very well, but it basically uses webkitSlice function for uploading the files in javascript. You can use the callback functions to display the progress of the files.
This would be a javascript/php solution.
I have a general question about this.
When you have a gallery, sometimes people need to upload 1000's of images at once. Most likely, it would be done through a .zip file. What is the best way to go about uploading this sort of thing to a server. Many times, server have timeouts etc. that need to be accounted for. I am wondering what kinds of things should I be looking out for and what is the best way to handle a large amount of images being uploaded.
I'm guessing that you would allow a user to upload a zip file (assuming the timeout does not effect you), and this zip file is uploaded to a specific directory, lets assume in this case a directory is created for each user in the system. You would then unzip the directory on the server and scan the user's folder for any directories containing .jpg or .png or .gif files (etc.) and then import them into a table accordingly. I'm guessing labeled by folder name.
What kind of server side troubles could I run into?
I'm aware that there may be many issues. Even general ideas would be could so I can then research further. Thanks!
Also, I would be programming in Ruby on Rails but I think this question applies accross any language.
There's no reason why you couldn't handle this kind of thing with a web application. There's a couple of excellent components that would be useful for this:
Uploadify (based on jquery/flash)
plupload (from moxiecode, the tinymce people)
The reason they're useful is that in the first instance, it uses a flash component to handle uploads, so you can select groups of files from the file browser window (assuming no one is going to individually select thousands of images..!), and with plupload, drag and drop is supported too along with more platforms.
Once you've got your interface working, the server side stuff just needs to be able to handle individual uploads, associating them with some kind of user account, and from there it should be pretty straightforward.
With regards to server side issues, that's really a big question, depending on how many people will be using the application at the same time, size of images, any processing that takes place after. Remember, the files are kept in a temporary location while the script is processing them, and either deleted upon completion or copied to a final storage location by your script, so space/memory overheads/timeouts could be an issue.
If the images are massive in size, say raw or tif, then this kind of thing could still work with chunked uploads, but implementing some kind of FTP upload might be easier. Its a bit of a vague question, but should be plenty here to get you going ;)
For those many images it has to be a serious app.. thus giving you the liberty to suggest a piece of software running on the client (something like yahoo mail/picassa does) that will take care of 'managing' (network interruptions/resume support etc) the upload of images.
For the server side, you could process these one at a time (assuming your client is sending them that way)..thus keeping it simple.
take a peek at http://gallery.menalto.com
they have a dozen of methods for uploading pictures into galleries.
You can choose ones which suits you.
Either have a client app, or some Ajax code that sends the images one by one, preventing timeouts. Alternatively if this is not available to the public. FTP still works...
I'd suggest a client application (maybe written in AIR or Titanium) or telling your users what FTP is.
deviantArt.com for example offers FTP as an upload method for paying subscribers and it works really well.
Flickr instead has it's own app for this. The "Flickr Uploadr".
I accept file uploads from users. Each file has a pointer in the db which has info on the file location in the filesystem.
Currently, I'm storing the files in the filesystem non categorically, and each file is currently just named a unique value. All categorisation and naming etc is done in the app using the db.
A factor that I'm concerned about is that of file synchronization issues.
If I wanted to set up file system synchronization where, for example, the user's files are automatically updated by bridging with a pc app, would this system still work well?
I have no idea how such a system would work so hopefully I can get some input.
Basically, is representing a file's name and location purely in the database optimal, especially if said file may be synchronized with a pc application?
Yes, the way you are doing this is the best way to do it. You are using a file system to store files and a database to sore structured data.
One suggestion I would make is that you create a directory tree on the file system. You may one day run up against a maximum files per directory limitation of your file system. I have built systems that create a new sub directory for each day or week.
Make sure you have good backups of the database as well as the document repository.
All you need to make such a system work is to make sure the API you use (or, more likely, create) can talk to the database and to the filesystem in a sensible way. Since this is what your site is already doing anyway, it shoudn't be hard to implement.
The mere fact that your files are given identifiers instead of plain-English names is mostly irrelevant with regard to remote synchronization.
Store a file hash in the database rather than a path (i.e. SHA1) and have a separate database connect the hash with the path. Write a small app that will synchronize the hash database so that when you move your files to a different location it'll be easy to build a new database with updated paths.
That way you can also have the system load the file from a different location depending of which hash database you use to locate the file so it offers some transparency if you need people to be able to access the same file from diverse locations (i.e. nfs or webdav).
We use exactly this model for file storage, along with (shameless plug) SabreDAV to make it seem to the end-user it's a normal filesystem.
I think this is a perfectly fine model, as long as looking up the file is documented and easily retrieved there shouldn't be an issue. Just make backups of your DB :)
One other advice I can give, we use an md5() on the file-id to generate a unique filename. We use parts of the files to generate a directory structure, for example.. id 1 will yield: b026324c6904b2a9cb4b88d6d61c81d1, the resulting filename will become:
b02/632/4c6/904b2a9cb4b88d6d61c81d1 The reason for this is that most stable filesystems can become very slow after a high number of files (or directories) in one directory. It's much, much faster too traverse a few sub-directories.
The Boring Answer™:
I think it depends on what you wanna do, as always :)
I mean take your regular web hosting company. Developers are synching files to web servers all the time. Would it make sense for a web server to store hash-generated file names in a db that pointed to physical files? No. Then you couldn't log in with your FTP-client and upload files like that, and you'd have to code a custom module to get Apache to work etc. Instant headache.
Does it make sense for Flickr to use a db? Yes, absolutely! (Then again, you can't log in with an FTP-client and manage your photos—and that's probably a good thing!)
Just remember, a file system is a (very simple) db too. And it's a db that comes with a lot of useful free tools.
my 2¢
/0