I wish to download a lot of jpegs keeping the original name and their paths.
Example: http://www.somesite.org/path1/image1.jpg, should be downloaded on www.mysite.com/path1/image1.jpg (and creating the "path1" directory if it does not exist) This is repeated a lot of times, fetching the original download site from a field on a database.
Is it possible? Keep in mind that I can't use cURL nor wget since I am on a "limited" hosting service.
Well, first of all you have to find a method to get all download links. There are several ways to fetch data from a database, depending on what database it is (mysql, exel, textfile...)
Then you need to use ftp to upload these files into your database. Strip down the original link to the path (remove the www.somesite.org) and use your site + the path for storing.
This is very manageable, but we are not here to do all the thinking for you. Use google and try methods first, then come back if you are facing a specific problem.
Related
Folks
I have an image at some server (SOURCE)
i.e. http://stagging-school-images.s3.amazonaws.com/2274928daf974332ed4e69fddc7a342e.jpg
Now I want to upload it to somewhere else (DESTINATION)
i.e. example.mysite.com/receiveImage.php
First, I am copying image from source to my local server and then uploading it to destination.
It's perfectly working but taking too much time as it copy the image and then uploads...
I want to make it more simple and optimized by directly uploading image from source URL to destination URL.
Is there a way to handle this ?
I am using php/cURL to handle my current functionality.
Any help would be very much appreciated.
Cheers !!
If example.mysite.com/receiveImage.php is your own service, then you may
pass SOURCE URL to your PHP script as GET or POST parameter
in PHP script, use file_get_contents() function to obtain image by URL, and save it to your storage
Otherwise it's impossible by means of HTTP.
However, there are some ways to increase files uploading speed a little:
If files are huge, you may use two threads: one for downloading (it will store all downloaded data to some buffer) and one for uploading (it will get all available data from buffer and upload it to site). As far as I know, this can't be done easily with PHP, because multi-threading is currently not supported yet.
If there are too many files, you may use many threads / processes, which will do download/upload simultaneously.
By the way, these means do not eliminate double traffic for your intermediate service.
One of the services may have a form somewhere that will allow you to specify a URL to receive from/send to, but there is no generic HTTP mechanism for doing so.
copy($url, $uploadDir.'/'.$fileName);
The only way to transfer the image directly from source to destination is to initiate the transfer from either the source or the destination. You can't magically beam the data between these two locations without them talking to each other directly. If you can SSH login to your mysite.com server you could download the image directly from there. You could also write a script that runs on mysite.com and directly downloads the image from the source.
If that's not possible, the best alternative may be to play around with fread/fwrite instead of curl. This should allow you to read a little bit from the source, then directly upload that bit to the destination so download and upload can work in parallel. For huge files this should make a real difference, for small files on a decent connection it probably won't.
create two textfield one url, other filename
in php, use :
uploadDir is path to your file directory ;)
copy($url, $uploadDir.'/'.$fileName);
I have a flash .swf file that I embed on my webpage. On my server I have the .swf file and multiple image folders. I would like to load every file in one of those folders into the flash slideshow. How should I go about doing this? I tried used Air but it doesn't work on my system as an application so I doubt it will work online. Eventually I plan on making a menu where you can select different folders to display and since they are of different sizes, a foreach loop would be optimal. Keeping a txt file with the number of images is also possible if theres a way to read that in, but I would prefer the more dynamic approach. I am working towards using php for the website if that helps find a solution.
Thanks,
-Mike
Also my slideshow works great online currently but i have to hardcode in the number of files.
I would suggest to have a PHP script on your server that takes care of parsing those folders, and return the list of files to Flash (with a valid public URL).
Basically at your application startup, you would call the PHP script to retrieve the full list of file (XML is a good format to be returned, or AMF if you have a lot of folders/files).
After all you have to do is manipulate that data to load whatever folder/files the user is willing to see.
Just for your information, Flash doesn't have access to the Filesystem, so it's impossible to parse folders directly from Flash. (However It is possible with an Air Application)
we are creating a website for hotel booking. we need to store a large number of images. we think it would be a better option to store images in the filesystem and store the path in the database. But do we have to manually save them? We are using web services from another website to get the images. is there a way to save the images dynamically in the filesystem??
You can use PHP's file get contents function or CURL to download all the images you want to the disk or simply refering the foreign image to your clients and you won't need to store them locally on the server.
If you like Python check the Mecanize lib and BeautifulSoup to parse XML if you need.
Storing in disk vs storing in database has it's beneficts. If you need to scale, it's easier and you can always have a lighttpd or a nginx http servers dedicated to images or simply put it out on other server to balance bandwidth.
It depends on the database, and how you are serving up the images. In general it is better to save the images to disk, depending on how you are delivering them to the client.
Getting the images is usually a matter of some process on the server downloading them from websites and saving them. On many systems you could use wget or curl to download the images and save them.
It also depends on how you are getting the data. If it is some inline binary via XML or something, then you will need to extract that using the features of your application language, and save it to disk.
The mechanics of how to do that vary wildly depending on the implementation language and the hosting operating system.
I think that you should store both remote (web service) and local (filesystem) location in the database, with initial file system location blank. If a user requests an image for the first time, download it, update the file field and show it. With this concept you will only have images your clients need.
I accept file uploads from users. Each file has a pointer in the db which has info on the file location in the filesystem.
Currently, I'm storing the files in the filesystem non categorically, and each file is currently just named a unique value. All categorisation and naming etc is done in the app using the db.
A factor that I'm concerned about is that of file synchronization issues.
If I wanted to set up file system synchronization where, for example, the user's files are automatically updated by bridging with a pc app, would this system still work well?
I have no idea how such a system would work so hopefully I can get some input.
Basically, is representing a file's name and location purely in the database optimal, especially if said file may be synchronized with a pc application?
Yes, the way you are doing this is the best way to do it. You are using a file system to store files and a database to sore structured data.
One suggestion I would make is that you create a directory tree on the file system. You may one day run up against a maximum files per directory limitation of your file system. I have built systems that create a new sub directory for each day or week.
Make sure you have good backups of the database as well as the document repository.
All you need to make such a system work is to make sure the API you use (or, more likely, create) can talk to the database and to the filesystem in a sensible way. Since this is what your site is already doing anyway, it shoudn't be hard to implement.
The mere fact that your files are given identifiers instead of plain-English names is mostly irrelevant with regard to remote synchronization.
Store a file hash in the database rather than a path (i.e. SHA1) and have a separate database connect the hash with the path. Write a small app that will synchronize the hash database so that when you move your files to a different location it'll be easy to build a new database with updated paths.
That way you can also have the system load the file from a different location depending of which hash database you use to locate the file so it offers some transparency if you need people to be able to access the same file from diverse locations (i.e. nfs or webdav).
We use exactly this model for file storage, along with (shameless plug) SabreDAV to make it seem to the end-user it's a normal filesystem.
I think this is a perfectly fine model, as long as looking up the file is documented and easily retrieved there shouldn't be an issue. Just make backups of your DB :)
One other advice I can give, we use an md5() on the file-id to generate a unique filename. We use parts of the files to generate a directory structure, for example.. id 1 will yield: b026324c6904b2a9cb4b88d6d61c81d1, the resulting filename will become:
b02/632/4c6/904b2a9cb4b88d6d61c81d1 The reason for this is that most stable filesystems can become very slow after a high number of files (or directories) in one directory. It's much, much faster too traverse a few sub-directories.
The Boring Answer™:
I think it depends on what you wanna do, as always :)
I mean take your regular web hosting company. Developers are synching files to web servers all the time. Would it make sense for a web server to store hash-generated file names in a db that pointed to physical files? No. Then you couldn't log in with your FTP-client and upload files like that, and you'd have to code a custom module to get Apache to work etc. Instant headache.
Does it make sense for Flickr to use a db? Yes, absolutely! (Then again, you can't log in with an FTP-client and manage your photos—and that's probably a good thing!)
Just remember, a file system is a (very simple) db too. And it's a db that comes with a lot of useful free tools.
my 2¢
/0
A while a go I had to developed a music site that allowed audio files to be uploaded to a site and then converted in to various formats using ffmpeg, people would then download the uploaded audio files after purchasing them and a tmp file would be created and placed at the download location and was only valid for each download instance and the tmp file would then get deleted.
Now I am revisiting the project, I have to add pictures and video as upload content also.
I want to find the best method for storing the files,
option 1 : storing the files in a folder and reference them in the database
option 2 : storing the actual file in the database(mysql) as blob.
I am toying around with this idea to consider the security implications of each method, and other issues I might have not calculated for.
See this earlier StackOverflow question Storing images in a database, Yea or nay?.
I know you mentioned images and video, however this question has relevance to all large binary content media files.
The consensus seems to be that storing file paths to the images on the filesystem, rather then the actual images is the way to go.
I would recommend storing as files and storing their locations in the database.
Storage the files in a database requires more resources and makes backing up/restoring databases slower.
Do you really want to have to transfer lots of videos every time you do a database dump?
File systems work very well for dishing out files, and you can back them up/sync them very easily.
I would go for the database option. I've used it on a number of projects, some very larger 100+GB. The storage implementation is key, design it poorly and your performance will be punished. See this example for some good implementation ideas:
Database storage allows more scalability and security.
I would go for storing files directly on the disk, and database holding only their ID/url.
This way accessing those files (that can be large, binary files) doesnt require any php/database operation, and it's done by the webserver directly.
Also it will be easier to move those files to another server if you'd want to.
Actually only one upside I can see atm of storing them in database is easier backup - you wanna backup your DB anyway, this way you'll have all data in one place and you can be sure that each backup is full (i.e. you don't have files on disk that aren't used by database entries; and you don't have image IDs in your database that point to nowhere)
I asked a similar question using Oracle as the backend for a Windows Forms application.
The answer really boils down to your requirements for backing up and restoring the files. If that requirement is important then use the database as it'll be easier (as you're backing up the database anyway, right? :o)