Check for Valid/Active URLs before download - Laravel 7 - php

Question

I'm trying to check download images and store locally in my assets folder.
Before start downloading, I want to check and make sure the link is still live.
I only want to start my download if the link is live 200 Ok.
Try #1
public function handle()
{
$skills = Skill::all();
if($skills != null){
foreach($skills as $i=>$skill){
if (strpos($skill->img_path, 'http') !== false) {
if(!isset($exception)) {
//update the path in DB
$image_path = '/assets/fe/img/skill/';
$img_name = $skill->name.'.png';
$path = public_path() . $image_path . $img_name;
$uploadSuccess = file_put_contents($path, file_get_contents($skill->img_path));
// dd($uploadSuccess);
if($uploadSuccess) {
$skill->img_path = $image_path . $img_name;
}
}
}
$skill->save();
}
}
}
I seems to get so many issues
One of them is
curl: (6) Could not resolve host: thumbsplus.tutsplus.com
Another one is
file_get_contents(https://assets-cdn.github.com/images/modules/logos_page/Octocat.png): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found
What is the cleaner way ? Please suggest, I will try now.
Try #2
public function handle()
{
$skills = Skill::all();
if($skills != null){
foreach($skills as $i=>$skill){
if (strpos($skill->img_path, 'http') !== false) {
$file_headers = #get_headers($skill->img_path);
if(!$file_headers || $file_headers[0] == 'HTTP/1.1 404 Not Found') {
$exists = false;
}
else {
$exists = true;
if(!isset($exception)) {
//update the path in DB
$image_path = '/assets/fe/img/skill/';
$img_name = $skill->name.'.png';
$path = public_path() . $image_path . $img_name;
$uploadSuccess = file_put_contents($path, file_get_contents($skill->img_path));
// dd($uploadSuccess);
if($uploadSuccess) {
$skill->img_path = $image_path . $img_name;
}
}
}
}
$skill->save();
}
}
}
Try #3
public function handle()
{
$skills = Skill::all();
$failCount = 0;
$successCount = 0;
$failList = [];
if($skills != null){
foreach($skills as $i=>$skill){
if (strpos($skill->img_path, 'http') !== false) {
$file_headers = #get_headers($skill->img_path);
if(!$file_headers || strpos($file_headers[0], '404') !== false) {
$exists = false;
$failCount++;
array_push($failList,$skill->name);
// break;
}
else {
$exists = true;
$successCount++;
//DEBUG
// dd($file_headers[0]);
if(strpos($file_headers[0], '200')) {
//update the path in DB
$image_path = '/assets/fe/img/skill/';
$img_name = $skill->name.'.png';
$path = public_path() . $image_path . $img_name;
$uploadSuccess = file_put_contents($path, file_get_contents($skill->img_path));
// dd($uploadSuccess);
if($uploadSuccess) {
$skill->img_path = $image_path . $img_name;
}
}
}
}
$skill->save();
echo ".";
}
}
echo "\r\n";
$this->info('=========================');
$this->info('Success :'. $successCount);
$this->info('=========================');
$this->info('Fail :'. $failCount);
$this->info('List :'. print_r($failList));
$this->info('=========================');
}
seems to work
but it hang, sometimes more than 1 minute, at a certain dot
⚡️ php artisan skillIcons:download
..............................................................................................................................
=========================
Success :12
=========================
Fail :7
Array
(
[0] => GitHub
[1] => Geolocation API
[2] => Xcode
[3] => Protractor
[4] => Sketch
[5] => Amazon ECR
[6] => WinSCP
)
List :1
=========================
All images seems to be downloaded successfully
⚡️ ls public/assets/fe/img/skill/
AWS Console.png Digital Ocean.png Javascript.png PayPal.png Terminal.png
AWS.png Disqus.png Jest.png Photoshop.png TextMate.png
Alimofire.png Divvy.png Jira.png Pod.png TextWrangler.png
Amazon ECS.png Docker.png Kamar.png PostgreSQL.png Transmit.png
Amazon RDS.png Duet.png LESS.png PyCharm.png Twitter.png
Angular.png EC2.png Laravel Elixir.png Python.png Ubuntu.png
AngularJS.png Evernote .png Laravel.png QuickBooks.png VMWare Fusion .png
Apache.png Express.png Linode.png React Native.png VS Code.png
Atom.png Facebook.png Mac OS X.png Realm.png Vagrant.png
Bash.png Final Cut.png Markdown.png Redis.png Virtual Machine.png
BitBucket.png FusionCharts.png MobaXTerm.png RequireJS.png Virtualbox.png
Bower.png GitLab.png Mocha.png S3.png Webpack.png
CKEditor.png Go Daddy.png MySQL.png SAML 2.0.png Windows.png
CSS.png Google Chart.png NPM.png Salesforce.png Wireshark.png
Camtasia.png Google Map.png Navicat Premium.png Sass.png Word.png
Cent OS.png Google Translation.png Nginx.png Secure Shell.png Yarn.png
Chai.png Gulp.png Node.png Selenium.png iMovie.png
Chat.io.png HTML.png Noteability.png Shopify.png iOS.png
Coda.png Heroku.png OAuth 2.0.png SinonJS.png jQuery.png
CodeBox.png Illustrator.png Open Stack.png Siteground.png
Composer .png Instagram.png OpenID Connect.png Sublime Text.png
Confluence .png J Player.png PHP.png Swagger.png
3 seconds
How do I decrease the wait time to only 3 seconds ?

Answer 1

You have to handle error in some way.
You can try
try {
...
} catch () {
...
}
But I prefer doing things this way
public function handle()
{
Skill::get()->map(function($skill){
if(strpos($skill->img_path, 'http')) return;
$img = $this->getImageFromUrl($skill->img_path)
if(!$img == null) return;
$image_path = '/assets/fe/img/skill/';
$img_name = $skill->name.'.png';
$path = public_path() . $image_path . $img_name;
$fp = fopen($path,'x');
fwrite($fp, $img);
fclose($fp);
if($uploadSuccess) {
$skill->img_path = $image_path . $img_name;
$skill->save();
}
});
}
public function getImageFromUrl($url){
$ch = curl_init ($url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0); // The number of seconds to wait while trying to connect. Use 0 to wait indefinitely.
curl_setopt($ch, CURLOPT_TIMEOUT, 2); // The maximum number of seconds to allow cURL functions to execute
$img = curl_exec($ch);
$err = curl_error($ch);
if($err) {
echo $err;
return null;
}
curl_close($ch);
return $img;
}
I refactored some code and if you prefer this keep it if not get only logic and do it as you prefer
Hope this helps

Answer 2

You may use active_url validation rule to check if the given URL is alive. According to the docs:
The field under validation must have a valid A or AAAA record according to the dns_get_record PHP function. The hostname of the provided URL is extracted using the parse_url PHP function before being passed to dns_get_record.
if (validator([$skill->img_path], ['active_url'])->fails()) {
// URL is not valid/active
}
else {
// URL is valid/active
}