Author Topic: Deleting large number of items  (Read 576 times)

Flashman

  • Sr. Member
  • ****
  • Posts: 51
Deleting large number of items
« on: July 20, 2017, 04:39:29 PM »
I have been using a cloud backup service called SquirrelSave for the last few years, which has just announced they will be closing their doors next month. I've started backing up to an alternative, however I wanted to remove everything from SquirrelSave, which can be done by FTP.

I connected via Yummy and selected all, before hitting the delete button. Fast forward five hours and it's still preparing to delete as it counts every item on the server. Is there any way to make this sort of process faster? Does it really have to count every item? I tried ticking the ssh option, but that isn't permitted.

JD

  • Administrator
  • FTP Guru
  • *****
  • Posts: 2823
Re: Deleting large number of items
« Reply #1 on: July 21, 2017, 09:25:58 AM »
Sorry for the delayed reply.

Just to clarify what's happening, Yummy determines all the file listings of everything you asked to delete before proceeding so that it can provide a progress indicator. While this may seem like an overhead, the delete process requires that all the file listings are fetched in any case, so this is just happening up front rather than one-at-a-time as you're suggesting, and the benefit of the progress indicator is very valuable for many people.

Having said that, 5 hours seems like a long time. Maybe the server has symlinks that resolve to parent directories? This can lead to an infinite loop trying to delete everything.

You'd need to connect via SFTP to use the SSH features such as remote delete. That would be the fastest way to do it, though. Otherwise, there's usually some sort of file manager in the site control panel that might let you delete the contents more easily.

At the end of the day, FTP is a long winded way of deleting large amounts of data because it can't delete a whole folder in one go, every single file needs to be deleted individually first. That takes a lot of time.

Flashman

  • Sr. Member
  • ****
  • Posts: 51
Re: Deleting large number of items
« Reply #2 on: July 21, 2017, 02:47:12 PM »
Thanks for the explanation. This was a Cloud backup so there was contents from my computer going right back to 2014. Backups normally took place via a specific app, but they also allowed connections using an app like Yummy via SFTP if needed. 

About an hour later it stopped counting after something like 500,000 files and was ready to delete, so I hit the button and tried the SSH option, but it wasn't permitted. Error messages about not having sufficient file permissions started popping up, so I selected the option to ignore that and it just carried on like that for about an hour, apparently suppressing errors, then reconnecting every time. Eventually I got bored and stopped the whole process.

My suspicion is that I set off some kind of firewall setting because I cannot even log in now. It tells me permission is denied (public key). Obviously this was an unusual situation and SquirrelSave say they'll delete everything when the service closes next month. It did strike me though, that counting files for 5 hours was a process that was crying out for a workaround, so they could just be deleted without the wait.