Automated Downloading and Notification

[gemillam]

At my work we have a need for a solution. We have an FTP site that our clients use to upload files for us. They are supposed to call the CSR about a job first, so that we know what to look for, but increasingly people are just dropping files in unbidden. What ends up happening is that some files will sit for days before acted upon.

We need a Mac-friendly, inexpensive way to automatically look at our FTP site on a regular, scripted basis to check for new files. When new files are detected, the files get automatically downloaded and some kind of notification to humans occurs. Can Yummy do this with its large bag of tricks? Even if it required additional scripting, plug-ins, or other stuff, if it can happen, it would definitely be worth our while.

[spacific]

How about a simple web script that looks in the relevant folder periodically (like when you visit your home page, or wherever) and reports recent activity.  If found it can create a web page with a list of files for you to download as required. The only problem might be if your web server software doesn't have access to the ftp folder, but that is easily fixed by setting the correct permissions.
E.g. all this can be easily done using a PHP script.

[gemillam]

We have a separate "web-based FTP" site where clients can upload files to us via their web browser. This site gives us full notifications, even allowing clients to send us notes about the files they upload. It has its issues, however. It tends to be slow, and has a max upload size of 30 mb. (The upload limit was arbitrarily set by our ISP, which developed and hosts the site.)

The site we are having the problems with is our "traditional" FTP site. It requires a standard FTP client like Yummy to access, and has pretty good security. We want similar notification features for this site that we have with the web-based one.

[spacific]

Yes, I wasn't suggesting you change the way users upload files. That stays the same using ftp.

But you can still use a web-based script to report the activity, and allow you to see the files and download the ones you want.  However if you don't want to use a web-based script, perhaps because the web and ftp systems are not easily linked, then fine. Worth considering.
 

[JD]

Hmmm, I can't think of a really slick solution, but I did come up with this:

1. Create a Bookmark of your server, and set the default Mac folder to a place where you can download file server files, and set the Server folder to the location that people send files.

2. Install Growl (www.growl.info) and set it up so that Yummy will send an email (to whoever is interested) whenever a download completes

3. Create an AppleScript using Script Editor (HD -> Applications -> AppleScript) and copy the following into it:

tell application "Yummy FTP"
   connect "Bookmark Name"
   
   if result is 0 then
      using connection "Bookmark Name" set mode "Replace"
      
      repeat
         delay 60
         with timeout of 32000 seconds
            using connection "Bookmark Name" download "*"
         end timeout
      end repeat
   end if
end tell

4. Replace "Bookmark Name" with the name of your Bookmark.

5. Quit Yummy FTP

6. Run the Script.

What will happen:

Yummy will be launched and a connection will be opened to your server. Every 60 seconds, Yummy will try to download everything in the server folder. If something is downloaded an email notification will be sent. If nothing is downloaded, no notification will be sent. Yummy will check again in 60 seconds.

Obviously you'll need to move the files out of the server folder otherwise you'll get constant notifications, but I expect that happens as you process them anyway, and you'll also need to cleanup the downloaded files, but it will give you the notification you're looking for. If needed, I could add some script to move the server files somewhere else after being downloaded, and also clean up the downloaded files, but let's see what you think of my proposed solution first

Any good?

[gemillam]

It works... the first time. The script does not refresh the list of the directory being looked at by Yummy on the FTP site. Thus, if there is one file on the FTP site when the script is ran, it will download the file and Growl will send a notification to the department email. But, while it is running, if a new file is uploaded, Yummy (and thus the script) will not recognize the new file in the next if-then cycle, since it doesn't refresh. Instead, it downloads the first file again, and sends out another set of notifications for the first file.

Which brings up another weird item. While Growl does a good job of notifying via email that a new file has been downloaded, it does it multiple times.

After opening the script dictionary for Yummy, I see that there is only a limited amount of commands for controlling the app (although they are very powerful). The way I see around there not being a "refresh" command would be to force a disconnect. This would make the script reconnect each time it cycled. Also, since there is a non-activity time-out on our FTP site, it might be possible to just set the delay on the if-then loop to be greater than the time-out. I am going to try that and see if that works. The only thing I wonder about is what will happen when the loop discovers that it has been disconnected. Will it reconnect automatically or will it crash and burn?

[gemillam]

I just found the command for "refresh", or update listing, as I should have recognized it. I am trying this command out. I am not sure what it wants for its string. I am going to use the bookmark for he site.

[gemillam]

Here's what I did to fix the refresh problem.

tell application "Yummy FTP"
   connect "Bookmark Here"
   
   if result is 0 then
      using connection "Bookmark Here" set mode "Replace"
      
      repeat
         delay 60
         with timeout of 32000 seconds
            using connection "Bookmark Here" update listing "remote"
            using connection "Bookmark Here" download "*"
         end timeout
      end repeat
   end if
end tell

Thanks for your help. I am going to continue to play with this to see if I can be more sophisticated with it (like pass the names of the files that have downloaded to an array or something so that the script doesn't keep downloading them between human checks). Still, in its current incarnation, it is quite usable. Thanks again.

[JD]

Oh, yes, I am sorry - I typed the script out off the top of my head. The refresh command was indeed needed here and I didn't think of that. Anyway, I'm glad it's working for you now

As far as the multiple email notifications is concerned, it might be that you're seeing one per file, or group of files, since Yummy uses batch transfers by default. I suggest to enable the 'Use single connection' checkbox in the Advanced Options tab of the Bookmark settings for your server. This will eliminate the problem - you will receive only one notification each time the download takes place.