Yummy Software

howdy

firstly..... great software

i am having some issues sync'ing a folder on my server with a remote ftp server

i am wanting to setup 2 sync's... the first uploading everything from 12.01.2007 forward (excluding a number of file types) and a second sync... of the same folder from 11.30.2007 backwards

is this possible to do? (i have set something up... but it does not appear to be working the way that i had hoped)

also.... not sure if these exist and i have overlooked them.... but would be good to have:

- is there anyway that when sync'ing the application can refer to some sort of catalogue that it has generated since the last sync... as it seems to take a good few hours of "preparing" prior to the sync actually taking place (it is on a server of over 300gb)

- a way to set a start and pause/stop time for a schedule (ie. the above scenario... would be good if i could run schedule 01.... pause it 5 min before schedule 02 starts... and then have it resume when schedule 02 is finished)

- set-up email notification for when a sync is finished (or encounters errors like a server disconnect etc)

- report on time, number of folders/ files uploaded, the amount of data (mb/gb) uploaded, average upload/download speed

- rename schedules

- if i use a filter to exclude files/folders and still have yummy set to "delete orphaned items" it would be good to either NOT have it delete from the server if that file/folder is simply "excluded" rather than actually "missing/orphaned" or have a toggle to allow you to decide whether or not you want it deleted

thanks

yukioMishima

Hi,

Yes, it should be possible to do that. Please send a screenshot of your filter setup to

support at yummysoftware dot com

I'll take a look and see what's going on.

To answer your questions:

There is no cached information kept about the files and folders that have been synched. Unfortunately, the only really reliable way to ensure only the required items get transferred is to not make any assumptions about the directory structure, both remote and local. Several hours does sound like a long time, but there are a lot of factors such as actual number of files, server speed, your line speed, even your Mac's speed, etc. that need to be taken into account.

There is currently no way to set a pause and resume time in a schedule. If you want to avoid overlapping synchs, I suggest limiting the number of simultaneous connections to one, which will force a scheduled item to wait if another has not yet completed. You can set this restriction in the Advanced Settings tab of the editor for your Bookmark.

For email notification of any event, including synchs and errors, install Growl (http://www.growl.info) and use the Email Me notification.

There is already a Synch Completion report displayed at the end of synchronization. It would be reasonable to add other statistics to it :)

There are no options to name schedules at the moment.

For the delete orphans problem, Yummy doesn't make any assumptions with your filters. You need to exclude both the local and remote items, otherwise it will appear to the synch engine that there are items which are orphaned.

Thanks :)

JD

thanks for the reply

on the 2 sync's that i have set-up.... sync_01 backups everything since 12.07.2007 and sync_02 backups everything on the server prior to 12.07.2007.... obviously i need 2 filters for that

i believe i set both schedules up correctly (exactly the same... except for the date range).... however... when sync_01 runs... it does not seem to sync (are the filter settings saved within the schedule... or do i need to clear out the filter settings)

also.... as for the caching.... even though i only need to backup say 3gb of data per day... currently yummy needs to scan the entire drive to compare with the off-site backup to determine which files need backing up.... as... with a 300gb drive... this simple operation... which should be relatively quick... takes hours (just in the "preparing" stage)

does it not make sense to cache sync'ed info?.... or at least have it as an option... esp if you are only sync'ing between a single machine + a single server... it woulod certainly facilitate a faster sync operation

the problem with only allowing a single connection is that... in my scenario.... i need to sync the same server to an offsite backup.... sync_01 backups everything done over the past 24 hrs.... whilst sync_02 is currently backing up the rest of the server (±300 gb)..... given that sync_02 is going to take weeks to complete... i cannot really go down that route

cool... will install growl and see if that provides what i need

additional info on the sync completion report would be super useful

naming/saving/editing schedules would be helpful... it appears at the moment that if you quit the application you need to set them all up again

i believe that Transmit allows you to "filter" items to be sync'ed WITHOUT deleting the files on the server... however... if i can filter both the server and the local machine in the same way and ensure that items on the server are not deleted that should work nicely

thanks again

yukioMishima

It sounds like you're not using a Bookmark. Schedules and filters are saved to the Bookmark. If you close the connection without saving it first you lose the schedule and filter settings.

To save a Bookmark, select 'Add Bookmark' from the Bookmarks menu when you are connected to your server. When you wish to reconnect to your server, select it's Bookmark name from the Bookmarks menu.

The first thing to remember about filters is that they exclude items, so for example, your sync_01 should be set to 'Exclude items whose: Date is older than 12.07.2007'

Secondly, the filter does not get saved to the schedule, it is saved to the Bookmark, so you can't have two different filters active unless you have two different Bookmarks. This probably explains why you're getting unexpected results. If you have two Bookmarks to your server, one set with the filter for synch_01 and the other set with the filter for sync_02 and then add a scheduled sync to each of the Bookmarks then you should get what you're after.

For the large scanning time, is it possible to narrow down your synch to some more specific subfolders? Or is it usual/expected that any file anywhere on the 300GB drive could be changed? For the latter, I don't have a solution for you at the moment, I'm afraid.

I understand about the one connection idea, but are you sure it will really take weeks to complete? What speed is your Internet connection?

One feature you might want to investigate on top of, or instead of, filters is the 'Exclude From' option in the listing context menu. This allows you to exclude specific items from synchs and or transfers. The settings are again stored per Bookmark, but they can be used in addition to the filters.

JD

thanks again for the reply

so

have been playing about with bookmarks + the listing>filters function

here is what i have discovered

for some reason my local machine does not always seem to update the folder>modification date when a file has been modified within it

so... when i use the filter to filter out folder/files prior to a specific date.... then some of the enclosing folders which have not had their modification date altered are holding files which are newer than the date on the folder

so... it would appear that the enclosed files do not get updated... even though they do not have a red "X" thru them (and the enclosing folder does)

is this normal?

is there a way to get the results in a sync that i am looking for

thanks

yukioMishima

This is normal behavior of the operating system. A folder mod date will only get changed if its contents have changed - a new file/folder or a removed file/folder. The mod date will not get changed if the contents of a file are changed.


Yes, because of the incorrect assumption about how the file system works.


Yes, it is normal behavior - your filter setting excluded the whole folder, and contents too. The items within the folder may not match the filter exclusion and therefore do not have the red X, but because their parent directory is excluded by the filter, their exclusion is a direct result.


Without being there to try and test, from the sounds of it you might have to disable the 'Apply to folders' setting and put up with folder modification dates being out of sync. The files should sync as you expect, though.

JD

thanks for the prompt response

the weird thing is..... the files inside the folders are being modified... but the enclosing folder's modification date is not always updated auto by the system... so am getting weird results

if i exclude folders... what happens if i have files buried say 3 folders deep... will the file/ folder structure remain in tact?

thanks again

yukioMishima

Well, maybe it seems weird, but that's just the way it is - no way to change that.

If you don't filter folders, but a file 3 levels deep changes (for example), then yes, the directory structure will still be maintained, creating folders on the server as necessary.

Give that a shot and see how you go.

JD

major issues

i am attempting to sync as per above

however...... even though i seem to have both folders (server/ client) selected correctly, it would seem that when i sync it seems that all of the files/folders that need to be sync'ed are uploaded in COMPLETELY the incorrect directory.... not even close

it has been happening for ages and i thought it might have been something i did..... but it would appear that it is the application

this is proving to be HUGELY problematic

any thoughts?

thanks

yukioMishima

any thoughts?

Have you read this thread:

http://www.yummysoftware.com/forum/index.php/topic,581.0.html

Make sure you are using the latest version of Yummy.

thanks for the reply

downloaded and installed 1.7.1 b5

HOWEVER.... i have found the problem to be still there

after 24 hours of waiting for the "preparation" to sync (i have 300 gb of data that i am sync'ing with an offsite ftp server)..... i find that the data is being transferred to completely and utterly the incorrect folder

this is a MAJOR issue... as i now seem to have HUGE amounts of data both un-sync'ed as well as duplicated

any thoughts..... this is proving to be hugely problematic and a very big waste of time.... when i can't simply rely on the sync process and need to check with the application every few minutes to ensure that it is reading/writing the correct data there is something critically wrong with the process

thanks in advance

yukioMishima

a further update

just going thru now and deleting the more than 15000 files that were uploaded into completely the wrong folders structure (and i stopped it even though it said it had TWO days left for the sync.... so who knows how many of the files were going to be uploaded)

it is not syncing.... but uploading everything again..... as it doesn't seem to sunc the 2 folders.... but simply find some random place to upload stuff and then merrily go about its business

this is becoming a little catastrophic..... i cannot spend my time second guessing something as simple as sync'ing 2 folders

a response/solution/workaround at your earliest would be hugely appreciated

thanks

yukioMishima

The problem with folders in the wrong place after a synch has been confirmed as fixed by three people who reported it, so I am very surprised if you really are experiencing it and I am very sorry that I did not mention it before - I assumed you were already using the latest version.

Please try to delete all the duplicates (this is very important) and then try again, but this time, if something does go wrong, please select 'Save FTP Transcript' from the File menu and send me the resulting file.

Also please make sure that you have the correct client + server folders selected - the Synch setup window will show the names of the two folders to synch. Double check it.

Send the file to:

support at yummysoftware dot com

.. and tell me which folders are in the wrong place.


(I am sorry for the delayed response - this is due to Xmas time)

JD

thanks for the reply

i did not realise til recently that this was happening... i have hundreds of thousands of files in the incorrect place... and it is only when i sit and watch the sync operation that i notice that things are in the incorrect place (i have multiple>multiple>multiple embedded files/folders.... so it would take me a week to go thru and verify that everything is in the correct location)... and trust me... i have WAY BETTER things to do with my time

maybe the fact that there is stuff in the incorrect location that it is throwing it off?

but... for example.... with the latest sync.... i noticed that something was in the incorrect place (after 18 hrs of "preparing") so thought it best to leave it and have yummy delete it with the orphan command

the incorect files/folders were in... say folder A>subfolder B

however... with the latest sync.. ALL of the stuff that was being "sync'ed" (and i use that term VERY loosely) were inside folder D> subfolder X

how does that happen?

i have verified that i am sync'ing the top level of BOTH server & client

thanks

yukioMishima

another thing

seems ludicrous that i now need to go thru and remove all of the stuff that yummy "sync'ed" incorrectly and then run the sync again

i thought the idea of sync'ing was that it sync'ed two folders... and as per your above comment... the reason that the "preparation" stage takes so long is that it checks both directories each time it runs (no cache file etc)

so

if it was doing it's job correctly why wouldn't it upload what i want it to... and delete the stuff on the server that is orphaned?

(just out of interest... i still have it set to do "recursive batch transfers"... is that OK?)

to spend days and days going thru all of the files/folders on the server to verify that they are in the correct place doesn't seem like something i should have to do manually

thanks

yukioMishima

Gosh, I am sorry - no wonder it is taking so long for Yummy to prepare for synchronization!


It definitely will not help, yes.


Unfortunately, without being there to see what is happening and make tests, I don't know. It sounds like everything is messed up still from previous synchs which had the bug. All I can suggest is to start again from scratch with the latest version of Yummy FTP


Ok, good. I wanted to be certain of that.

Yes, you are right. If Yummy is working correctly then only the required folders/files would be uploaded and all the extra items on the server would be deleted as 'orphans'.

I think you should do this - a complete re-synch from local to server - but probably I would suggest to do so without the Filters applied, just to be certain that they have no negative effect. Then when we are sure that the local and server directories are really the same, then we can go ahead with the Filtered synch again.


Yes, this setting does not currently apply to synchronizations anyway.


Of course, yes. I did not realize the extent of the extra items on your server.  I thought just maybe a few, but if there are thousands and all in various places then it makes sense for Yummy to do the work for you

JD

thanks for the reply

a complete re-sync will take upwards of 3 weeks.....  (over 300 gb of data)

we are using this to do daily backups of data that has been changed on the local machine

having to re-sync everything... from scratch... is not really an option.... it took long enough to get the data there in the Ist place.... and was hoping that YFTP would simply keep them in sync... with daily changes being uploaded

boy... was i wrong

so instead of having a reliable offsite backup... we now have a HUGE mess

anything else i can try?

as i have mentioned before.... even doing a simple sync for the past 24 hrs... it takes YFTP 12 hours or so just to "prepare" for the sync.... so it seems like it is days and days of preparing... deleting stuff off the server that is in the incorrect place... starting the suync again... etc etc

there has to be a better way?

thanks

yukioMishima

Firstly, I'm sorry for the situation you have got into. Unfortunately no software is perfect, and Yummy is no exception. In this case, the problems have been caused by a bug in previous versions, but I sincerely believe that bug has been fixed.

We must get back to a known 'good' state before further synchs can be performed. Ideally this would mean a complete re-synch and without filters, because I have never seen your filters, so I don't know if they would negatively affect the synch. But if that would really take over 3 weeks just to upload 300GB of data then I understand it is not an option for you.

So then, I suggest to try the sync again, using 1.7.1b5 with Filters enabled and with Delete Orphans enabled. It will take Yummy a long time to prepare the sync because there are so many orphaned items on the server that it must still scan. We cannot avoid that now. But once the synch completes and the orphaned files/folders are deleted, subsequent synchs should work a lot quicker.

I think this is the only choice at the moment.

JD

as per above.... this is the option that i took yesterday... exactly as you have outlined... and.. as per my prev post.... ALL of the files that were sync'ed were sync'ed to completely the wrong location

so... unfort... that did not seem to work for me

the olnly filters i have running are:
local:
- exclude with suffix: .lbk
- exclude with suffix: .bbk
- exclude with suffix: .bpn
- excludes dates prior to 12.16.2007

on the server side:
- exclude dates prior to 12.16.2007

i have it set to delete orphans
check network time
sync the top level of a 300gb drive

why would it be selecting a random location to upload all of the files that it believes need sync'ing..... and it appears that as it sync'ing to a different location to where it should... it is uploading way more than it should

thanks

yukioMishima

so

completely uninstalled and reinstalled YFTP incl. plist etc.....

ran a new sync...... on one of the subfolders (that was already on the server)

first time thru it seemed to perform ok (thou i did not delve down into every folder)

second time thru... it started to upload stuff into subfolders..... at random

this is driving me nuts

thoughts?

yukioMishima

Reinstalling will make no difference, but your Bookmark might have an unexpected setting, although I don't think it's possible to affect the synch. Did you try re-creating your Bookmark?

Again with a test subfolder:

1. What happens if you use the Synch Preview first? Does it show all the incorrect uploads before uploading them?

2. Can you retry that same test without the Filters enabled?

JD

as much as it pains me to say this.... cause i think YFTP is a great program... nice interface, some great features.... but this whole "sync" thing is driving me nuts.... i really cannot afford to have no confidence in the application doing what it should be doing

i just ran a competitors FTP application on the same set of files/folders and experienced no sync issues...... i would rather stick with YFTP... but for the moment i really cannot afford to spend any more time trying to problem solve a bug that really should not be there

i will keep an eye on what/ how YFTP progresses with this and other issues i have outlined earlier in this thread... but for now... HUGE thanks for all of your time/ effort in trying to solve this problem... but i really have to jump ship

good luck with it all

happy new year

yukioMishima

Thank you for your patience. I understand your reluctance to try to find the bug. It is especially difficult for me to narrow down because of the huge amount of data involved, with special filters, and so on, plus this being a very busy time of year I haven't been able to spend as much time as usual debugging it. But I will continue to investigate by myself and as soon as I have fixed it (and I will fix it) I will be in contact with you.

First thing this morning I set up a test environment, with 300GB of data to synch, an FTP server on my LAN (across the Internet would be too slow for testing) and with very similar filters to those that you described.

I have spent all day making batch changes to local files so that their mod date passes the filters, then performing synchs. All the synchs have worked perfectly. Every single one.

So, this leads me to believe there is still a problem with your setup:  I know I asked you to check that the folders to be synched were correct, but it's easy to make a mistake with that setting, and if that is set incorrectly then it would explain the odd behavior - One thing that quite often throws people is that when you choose the Synch option it is context sensitive, which means that if you have a sub folder selected in the server listing then the synch destination would be that sub folder, not the folder whose name is displayed in the menu at the top of the server listing. Without being there to see what is happening, this is the only thing I can see which might be the cause of the problems you've had.

I admit that you were originally using a version which had a bug that could upload to the wrong place, but that has been fixed and verified as so by all the people who originally reported the problem. Of course this has made matters a lot worse for you because subsequent synchs required progressively longer and longer 'preparing' times, where Yummy built the two directory trees to make comparisons.

Assuming that your alternative FTP app has synched correctly and removed the erroneous uploads, I would ask that you try a sync one more time, bearing in mind my suggestion about the context-sensitive nature of the Synch tool, and making sure to enable the Preview option so that no items are actually transferred or deleted - we can then see if the correct items for synchronization have been detected or not.

JD

thanks for the reply

so..... followed your advice

set-up a brand new sync between the local server and the remote server

created a new bookmark with the filters as before

synced about 6gb on the Ist sync (filter sync'ed everything from 12.16.2007 forward)

sync'ed about 5 gb on the 2nd sync (filter sync'ed everything from 12.10.2007 forward)

am just syncing again (using the same filters etc as the 2nd sync) and files/folders are being sync'ed INTO THE INCORRECT PLACE!!!!!

this is using 1.7.1b5.... brand new sync..... brand new bookmark... brand new filters

the 3 sync's ran from a scheduled bookmark.... so there is no way that i was in the incorrect directory when the sync occured (and the location that it is uploading to is not one that i have navigated down to on the remote server)

this problem.... at least from my end.... has not been solved

thanks

yukioMishima

JD

i would send you the transcript (incomplete as i stopped the sync once i saw that is was not performing correctly)

however.... the .txt file is ±43mb

thanks

yukioMishima

Hi again. I'm glad to see you're trying again with Yummy :)

Did the other solution not work out? I'm interested to know if the erroneous uploads got cleaned up or not.

For Yummy, I haven't said that the problem was solved, but only that I could not reproduce it after trying all day. I offered another possible cause in the shape of the context-sensitive nature of the Synch tool. At least that can be eliminated now.

So far I haven't had any real data from you, only your descriptions. I really need to see the FTP transcript file and your Bookmark file, otherwise I don't have information to go on. The Bookmark file contains your exact filter setup plus your scheduled sync. Obviously your password will be stored in your keychain, not in your Bookmark file, so I won't be able to connect to your server. I will email you connection details to upload the information to me.

For the results you have seen, are you saying that the first two syncs put the files/folders into the correct locations, but only the third one displayed the problem?

JD

thanks

shoot me an email with instructions on getting the transcript file and the bookmark to you and i will shoot it off asap

thanks

yukioMishima

I have emailed you.

So, did the orphaned items get cleaned up already or not?

Also, was it the third sync only that had the problem, and the first two worked fine?

Thanks.

JD

the Ist 2 "appeared" to be OK

the 3rd one is the one that i started noticing that everything being sync'ed was being sync'ed to the incorrect location

at your advice.... i started a whole new offiste file/folder structure (as outlined above)

the old folder/file structure is still on the remote ftp server.... but was only partially "cleaned-up" as i stopped the other application before it had finished

btw

is the 43mb transcript file just for the most recent sync... or from when i first started using YFTP or since the last time YFTP was started?

do i need to sort thru it and just send you the most recent stuff?

thanks

yukioMishima

Ok, so for all you know it could have been going wrong in the first one and you just didn't notice?


Ok, so delete this new one and start again. This time perform only one sync, then check thoroughly for errors. If there are errors, then send me the transcript.

We need to test that a single synch is successful. It doesn't make any sense that two synchs would work and the third not. Either it works or it doesn't.


It depends how you use the scheduled Bookmark:

If you have the a FTP Browser window open from that Bookmark, then the scheduled synchs execute within it.  So if you select Save Transcript then you get the log from the time you opened the FTP Browser window, including any scheduled synchs that have executed and any other transfers or browsing you have done.

If you don't have an FTP Browser window open from that Bookmark, then the scheduled synchs execute in a small progress window which auto-dismisses after the synch has finished. In this case, selecting Save Transcript will only get the last synch log data.

I would advise the open FTP Browser window for our test.

Which do you normally use?


No, definitely don't touch the data at all.

JD

the Ist 2 sync's worked correctly...... the Ist one built the folder structure and then populated with sub-folders and files

the 2nd sync worked correctly (though have not fully verified) and added files folders as necessary

the 3rd sync definately DID NOT work and sync'ed into the incorrect folders

i normally open the bookmark and have the FTP Browser window open

can i delete the current data in the transcript and start this all again?.... so that i have can shoot you thru relevant data?

i will set it up to only sync data since yesterday.... the 2nd sync will be data from 1 week ago... the 3rd sync will be data from 2 weeks ago

each sync will be drawing from the same local 300gb folder

should i create a bookmark for each sync... or just run the Ist sync and then modify the filter parameters

it will probably be a few days before i can get you the transcript as it will take that long to get the data from the server to the remote ftp site

please let me know how i should attack this new "test" and i will get it under way asap

thanks

yukioMishma

Ok, with not fully verified I still don't feel very confident, but let's assume that only the last synch went wrong then :)

If that's the case then the existing 43mb transcript will show the mistaken uploads. It will also show the previous 2 synchs if you didn't close the FTP Browser window. If you re-opened the FTP Browser window for the last synch then only the last synch's transcript will be in there.

You will need to tell me which files/folders have been uploaded to the wrong places.

Please send it, plus the Bookmark and Preferences file as I described in my email.


Ok, good.


The transcript data is discarded immediately after you close the FTP Browser window. When you open the connection again, a new file is started.


Well, I have to wonder why you are doing it like this in the first place. Why 3 synchs when the last one will back up all the data. And also, why you are using a scheduled synch. Is it intended take place overnight usually, but you're manually starting them to test?

If I were testing this, I would do perform each sync manually, using the preview option to see what will be uploaded - that is one test I have asked for, in fact - and then I would thoroughly check everything to make sure the correct uploads were made and to the correct places. That way, it can be narrowed down to exactly which point the failure occurs.


I'd actually prefer it if you closed the browser after each verified as successful synch and only send me the transcript of the synch that fails. It's a lot of data to wade through otherwise...

Again, you will need to tell me which files/folders have been uploaded to the wrong places.


No problem. I am ready and waiting :)


If you can get me a single failed synch session transcript, plus the Bookmark and Preferences file, that will be what I need. Failing that, then 43mb file should prove interesting, but it will take me a few days to get through ;)

JD

the reason i am sync'ing the way i am is purely for testing

with 300gb of data locally and only a 768k line to the remote FTP server..... to sync the entire 300gb would take weeks and weeks... or.... months and months

so.... i am trying to get it down to bite size pieces for testing... hence backing up a day/week at a time going backwards

i will setup a brand new folder.... and sync all work done today

sync 02 will backup the last 7 days

sync03 the last 14

as there is ±50 to 10 gb of data / sync that will be backed up..... with hundreds of folders/sub-folders... it will take some time for the data to be uploaded AND for me to identify exactly when it is that it all goes belly up

but..... will get it all underway now.... and shoot thru the files requested files when i have them available

thanks again for all

yukioMishima

Ok understood. Thank you for making this clear.


Ok sounds good.


Yes, I am sorry. I know it's a pain, but it's the only way to narrow this down. I appreciate your patience. Thank you.


You're welcome :)

JD

email has been sent + files uploaded as instructed

please let me know if you require further

thanks

yukioMishima

JD

just to follow-up

as you are probably aware by now...... the Ist sync worked correctly... setting up all of the approp folders/sub-foldrs and populating with the filtered files into a brand new folder on the remote FTP server

the 2nd sync (or.. in fact... the Ist actual sync between the data sets) got all funked up...... so it appears to happen immediately that the actual data is sync'ed and not... as i had mentioed in a previous message on the 3rd sync

hope this helps

looking forward to your findings and a solution to this issue

thanks again for all

yukioMishima

Just a generic FYI... text files like ftp transcripts compress HUGELY. I have seen such file shrink by 95+% on occasion. Part of the reason is that there are HUGE amounts of common data in such listings.

so... after alot of testing, alot of very hard work and multiple beta versions it looks as if the problem has been resolved

JD has worked tirelessly on tracking down and solving the problem

i cannot praise him enough for all the hard work, dedication, persistence and tireless effort he has put in to solving this issue

after having tested all of Yummy's major competitors i have no hesitation in recommending yummy for its unmatched feature set, interface, ease of use and fantastic tech support.

once again, JD, HUGE HUGE thanks for all your time and effort in sorting this out

yukioMishima

You're welcome! And thanks to you for all the help tracking it down ... it's been a pleasure working with you ;D