sync'ing using multiple filters

[JD]

i did not realise til recently that this was happening... i have hundreds of thousands of files in the incorrect place... and it is only when i sit and watch the sync operation that i notice that things are in the incorrect place (i have multiple>multiple>multiple embedded files/folders....

Gosh, I am sorry - no wonder it is taking so long for Yummy to prepare for synchronization!

maybe the fact that there is stuff in the incorrect location that it is throwing it off?

It definitely will not help, yes.

but... for example.... with the latest sync.... i noticed that something was in the incorrect place (after 18 hrs of "preparing") so thought it best to leave it and have yummy delete it with the orphan command

the incorect files/folders were in... say folder A>subfolder B

however... with the latest sync.. ALL of the stuff that was being "sync'ed" (and i use that term VERY loosely) were inside folder D> subfolder X

how does that happen?

Unfortunately, without being there to see what is happening and make tests, I don't know. It sounds like everything is messed up still from previous synchs which had the bug. All I can suggest is to start again from scratch with the latest version of Yummy FTP

i have verified that i am sync'ing the top level of BOTH server & client

Ok, good. I wanted to be certain of that.

[JD]

another thing

seems ludicrous that i now need to go thru and remove all of the stuff that yummy "sync'ed" incorrectly and then run the sync again

i thought the idea of sync'ing was that it sync'ed two folders... and as per your above comment... the reason that the "preparation" stage takes so long is that it checks both directories each time it runs (no cache file etc)

so

if it was doing it's job correctly why wouldn't it upload what i want it to... and delete the stuff on the server that is orphaned?

Yes, you are right. If Yummy is working correctly then only the required folders/files would be uploaded and all the extra items on the server would be deleted as 'orphans'.

I think you should do this - a complete re-synch from local to server - but probably I would suggest to do so without the Filters applied, just to be certain that they have no negative effect. Then when we are sure that the local and server directories are really the same, then we can go ahead with the Filtered synch again.

(just out of interest... i still have it set to do "recursive batch transfers"... is that OK?)

Yes, this setting does not currently apply to synchronizations anyway.

to spend days and days going thru all of the files/folders on the server to verify that they are in the correct place doesn't seem like something i should have to do manually

Of course, yes. I did not realize the extent of the extra items on your server.  I thought just maybe a few, but if there are thousands and all in various places then it makes sense for Yummy to do the work for you

[yukiomishima]

JD

thanks for the reply

a complete re-sync will take upwards of 3 weeks.....  (over 300 gb of data)

we are using this to do daily backups of data that has been changed on the local machine

having to re-sync everything... from scratch... is not really an option.... it took long enough to get the data there in the Ist place.... and was hoping that YFTP would simply keep them in sync... with daily changes being uploaded

boy... was i wrong

so instead of having a reliable offsite backup... we now have a HUGE mess

anything else i can try?

as i have mentioned before.... even doing a simple sync for the past 24 hrs... it takes YFTP 12 hours or so just to "prepare" for the sync.... so it seems like it is days and days of preparing... deleting stuff off the server that is in the incorrect place... starting the suync again... etc etc

there has to be a better way?

thanks

yukioMishima

[JD]

Firstly, I'm sorry for the situation you have got into. Unfortunately no software is perfect, and Yummy is no exception. In this case, the problems have been caused by a bug in previous versions, but I sincerely believe that bug has been fixed.

We must get back to a known 'good' state before further synchs can be performed. Ideally this would mean a complete re-synch and without filters, because I have never seen your filters, so I don't know if they would negatively affect the synch. But if that would really take over 3 weeks just to upload 300GB of data then I understand it is not an option for you.

So then, I suggest to try the sync again, using 1.7.1b5 with Filters enabled and with Delete Orphans enabled. It will take Yummy a long time to prepare the sync because there are so many orphaned items on the server that it must still scan. We cannot avoid that now. But once the synch completes and the orphaned files/folders are deleted, subsequent synchs should work a lot quicker.

I think this is the only choice at the moment.

[yukiomishima]

JD

as per above.... this is the option that i took yesterday... exactly as you have outlined... and.. as per my prev post.... ALL of the files that were sync'ed were sync'ed to completely the wrong location

so... unfort... that did not seem to work for me

the olnly filters i have running are:
local:
- exclude with suffix: .lbk
- exclude with suffix: .bbk
- exclude with suffix: .bpn
- excludes dates prior to 12.16.2007

on the server side:
- exclude dates prior to 12.16.2007

i have it set to delete orphans
check network time
sync the top level of a 300gb drive

why would it be selecting a random location to upload all of the files that it believes need sync'ing..... and it appears that as it sync'ing to a different location to where it should... it is uploading way more than it should

thanks

yukioMishima

[yukiomishima]

so

completely uninstalled and reinstalled YFTP incl. plist etc.....

ran a new sync...... on one of the subfolders (that was already on the server)

first time thru it seemed to perform ok (thou i did not delve down into every folder)

second time thru... it started to upload stuff into subfolders..... at random

this is driving me nuts

thoughts?

yukioMishima

[JD]

Reinstalling will make no difference, but your Bookmark might have an unexpected setting, although I don't think it's possible to affect the synch. Did you try re-creating your Bookmark?

Again with a test subfolder:

1. What happens if you use the Synch Preview first? Does it show all the incorrect uploads before uploading them?

2. Can you retry that same test without the Filters enabled?

[yukiomishima]

JD

as much as it pains me to say this.... cause i think YFTP is a great program... nice interface, some great features.... but this whole "sync" thing is driving me nuts.... i really cannot afford to have no confidence in the application doing what it should be doing

i just ran a competitors FTP application on the same set of files/folders and experienced no sync issues...... i would rather stick with YFTP... but for the moment i really cannot afford to spend any more time trying to problem solve a bug that really should not be there

i will keep an eye on what/ how YFTP progresses with this and other issues i have outlined earlier in this thread... but for now... HUGE thanks for all of your time/ effort in trying to solve this problem... but i really have to jump ship

good luck with it all

happy new year

yukioMishima

[JD]

Thank you for your patience. I understand your reluctance to try to find the bug. It is especially difficult for me to narrow down because of the huge amount of data involved, with special filters, and so on, plus this being a very busy time of year I haven't been able to spend as much time as usual debugging it. But I will continue to investigate by myself and as soon as I have fixed it (and I will fix it) I will be in contact with you.

[JD]

First thing this morning I set up a test environment, with 300GB of data to synch, an FTP server on my LAN (across the Internet would be too slow for testing) and with very similar filters to those that you described.

I have spent all day making batch changes to local files so that their mod date passes the filters, then performing synchs. All the synchs have worked perfectly. Every single one.

So, this leads me to believe there is still a problem with your setup:  I know I asked you to check that the folders to be synched were correct, but it's easy to make a mistake with that setting, and if that is set incorrectly then it would explain the odd behavior - One thing that quite often throws people is that when you choose the Synch option it is context sensitive, which means that if you have a sub folder selected in the server listing then the synch destination would be that sub folder, not the folder whose name is displayed in the menu at the top of the server listing. Without being there to see what is happening, this is the only thing I can see which might be the cause of the problems you've had.

I admit that you were originally using a version which had a bug that could upload to the wrong place, but that has been fixed and verified as so by all the people who originally reported the problem. Of course this has made matters a lot worse for you because subsequent synchs required progressively longer and longer 'preparing' times, where Yummy built the two directory trees to make comparisons.

Assuming that your alternative FTP app has synched correctly and removed the erroneous uploads, I would ask that you try a sync one more time, bearing in mind my suggestion about the context-sensitive nature of the Synch tool, and making sure to enable the Preview option so that no items are actually transferred or deleted - we can then see if the correct items for synchronization have been detected or not.

[yukiomishima]

JD

thanks for the reply

so..... followed your advice

set-up a brand new sync between the local server and the remote server

created a new bookmark with the filters as before

synced about 6gb on the Ist sync (filter sync'ed everything from 12.16.2007 forward)

sync'ed about 5 gb on the 2nd sync (filter sync'ed everything from 12.10.2007 forward)

am just syncing again (using the same filters etc as the 2nd sync) and files/folders are being sync'ed INTO THE INCORRECT PLACE!!!!!

this is using 1.7.1b5.... brand new sync..... brand new bookmark... brand new filters

the 3 sync's ran from a scheduled bookmark.... so there is no way that i was in the incorrect directory when the sync occured (and the location that it is uploading to is not one that i have navigated down to on the remote server)

this problem.... at least from my end.... has not been solved

thanks

yukioMishima

[yukiomishima]

JD

i would send you the transcript (incomplete as i stopped the sync once i saw that is was not performing correctly)

however.... the .txt file is ±43mb

thanks

yukioMishima

[JD]

Hi again. I'm glad to see you're trying again with Yummy

Did the other solution not work out? I'm interested to know if the erroneous uploads got cleaned up or not.

For Yummy, I haven't said that the problem was solved, but only that I could not reproduce it after trying all day. I offered another possible cause in the shape of the context-sensitive nature of the Synch tool. At least that can be eliminated now.

So far I haven't had any real data from you, only your descriptions. I really need to see the FTP transcript file and your Bookmark file, otherwise I don't have information to go on. The Bookmark file contains your exact filter setup plus your scheduled sync. Obviously your password will be stored in your keychain, not in your Bookmark file, so I won't be able to connect to your server. I will email you connection details to upload the information to me.

For the results you have seen, are you saying that the first two syncs put the files/folders into the correct locations, but only the third one displayed the problem?

[yukiomishima]

JD

thanks

shoot me an email with instructions on getting the transcript file and the bookmark to you and i will shoot it off asap

thanks

yukioMishima

[JD]

I have emailed you.

So, did the orphaned items get cleaned up already or not?

Also, was it the third sync only that had the problem, and the first two worked fine?

Thanks.