sync'ing using multiple filters

[yukiomishima]

howdy

firstly..... great software

i am having some issues sync'ing a folder on my server with a remote ftp server

i am wanting to setup 2 sync's... the first uploading everything from 12.01.2007 forward (excluding a number of file types) and a second sync... of the same folder from 11.30.2007 backwards

is this possible to do? (i have set something up... but it does not appear to be working the way that i had hoped)

also.... not sure if these exist and i have overlooked them.... but would be good to have:

- is there anyway that when sync'ing the application can refer to some sort of catalogue that it has generated since the last sync... as it seems to take a good few hours of "preparing" prior to the sync actually taking place (it is on a server of over 300gb)

- a way to set a start and pause/stop time for a schedule (ie. the above scenario... would be good if i could run schedule 01.... pause it 5 min before schedule 02 starts... and then have it resume when schedule 02 is finished)

- set-up email notification for when a sync is finished (or encounters errors like a server disconnect etc)

- report on time, number of folders/ files uploaded, the amount of data (mb/gb) uploaded, average upload/download speed

- rename schedules

- if i use a filter to exclude files/folders and still have yummy set to "delete orphaned items" it would be good to either NOT have it delete from the server if that file/folder is simply "excluded" rather than actually "missing/orphaned" or have a toggle to allow you to decide whether or not you want it deleted

thanks

yukioMishima

[JD]

Hi,

Yes, it should be possible to do that. Please send a screenshot of your filter setup to

support at yummysoftware dot com

I'll take a look and see what's going on.

To answer your questions:

There is no cached information kept about the files and folders that have been synched. Unfortunately, the only really reliable way to ensure only the required items get transferred is to not make any assumptions about the directory structure, both remote and local. Several hours does sound like a long time, but there are a lot of factors such as actual number of files, server speed, your line speed, even your Mac's speed, etc. that need to be taken into account.

There is currently no way to set a pause and resume time in a schedule. If you want to avoid overlapping synchs, I suggest limiting the number of simultaneous connections to one, which will force a scheduled item to wait if another has not yet completed. You can set this restriction in the Advanced Settings tab of the editor for your Bookmark.

For email notification of any event, including synchs and errors, install Growl (http://www.growl.info) and use the Email Me notification.

There is already a Synch Completion report displayed at the end of synchronization. It would be reasonable to add other statistics to it

There are no options to name schedules at the moment.

For the delete orphans problem, Yummy doesn't make any assumptions with your filters. You need to exclude both the local and remote items, otherwise it will appear to the synch engine that there are items which are orphaned.

Thanks

[yukiomishima]

JD

thanks for the reply

on the 2 sync's that i have set-up.... sync_01 backups everything since 12.07.2007 and sync_02 backups everything on the server prior to 12.07.2007.... obviously i need 2 filters for that

i believe i set both schedules up correctly (exactly the same... except for the date range).... however... when sync_01 runs... it does not seem to sync (are the filter settings saved within the schedule... or do i need to clear out the filter settings)

also.... as for the caching.... even though i only need to backup say 3gb of data per day... currently yummy needs to scan the entire drive to compare with the off-site backup to determine which files need backing up.... as... with a 300gb drive... this simple operation... which should be relatively quick... takes hours (just in the "preparing" stage)

does it not make sense to cache sync'ed info?.... or at least have it as an option... esp if you are only sync'ing between a single machine + a single server... it woulod certainly facilitate a faster sync operation

the problem with only allowing a single connection is that... in my scenario.... i need to sync the same server to an offsite backup.... sync_01 backups everything done over the past 24 hrs.... whilst sync_02 is currently backing up the rest of the server (±300 gb)..... given that sync_02 is going to take weeks to complete... i cannot really go down that route

cool... will install growl and see if that provides what i need

additional info on the sync completion report would be super useful

naming/saving/editing schedules would be helpful... it appears at the moment that if you quit the application you need to set them all up again

i believe that Transmit allows you to "filter" items to be sync'ed WITHOUT deleting the files on the server... however... if i can filter both the server and the local machine in the same way and ensure that items on the server are not deleted that should work nicely

thanks again

yukioMishima

[JD]

It sounds like you're not using a Bookmark. Schedules and filters are saved to the Bookmark. If you close the connection without saving it first you lose the schedule and filter settings.

To save a Bookmark, select 'Add Bookmark' from the Bookmarks menu when you are connected to your server. When you wish to reconnect to your server, select it's Bookmark name from the Bookmarks menu.

The first thing to remember about filters is that they exclude items, so for example, your sync_01 should be set to 'Exclude items whose: Date is older than 12.07.2007'

Secondly, the filter does not get saved to the schedule, it is saved to the Bookmark, so you can't have two different filters active unless you have two different Bookmarks. This probably explains why you're getting unexpected results. If you have two Bookmarks to your server, one set with the filter for synch_01 and the other set with the filter for sync_02 and then add a scheduled sync to each of the Bookmarks then you should get what you're after.

For the large scanning time, is it possible to narrow down your synch to some more specific subfolders? Or is it usual/expected that any file anywhere on the 300GB drive could be changed? For the latter, I don't have a solution for you at the moment, I'm afraid.

I understand about the one connection idea, but are you sure it will really take weeks to complete? What speed is your Internet connection?

One feature you might want to investigate on top of, or instead of, filters is the 'Exclude From' option in the listing context menu. This allows you to exclude specific items from synchs and or transfers. The settings are again stored per Bookmark, but they can be used in addition to the filters.

[yukiomishima]

JD

thanks again for the reply

so

have been playing about with bookmarks + the listing>filters function

here is what i have discovered

for some reason my local machine does not always seem to update the folder>modification date when a file has been modified within it

so... when i use the filter to filter out folder/files prior to a specific date.... then some of the enclosing folders which have not had their modification date altered are holding files which are newer than the date on the folder

so... it would appear that the enclosed files do not get updated... even though they do not have a red "X" thru them (and the enclosing folder does)

is this normal?

is there a way to get the results in a sync that i am looking for

thanks

yukioMishima

[JD]

for some reason my local machine does not always seem to update the folder>modification date when a file has been modified within it

This is normal behavior of the operating system. A folder mod date will only get changed if its contents have changed - a new file/folder or a removed file/folder. The mod date will not get changed if the contents of a file are changed.

so... when i use the filter to filter out folder/files prior to a specific date.... then some of the enclosing folders which have not had their modification date altered are holding files which are newer than the date on the folder

Yes, because of the incorrect assumption about how the file system works.

so... it would appear that the enclosed files do not get updated... even though they do not have a red "X" thru them (and the enclosing folder does)

is this normal?

Yes, it is normal behavior - your filter setting excluded the whole folder, and contents too. The items within the folder may not match the filter exclusion and therefore do not have the red X, but because their parent directory is excluded by the filter, their exclusion is a direct result.

is there a way to get the results in a sync that i am looking for

Without being there to try and test, from the sounds of it you might have to disable the 'Apply to folders' setting and put up with folder modification dates being out of sync. The files should sync as you expect, though.

[yukiomishima]

JD

thanks for the prompt response

the weird thing is..... the files inside the folders are being modified... but the enclosing folder's modification date is not always updated auto by the system... so am getting weird results

if i exclude folders... what happens if i have files buried say 3 folders deep... will the file/ folder structure remain in tact?

thanks again

yukioMishima

[JD]

Well, maybe it seems weird, but that's just the way it is - no way to change that.

If you don't filter folders, but a file 3 levels deep changes (for example), then yes, the directory structure will still be maintained, creating folders on the server as necessary.

Give that a shot and see how you go.

[yukiomishima]

JD

major issues

i am attempting to sync as per above

however...... even though i seem to have both folders (server/ client) selected correctly, it would seem that when i sync it seems that all of the files/folders that need to be sync'ed are uploaded in COMPLETELY the incorrect directory.... not even close

it has been happening for ages and i thought it might have been something i did..... but it would appear that it is the application

this is proving to be HUGELY problematic

any thoughts?

thanks

yukioMishima

any thoughts?

[spacific]

Have you read this thread:

http://www.yummysoftware.com/forum/index.php/topic,581.0.html

Make sure you are using the latest version of Yummy.

[yukiomishima]

thanks for the reply

downloaded and installed 1.7.1 b5

HOWEVER.... i have found the problem to be still there

after 24 hours of waiting for the "preparation" to sync (i have 300 gb of data that i am sync'ing with an offsite ftp server)..... i find that the data is being transferred to completely and utterly the incorrect folder

this is a MAJOR issue... as i now seem to have HUGE amounts of data both un-sync'ed as well as duplicated

any thoughts..... this is proving to be hugely problematic and a very big waste of time.... when i can't simply rely on the sync process and need to check with the application every few minutes to ensure that it is reading/writing the correct data there is something critically wrong with the process

thanks in advance

yukioMishima

[yukiomishima]

a further update

just going thru now and deleting the more than 15000 files that were uploaded into completely the wrong folders structure (and i stopped it even though it said it had TWO days left for the sync.... so who knows how many of the files were going to be uploaded)

it is not syncing.... but uploading everything again..... as it doesn't seem to sunc the 2 folders.... but simply find some random place to upload stuff and then merrily go about its business

this is becoming a little catastrophic..... i cannot spend my time second guessing something as simple as sync'ing 2 folders

a response/solution/workaround at your earliest would be hugely appreciated

thanks

yukioMishima

[JD]

The problem with folders in the wrong place after a synch has been confirmed as fixed by three people who reported it, so I am very surprised if you really are experiencing it and I am very sorry that I did not mention it before - I assumed you were already using the latest version.

Please try to delete all the duplicates (this is very important) and then try again, but this time, if something does go wrong, please select 'Save FTP Transcript' from the File menu and send me the resulting file.

Also please make sure that you have the correct client + server folders selected - the Synch setup window will show the names of the two folders to synch. Double check it.

Send the file to:

support at yummysoftware dot com

.. and tell me which folders are in the wrong place.


(I am sorry for the delayed response - this is due to Xmas time)

[yukiomishima]

JD

thanks for the reply

i did not realise til recently that this was happening... i have hundreds of thousands of files in the incorrect place... and it is only when i sit and watch the sync operation that i notice that things are in the incorrect place (i have multiple>multiple>multiple embedded files/folders.... so it would take me a week to go thru and verify that everything is in the correct location)... and trust me... i have WAY BETTER things to do with my time

maybe the fact that there is stuff in the incorrect location that it is throwing it off?

but... for example.... with the latest sync.... i noticed that something was in the incorrect place (after 18 hrs of "preparing") so thought it best to leave it and have yummy delete it with the orphan command

the incorect files/folders were in... say folder A>subfolder B

however... with the latest sync.. ALL of the stuff that was being "sync'ed" (and i use that term VERY loosely) were inside folder D> subfolder X

how does that happen?

i have verified that i am sync'ing the top level of BOTH server & client

thanks

yukioMishima

[yukiomishima]

another thing

seems ludicrous that i now need to go thru and remove all of the stuff that yummy "sync'ed" incorrectly and then run the sync again

i thought the idea of sync'ing was that it sync'ed two folders... and as per your above comment... the reason that the "preparation" stage takes so long is that it checks both directories each time it runs (no cache file etc)

so

if it was doing it's job correctly why wouldn't it upload what i want it to... and delete the stuff on the server that is orphaned?

(just out of interest... i still have it set to do "recursive batch transfers"... is that OK?)

to spend days and days going thru all of the files/folders on the server to verify that they are in the correct place doesn't seem like something i should have to do manually

thanks

yukioMishima