Constant disk activity in C1 20
I have recently decided to migrate from Lightroom to C1. I have a fairly small image database - about 6500 images residing on a NAS drive. I imported this into C1 with the catalog on my local drive (on a Mac Mini). The import seemed to go OK but I found that C1 was continually accessing my NAS drive - even when C1 was idle, ie I was not doing any editing. It is not updating Previews or Thumbnails - I have been monitoring these folders in the .cocatalog package. This behaviour is a major problem because the continuous actvity is hammering my drive which I am sure will increase the likelihood of failure.
I have contacted Support but they have not been very helpful - the only suggestion I have had has been I should split up my C1 catalog - but 6500 images in one catalog doesn't sound like a big deal - Lightroom can cope with no problem.
Willing to try anything I split the catalog on 'year' boundaries. I ended up with several smaller catalogs, the largest was 1800 images. Surprisingly when I load any of these smaller catalogs I don't see the continuous activity I see with the combined catalog.
So I decide to try to import them one at a time into a new catalog. Each time I imported a smaller catalog I tested the disk behaviour - all seemed good until I got to over 4500 images when activity started up again.
Being somewhat familiar with Linux I decided it was time to do some serious digging into process activity. Using various diagnostic, eg dtruss, opensnoop, fs_usage on the 'Capture One 20' process (I'm running High Sierra) I have established that C1 seems to be continuously iterating over my image folders. To be precise it is calling getattrlist() on all files in the image folders referenced in the catalog, regardless of whether they are images or not. It is not calling read() and not writing anything (backing up my assertion that it is not updating Previews and Thumbnails) - it just seems to be gathering file stats on everything. This is happening regardless of the catalog size but the reason I don't see activity on smaller catalogs is that, I am assuming, the smaller number of getattrlist() calls access the disk cache and the disk doesn't physically have to search for the file. I can see no sensible reason why C1 is doing this - catalogs are not supposed to monitor their folders.
Has anyone else seen similar behaviour and got any ideas for a remedy? I have tried all the usual suspects, like validating the catalog, setting the preview cache size, etc.
If you want to see what your C1 is doing on a Mac open a terminal window and type this:
sudo dtruss -t getattrlist -n 'Capture One 20' | & tee co.log
Then fire up C1. After you close C1 you can browse co.log to see what files were accessed.
FYI, there is a post with a similar issue to what I've described above but I don't think it is the same problem.
-
Hi Paul,
It remembers to me that some people already mentioned, in the previous forum, this continuous iteration (mostly for Windows) that I haven't check as far as I am concerned.
And the second remark is that Capture One doesn't like very much NAS. You should also find some threads on the web about this issue.
Robert
0 -
Hi Robert - thanks for your feedback. I'll see if I can find the threads you mentioned. I guess the central issue for me is that C1 is iterating over catalog files and accessing file attributes. I can see no reason for this since catalogs are not 'smart' like sessions are supposed to be, ie they are not supposed to monitor image folders for any activity. Unless C1 s/w engineering can come up with a valid reason for this I would say it was a bug.
Paul
0 -
I suggest sending these details into a support case you provided here (or just link to your post) with the request to send it to the test department. This is not a case for the support department.
0 -
Interesting. I am trying
sudo dtruss -t getattrlist -n 'Capture One 20' | & tee co.log
but I'm getting "-bash: syntax error near unexpected token `&' "
I think what was meant was this
sudo dtruss -t getattrlist -n 'Capture One 20' | tee co.log &
In 10.14.6 I would have to turn System integrity Protection off for dtrace and dtruss to fully work, right now I'm just getting empty logs.
0 -
The syntax error was the space between | and &. It should read like this.
sudo dtruss -t getattrlist -n 'Capture One 20' |& tee co.log
Without the "&" the log file will never be written.
FWIW I am not seeing getattrlist calls to my original files unless I'm actually doing something like perusing image folders. My images live on an external disk -- catalog is on my main internal disk. Catalog contains about 48,000 images.
I suspect the issue is related to the network store.
0 -
Hi Folks - thanks for the various comments.
<ericnepean> - Sorry for the extra space typo - hope you have managed to run the command now. FYI, I did not have to change my Mac's System Integrity Protection to sudo dtruss on Capture One under High Sierra. Not sure how later OS's will behave.
<marchyman> - Interesting you are not seeing the same kind of file activity that I am - though I assume that if this was a bug manifesting all the time R&D would have seen it before. It could be network related but since Capture One help talks about storing images on network drives to allow multi-user access one would expect this method of working would have been thoroughly tested!
<Walter> - Do you mean the Preferences setting 'Auto Sync Sidecar XMP'? Mine is set to None.
I'll follow Paul Steunebrink's suggestion and file a support ticket and ask for it to be routed it to Test.
Thanks
Paul
0 -
Hi Paul - as of MacOS 10.14.5 Apple introduced heightened security restrictions which disables certain privileged operations, even if you are root. I gather it is even more so on 10.15. Since you are still on 10.13, your won't see this until or unless you upgrade your OS. I plan to stay on 10.14.6 for another couple of years.
0 -
@Paul Traynar,
Out of interest if you move the image files onto the same internal drive as the catalogue file do you see the same disk activity?
That you do NOT see the same activity with smaller catalogues might suggest that their files are all being read into RAM eliminating any potential need to monitor the source file disk(s) although periodic saves might be applied anyway.
How much of the available memory is C1 able to use?
It's possible the log files might throw some light on what is going on and it sounds like your technical expertise should allow you to interpret what they report.
Grant
0 -
A little research tells me that
|&is a shorthand for2>&1 |Fotr some reason my Bash shell doesn't recognise this combination, I don't know if it's due to a missing alias or a missing setting.
Now that I have it working, I can see both COP 12 and COP 20 scanning a lot of stuff, not just the image files. I would seem reasonable that it scans every image in the current collection. If the current collection is All Images, that's a lot of files.
0 -
<ericnepean> - OK, I'm generally reluctant to upgrade so your observation about 10.14 is another reason not to. Thanks for the heads-up.
<Grant> - I have not tried moving the images onto my local drive - I'd have to free up a lot of space to do the experiment. If that fixed the problem it would be interesting but I am not sure it would solve anything since I would still need C1 to behave properly with images on my NAS. Lightroom has not had any problems with my NAS image storage. If I had to buy another large (external since I can't upgrade my internal) drive for my Mac to fix C1's problems I might just as well put the cash towards a few years of Lightroom subscriptions - which I'd hate but I'd be using a storage/backup setup that I know.
Also I do see the getattrlist() activity on smaller catalogs, but it seems that because the catalog is smaller the number of images it's polling is smaller and is probably hitting the cache and so I don't see/hear the drive being hammered. FYI, my machine has 16G RAM.
BTW I have just filed a new support ticket and asked for it to be redirected to the test department.
Thanks
0 -
Hi <ericnapean> - just out of interest if you:
- Setup C1 to view a small (ie 2 or 3) image collection and then close it down.
- Then run the dtruss command at your terminal prompt.
- Then start up C1 and after it has settled down, let it run for say 30secs then close it.
Do you see file activity in C1 on images beyond what you selected in the small collection? I'm sure you've had a look at the full dtruss output (ie by removing the -t option) - you might see a bunch of other file related system calls indicating C1 is polling files.
0 -
I've already done something like that, I'm using my Incoming Catalog with 4300 images, but I've set C1 to select a collection with 10 or less images. I don't see any polls of 4300 image files, it seems to be for the images in the selected collection.
Fortuitously, I have catalogs of 4300 and 16000 images, for Capture One 12.1 and Capture One 13.04 and a version I'm unable to discuss. The comparison should be interesting.
I'll have a better look at it after work.
0 -
I admit I did not fully understand everything here especially the Mac specifics. If my comment is useless then ignore.
Especially in a (potential) multi-user environment, doesn't it make sense to scan the common ressources regularly? I don't know though how exactly the multiuser setup of C1 works...
LR cannot run multi-user so no need to do so. Likewise no need to scan periodically if everything is located locally.
However, I think it would be sufficient to scan a files' "online" state (if that would be the reason) when you access it, or the collection it is in.
0 -
Hi BeO - thanks for your comments. My understanding of C1 catalogs is that they are dumb, ie they reference read-only images and are not supposed to respond to images being dumped (by some other application) in the folders referenced in the catalog, unlike sessions. So assuming that the thumbnails and previews have been generated, the only reason C1 would need to keep polling an image location would be to determine whether it's online or not, and it would only need to do that for images that are visible in the viewer which is generally a small number. So even large collections would not need to be completely scanned. Of course I'm speculating how things might work in C1! Software development takes many turns and sometimes things don't work the way one might expect.
0 -
Paul T,
A couple of thoughts.
My NAS, unless it has decided to go into some sort of hibernation for a while, can be very busy doing internal stuff much of the time. I assume it is managing disks and checking for "indexing" and reporting back to base since I receive test messages form time to time if it has stopped communicating with base and then later when, usually, it starts communicating again.
It connects to everything via WiFi and I think its cabled connection point, a WiFi extender device, may have some issues these days. Either that or the NAS has some issues that "freeze" the WiFi. I have a replacement device to see if that makes a difference but have yet to get around to testing it. (As yet it is still a somewhat mild annoyance.)
My notebook has 24Gb ram. WiFi connection speed is good enough, most of the time, to run an edit session (Usually a C1 Session but some times a small test catalogue with Catalogue file and images on the NAS) with up to several thousand images at a reasonable speed most of the time. Good enough for a quick visit. I would not want to work on full scale projects using the NAS (it would be usable but not as good as a local drive) and would probably not appreciate a very large catalogue either. I have 2 internal SSD drives for normal operations.
Also I am Windows 7 not MAC. The notebook is just over 7 years old.
I note that the log file for Application has repeating activities even when nominally idle.
Approximately every 5 minutes the log shows "Autosaving Documents" but does not obviously specify which "documents" it refers to.
Also, approx every 3 minutes, it records current memory statistics.
The section of log file I have quickly glanced at relates to a low activity time working on a session with about 2200 images in the current selection.
I don't think the messages are significant (minor disk activity) but you may see something different for a Mac working with a catalog.
Grant
0 -
Hi Grant - thanks for your observation. I seem to be seeing much different activity to you. I guess I'm going to have to wait for Capture One Test/R&D people to look at my data and decide what to do.
Thanks
0 -
Paul T,
A long time and many C1 versions ago I seem to recall reading an article that outlined the C1 philosophy for presenting images from a selected folder in a Session (this was before the catalogue concept was introduced but the principles would likely be the same) with a view to making the edit session as smooth and delay free as possible so far as the user's actions could be predicted.
So the images would be pre-loaded into memory and pre-presented according to the selected "Process Recipe" (where used) and some effort was made to assess and pre-prepare the next few images that the user might view perhaps depending on the sort order used for the browser and the way that the user was working.
The idea described was to make it the "edit and move to the next image" as smooth as possible. A little (or maybe sometimes a lot) of pre-preparation in the background in order to try to make the moment of edit a smooth experience.
Obviously the smooth experience will be difficult to always guarantee. People do not necessarily always work consistently! But where there is a logical flow or perhaps a need to refer back to recently edit images, it would be possible to manage behind the scenes to keep memory populated with pre-prepared thumbnails and previews ready to be loaded to the screen.
As processing power and memory availability have improved the benefits of such an approach, assuming it is still employed, would likely be less obvious to the user but presumably no less useful.
If one can get everything into available memory then a one time read batch job and some perpetual write activity (for instant storage of changes as they are applied thus largely avoiding any work loss in the event of a system crash) would seem to be logical. Likewise some of the changes may affect things like filters - especially "smart" filters - and so require constant monitoring to be fully effective.
Constant activity seems to be a normal state when part of the on-line real-time world.
It would be interesting to hear what the support team have to say.
Grant
0 -
Hi Folks - I have some follow up results to report on this issue. I normally mount my NAS drive using the AFP protocol - this seems to be the recommend method for NAS drives mounted on Mac's. I have discovered that if I mount the drive using SMB/CIFS the continuous disk activity ceases. In all other respects the images are fully accessible to C1, but my NAS drive is no longer being hammered. The only downside to this change is that SMB/CIFS (at least earlier versions of SMB) is supposedly less efficient than AFP so reads and writes are likely to be slower.
I have reported this finding to Capture One as a followup to my extant support request since this would seem to demonstrate a bug.
0 -
Apple is evolving towards SMB for sharing.
For example, this Apple support document states that APFS volumes can only be shared with SMB in Catalina, AFP not supported. https://support.apple.com/en-ca/guide/mac-help/mh17131/mac
It might be an idea to run a speed test on the AFP share and on the SMB share, if the SMB data transfer rate is comparable or better and the stability is good, .....
0
Post is closed for comments.
Comments
19 comments