Catalog "Speed optimizing"
Hi all,
I have been having some issues with "ghosts" in the catalog. I noticed that the backup size was the same size, even when removing all the images........The issue has been logged with support.
But today i created a new catalog (9.0.3), imported my main catalog, generated all the previews. Did the backup, and
the backup size is now 1/6 size. And the new catalog opens within 15 seconds. The "old" one, took 1.30 min.
Apparently some cleaning is going on, and i have double checked that all images etc. is present......
So for those of you, whom are having issues with speed, this might also work for you......
Best...Jan
I have been having some issues with "ghosts" in the catalog. I noticed that the backup size was the same size, even when removing all the images........The issue has been logged with support.
But today i created a new catalog (9.0.3), imported my main catalog, generated all the previews. Did the backup, and
the backup size is now 1/6 size. And the new catalog opens within 15 seconds. The "old" one, took 1.30 min.
Apparently some cleaning is going on, and i have double checked that all images etc. is present......
So for those of you, whom are having issues with speed, this might also work for you......
Best...Jan
0
-
Were those catalogs upgraded from CaptureOne version 8?
All of my larger catalogs perform poorly when using the keyword library and I have a support ticket open for that. But those have all been upgraded catalogs. Have already been wondering if that might be related but didn't have the time yet to experiment with a new catalog and import one of the old ones (and actually feel it is the job of Phase One to identify and fix these problems).0 -
Those catalogs came from V 8, but had been upgraded til V 9.
But still had the same issues, speed, ghosts, large backup files etc.
But when importing into a new catalog, it seems to only import the necessary stuff,
and not all "ghosts", which should not be present.....0 -
[quote="NN635396322977093750UL" wrote:
it seems to only import the necessary stuff,
and not all "ghosts", which should not be present.....
What are "ghosts"?
I am currently creating a new catalog and am busy importing a 14K catalog. That alone is taking hours.
Will let you know if I see any differences.0 -
I have no idea how PhaseOne managed the catalogs programming wise, so it will be "a hunch".
I noticed that previous deleted files and adjustments, was still available in the catalog, long after the files where deleted.
The reason i call them ghosts, is that you cannot see them, but they exist somewhere in the catalog, where you as the user cannot see them.
I had a catalog, where i deleted all the images, and the backup catalog size was still 7gb. So something was hiding in the catalog, some where....0 -
[quote="NN635396322977093750UL" wrote:
I had a catalog, where i deleted all the images, and the backup catalog size was still 7gb. So something was hiding in the catalog, some where....
Did you empty the catalog trash can?0 -
Yep, as i mentioned, the bug has been logged and confirmed by Phase One 😉 0 -
I am finally done with the import of 14K images.
The database file in the new catalog is larger (89.2 MB vs. 86.7 MB).
Cache for thumbs and previews is smaller (17.5 GB vs. 18.94 GB). I might have changed preview size a while ago, which would explain why the new previews need less space.
So far the new catalog does not behave better, in contrary. As I am writing this C1 is just showing me the unresponsive spinning ball with no information in activity window after clicking on 'All images'.
Had to force quit and then did a few more tests. Performance is as poor as with the original catalog that was converted from version 8. It is pathetic.0 -
In my case on 9.0.3 performance of opening 15K catalog, still slow, like 3 min to open catalog so I do preffer not to close it at all. Changing preview size helped a little.
BTW are using SSD or regular HDD?0 -
I have all my raw images on a single NAS (despite the very latest ones, for a while), all beneath a follder structure in the same main folder. If the NAS is offline then C1 would theoretically be able to determine that ALL images are offline.
But from in the application.log file, as it takes many minutes and there are dozens of log warnings I believe C1 checks for every single image (or subfolder) whether or not the image is online.
If the NAS in online, it takes also several minutes (5 min. for 4k images) to load the collection "All images".
I think C1 has room for improvements here, maybe a on-demand online check on an image per image basis.0 -
Yeh, NAS setup is completely not reccomended if you want fast workflow, that is why my active catalog is always on my SDD.
Then it works.
Otherwise you getting to issues that you just described.0 -
[quote="Andriy.Okhrimets" wrote:
Yeh, NAS setup is completely not reccomended if you want fast workflow, that is why my active catalog is always on my SDD.
Then it works.
Otherwise you getting to issues that you just described.
Depends on how you use the catalog. You are correct, that the catalog should be on a fast drive (like internal SSD or very fast external drive). But the raw files could be on a NAS as referenced files as they are only needed for processing (export, generate previews, etc.) or individual image rendering. But XMP sync should be turned off then as those XML files are a the same location as the raw files.
In the catalog you have the sql database, thumbnails and previews (and original raw files if not using referenced files). Problem is that many of the performance problems (keywords, 4K/5K iMacs, ....) some of us have been reporting are related to the catalog itself and it is unbearably slow even on an internal SSD.0 -
[quote="BeO" wrote:
I have all my raw images on a single NAS (despite the very latest ones, for a while), all beneath a follder structure in the same main folder. If the NAS is offline then C1 would theoretically be able to determine that ALL images are offline.
But from in the application.log file, as it takes many minutes and there are dozens of log warnings I believe C1 checks for every single image (or subfolder) whether or not the image is online.
If the NAS in online, it takes also several minutes (5 min. for 4k images) to load the collection "All images".
That is an interesting thought. Theoretically it should not even access the NAS at all just to show the thumbs of 'all images' since the thumbs are in the catalog. But in order to display the 'offline' warning it would probably do that and that is of course slowing it down significantly then if it has to access the NAS for the status check. Disabling the online/offline check for the browser view should then improve performance significantly. You can change the browser labels to off but I doubt that that will skip the online status check.0 -
So, here are some figures:
Catalog on internal SSD on my computer, raw images on HDD based NAS. 4000 images.
I checked both v8.3.3 and 9.0.3, it's roughly the same. I've done the tests several times.
Loading "All images" in the browser at startup
Computer not connected to network:
80 sec.
Computer connected to network but NAS OFFline:
130 sec. *)
Computer connected to network and NAS ONline:
320 sec.
So, whatever is done for online images, why please does this need to be done when a collection is loaded??
Per image, it is roughly 0.05 sec. additional time. I would be willing to wait 0.05 seconds longer if I load an image the first time to the viewer and save several minutes at collection loading!
I will switch to faster hardware very soon, but what about 40k images?
*) application..log:[2016-02-06 01:20:09.874][736][ID:034, Persistenc]{FS } | WARNING: Sync task takes more than 50000 ms to execute... (GetDirectoryFromPath - True)
[2016-02-06 01:20:11.138][264][ID:032, [Collectio]{FS } | WARNING: Sync task takes more than 50000 ms to execute... (GetFileFromPath - True)
I don't know what C1 does for all online images, but it this needed for all images? I have no sidecar xmp files, btw.
Not all edits can be done offline, as far as I know, but I think it would be totally sufficient to check online status if
- an image is loaded into the viewer,
- or the image is processed,
- or when starting a process recipe for a batch of images.
In addition, this would have the benefit that the online status could be updated; currently, if I open the catalog with NAS offline or withouth network, then start the NAS afterwards (or connect to the network), the images will still show "offline" and I have to close and restart C1 to get images online.
For catalog/folder management, or just because you want to check the online status, when you'd maybe want to see the online status for all (or a collection of) images, there could be a button to start the online check for all, or for the currently selected collection or folder.
Also, a checkbox in the preferences would be possible to retain the current behaviour.
Last but not least I think if it benefits a NAS "setup" for the raw images, it might likely benefit an internal HDD and SSD setup too, probably to a lesser extent but stil...
Cheers
BeO
P.S. btw, what means "Mask corrupted, regenerating." in the application.log? I see this from time to time in the log.0 -
I imported my main catalog into a new catalog. The original images are referenced. The new catalog is still loading up thumbnails and so I cannot yet tell for sure if there is a speed improvement. However the process seems to have omitted to import 501 out of 43,613 images and my question is how do I identify the missing images? There does not seem to be a function to exclude import of duplicates. 0 -
Result was no improvement either in startup speed or speed of opening "all images" in catalog. So I have deleted the new catalog and resumed use of the previous one. 0
Post is closed for comments.
Comments
15 comments