Large Catalog performance
Hello,
I have some issus with larges catalogs, i have freeze, excessive memory used, etc. I need some time to kill Capture One because nothing append, memory is at 70GB, i check if the process do something with fs_usage command (and i see nothing, so Capture One sleep) and some time i have the wheel of death a very long time.
I have a good computer (MacPro 2013, 32GB RAM, Promise Pegasus). My catalog have between 20.000 and 80.000 pictures. I have work before with Aperture 3 and big catalog with 300.000 has never been a problem.
Strange thing my old Capture One 8 catalog converted to CO 9 don't have problem.
Anything to optimize the SQLite3 Database (because i really thing it is a problem with the database).
Best regards
I have some issus with larges catalogs, i have freeze, excessive memory used, etc. I need some time to kill Capture One because nothing append, memory is at 70GB, i check if the process do something with fs_usage command (and i see nothing, so Capture One sleep) and some time i have the wheel of death a very long time.
I have a good computer (MacPro 2013, 32GB RAM, Promise Pegasus). My catalog have between 20.000 and 80.000 pictures. I have work before with Aperture 3 and big catalog with 300.000 has never been a problem.
Strange thing my old Capture One 8 catalog converted to CO 9 don't have problem.
Anything to optimize the SQLite3 Database (because i really thing it is a problem with the database).
Best regards
0
-
I'm very interested in your experience.
My 30 day trial with v9.1 finished yesterday, and was generally very successful. I worked my way though a long list of issues and found ways to deal with just about everything of significance. But the one thing I couldn't conclusively satisfy myself about was the performance of catalogues of more than 1,000 images or so. My test catalogue of that size was taking about 10-15 seconds to open (I also have a late-2013 MacPro). My LR catalogue containing around 250,000 images opens and presents images ready to go in about half that time, so the portent is ominous. Assuming my transition strategy was just to begin building a CO catalogue with new images only, as I shoot around 50,000 per year I could be in trouble in just a few months.
Leaving aside the preliminary nature of the new and very promising DAM functionality introduced in v9.0, the catalogue performance issue seems to be the most significant outstanding problem with CO at this point. Surely there must be serious work going on to remedy this situation, but I have to conclude the required rewrite will not see the light of day before v10.0 - not in a v9.x point release. So I would need to buy v9.x then outlay the further cost of upgrading to 10.0 for CO to be viable for me. That's one hell of a lot of money.
So I'm compelled to wait until PO delivers industrial strength catalogue performance, then I'll check it out with another trial. In the meantime I'll continue to follow this issue with keen interest in the positive expectation that moving to CO will not be a case of "if" for me, but "when".0 -
[quote="yann.queniart" wrote:
Anything to optimize the SQLite3 Database (because i really thing it is a problem with the database).
Yes, I totally agree. All the efforts should be focused now towards the implementation of a good and well managed SQLite database. My catalog contains 16000 references (my RAW are on an external HDD). C1 needs about 10-15 secondes to open (compared to 5 seconds in Lightroom, when I use it sometimes), and a search with the generic filter takes about 10 seconds when I make it for the first time (it's almost without delay thereafter), compared to almost nothing in Lightroom.
So, Phase One has to improve the SQLite database !!0 -
[quote="yann.queniart" wrote:
I have a good computer (MacPro 2013, 32GB RAM, Promise Pegasus). My catalog have between 20.000 and 80.000 pictures. I have work before with Aperture 3 and big catalog with 300.000 has never been a problem.
This is a well documented problem of C1 here in this forum. Many former Aperture users complain about this and I have told this story many times, that there are Aperture databases out there in the wild which easily handle a six digit figure of images and perform flawlessly. Your 300.000 images example is the best case in point which could be made. And this all with a piece of software, that has not been updated for I don't know how long, is it five years now?
And yet we are told again and again by some C1 apologists here, that this is all insane and NOBODY would ever use a RAW software to catalog that many images and so on...
Fact is: C1 is YEARS behind the competition in this area and needs to seriously step up its game.0 -
[quote="yann.queniart" wrote:
Hello,
I have some issus with larges catalogs, i have freeze, excessive memory used, etc. I need some time to kill Capture One because nothing append, memory is at 70GB, i check if the process do something with fs_usage command (and i see nothing, so Capture One sleep) and some time i have the wheel of death a very long time.
I have a good computer (MacPro 2013, 32GB RAM, Promise Pegasus). My catalog have between 20.000 and 80.000 pictures. I have work before with Aperture 3 and big catalog with 300.000 has never been a problem.
Strange thing my old Capture One 8 catalog converted to CO 9 don't have problem.
I had exactly the same with 9.0.1 -9.0.3 and it was getting more and more frustrating, but not anymore with 9.1.
C1 9.1 is fast (except for the amount of time it takes to load the catalog at startup) and reliable ever since to me.
(45'000 images referenced on Promise Thunderbolt DAS, C1 Catalog (170 GB) on internal SSD on Mac Pro 2013 with 32 GB RAM)
I'd recommend you to open a support case with Capture One Support.0 -
I have been seeing improvements in catalog use, but I have a catalog right now that is 13K images, 70GB in size, referenced files that simply will not open. Every time I try it hangs and I have to reboot.
So a couple of questions to people with large catalogs:
How much ram do you have installed?
Is your catalog local referring to files on an external hard disk?0 -
[quote="UCS308" wrote:
So a couple of questions to people with large catalogs:
How much ram do you have installed?
Is your catalog local referring to files on an external hard disk?
I have 16 MB RAM and my files (about 16000) are on an external HDD.0 -
[quote="UCS308" wrote:
I have been seeing improvements in catalog use, but I have a catalog right now that is 13K images, 70GB in size, referenced files that simply will not open. Every time I try it hangs and I have to reboot.
So a couple of questions to people with large catalogs:
How much ram do you have installed?
Is your catalog local referring to files on an external hard disk?
Are you telling us your files are on an external hard disk?
What happens if you disconnect the external drive and try to start C1?
Grant0 -
[quote="SFA" wrote:
[quote="UCS308" wrote:
I have been seeing improvements in catalog use, but I have a catalog right now that is 13K images, 70GB in size, referenced files that simply will not open. Every time I try it hangs and I have to reboot.
So a couple of questions to people with large catalogs:
How much ram do you have installed?
Is your catalog local referring to files on an external hard disk?
Are you telling us your files are on an external hard disk?
What happens if you disconnect the external drive and try to start C1?
Grant
In this instance the referenced files live on a raid box attached via thunderbolt. I created the catalog yesterday, on the same raid drive as the referenced files. I then moved the catalog to my local drive to see if the performance improved. The catalog opened once, I told C1 where to find the referenced files (updating the references). That went ok, but after closing and attempting to reopen the catalog, it never opened again.0 -
I have check my old Aperture library and Apple use SQLite 3 database. So I think it is really an optimisation of SQL query to do. 0 -
[quote="UCS308" wrote:
[quote="SFA" wrote:
[quote="UCS308" wrote:
I have been seeing improvements in catalog use, but I have a catalog right now that is 13K images, 70GB in size, referenced files that simply will not open. Every time I try it hangs and I have to reboot.
So a couple of questions to people with large catalogs:
How much ram do you have installed?
Is your catalog local referring to files on an external hard disk?
Are you telling us your files are on an external hard disk?
What happens if you disconnect the external drive and try to start C1?
Grant
In this instance the referenced files live on a raid box attached via thunderbolt. I created the catalog yesterday, on the same raid drive as the referenced files. I then moved the catalog to my local drive to see if the performance improved. The catalog opened once, I told C1 where to find the referenced files (updating the references). That went ok, but after closing and attempting to reopen the catalog, it never opened again.
To close this issue off. I moved the catalog back to the raid array and now it opens.0 -
[quote="yann.queniart" wrote:
I have check my old Aperture library and Apple use SQLite 3 database. So I think it is really an optimisation of SQL query to do.
There are two design principles in software development. One is to retrieve data freshly from the database on any action, the other is to use the database as a place to persist information which is otherwise loaded to memory at startup and then most actions are done in the program/memory.
I'd bet that C1 does not use the database very much but rather load everything into memory (into the internal program data structures) and performs most operations in memory/ on the internal data structure. E.g. if I filter I can see thumbnails vanish (those which do not meet the filter condition) rather than the browser is freshly filled with those images which meet the filter condition. The latter would indicate that a SQL statement is sent to the database and the dataset is freshly filled with the result.
Another reason for me to believe this is because C1 started as a great raw converter and editing tool which probably means a lot of C++ skills, and catalog / database applications require a different skillset and software design (and catalogs was / is a minor part of the software).
This is also backed by the heavy use of CPU and RAM.
I also see a lot of "one-by-one" treatment of images, e.g. check for online/offline status, application of adjustments to thumbnails etc.
And if you open a C1 and e.g. a Lightroom catalog you see a massive difference in the number of tables they have, and "database normalization" in C1 seems to be rather low. Although I can't tell for sure that there are SQL queries to optimize, I'd rather think the database structure/design and design pattern for the software as outlined above makes the difference.
The longer a user uses C1 the bigger his catalog becomes (provided that he uses a single catalog), hence the performance problems will get bigger. Therefore I believe P1 will see an increase in complaints and needs to react accordingly in mid future.
Just my thoughts, for what it is worth...0 -
You can actually optimise a bit the catalog.
- issue a backup with the optimise function.
- copy the backup'ed/optimised .cocatalogdb back into the original place.
This works a bit for me.
Maybe C1 could have a function to optimise it directly. Anyway, I have no idea what is doing this function exactly..
Take care[quote="yann.queniart" wrote:
Hello,
I have some issus with larges catalogs, i have freeze, excessive memory used, etc. I need some time to kill Capture One because nothing append, memory is at 70GB, i check if the process do something with fs_usage command (and i see nothing, so Capture One sleep) and some time i have the wheel of death a very long time.
I have a good computer (MacPro 2013, 32GB RAM, Promise Pegasus). My catalog have between 20.000 and 80.000 pictures. I have work before with Aperture 3 and big catalog with 300.000 has never been a problem.
Strange thing my old Capture One 8 catalog converted to CO 9 don't have problem.
Anything to optimize the SQLite3 Database (because i really thing it is a problem with the database).
Best regards0 -
I have a catalog of 128000 images.
Up and until v9.1 I could not build a full catalog in Capture One Pro.
The latest version 9.1.1 improves the stability of the catalog.
OK now the bad news.
It takes approximately two to three minutes for my full catalog to load.
In Lightroom a catalog for the same image set takes approx 1 minute to load.
In MediaPro a catalog of the same images takes a similar length of time to Lightroom.
If I go to a Windows machine and try the same image set catalog in iMatch v3.6 then it loads in 15-20 seconds. (I dont use Windows these days except for some rare testing)
I think that large catalogs (>50000 images) are probably something I would recommend people to avoid. Split your image sets into small catalogs of either types of images or years.
PhaseOne probably need to get some SQLlite database experts in to examine their data model and also their database optimisation methods and then implement this advice. I believe that there should be little or no performance difference between Lightroom and COP9.x catalogs if this was done.
On a positive note I know from helping in the beta testing that PhaseOne are making efforts to address this so I recommend that we try to give as much feedback to PhaseOne as possible on this item. Also we need to be patient as it wont happen overnight!0 -
[quote="jknights" wrote:
On a positive note I know from helping in the beta testing that PhaseOne are making efforts to address this so I recommend that we try to give as much feedback to PhaseOne as possible on this item. Also we need to be patient as it wont happen overnight!
Good news !!0 -
[quote="BeO" wrote:
[quote="yann.queniart" wrote:
I have check my old Aperture library and Apple use SQLite 3 database. So I think it is really an optimisation of SQL query to do.
There are two design principles in software development. One is to retrieve data freshly from the database on any action, the other is to use the database as a place to persist information which is otherwise loaded to memory at startup and then most actions are done in the program/memory.
I'd bet that C1 does not use the database very much but rather load everything into memory (into the internal program data structures) and performs most operations in memory/ on the internal data structure. E.g. if I filter I can see thumbnails vanish (those which do not meet the filter condition) rather than the browser is freshly filled with those images which meet the filter condition. The latter would indicate that a SQL statement is sent to the database and the dataset is freshly filled with the result.
Another reason for me to believe this is because C1 started as a great raw converter and editing tool which probably means a lot of C++ skills, and catalog / database applications require a different skillset and software design (and catalogs was / is a minor part of the software).
This is also backed by the heavy use of CPU and RAM.
I also see a lot of "one-by-one" treatment of images, e.g. check for online/offline status, application of adjustments to thumbnails etc.
And if you open a C1 and e.g. a Lightroom catalog you see a massive difference in the number of tables they have, and "database normalization" in C1 seems to be rather low. Although I can't tell for sure that there are SQL queries to optimize, I'd rather think the database structure/design and design pattern for the software as outlined above makes the difference.
The longer a user uses C1 the bigger his catalog becomes (provided that he uses a single catalog), hence the performance problems will get bigger. Therefore I believe P1 will see an increase in complaints and needs to react accordingly in mid future.
Just my thoughts, for what it is worth...
As someone who used to work in IT designing software I agree with these comments.
I think that the Sessions method of working is very much better as it means that only a small number of images are actually referenced and probably held in memory.0 -
With sqlite, the entire database must be in memory. That is the only way sqlite works.
In sessions and catalogs, images (the full image data), previews, and thumbnails are not stored in the database. You will find them in the Capture One Settings folders. Masks are also saved in the file system for both sessions and catalogs.
In sessions, adjustments are saved in the file system.
In catalogs, adjustments are saved in the sqlite database,
In both cases, keywords and other metadata are stored in the database and, optionally, in xmp sidecar files.
In sessions, favorited folders are in the database.0
Post is closed for comments.
Comments
16 comments