Skip to main content

⚠️ Please note that this topic or post has been archived. The information contained here may no longer be accurate or up-to-date. ⚠️

Upgrading large catalogs (>50k images) - takes some time...

Peter Grüner
Hi,

I do have a catalog with referenced files on my internal hard disks on my MacPro late 2010 and I am running the update processing method from Capture One x to Capture One 10. My machine tells me that it takes 5+ hours (it runs already for 2 hours) to complete. I must admit that I have not performed these tasks previously but I am wondering if someone could share his thoughts/workflow managing large catalogs and sessions in Capture One?

Thanks,
Peter
0

Comments

11 comments

Date Votes
  • HansB
    I have just loaded a 10k images referenced (images on mobile USB HDD, online) v9 catalog in v10. It took less than a minute. CO took a few minutes to save it (slow old MBP HDD), then I was ready to go.

    Are you upgrading the processing engine for all image variants?

    If yes, be aware that this can change the look of already adjusted images dramatically. During evolution of the processing engine, tools like highlight recovery have changed significantly. For the images I already like, I would create a clone and upgrade the engine of only this clone.


    Regards,
    Hans
    0
  • peter Frings
    [quote="HansB" wrote:
    I have just loaded a 10k images referenced (images on mobile USB HDD, online) v9 catalog in v10. It took less than a minute. CO took a few minutes to save it (slow old MBP HDD), then I was ready to go.


    Would you like to share how you get those crazy short times? That must be some serious iron you have. I don't even dare to make a catalog that big anymore. I tried converting my 60K catalog, but that's an absolute failure. I gave up on that.

    Cheers,
    Peter.
    0
  • SFA
    [quote="peter.f" wrote:
    [quote="HansB" wrote:
    I have just loaded a 10k images referenced (images on mobile USB HDD, online) v9 catalog in v10. It took less than a minute. CO took a few minutes to save it (slow old MBP HDD), then I was ready to go.


    Would you like to share how you get those crazy short times? That must be some serious iron you have. I don't even dare to make a catalog that big anymore. I tried converting my 60K catalog, but that's an absolute failure. I gave up on that.

    Cheers,
    Peter.


    Peter,

    Your signature suggests that your referenced images are on an external USB2 drive.

    Is that still the case?

    What sort of data transfer rate do you get from the external drive?


    Grant
    0
  • HansB
    [quote="peter.f" wrote:
    ...
    Would you like to share how you get those crazy short times? That must be some serious iron you have.
    ...

    The 'serious iron' i have used is a Macbook Pro, mid2009, 2.8GHz Dual Core, 8GB RAM, 1TB HDD with USB2 ports and no OpenCL support. Catalogs are on the internal HDD with a maximum of 70MB/s writing and 100MB/s reading.

    Referenced images are on external USB3 disk, simple WD Elements, with a maximum of 35MB/s reading+writing connected to the USB2 port.

    All catalogs were v9.3 compatible before upgrading to v10. No new previews created.

    I upgraded all my catalogs yesterday. 500 to 10k images per catalog. Looking at the 'Date modified', none took more than a minute.


    Regards,
    Hans


    Edit: a typo correction.
    Edit/PS: CO took the extra minutes for saving only for the catalog I copied to a new location to test upgrading.
    Edit: corrected the author of the quote. See Grant's reply below.
    0
  • SFA
    Most interesting Hans.

    The comment you quoted was from Peter not me BTW.

    SQL databases can suffer performance issues that seem quite dramatic once you reach about 40k records - or so I have read and been told by sources connected with other types of application.

    In a corporate world one can move up the scale of SQL options provided one have the budget to afford that!

    However even extending the time estimate considerably for larger catalogues it does seem that some installations are much slower than others for some as yet unidentified reason.

    Grant
    0
  • HansB
    [quote="SFA" wrote:
    ...
    The comment you quoted was from Peter not me BTW.
    ...

    Oops, sorry. My fault. I'll fix that.


    Regards,
    Hans
    0
  • Peter Grüner
    Dear all,

    Thanks for you input. I envy the performance that Hans experiences. Maybe I should explain my setup a little further. I primarily work with sessions as preferred workflow because that is the most convenient and quickest way to work on one specific project. All sessions including the files are stored internally on two WD 4TB hard drives running in RAID0 mode for performance and storage capacity. All images are stored on a separate eSATA connected G-Technology GSpeed ES drive as well as on my Synology NAS.
    However, my workflow has one constraint: If I want to search across my images for a certain lens, camera or whatever you could think of I can't do this across sessions.
    Therefore, I had the following idea to import all images from 2014-2016 as referenced files into Capture One and do the same for 2009-2013, etc.
    Now I made the decision to do this year by year and I am currently creating a catalog of images for 2016, then go back year by year.
    Let's see how this turns out in general performance. The good news are that all images are now using the default Capture One 10 processing engine and yes I am aware of slight differences but that does not matter too much for me.

    Peter
    0
  • OddS.
    [quote="SFA" wrote:
    ...SQL databases can suffer performance issues that seem quite dramatic once you reach about 40k records - or so I have read and been told


    As a general property of a RDBMS? Your source was pulling your leg.
    0
  • peter Frings
    Hi Grant,

    [quote="SFA" wrote:
    [quote="peter.f" wrote:
    [quote="HansB" wrote:
    I have just loaded a 10k images referenced (images on mobile USB HDD, online) v9 catalog in v10. It took less than a minute. CO took a few minutes to save it (slow old MBP HDD), then I was ready to go.


    Would you like to share how you get those crazy short times? That must be some serious iron you have. I don't even dare to make a catalog that big anymore. I tried converting my 60K catalog, but that's an absolute failure. I gave up on that.

    Cheers,
    Peter.


    Peter,

    Your signature suggests that your referenced images are on an external USB2 drive.

    Is that still the case?

    What sort of data transfer rate do you get from the external drive?


    Grant


    Painfully slow.: 26.7 MB/s write, 30.8 MB/s read. It's also a Western Digital Elements, 2TB. But, the catalog is on the internal SSD which fares a little better: 70MB/s write, 266MB/s read. Not mindblowing, I admit.

    It's just two things
    - On the very same hardware, both LR and Aperture are handling large catalog (40K) without sweating.
    - C1 consumes huge amounts of memory; I've had C1 take 24GB of RAM before my mac gave up. 24GB!

    So, what's the difference between Hans' set-up and mine? His mac is even a tad older, but the RAM, disk config seems to be the same. OpenCL is off (it wouldn't even help because of the X-trans sensor...).

    Cheers,
    Peter — being baffled...
    0
  • SFA
    [quote="OddS" wrote:
    [quote="SFA" wrote:
    ...SQL databases can suffer performance issues that seem quite dramatic once you reach about 40k records - or so I have read and been told


    As a general property of a RDBMS? Your source was pulling your leg.


    It depends whether the database set up is tuned for speed (anticipating relatively small number of records mainly and users demanding instant responses) or volume (the opposite). Or somewhere in the middle.

    Also what indexation is going on and how it needs to be applied at creation.

    By "quite dramatic" I mean that what may be instant with 100 records takes a little noticeable time with 10k records and slightly more time with 40k records (normal business database type stuff rather than images). Once you head towards 100k records it takes a little long again if back calculated to a "per record" value.

    Throw in a few million records and our perception of processing time may lead to some dissatisfaction - depending on the level of sql database deployed.

    Somewhere I may still have the test results from a few years ago. They covered a period of experimentation with settings (on behalf of a development company as they tuned their application during beta testing) and that was the explanation of what they were doing at that time. It's a source I trust.


    Grant
    0
  • OddS.
    [quote="SFA" wrote:
    ...Somewhere I may still have the test results from a few years ago.


    No worries, I already have my fair share of tests and data sets for several database systems.
    0

Post is closed for comments.