Skip to main content

⚠️ Please note that this topic or post has been archived. The information contained here may no longer be accurate or up-to-date. ⚠️

How long does it take to build a catalog.

paintbox
I'm asking anyone who has built a catalog OVER 30,000 images.
I'll start out by saying it took me over 36 hrs.
0

Comments

21 comments

Date Votes
  • Drew Altdo
    Mine took 4 hours.

    Now what do we do? Do I win? 😉

    And if the files are local... it's less than an hour!

    http://v8.tinypic.com/player.swf?file=348j7cy&s=8
    0
  • paintbox
    Well yeah, you win big time!!!!
    I'm trying again to build a catalog. This time i'm doing it with no sidecars at all.
    I'm at 18,000 images in five hours. This makes me sad.

    Would you mind divulging your current setup?
    0
  • Drew Altdo
    Nothing special...
    http://i58.tinypic.com/313k36b.png

    Files are all RAW IIQ Phase One Files from varying models.
    4 Hours = Network mounted volume
    45 Minutes = Local SSD

    I think the main change to achieve the speed is the import option to restrict the software for import only, and not also try to multitask.... http://i58.tinypic.com/b4vqdi.png

    When you use this, it just imports the files and links them to the Catalog. If you try to open the files while it's importing, it will take hours as it's multitasking. A big import while you have a coffee is much better than a snails pace system trying to do two things at once.
    0
  • Paul Steunebrink
    That is a great tip (not try to multi task), Drew. I think many users will benefit from this. Thanks for sharing.
    0
  • paintbox
    [quote="Drew" wrote:
    Nothing special...
    http://i58.tinypic.com/313k36b.png

    Files are all RAW IIQ Phase One Files from varying models.
    4 Hours = Network mounted volume
    45 Minutes = Local SSD

    I think the main change to achieve the speed is the import option to restrict the software for import only, and not also try to multitask.... http://i58.tinypic.com/b4vqdi.png

    When you use this, it just imports the files and links them to the Catalog. If you try to open the files while it's importing, it will take hours as it's multitasking. A big import while you have a coffee is much better than a snails pace system trying to do two things at once.


    Thanks so much! I'll try this next.
    0
  • paintbox
    Well that didn't do any better.
    I gave up on that one.
    After about 13,000 it slowed to a crawl.
    0
  • Drew Altdo
    [quote="Jimmy D Uptain" wrote:
    After about 13,000 it slowed to a crawl.


    And what would your system setup be?
    0
  • paintbox
    Mac Pro 2012
    Six core 3.33
    24Gb Ram
    7950 3GB Mac Edition video card
    Boot/App drive: Samsung 840 Pro SSD 512 in the PCI Express slot.
    Catalog is on the SSD
    Raw images on a 7200 RPM HDD
    Yosemite OS
    0
  • SFA
    Hmm.

    I use the same SSD with Windows but not catalogs.

    Performance seems to hold up for my sessions BUT there are potential Write performance issues with SSDs whereby the performance suddenly drops dramatically. I believe there are ways to minimise this effect - you may have addressed this already.

    My guess would be that a large catalog could well be affected by the write performance challenge. Might be worth trying the creation on the HDD if it is internal (or on a fast external connection) to see if anything is different.


    HTH.



    Grant
    0
  • Ario
    [quote="Jimmy D Uptain" wrote:
    Mac Pro 2012
    Six core 3.33
    24Gb Ram
    7950 3GB Mac Edition video card
    Boot/App drive: Samsung 840 Pro SSD 512 in the PCI Express slot.
    Catalog is on the SSD
    Raw images on a 7200 RPM HDD
    Yosemite OS

    If you are on Yosemite most probably TRIM is disabled and this might be the case of the low writing speed on your SSD Samsung PRO. As far as I know with Yosemite TRIM is available only for the SSD supplied by Apple.
    0
  • paintbox
    According to the Black Magic speed tests, my read/writes are perfectly fine at this point.
    Also, my LR works without issue. Same setup.
    Media Pro has no issues.
    0
  • paintbox
    Just wanted to say this started with me having rather poor catalog performance. I would try different ways of building catalogs, hoping to increase my overall catalog performance. Pretty much opening, searching and filtering being the worst
    performing tasks.

    Tried another (rather expensive experiment).
    I bought a 1Tb SSD, copied all my originals to it.

    Now the Catalog and originals are on SSD's. Separate but still, they are SSD's

    Now C1 didn't find the originals right away. Once I told it where they were, it redirected. This only took seven minutes or so. I was impressed.

    Lightroom also had to be told where the originals were. Its redirection was instantaneous. Something to think about.

    These SSD's are both on the same Sonnet SSD Pro PCI card.
    No noticeable catalog performance.

    I then moved the catalog to the same SSD as the originals.
    There was no noticeable improvement in catalog performance.

    So I decided to start from scratch again. I mean deleting the catalog and all files associated with C1.

    Six hours and I'm at 20,000 images imported. So the catalog building performance has improved. I wouldn't say a lot, but there is a difference.


    If the overall catalog experience isn't much better, my next experiment will involve building the catalog 10,000 images at a time. The first 10,000 only takes about an hour, just curious if building it piecemeal will make things smoother.

    Please bear in mind that this isn't a C1 bashing thread. I'm just trying to get to the root cause of my performance issues.
    Maybe someone with similar issues won't have go down the trial and error path that I had to.
    0
  • mli20
    I'd take a long hard look at the database structure, ...

    Information about image file location appear to be stored in a table by the name of zpathlocation, in a field called zrelativepath, while the drive id is in a field zwinroot. Neither of the fields are indexed!!! This fact has a major impact on database performance. Without an index the DBMS must resort to sequential searches, several orders of magnitude slower than a index look-up once you have a sizeable catalog.

    This analysis was done under Windows, so needs to be replicated for OS X, but experimenting with the database indexing is certainly a whole lot cheaper than a SSD and whatnot.

    Cheers,
    Mogens
    0
  • H. Cremers
    [quote="Jimmy D Uptain" wrote:
    Mac Pro 2012
    Six core 3.33
    24Gb Ram
    7950 3GB Mac Edition video card
    Boot/App drive: Samsung 840 Pro SSD 512 in the PCI Express slot.
    Catalog is on the SSD
    Raw images on a 7200 RPM HDD
    Yosemite OS


    I'm on a very similar system as yours. When i read your OP, i tried this for myself, with CO8.1.

    While i wasn't there to watch it, i imported 35K images over night. It may have been finished sooner, but at least when i woke up in the morning.

    Something must not be right.

    Wishing you success in ironing this out.
    0
  • H. Cremers
    [quote="mli20" wrote:
    I'd take a long hard look at the database structure, ...

    Information about image file location appear to be stored in a table by the name of zpathlocation, in a field called zrelativepath, while the drive id is in a field zwinroot. Neither of the fields are indexed!!! This fact has a major impact on database performance. Without an index the DBMS must resort to sequential searches, several orders of magnitude slower than a index look-up once you have a sizeable catalog.

    This analysis was done under Windows, so needs to be replicated for OS X, but experimenting with the database indexing is certainly a whole lot cheaper than a SSD and whatnot.

    Cheers,
    Mogens


    Wow, you guys, analyzing the DB 😄

    I do know that typically with relational databases without index, the scan time grows more or less exponentially. Even if that behaviour didn't show up for me (see the above post).
    0
  • Permanently deleted user
    Mac Pro 2012
    Eight core 2.8GHz
    16Gb Ram
    ATI Radeon HD 2600 256MB video card
    Yosemite OS 10.10.1

    Boot/App drive: Crucial 512GB in the PCI Express slot.
    Secondary Drive: 1TB Western Digital Green
    Catalog is on the secondary drive
    Raw images on USB/Firewire (makes no difference) attached 2TB 7200 RPM HDD drive.

    I have 128000+ images (TIFFs, RAW, JPG)
    I have never ever managed to build a full catalog using CaptureOne Pro (COP). I have tried many times when in COP7 and COP8. The application builds the first 12000-20000 images relatively quickly and then slows down from 40000-46000 before locking or stopping. After two days I then abort the build. Fortunately I have a second slightly slower machine that I can use.

    Instead I use PhaseOne Media Pro v1 which built a full catalog in about 14 hours. This is a much better piece of software and it imports catalogs from iMedia View which is its parent software. The new version of Media Pro is better than iMedia View so updating is advised. (No I dont work for Phase One).
    0
  • paintbox
    [quote="HCS" wrote:
    [quote="Jimmy D Uptain" wrote:
    Mac Pro 2012
    Six core 3.33
    24Gb Ram
    7950 3GB Mac Edition video card
    Boot/App drive: Samsung 840 Pro SSD 512 in the PCI Express slot.
    Catalog is on the SSD
    Raw images on a 7200 RPM HDD
    Yosemite OS


    I'm on a very similar system as yours. When i read your OP, i tried this for myself, with CO8.1.

    While i wasn't there to watch it, i imported 35K images over night. It may have been finished sooner, but at least when i woke up in the morning.

    Something must not be right.

    Wishing you success in ironing this out.


    Well I am not ruling out some sort of software conflict. The fact that your database is built so much quicker makes me think that software could very well be the issue.

    Could you contact me off line? I'd like to have a list of your applications via the activity monitor.

    my email addy is: paintbox@bellsouth.net
    0
  • paintbox
    [quote="jknights" wrote:
    Mac Pro 2012
    Eight core 2.8GHz
    16Gb Ram
    ATI Radeon HD 2600 256MB video card
    Yosemite OS 10.10.1

    Boot/App drive: Crucial 512GB in the PCI Express slot.
    Secondary Drive: 1TB Western Digital Green
    Catalog is on the secondary drive
    Raw images on USB/Firewire (makes no difference) attached 2TB 7200 RPM HDD drive.

    I have 128000+ images (TIFFs, RAW, JPG)
    I have never ever managed to build a full catalog using CaptureOne Pro (COP). I have tried many times when in COP7 and COP8. The application builds the first 12000-20000 images relatively quickly and then slows down from 40000-46000 before locking or stopping. After two days I then abort the build. Fortunately I have a second slightly slower machine that I can use.

    Instead I use PhaseOne Media Pro v1 which built a full catalog in about 14 hours. This is a much better piece of software and it imports catalogs from iMedia View which is its parent software. The new version of Media Pro is better than iMedia View so updating is advised. (No I dont work for Phase One).


    I have Media Pro. Unfortunately there is some weird thing going on where the colors aren't rendered correctly. I believe there is some sort of conflict with my monitor. Anyway, its useless to me.
    I have tried using Media Pro to build a catalog, then importing that catalog into C1, no dice.
    The Media Pro catalog was built in no time at all. C1, well, not so much.
    0
  • paintbox
    Good morning guys.
    Just wanted to let you know of my progress.
    My last few tests involved the new 1TB SSD for the originals.

    After some talking with another member on the forum, I noticed that his organization was quite different than mine.
    In my setup, the originals were all sitting in one folder. His were organized in separated folders of about 300 images (or 4GB) each.

    So I then used Photo Mechanic to separate mine by date, with each date getting its own folder. I then went through each folder and made sure that no single folder contained over three hundred images.

    *(Edit) I just found that atlas one folder was over 5GB There are probably more, so I'm gonna fix that today.*

    I then started the import.
    It was screaming with very little slowdown until it hit about 30,000 images.
    It only took 177 minutes to get the first 25,760 images.
    Overall, the time for catalog creation went from about three days, to 23hrs or so. Still unacceptable but revealing.

    Actual catalog performance was slightly better but again, unacceptable.

    Now here's something interesting:
    I found that by removing the Filter Tool, catalog performance was much better. Searching in the browser was fairly quick.
    Loading of the catalog hasn't really improved in any case, taking about two minutes to open.
    0
  • H. Cremers
    [quote="Jimmy D Uptain" wrote:
    After some talking with another member on the forum


    Jimmy, the address you've mentioned above is bouncing.
    0
  • paintbox
    Yeah, I'm sorry.
    I just tried to check my mail and nothing is coming through.
    I am getting mail on my phone so it may be my internet service.
    Try again
    0

Post is closed for comments.