Import Catalog from Lightroom catalog unbearably slow
I'm exploring your CaptureOne as a potential replacement for Lightroom. Over the years I accumulated almost 200k pictures in my Lightroom catalog, organized by years, albums, tags etc.
I'm trying to import the Lightroom catalog in CaptureOne and it looks like there's some exponential algorithm going on. The first few tens of thousands of photos imported took 24 hours to import, then it slowly decayed. The import process has been going on in the background on my MacPro trashcan for more than 7 days now.
The photos are on a RAID0 SSD drive accessible over Thunderbolt 2, which is plenty fast. The machine has a Xeon 3GHz processor, with 64GB RAM, with two AMD FirePro D700 GPUs.
What's going on? Using sqlite3 in the command line is lighting fast, I see no reason why reading the actual database could take so long. The only thing I can think of is generating thumbnails. Could that process be postponed to after the catalog has been imported, and be done on-demand, as pictures are requested?
-
I’ve had the same exact problem. I reached out to support but was sent chasing ghosts. Frustratingly, the same versions in Windows works very, very, very quickly.
To solve the problem you’re facing, I had to revert to version 13.0. But to caution you: I never was successful in migrating away from Lightroom. The performance with 200k+ photos was absolutely terrible on my iMac Pro.
The only way I was able to move forward (and this is not the long term solution) was to start the capture one library as a fresh new library, and keep the old photos in Lightroom.
This is far from ideal, but I never found a solution to make the large Lightroom catalog migration a success.
I suspect this program may have not been built for this use case (large import of 100k+ photos from Lightroom). It’s just so damn slow.
0 -
Matthew, thanks for your observation. The fact that the same version of the app works very quickly indicates there is no inherent problem importing that many pictures from a catalog.
12 hours later, it imported only 7000 files, and shows 10 hours left.
0 -
Bear in mind there will be a lot of work to do discovering the LR edit values for which interpretation to C1 values might be possible and then apply the adjustments and building previews and thumbnails.
If the LR database has not been pre-prepared and "cleaned" there could be all sorts of tricky assessments to be discovered and dealt with.
It might not be unrealistic to think in terms of 3 to 4 seconds per image as a reasonable process speed. So 100k images would be 300k seconds - although typically such volumes will tend to run in memory management challenges.
So that would be 5,000 minutes to run the conversion.
If you were to manage 1 image per second the conversion would still take 1666 minutes or almost 28hours.
0 -
One observation here. When the import starts, it looks like the app can easily process 5 images/second. As the number of imported files grows, this process slows dramatically, to 0.15 images/second after 8 days of running. Could this be an indication of a linear search somewhere in CaptureOne's algorithms or in the structure of their database?
On a different note, CaptureOne should probably not try to eagerly apply all the edits to an imported picture. Instead it should mark them in its internal SQLite database as unapplied edits, and only apply them when the user wants to view the picture. This way you delay the initial computation until it's actually needed, and the initial import becomes much faster.
My older MacPro machine has 8 cores, yet CaptureOne only uses some 220% CPU. It looks like the import process is not fully multithreaded, so it cannot take advantage of the increased computing power of modern computers.
I work in software too, and if my algorithms would take that long to process some 200k items, my users would be very upset.
0 -
I'm tracking similar behavior: iMac Pro, catalogs on internal SSD, referenced images on external Samsung SSD X5. Upgraded from 11 to 20 yesterday. Decided to merge my catalogs into a single. Performance seemed fine (honestly didn't notice the rate) until importing a 15,000-file catalog into a 30,000-file catalog. Now it seems really slow (1 image per second).
I was thinking Spotlight indexing might be the hidden factor, but adding my catalog containing folder to the Privacy exclude list didn't seem to make a significant difference. I've got another couple hours, then I'll try loading that 15,000-file catalog into an empty one without making any other changes, and maybe some other benchmarks to try.
But just looking at stats, it doesn't look i/o bound on either the catalog or the referenced images.
0 -
I stopped the import process after 8 days. After restarting importing the same Lightroom catalog in the same CaptureOne catalog, it goes even slower. It appears the algorithm uses some linear scan to determine whether a picture is in the catalog or not.
Sorry, this is unacceptable and totally unusable.
0 -
I'm starting to see evidence that it's a busted OpenCL pipeline.
https://support.captureone.com/hc/en-us/articles/360002404937
I have a monitoring program that shows how much processor my Radeon Vega is using: 0%. But following the tests above I confirm that C1 is running in OpenCL enabled mode. And UI has horrible response, too.
I'm thinking that something in the OpenCL pipeline "crashed/corrupted/etc" for me earlier today, and now it's fallen back on software emulation inside the pipeline.
I wonder if you disable hardware acceleration and start again if there is a difference. Obviously you want OpenCL to work, but that might be a reboot and a clear of the ImageCore (see above).
0 -
But the linear scan theory is also good, and fits better with gradual decline over time.
0 -
Following the instructions above to reset OpenCL (delete directory and reboot) got my UI responsive again, and I can see load on the GPU again. It did not seem to impact import speed. I now have a 45,000 image catalog. I re-imported a 100 image catalog and was getting about 5 images per second. I started re-importing a 15,000 image catalog and was getting about 1 image per second. This seems to suggest the slow import is impacted by SOURCE catalog size.
Starting with a new catalog, I imported the 100 image catalog and it must have been 30 images per second. I re-imported the 100 image catalog and it honestly was instant... 100 images per second.
Importing the 15,000 image catalog into the 100 image catalog is giving me 15 images per second. So clearly DESTINATION catalog size matters as well.
These are all C1 20 catalogs.
The import speed is clearly based on both the size of the source catalog and the size of the destination catalog.
0 -
I've spent a week trying to import a 70k image LR catalog; I came to the same conclusions as you did about the import speed. C1 crashed midway through the import process multiple times as well.
0 -
I have switched to Mylio and have been impressed by their handling of large image catalogs without skipping a beat. So far I've only cataloged my iOS images/movies, not my DSLR images, but that was doubly-lacking by C1, since C1 still hasn't jumped on the HEIC support. Mylio doesn't have nearly the same level of image manipulation that C1 does - and I'll miss that. But for my iOS images - Mylio sync and organize is a game-changer. I still can't believe how long it has been out there and I'm just now finding it. My advice: use C1 only with small catalogs (think by event or by topic) where you plan on taking lots of time with fine adjustments to the image quality (DSLR). But for your phone image catalog that spans years and has everything in it - which is an organization problem: use Mylio. No reason to be upset with two solutions once you realize they are two use cases. Ideal: no. But I regret merging my C1 images into a single catalog. I'm going to have to split them up again. One-big-catalog in C1 just doesn't work; probably never will.
0 -
John, I wish this was the case with Mylio. However the import of a smaller directory with just 20,000 iPhone-only images takes forever! And with a price tag of $100/year for the unlimited app, I might as well pay for Lightroom and avoid the hassle of migrating in the first place.
0
Post ist für Kommentare geschlossen.
Kommentare
12 Kommentare