Looking for fastest CO9 system
First please let me explain why I need a very fast system:
I need to develop up to 1.500 Raw photos coming from a Canon 1Dx in about 5 hours (yes: hours, not days). So we're not talking about sophisticated image processing. Basically white balance, colour, sharpen. All done with shortcuts. The main bottleneck so far is the creation of the cache (preview, thumbnails).
In another thread I've already been told that CO9 will use all cores available. So CO9 will benefit from a full blown dual Xeon setup.
And what about the graphics card? Does CO9 use Cuda on Nvidia cards? Or is it just about OpenGL performance? Would CO9 benefit from two or more cards?
How many SSD should be used? One for Raw? One for Cache? One for output?
Please feel free to recommend everything which will speed up everything.
I need to develop up to 1.500 Raw photos coming from a Canon 1Dx in about 5 hours (yes: hours, not days). So we're not talking about sophisticated image processing. Basically white balance, colour, sharpen. All done with shortcuts. The main bottleneck so far is the creation of the cache (preview, thumbnails).
In another thread I've already been told that CO9 will use all cores available. So CO9 will benefit from a full blown dual Xeon setup.
And what about the graphics card? Does CO9 use Cuda on Nvidia cards? Or is it just about OpenGL performance? Would CO9 benefit from two or more cards?
How many SSD should be used? One for Raw? One for Cache? One for output?
Please feel free to recommend everything which will speed up everything.
0
-
A decent graphics card (or two) will outperform any CPU(s).
However should there be issues/glitches with the image output, which can occur with some graphics cards/drivers/otherwise unclear occasions, then sometimes the only way is to turn the graphics card (OPENCL, hardware accerelation) off, which means the burden is on the CPUs (which are more reliable in this respect but slower)
See also here
For Canons files, the graphics card will be used (unlike with Fuji Xtrans), and also all CPU cores will be used for the raw conversion step in the process pipeline (unlike with Sony files)
Have enough RAM (16 GB or more)
I have multiple SSDs as you described but noticed only minor differences, if at all.0 -
As for the graphics card(s), Xeon workstations mostly come with NVidia Quadro cards which are quite expensive but which doesn't mean they are fast (for the dollar spent), Geforce or AMD gamer cards are faster. It is the number of Cuda cores (or stream processors) which matter for speed. 0 -
Is this the fastest? 😂
Intel Xeon Processor E5-2699 v3 - 18 cores0 -
[quote="BeO" wrote:
As for the graphics card(s), Xeon workstations mostly come with NVidia Quadro cards which are quite expensive but which doesn't mean they are fast (for the dollar spent), Geforce or AMD gamer cards are faster. It is the number of Cuda cores (or stream processors) which matter for speed.
Aren't workstation GPUs used more for 3D modeling & rendering work?
I suppose they're more of a pickup truck while gamer cards are sports cars.0 -
Thank you for your replies so far. I'm still hoping that someone from Phase One will look at the thread.
Currently I'm using 2 x Intel Xeon E5-2695 v3 (14 cores) and looking at 2 x Intel Xeon E5-2687W v4 (12 cores). The workstations are custom built. Using one GTX980 Ti right now. 64GB Ram. SSD only (no Raid).
So how to speed up creating of cache (preview, thumbnails) files significantly?
And which combination of CPU and GPU to choose for fastest processing?0 -
[quote="DV_Chris" wrote:
Thank you for your replies so far. I'm still hoping that someone from Phase One will look at the thread.
Currently I'm using 2 x Intel Xeon E5-2695 v3 (14 cores) and looking at 2 x Intel Xeon E5-2687W v4 (12 cores). The workstations are custom built. Using one GTX980 Ti right now. 64GB Ram. SSD only (no Raid).
So how to speed up creating of cache (preview, thumbnails) files significantly?
And which combination of CPU and GPU to choose for fastest processing?
You might be better to create a Support Case with Phase if you want to ensure a response - this is a User to User forum facility kindly provided by Phase One.
That said I don't think Phase are really in a position to offer specific help in terms of naming processors and cards.
When you say you need to process 1500 shots in less than 5 hours but mainly use presets does that mean you can apply presets in bulk to a mass import or are you shooting tethered and processing as you go?
Your numbers suggest you have about 12 seconds per image to work with. If loading in bulk and editing in bulk and processing to output in bulk that is likely feasible. For tethered work some of the efficiencies of bulk loading will be lost so other parts of the workflow will need to make up the time.
I suspect that components other than the obvious processor/GPU/Memory/SSD capacity may be important. The boards and buses and all the peripheral's in the machine are likely to have an influence.
If you look at information provided by specialist systems suppliers who target niche markets you will see that they often provide specifications for those working in specific industries or hobbies areas (for gaming for example). Components can be selected and combined based on anticipated needs.
In your situation I would be tempted to talk to that sort of supplier since they will likely have wider experience and feedback for users to work with when assessing your needs. They will also be more up to speed with the latest hardware developments and their strengths and weaknesses.
If you are already doing that (and from the description of your specs it seems like you might already be there) then you are probably ahead of the game as far as the experience of most people on the forum are concerned.
Moreover I would suspect that possible time savings form the system would be small compared to any time savings you can develop during the "human interaction" steps in the workflow! 😉
HTH.
Grant0 -
[quote="DV_Chris" wrote:
Thank you for your replies so far. I'm still hoping that someone from Phase One will look at the thread.
Currently I'm using 2 x Intel Xeon E5-2695 v3 (14 cores) and looking at 2 x Intel Xeon E5-2687W v4 (12 cores). The workstations are custom built. Using one GTX980 Ti right now. 64GB Ram. SSD only (no Raid).
So how to speed up creating of cache (preview, thumbnails) files significantly?
And which combination of CPU and GPU to choose for fastest processing?
Basic advise: Get the fastest you can buy.
Little more in-depth:
- Preview: CPU only, as many cores as possible, as fast individual cores as possible, fast disk.
- Processing: Fastest GPU (number of cuda cores (nvidia) or stream processors (ati/amd) and CPU possible (many cores), fast disk.
For what specifically to chose, please look at openCL benchmark pages, and CPU benchmark pages. I can recommend this one for OpenCL: https://compubench.com/result.jsp?benchmark=compu15d0 -
Now we're talking 😉
(Some background info: since 1999 I'm a VAR for video editing systems. We build the workstations in house. Video wise I might know everything but when it comes to Raw processing with CO9 I need some advice).
@Christian Gruner:
Would it make sense to use more than one graphics card? If so how will CO9 address them? Do I need SLI?
And MOST important since I do not see any increase of speed right know, my No.1 bottleneck:
How to speed up creating of cache (preview, thumbnails) files significantly?0 -
[quote="DV_Chris" wrote:
Now we're talking 😉
(Some background info: since 1999 I'm a VAR for video editing systems. We build the workstations in house. Video wise I might know everything but when it comes to Raw processing with CO9 I need some advice).
@Christian Gruner:
Would it make sense to use more than one graphics card? If so how will CO9 address them? Do I need SLI?
And MOST important since I do not see any increase of speed right know, my No.1 bottleneck:
How to speed up creating of cache (preview, thumbnails) files significantly?
I had a feeling you might know something about video .....
The question about tethered operation or bulk import still has relevance I think.
Workflow management MIGHT be more significant for time optimisation than hardware.
For an extremely critical requirement (provided there is enough budget to justify the solution) one might consider splitting the operation across 2 or more systems - but perhaps tricky if tethered. Saving in the system spec. might offset the cost of having to dual up.
Grant0 -
Fastest cache speed? I'm going to assume dual 18-core Xeons & RAID 0 SSD (2-4 drives).
Finish it off with quad 980 TI SLI setup 😲
I know CO9 uses the dual GPUs in the current Mac Pro so don't see why it wouldn't do SLI on Windows.0 -
For clarification:
Even on my Xeon setup CO9 is slow when creating the previews and thumbnails. There's no difference compared to my old i7-3930K setup.
Furthermore it takes 2-3 seconds before I actually see the various changes applied.
So which hardware is needed to minimize any time lag?0 -
I only see going dual CPU & SSD as the main options to speed cache up.
Perhaps CO9 itself is the bottleneck (optimization?)0 -
From what camera are the raw-files? 0 -
@Christian: the Raws are coming from a Canon 1Dx, size is about 25-30MB each. 0 -
I've processed (WB, Crop, Straighten and output JPG) over 3K of CR2 files (32MB files) in just under 7 hours with my system
i5 2500K OC'd to 4.3 and a single XFX R9 390 8GB GPU running win 7 Pro with 16GB at the time. RAW files are on a Corsair 240GB SSD on SATA20 -
@Bob:
How long did it take to create the preview files?
Using an extra SSD for the cahce files?
How long does it take to actually see any change eg after applying white balance?0 -
[quote="DV_Chris" wrote:
@Christian: the Raws are coming from a Canon 1Dx, size is about 25-30MB each.
In that case, with your R9, it should be pretty much fluent, unless you are on a 4k-5k system.
With regards to preview generation, then what really matters here, are numbers of CPU cores and their individual speed, and the disk speed (in the mentioned order). The GPU is not used, as it not really advantageous in this operation.0 -
[quote="Christian Gruner" wrote:
[quote="DV_Chris" wrote:
@Christian: the Raws are coming from a Canon 1Dx, size is about 25-30MB each.
In that case, with your R9, it should be pretty much fluent, unless you are on a 4k-5k system.
With regards to preview generation, then what really matters here, are numbers of CPU cores and their individual speed, and the disk speed (in the mentioned order). The GPU is not used, as it not really advantageous in this operation.
Dear Christian, thank you for reply.
So the preview relies on the CPU cores and speed.
Which steps are GPU intensive? Would a system benefit from two graphics cards?
And how would you store the files (Raw, preview and Jpeg output)?0 -
[quote="DV_Chris" wrote:
@Bob:
How long did it take to create the preview files?
Using an extra SSD for the cahce files?
How long does it take to actually see any change eg after applying white balance?
Cache files are in the same folder as the RAW files takes approx 15 minutes for all the previews to render. but I pretty much start working on the files as soon as they start rendering in (I also have custom pre-set for my camera profile - so all those are applied too)
How long does it take to change the white balance? it's instant.. everything i do is pretty much live view, if i shift a slider, i see the change as it moves..
I've made a vid so you can see
https://youtu.be/L6X8JP9X33k0 -
Bob, thank you so much!
Is there any setting to optimize the creation of the cache, preview files?0 -
So here's another visible bottleneck:
Import of 100 Raw photos from a Canon 1Dx works ok. No I define a shortcut for white balance (up and down). Now I start to use the shortcut on all Raws. I see the changes on the first few immediately but after a while it takes up to three seconds that I actually see the changes. So what causes the delay?0 -
One further observation: up to CO6 there was no delay. 0 -
[quote="DV_Chris" wrote:
I see the changes on the first few immediately but after a while it takes up to three seconds that I actually see the changes. So what causes the delay?
FWIW sometimes i do multiple corrections on all the files too and the thumbnails don't change until i've made a change on that particular file.. all changes have been made, just the thumbnails are left untouched/updated.
Just ran a test on 1923 RAW files (59GB), C1 V9.1.1 took 85 seconds to render all thumbs. generating all the previews however is taking a little longer..0 -
I moved the directory over to my RAID0 Samsung EVO840 array on my SATA3 and got the following.
thumbs took just under 50 seconds and the previews started appearing about 30 seconds into the thumbnail creation..
working on the first 20 images, changes on each took roughly .25-.5 second to complete..
in all using a couple of SSD on SATA3 in RAID0 seems to speed up the creating process greatly, I'd say over double the speed of 1 SSD..0 -
I have experienced poor preview file creation performance as well. I'll just share.
Using Win10, i7 @3,5GHz (4 cores - 8 threads) 32 GB RAM, 2xSSD, C1 9,1,1.
Importing 3.500 old jpegs. (15 GB)
Those are placed on a SATA 3.0 SSD (Sandisk Extreme Pro 960)
This SSD should be one of the fastest SATA SSD disks, with good I/O as well.
I did some tests to make sure my system is ok. My other SSD is a little older SSD (OCZ Vertex3) - which is then the bottleneck here.
I can still copy almost 300 MByte/sec. (one big file) The monitor shows that this is real disk I/O, and not caching in memory.
If I copy the jpegs (like 1.000 files, 20 GB), I'll get about 270 MByte/sec.
This is fine, and this is matching what I calculated for my most "slow" SSD.
When I tested image import, by importing the 3.500 jpegs, I had the images on the new disk. So basically the system did both read and write the files. In the task manager I could see, that the speed was more then 100 MBytes/sec both read and write at the same time. The import is done very fast. C1 task CPU using almost 100%
BUT...
Then there are preview file generating. This is just slow. Very slow.
But at the same time, when C1 is generating previews, I can see that my disk is doing almost nothing. Max 10-12 MByte/sec, and my CPU is doing very little as well. The C1 task is utilizing about 13% only, and there are plenty of free power to use on all cores, so the C1 CPU usage are not on only one core or thread. My system also has plenty of RAM left.
Unfortunately I cannot figure out what is going on here. I have no reason to believe that my system is slow, and from what I understand in this thread, previews are generated by CPU only (no GPU).
Have any of you with preview file generation speed problems, had a look at the OS performance monitors while the system tried to generate previews?
Kind regards
BCH0 -
I'll do a test and see if any GPU is used when creating previews. 0 -
No GPU is used when making the preview files. 0 -
[quote="Bobtographer" wrote:
No GPU is used when making the preview files.
Nop, also stated earlier in this thread: viewtopic.php?f=62&t=22575&start=15#p1086630 -
[quote="BCH" wrote:
Then there are preview file generating. This is just slow. Very slow.
Generating jpeg previews are a single-core operations currently. If generating for say D810, IQ3 100mp or 5d m3, then it is multi-core (speed improvement in CO8)0 -
sounds more like a software driver problem.
Your System should run circles around my system (i5 3450 4c at 3,7 ghz, 16gb ram, ssds and gtx970).
yesterday I imported around 1300 d800 raw files, applied wb, custom colors, curves, hdr, and other stuff and exported jpgs this was done in under 1 hour.
the import was done ca 3 minutes, previews in ca 6 min and exporting took between 0,8 and 1,5 seconds per image.
this is the reason I ditched lightroom. with lightroom this task will take 3 hours or more on the same system0
Post is closed for comments.
Comments
31 comments