CN Resource Requirements
-------------------------------------------
This document provides an overview of estimates of the resources needed to run a CN node. It starts with an overview of the current resource baseline based on the cn-ucsb-1 processes. We need to add a more realistic estimate under load.
Current minimum baseline
-----------------------------------------
The development CN instances provide an absolute minimum baseline because they show that majority of processes but with minimal data and no load. Here's the baseline:
Storage:
14GB used out of 569G disk available
Memory:
Total: 6GB used out of 31GB total on machine, + 1 GB used for swap
Memory by major processes:
Process VirtualMem (MB) ResidentMem (MB)
Apache 1284 30
Tomcat 10481 5472
MySQL 261 53
Postgres 937 74
Total 12963 5629
Tomcat memory settings:
-XX:MaxPermSize=128m -Xms8192m -Xmx8192m
KNB Resource usage (for comparison)
-----------------------------------------------------------
Storage:
Metacat XML Documents
933MB on filesystem for 41482 XML documents, averaging 23Kb/document
Metacat data storage:
205GB for 33,466 data objects, averaging 6.1MB/object
Postgres DB size
56GB for 41482 documents + 33466 data objects, averaging 783Kb/object
Memory:
Total: 3GB used out of 15GB on machine, + 3GB used for swap
Memory by major processes:
Process VirtualMem (MB) ResidentMem (MB)
Apache 1129 30
Tomcat 21094 2969
MySQL 0 0
Postgres 6859 542
Total 29082 3541
Crude Projection
---------------------------
At these rates, if we project storage linearly out to 100K, 500K, and 1000K documents, and we project memory out as a function ofour needs will be:
41K 100K 500K 1000K
Storage (GB) 262 632 3158 6316
Memory (GB) 28 68 342 685
Why is Tomcat taking 6GB of memory on the development server? Does this have anything to do with the differences in software stacks being run? 3GB of memory difference seems huge.
Yes, probably -- in addition to Metacat, the CN runs mercury under tomcat, as well as the CN servlets. More servlets = more memory.
Also, even though the Postgres database is much larger in the case of KNB, the difference in the active Postgres process related memory (ResidentMem (MB)) between the two is only 400mb which is small in comparison with the memory usage by Tomcat.
Yep, and I suspect the postgres memory usage will not scale with db size, except maybe for indices. Mainly it will scale with the # of pg processes and size of the global area.
One more point, these numbers seem to be calculated using only VirtualMem (that's how I got the same numbers in the projection for Memory). If this is so, we won't have a good idea of the actual memory footprint of the service as a whole as this is swap and not actually in the RAM.
In the process table, VIRT = RES + SWAP. I reported VIRT and RES. Resident memory usage (RES) is important because it shows what is in memory now. Virtual memory usage is important because it represents the size of the whole image, and so may be in physical RAM at some points in time when swapped in. I omitted SWAP because it can be calculated and doesn't tell us any more.
Postgres difference: 542-74 = 468 MB
Tomcat difference: 2969-5472 = -2503 MB
This seems to suggest that the difference in deployed tomcat services between DataONE and KNB will have more of an effect on RAM footprint than the first 200K documents.
Yeah. I think a lot of the difference for Tomcat is that the CN also runs mercury in tomcat, which has a lot of memory for indices and SOLR, etc.
(Postgres difference)/41K files * 200K files 468/41*200=2282 MB
I don't expect PG memory to scale linearly with database size (see above).
Of course this does have quite an effect on the swap space used on the disk...
Postgres difference: 6859-937 = 5922 MB
Tomcat difference: 21094-10481 = 10613 MB
For the same 200K documents, you might expect to use
(Postgres difference)/41K files * 200K files 5922/41*200=29 GB for Postgres
That all assumes linear memory scaling, which I don't think will happen.
I can't think of a reason the Tomcat service's needs would grow with the # of documents, but if it did... scratch that, I think I can as more and more of the database was being accessed by users...
(Tomcat difference)/41K files * 200K files 10613/41*200= 52 GB for Tomcat on disk.
And as we approach 1e6 documents....
Postgres mem: 468/41*1000 = 11.4 GB
Postgres swap: 5922/41*1000 = 144 GB
can't estimate Tomcat's memory increase due to the difference between KNB and D1CN,
Tomcat swap: 10613/41*1000 = 258 GB