what's the recommended tps for SBOM upload

[andreeaButerchi]

Hello,

We've trying to use DT in a batch mode: meaning we upload a lot of SBOM files and then we wait for the processing to be done... but I think we're causing a bottleneck :(
I did not manage to find any recommendation regarding the SBOM ingestion ratio...
are there any recommendations/thresholds/benchmarks? Or the container resources is the limit?
knowing that an average SBOM is 6-700Kb and we're at a 5 TPS (sbom uploads)... but we can see the processing getting longer & longer & longer :(
we're considering uploading something like 20 and send another 20 only when the 1st ones are done... Knowing that 20 is just an example... we'll try to figure out the magic number later. Just wanted to know if it might be a good idea...
or maybe we're doing something wrong

Thank you very much for your help!

[nscuro]

It's very hard if not impossible to provide one-size-fits-all recommendations.

The symptom you're experiencing is one of the reasons we started working on Hyades, which eventually will become Dependency-Track v5. We wrote about this particular issue here: https://github.com/DependencyTrack/hyades/blob/main/WTF.md#why

Still, you can improve the current situation, but you will need to start monitoring application metrics, otherwise you'll be blind to where the actual limitation lies. We support exposition of Prometheus metrics, and provide an example Grafana dashboard that you can use: https://docs.dependencytrack.org/getting-started/monitoring/#grafana-dashboard

For a start, have a look at the eventing metrics:

You'll likely see high numbers of Events Queued and a constantly saturated Active Worker Threads. Depending on how large you worker pool is at the moment, you can try increasing it and see if it helps:

dependency-track/src/main/resources/application.properties

Lines 3 to 19 in b3f96ea

	# Required
	# Defines the number of worker threads that the event subsystem will consume.
	# Events occur asynchronously and are processed by the Event subsystem. This
	# value should be large enough to handle most production situations without
	# introducing much delay, yet small enough not to pose additional load on an
	# already resource-constrained server.
	# A value of 0 will instruct Alpine to allocate 1 thread per CPU core. This
	# can further be tweaked using the alpine.worker.thread.multiplier property.
	# Default value is 0.
	alpine.worker.threads=0
	# Required
	# Defines a multiplier that is used to calculate the number of threads used
	# by the event subsystem. This property is only used when alpine.worker.threads
	# is set to 0. A machine with 4 cores and a multiplier of 4, will use (at most)
	# 16 worker threads. Default value is 4.
	alpine.worker.thread.multiplier=4

For example, you can pass the following environment variable to increase the worker threads to 32:

ALPINE_WORKER_THREADS=32

You should be catious with this though, as more concurrency can cause more load / contention on the database as well. Have an eye on the CPU / memory and database metrics on the dashboard as well.

Separately, in DT v4.11 we introduced a new way of processing uploaded BOMs, which is significantly faster / more efficient. You can enable it under Configuration -> Experimental -> BOM Processing V2 in the settings. It will become the default in DT v4.12.

[nekhtan]

You're amazingly fast to answer, that's awesome !!

What did you configure for requests.memory and limits.memory?

On our last run, request and limit are the same, but they weren't previously.

try overriding the default MAXRamPercentage with a lower value

We'll do that and keep you posted on how it behaves ;)

The event queue is entirely in-memory and thus will lose its state upon reboot or crash. It's one of the reasons we started working on Hyades

That's what I thought, and yes we did look at Hyades (which looks great btw), but we think we're in too deep with v4 and business is expecting us to deliver soon... But rest assured that the plan is to migrate as soon as possible.

It doesn't look like it supports a max cache size

That would be datanucleus.cache.level2.maxSize (cache properties), or did I miss something ? As I understand it, its default value is -1 which means "no limit".

[nscuro]

That would be datanucleus.cache.level2.maxSize (cache properties), or did I miss something ?

Ha, you looked deeper into the docs than me. TIL that property exists! :D

If it turns out to work for you, perhaps we should add this to the docs as another knob to tweak when memory usage gets too high...

[nscuro]

If you're feeling adventurous:

In Hyades we have switched to the G1 garbage collector and enabled String deduplication. We found that G1 per default does a better job of keeping the heap manageable, while also having shorter pause times:

https://github.com/DependencyTrack/hyades-apiserver/blob/40dfe3b1da0aadc52aba508c4effc057171a618d/src/main/docker/Dockerfile#L35C19-L35C84

Saying this because I just now saw the average 8s GC pause times you're experiencing under load. That's a lot.

[nekhtan]

Hello @nscuro

I didn't try yet the G1GC but it's definitely on my todo list.

Since last time I tried to configure the datanucleus' maxSize. It works well on startup but not so much under high pressure... There's a TODO in the putAll method so depending on how it's being used it might have no effect to configure it. I'm not saying it's the case here, but right now I can't explain why setting a limit to 2M is ignored while setting a limit to 2k is taken into account.

I also tried to completely disable the cache (alpine.datanucleus.cache.level2.type=none) it looked ok overall (even though it was queuing more) but it was on the end of a run, so I might try that again later on a full run. That being said I'd really prefer not disable it entirely.

That's why I eventually tried with the weak references cache (alpine.datanucleus.cache.level2.type=weak) and it looks promising, even though you're advising against changing it:

The database operations / queries have slightly increased compared to a run with the soft cache while the cache is MUCH smaller since its values can be GCed. So it makes me wonder, are the cached entries really that reusable or do you expect it to be just a small subset of data that are actually queried more often than the vast majority?

[nscuro]

are the cached entries really that reusable or do you expect it to be just a small subset of data that are actually queried more often than the vast majority?

That pretty much hits the nail on its head. That is exactly what we have observed. The problem is that L2 caching is enabled globally per default, so everything ends up being put into the cache as soon as it's "touched" by the code.

But in order to get a cache hit, objects need to be queried by their ID (i.e. the ID database column). But DT mostly uses UUIDs (which is separate from the primary key) in its API. Generally most queries are not being performed based on ID, but some other fields instead. So the L2 cache ends up being useless for a lot of objects.

Usually you'd enable L2 caching for the areas where you know you need it, making it easier to control cache sizes. We did end up in the reverse situation, which makes it a bit harder to resolve.

I have been working on this for quite a while now, in the sense that for all new features, and code areas I am refactoring, I intentionally disable L2 caching. So one step after another, I'm positive we can drastically decrease the negative impact it has.

Sometime ago I did experiment with changing the L2 cache type to weak, but as you already figured out, it might be GCing entries too soon, to a point where you don't get much benefit out of the caching to begin with.

In our experience, G1 is a bit more aggressive with cleaning soft references, whereas the default ParallelGC tends to keep them around for longer.