Stored procedure

[chr4]

Use a stored procedure to generate asset_stats on the fly from the effects table. I'm not sure whether this is actually good or bad for performance (less writes, but probably a bit more calculations).
My guess is, that CPU is way less of a bottleneck than IO, therefore I'd go with this version.

This pull-request also adds a change (which we probably should backport in case we decide against this) to not include transfers and account stats from the asset issuer account. I'd argue that the issuer should be ignored when counting payments and involved accounts of customers.

[fabrik42]

Hmmm I don't think this is really beneficial to performance. It basically rebuilds a query with psql, including some N+1 sub-queries.

I read the file name of the migration and my question would be: Why not use a real materialized view for this? Next to a real table, this is probably the most performant solution.

[chr4]

I've considered materialized views initially, but the problem here is that we do not need view, but something that needs to loop over rows.
I'm not sure (and did no benchmark) whether a procedure is faster than a table (but it would at least safe some writes).

[chr4]

I've rebased this on the streamline_api_endpoint branch. It cancels out some of the problems with WHERE clauses, but I'm not sure if there's a way to use a materialized view instead of the procedure because of the LOOP statement.

[fabrik42]

The problem with this solution is that it will always be a sequential scan on the effects table that will also cause 4 additional queries per row in the table. So this is not a scalable/performant solution.

I think the best solution in terms of performance will be to persist every point of the accumulation. And every time you add another asset_stat you can get the last one and add the values of it.

[chr4]

The sub-selects are currently run on every INSERT as well (https://github.com/cndy-store/analytics/blob/master/models/asset_stat/asset_stat.go#L36-L44) - so instead of

INSERT INTO asset_stats(paging_token, asset_code, asset_issuer, asset_type, created_at, issued, transferred, accounts_with_trustline, accounts_with_payments, payments)
		                   VALUES ($1, $2, $3, $4, $5,
		                       (SELECT COALESCE(SUM(amount), 0) FROM effects WHERE type='account_debited' AND account=$6),
		                       (SELECT COALESCE(SUM(amount), 0) FROM effects WHERE type='account_debited' AND account!=$6),
		                       (SELECT COUNT(DISTINCT account) FROM effects WHERE type='trustline_created' AND account!=$6),
		                       (SELECT COUNT(DISTINCT account) FROM effects WHERE type='account_debited' AND account!=$6),
		                       (SELECT COUNT(*) FROM effects WHERE type='account_debited' AND account!=$6)
		                   )

It would be better to pass the new Effect into assetStat.New() and then just take the latest row and add the effect values? This wouldn't solve the DISCTINCT clauses though...