RFC: IIIF Metrics for efficiency, insight and bot defence

[tomcrane]

What does this change?

This RFC describes changes/enhancements to DLCS to tackle two distinct problems:

• dealing with increasing and increasingly complex automated traffic ("bots")
• ensuring the best image serving performance for human users (even if at the expense of response times for automated callers)

A side-effect is that it gives us a very rich store of reporting data for image traffic, for other analytic purposes.

It is very much a work in progress and is opened as a draft initially.

[donaldgray]

As a whole, I'm not sure I agree with deferred orchestration as the default unless the s3 tile serving is really efficient. Even then it's a pretty fundamental change in Orchestrator - we're no longer just looking at the incoming request to determine how to serve it, it needs to use something else to determine that. That could be the ImageLocation record - it's no longer where it is on S3, it's where it is but also how it should be served (from disk, via s3)? The nas column is potentially that but might need another column too.

'UX' wise it's a change, currently the first request takes a slight hit but then subsequent should be fast, we even have the notion of "Orchestrate after ingest" in the Engine so that it pre-warms the cache but this flips that. With deferred execution every asset starts cold, so every request takes a hit until proven populer. Not necessarily bad but doesn't feel quite right. Would it be a 'mode' we can run in?

Is an option to have normal running (standard request rates, hotdisk has plenty of room) we Orchestrate on first request. Under load (request rates high/suspected bots and/or hotdisk filling) switch to deferred orchestration? Or we can conditionally do this based on incoming request, CF can flag known bots with a specific HTTP header (via CustomRequestHandling) like x-suspected-bot: true or x-suspected-bot: gptbot, or just proxy the user-agent itself and we can have a known list to check for (bots and scripts)? If it's a bot or script, we'll serve you from fast disk if we have it but we won't orchestrate for you.

Orchestration, particularly at Wellcome as the images are small, isn't that expensive - we can replace the entire Lustre volume without there being any discernible drop in average performance. I think the overall issue isn't necessarily the churn of the hotdisk, it's the currently inefficient means of scavenging. The small images are the issue - there's lots of them. Finding the files to evict is relatively expensive without an external metrics tracker.
If we metrics could just say select path_on_disk from metrics order by last_access desc limit 100; we would be able to keep up.

[donaldgray]

Bursts of image requests for deep zoom tiles are normal IIIF traffic, generated by humans zooming into and panning around high resolution tiled images. We mustn't mistake these for automated traffic.

IMO if we get our limits correct and someone is zooming and panning enough to hit them then it's fine to block them. Being a human doesn't allow unrestricted use. It's unlikely someone would hit those limits but I think some accidental blocking is okay. Doesn't have much bearing on this overall.

[donaldgray]

The latter is possible: AWS WAF IP sets are updatable via API, and rate-based rules can be scoped to URI patterns. A DLCS Metrics Monitor (or whatever watches the metrics database) pushes offending IPs/CIDRs into an IP set with a TTL; CloudFront+WAF enforces it at the edge, before traffic ever reaches Orchestrator.

IP alone might be too blunt? Thinking of a dodgy script running at a uni, don't want to punish everyone - might need to use user-agent + IP combo.

This is much better than blocking in Orchestrator itself, because blocked traffic costs us nothing.

Tiny note - it is a small cost in comparison to compute but CF does still incur a charge for every inspected request - blocked or not. It needs to look at it to know whether to let it in or not.

[donaldgray]

It's not explicitly stated here but implied by "join". I don't think this should be in the actual DLCS DB, that should be owned, managed + accessed by the DLCS alone. If we did need to join to Images table we could do that by consuming asset created/updated events and getting the latest values and storing an external record of them, or using replication etc. It could bloat db, add resource overhead and could complicate rollouts where we do hot DB switching.

[donaldgray]

IMO the short answer is no. Did some digging as it's been a while since I looked at it but it still doesn't look like it's worth it.

• Lower latency, likely better perf.
• Needs to be provisioned iops (io1 or io2) EBS, 2.4TB is only marginally cheaper pretty than 2.4TB Lustre.
• Can provision more IOPS (more cost), independant of storage (Lustre can do same, again cost)
• Restricted to single AZ.
• Need to use cluster-aware file system like gfs2, then it looks like you need additional services on every instance that accesses it.

[donaldgray]

The Kinsesis -> Firehose -> S3 stuff looks sensible. One thing to mention is that the AWS docs (https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/real-time-logs.html) say "We recommend that you use the logs to understand the nature of the requests for your content, not as a complete accounting of all requests.". Kinesis logs can be dropped, it's more of an indicative sampling of requests than complete record, which should be fine but if we did need total accounting the S3 logging, albeit delayed, would be most accurate.

The key will be getting enough data that we can make informed decisions. Do we need to make a list of things we want to know, to then know how best to store + partition data?

[donaldgray]

Athena CTAS or a small batch job

Small, looks like Athena CTAS can only read/write to S3 so we'd need something to write that to PG. Maybe CTAS summarises then that's blatted into postgres?