Add FilterLimit, FilterBatch, FilterPresent, and per-fetch intervals

README.md

@@ -115,8 +115,8 @@ a plugin set every part of which still has somewhere to talk to. See
 
 - **Recipes in YAML.** A job is a file, not a program. No Ruby is written to
   wire a pipeline together.
-- **41 plugins** across seven categories: subscribe, custom feed, filter,
-  store, provide, notify, publish — and every one of them has a current use.
+- **Plugins across seven categories.** Subscribe, custom feed, filter, store,
+  provide, notify and publish plugins compose through the same pipeline contract.
 - **Markdown out of the box.** `PublishMarkdown` writes the result as a plain
   Markdown document, to a file or to standard output, with no service and no
   credential behind it. It is the natural end of a new Recipe.
@@ -392,24 +392,25 @@ like a shipped plugin replaces it.
 
 ### Which plugins still work
 
-41 plugins ship with the gem. Every one is classified in
+Every shipped plugin is classified in
 [`doc/PLUGINS.md`](doc/PLUGINS.md) section 6, with its settings and the reason
 for its status:
 
-| Status | Count | Meaning |
-| --- | --- | --- |
-| **Supported** | 26 | Works on the supported Rubies with current dependencies |
-| **Supported (external)** | 14 | Works, but needs something you provide: a service, a command, a credential, a data file |
-| **Needs rework** | 1 | The service exists; this plugin speaks a replaced interface |
+| Status | Meaning |
+| --- | --- |
+| **Supported** | Works on the supported Rubies with current dependencies |
+| **Supported (external)** | Works, but needs something you provide: a service, a command, a credential, a data file |
+| **Needs rework** | The service exists; this plugin speaks a replaced interface |
 
 Eleven plugins were removed in this release rather than kept as history: each
 talked to a service that has shut down, or through an API that has been
 withdrawn with no replacement. They are listed with their reasons in
 [`doc/PLUGINS.md`](doc/PLUGINS.md) section 8, and Git history holds the code.
 A Recipe naming one of them now fails at load, before anything runs.
 
-Restoring the one in **Needs rework** — `PublishHatenaBookmark` — is
-self-contained work and a good first contribution.
+`PublishHatenaBookmark` is currently classified as **Needs rework**; restoring
+it to the service's current interface is self-contained work and a good first
+contribution.
 
 No plugin here is stubbed, mocked or simulated to make a test pass. Where a
 plugin's gem is not installed its spec is skipped and says which gem is

doc/PLUGINS.md

@@ -866,6 +866,34 @@ match. Same three keys, same substring rule. An item whose field is missing is
 not matched, and says so; it used to end the run with a `NoMethodError`, which
 is not what its complement does with the same item.
 
+#### FilterPresent — **Supported**
+
+`filter/present.rb`. Keeps only items for which every configured field is
+present. It is an AND filter over field presence, not a keyword matcher.
+
+Recipe:
+
+```yaml
+  - module: FilterPresent
+    config:
+      fields:
+        - title
+        - description
+```
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `fields` | sequence | Fields that must all be present. Required and non-empty. |
+
+The fields that may be checked are `title`, `link`, `description`, `author`,
+`comments`, `source` and `content_encoded`. A field the item has no accessor
+for, a `nil` value, an empty string and a whitespace-only string are all
+absent; an RSS field that responds to `#content` — a parsed source, for
+instance — is judged on that content rather than on the element itself. An
+item is kept only when every configured field is present. An unknown field or
+a field named twice is a settings error, raised when the plugin is
+constructed. No network access, and no dependency on another plugin.
+
 #### FilterSort — **Supported**
 
 `filter/sort.rb`. Sorts each feed's items by date.
@@ -887,6 +915,31 @@ setting works; see section 2.6.1.
 | --- | --- | --- |
 | `pick` | string | `last` takes the last item. Anything else, including absent, takes the first. |
 
+#### FilterLimit — **Supported**
+
+`filter/limit.rb`. Limits how many items the whole pipeline passes downstream.
+The limit is shared across feeds rather than applied once per feed.
+
+Recipe:
+
+```yaml
+  - module: FilterLimit
+    config:
+      max_items: 20
+```
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `max_items` | integer | Maximum items passed by the whole pipeline. Required; must be greater than zero. |
+
+Items are selected in feed order, then item order, up to `max_items`; the
+grouping of the input feeds is kept in the output, and a feed that contributed
+no item under the limit is left out of it. Once the limit is reached, later
+feeds are not walked at all. Anything other than a positive integer —
+including zero, a negative number, and a non-numeric or fractional string — is
+a settings error, raised when the plugin is constructed; a numeric string such
+as `"20"` is accepted. No network access, and no dependency on another plugin.
+
 #### FilterRand — **Supported**
 
 `filter/rand.rb`. Shuffles each feed's items. Combined with `FilterOne`, picks
@@ -914,7 +967,16 @@ this filter will therefore keep links it used to blank.
 
 `filter/image_source.rb`. Replaces each item with one item per image found: the
 images in the description, or, if there are none, the images on the page the
-link points at. Fetching pages means network access. No settings.
+link points at. Fetching pages means network access.
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `interval` | integer | Seconds to wait after each page fetch attempt. Default `0`. |
+
+An item whose images come from the description is never fetched, and
+`interval` does not apply to it. Where a page is fetched, the wait follows the
+actual fetch attempt whether it succeeded or failed. A non-positive or
+non-numeric `interval` — including it being absent — means no wait at all.
 
 Needs `nokogiri`: `gem install nokogiri`, or the `html` group in a checkout.
 
@@ -974,9 +1036,14 @@ the body.
 | --- | --- | --- |
 | `clear_description` | `1` | Empty the description afterwards. Any other value leaves it. |
 | `get_title` | `1` | Fetch the new link and use its `<title>`. Any other value skips it. |
+| `interval` | integer | Seconds to wait after each title-page fetch attempt when `get_title` is `1`. Default `0`. |
 
 `get_title` makes one request per item; use `FilterOne` or a store plugin before
-it on a large feed.
+it on a large feed. When `get_title` is not `1`, no title page is fetched and
+`interval` does not apply. A URL that is not fetchable is not read either, and
+does not wait. Where a title page is read, the wait follows the actual fetch
+attempt whether it succeeded or failed. A non-positive or non-numeric
+`interval` — including it being absent — means no wait at all.
 
 **Both settings were being ignored in every real run.** The test that guarded
 them asked whether the settings mapping was a `Hash`, and the framework hands a
@@ -999,6 +1066,12 @@ page.
 | Key | Type | Meaning |
 | --- | --- | --- |
 | `siteinfo` | string | File name under the assets directory. Required. |
+| `interval` | integer | Seconds to wait after each article-page fetch attempt. Default `0`. |
+
+An item whose link matches no siteinfo record is never fetched, and `interval`
+does not apply to it. Where a link does match, the wait follows the actual
+fetch attempt whether it succeeded or failed. A non-positive or non-numeric
+`interval` — including it being absent — means no wait at all.
 
 Needs `nokogiri`: `gem install nokogiri`, or the `html` group in a checkout.
 
@@ -1045,6 +1118,35 @@ pipeline expects. Needed because GitHub publishes Atom, not RSS. No settings.
 A field that is already a string is taken as it stands, so a pipeline that has
 been through another filter first is no longer a `NoMethodError`.
 
+#### FilterBatch — **Supported**
+
+`filter/batch.rb`. Groups all items in the pipeline into fixed-size batches.
+Feed boundaries are intentionally discarded; each batch becomes one item in one
+output feed.
+
+Recipe:
+
+```yaml
+  - module: FilterBatch
+    config:
+      batch_items: 5
+```
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `batch_items` | integer | Maximum source items in one batch. Required; must be greater than zero. |
+
+The whole pipeline is collected in feed order, then item order, and sliced
+into batches of `batch_items` source items. Each batch becomes one item titled
+`Batch N`; a batch item carries no link. Its description lists the source
+items as `ARTICLE N`, `Title:`, `URL:` and the body, with the `ARTICLE`
+numbering starting over at 1 in every batch. An empty pipeline produces an
+empty pipeline. Anything other than a positive integer — including zero, a
+negative number, and a non-numeric or fractional string — is a settings error,
+raised when the plugin is constructed; a numeric string such as `"5"` is
+accepted. It is independent of `FilterJoin` — it does not require it, call it,
+or depend on it in any way — and reaches no network and no external service.
+
 #### FilterJoin — **Supported**
 
 `filter/join.rb`. Joins every item in the pipeline into one item. Many items
@@ -1832,21 +1934,19 @@ is a claim that the plugin works.
 
 ---
 
-## 7. Summary
-
-| Status | Count | Plugins |
-| --- | --- | --- |
-| Supported | 26 | `SubscriptionFeed`, `SubscriptionLink`, `SubscriptionXml`, `SubscriptionText`, `CustomFeedWeb`, `FilterIgnore`, `FilterAccept`, `FilterSort`, `FilterOne`, `FilterRand`, `FilterClear`, `FilterImage`, `FilterImageSource`, `FilterAbsoluteURI`, `FilterSanitize`, `FilterTumblrResize`, `FilterDescriptionLink`, `FilterGithubFeed`, `FilterJoin`, `StorePermalink`, `StoreFullText`, `StoreDigest`, `StoreFile`, `PublishMarkdown`, `PublishConsole`, `PublishConsoleLink` |
-| Supported (external) | 14 | `SubscriptionTumblr`, `CustomFeedSVNLog`, `FilterFullFeed`, `FilterOpenAI`, `FilterClaude`, `FilterGemini`, `FilterSakuraAI`, `ProvideFluentd`, `NotifyIkachan`, `PublishEject`, `PublishMemcached`, `PublishFluentd`, `PublishInstapaper`, `PublishAmazonS3` |
-| Needs rework | 1 | `PublishHatenaBookmark` |
+## 7. Catalogue maintenance
 
-Forty-one plugins. Every one of them either runs, or names the one thing it
-needs from the operator; the single exception says what is wrong with it and
-what fixing it would take.
+Section 6 is the single source of truth for the set of plugins that ship and
+for each plugin's current status. Current plugin totals, per-status totals and
+duplicate current plugin-name lists are not maintained here or in `README.md`;
+adding or removing a plugin changes its implementation, its specification and
+its Section 6 catalogue entry, not a second summary that has to be kept in
+sync.
 
-`spec/doc/plugins_catalogue_spec.rb` holds this table to the files in
-`plugins/`: an entry with no file, a file with no entry, and a count that has
-been left behind by an edit are all failures of the ordinary test suite.
+`spec/doc/plugins_catalogue_spec.rb` verifies that every shipped plugin has
+exactly one Section 6 entry, every Section 6 entry has a shipped plugin at the
+loader-derived path, and every entry uses one of the statuses defined in
+section 5.
 
 ## 8. Plugins that were removed
 

doc/VERSIONS

@@ -4,6 +4,9 @@ automaticruby Repository Version History
 v26.09 (Release Date: TBD)
 --------------------------
 - Refresh the bundled LDRFullFeed siteinfo database from the newer upstream snapshot.
+- Add FilterLimit, FilterBatch, and FilterPresent for pipeline-wide limiting, fixed-size batching, and required-field filtering.
+- Add per-fetch interval handling to FilterFullFeed, FilterImageSource, and FilterDescriptionLink.
+- Make the Section 6 plugin catalogue the single source of truth instead of duplicating current plugin counts and lists.
 
 v26.08 (2026-08-22)
 -------------------

plugins/filter/batch.rb

@@ -0,0 +1,97 @@
+# -*- coding: utf-8 -*-
+# Name::        Automatic::Plugin::Filter::Batch
+# Description:: Group the whole pipeline into fixed-size item batches.
+# Author:       id774 (More info: http://id774.net)
+# Source Code:: https://github.com/id774/automaticruby
+# License::     The GPL version 3, or LGPL version 3 (Dual License).
+# Contact::     idnanashi@gmail.com
+# Created::     Aug 24, 2026
+# Updated::     Aug 24, 2026
+# Copyright::   Copyright (c) 2012-2026 Automatic Ruby Developers.
+
+module Automatic::Plugin
+  class FilterBatch
+    require 'rss'
+
+    # Where one source item ends and the next begins inside a batch's
+    # description, in the manner of FilterJoin's own delimiter -- but built
+    # independently, since a batch item is not a joined item.
+    HEADING = 'ARTICLE'.freeze
+
+    def initialize(config, pipeline = [])
+      @config      = config || {}
+      @pipeline    = pipeline
+      @batch_items = validated_batch_items
+    end
+
+    # Collects the whole pipeline into one Array, discarding feed boundaries,
+    # and slices it into fixed-size batches. Each batch becomes one item in one
+    # output feed.
+    def run
+      items = collect
+      return [] if items.empty?
+
+      feed(items.each_slice(@batch_items).to_a)
+    end
+
+    private
+
+    def validated_batch_items
+      value = begin
+        Integer(@config['batch_items'].to_s, 10)
+      rescue ArgumentError
+        nil
+      end
+
+      if value.nil? || value < 1
+        raise ArgumentError, 'FilterBatch needs batch_items to be a positive integer'
+      end
+
+      value
+    end
+
+    def collect
+      @pipeline.each_with_object([]) do |feeds, items|
+        next if feeds.nil?
+
+        items.concat(feeds.items)
+      end
+    end
+
+    def feed(batches)
+      [RSS::Maker.make('2.0') { |maker|
+        maker.channel.title = 'Automatic Ruby'
+        maker.channel.description = 'Automatic::Plugin::FilterBatch'
+        maker.channel.link = 'https://github.com/id774/automaticruby'
+        maker.items.do_sort = false
+
+        batches.each_with_index do |batch, index|
+          item = maker.items.new_item
+          item.title = "Batch #{index + 1}"
+          item.description = description(batch)
+          item.date = Time.now
+        end
+      }]
+    end
+
+    def description(batch)
+      batch.each_with_index.map { |item, index| section(index + 1, item) }.join("\n\n")
+    end
+
+    def section(number, item)
+      ["#{HEADING} #{number}",
+       "Title: #{value(item, :title)}",
+       "URL: #{value(item, :link)}",
+       '',
+       value(item, :description)].join("\n")
+    end
+
+    # A field an item does not carry is empty rather than absent, so that every
+    # ARTICLE has the same shape whatever the feed it came from left out.
+    def value(item, name)
+      return '' unless item.respond_to?(name)
+
+      item.public_send(name).to_s.strip
+    end
+  end
+end

plugins/filter/description_link.rb

@@ -5,7 +5,7 @@
 # License::     The GPL version 3, or LGPL version 3 (Dual License).
 # Contact::     idnanashi@gmail.com
 # Created::     Oct 03, 2014
-# Updated::     Aug 15, 2026
+# Updated::     Aug 24, 2026
 # Copyright::   Copyright (c) 2012-2026 Automatic Ruby Developers.
 
 module Automatic::Plugin
@@ -63,10 +63,27 @@ def retitle(item)
     def fetch_title(url)
       return nil unless Automatic::Http.fetchable?(url)
 
-      Nokogiri::HTML.parse(Automatic::Http.read(url)).xpath('//title').text
+      Nokogiri::HTML.parse(page(url)).xpath('//title').text
     rescue StandardError => e
       Automatic::Log.puts('warn', "Failed in get title for: #{url}, #{e.message}")
       nil
     end
+
+    # The one place this plugin actually reaches the network. `wait` runs in
+    # the `ensure` so that a fetch attempt is waited out whether it succeeded
+    # or raised -- a URL that is not fetchable never gets here at all, and so
+    # never waits.
+    def page(url)
+      Automatic::Http.read(url)
+    ensure
+      wait
+    end
+
+    # `interval` seconds after a real fetch attempt, positive values only. See
+    # doc/PLUGINS.md section 6.3.
+    def wait
+      seconds = @config['interval'].to_i
+      sleep(seconds) if seconds.positive?
+    end
   end
 end