From df4c7611cea71fde7f1fe739f277b3a12738320e Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 07:46:52 +0000 Subject: [PATCH] Add FilterLimit, FilterBatch, FilterPresent, and per-fetch intervals Adds three pipeline-wide filters: FilterLimit caps total items passed downstream, FilterBatch groups all items into fixed-size batches regardless of feed boundaries, and FilterPresent keeps only items whose configured fields are all present. Adds an `interval` setting to FilterFullFeed, FilterImageSource, and FilterDescriptionLink so a Recipe can wait after each real page-fetch attempt, whether it succeeded or failed, without affecting existing behavior when the setting is absent. Makes doc/PLUGINS.md section 6 the single source of truth for the shipped plugin catalogue: README.md and section 7 no longer duplicate current plugin counts or per-status counts, and spec/doc/plugins_catalogue_spec.rb checks file/catalogue correspondence and duplicate entries without requiring manual count synchronization. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01U6QucmzFv12m9y4P6aL3zc --- README.md | 21 +-- doc/PLUGINS.md | 130 ++++++++++++++++--- doc/VERSIONS | 3 + plugins/filter/batch.rb | 97 ++++++++++++++ plugins/filter/description_link.rb | 21 ++- plugins/filter/full_feed.rb | 21 ++- plugins/filter/image_source.rb | 21 ++- plugins/filter/limit.rb | 57 ++++++++ plugins/filter/present.rb | 75 +++++++++++ spec/doc/plugins_catalogue_spec.rb | 67 +++------- spec/plugins/filter/batch_spec.rb | 101 ++++++++++++++ spec/plugins/filter/description_link_spec.rb | 78 ++++++++++- spec/plugins/filter/full_feed_spec.rb | 59 ++++++++- spec/plugins/filter/image_source_spec.rb | 58 ++++++++- spec/plugins/filter/limit_spec.rb | 70 ++++++++++ spec/plugins/filter/present_spec.rb | 86 ++++++++++++ 16 files changed, 885 insertions(+), 80 deletions(-) create mode 100644 plugins/filter/batch.rb create mode 100644 plugins/filter/limit.rb create mode 100644 plugins/filter/present.rb create mode 100644 spec/plugins/filter/batch_spec.rb create mode 100644 spec/plugins/filter/limit_spec.rb create mode 100644 spec/plugins/filter/present_spec.rb diff --git a/README.md b/README.md index 0414f24..b7a4a6c 100644 --- a/README.md +++ b/README.md @@ -115,8 +115,8 @@ a plugin set every part of which still has somewhere to talk to. See - **Recipes in YAML.** A job is a file, not a program. No Ruby is written to wire a pipeline together. -- **41 plugins** across seven categories: subscribe, custom feed, filter, - store, provide, notify, publish — and every one of them has a current use. +- **Plugins across seven categories.** Subscribe, custom feed, filter, store, + provide, notify and publish plugins compose through the same pipeline contract. - **Markdown out of the box.** `PublishMarkdown` writes the result as a plain Markdown document, to a file or to standard output, with no service and no credential behind it. It is the natural end of a new Recipe. @@ -392,15 +392,15 @@ like a shipped plugin replaces it. ### Which plugins still work -41 plugins ship with the gem. Every one is classified in +Every shipped plugin is classified in [`doc/PLUGINS.md`](doc/PLUGINS.md) section 6, with its settings and the reason for its status: -| Status | Count | Meaning | -| --- | --- | --- | -| **Supported** | 26 | Works on the supported Rubies with current dependencies | -| **Supported (external)** | 14 | Works, but needs something you provide: a service, a command, a credential, a data file | -| **Needs rework** | 1 | The service exists; this plugin speaks a replaced interface | +| Status | Meaning | +| --- | --- | +| **Supported** | Works on the supported Rubies with current dependencies | +| **Supported (external)** | Works, but needs something you provide: a service, a command, a credential, a data file | +| **Needs rework** | The service exists; this plugin speaks a replaced interface | Eleven plugins were removed in this release rather than kept as history: each talked to a service that has shut down, or through an API that has been @@ -408,8 +408,9 @@ withdrawn with no replacement. They are listed with their reasons in [`doc/PLUGINS.md`](doc/PLUGINS.md) section 8, and Git history holds the code. A Recipe naming one of them now fails at load, before anything runs. -Restoring the one in **Needs rework** — `PublishHatenaBookmark` — is -self-contained work and a good first contribution. +`PublishHatenaBookmark` is currently classified as **Needs rework**; restoring +it to the service's current interface is self-contained work and a good first +contribution. No plugin here is stubbed, mocked or simulated to make a test pass. Where a plugin's gem is not installed its spec is skipped and says which gem is diff --git a/doc/PLUGINS.md b/doc/PLUGINS.md index d242117..bce8dca 100644 --- a/doc/PLUGINS.md +++ b/doc/PLUGINS.md @@ -866,6 +866,34 @@ match. Same three keys, same substring rule. An item whose field is missing is not matched, and says so; it used to end the run with a `NoMethodError`, which is not what its complement does with the same item. +#### FilterPresent — **Supported** + +`filter/present.rb`. Keeps only items for which every configured field is +present. It is an AND filter over field presence, not a keyword matcher. + +Recipe: + +```yaml + - module: FilterPresent + config: + fields: + - title + - description +``` + +| Key | Type | Meaning | +| --- | --- | --- | +| `fields` | sequence | Fields that must all be present. Required and non-empty. | + +The fields that may be checked are `title`, `link`, `description`, `author`, +`comments`, `source` and `content_encoded`. A field the item has no accessor +for, a `nil` value, an empty string and a whitespace-only string are all +absent; an RSS field that responds to `#content` — a parsed source, for +instance — is judged on that content rather than on the element itself. An +item is kept only when every configured field is present. An unknown field or +a field named twice is a settings error, raised when the plugin is +constructed. No network access, and no dependency on another plugin. + #### FilterSort — **Supported** `filter/sort.rb`. Sorts each feed's items by date. @@ -887,6 +915,31 @@ setting works; see section 2.6.1. | --- | --- | --- | | `pick` | string | `last` takes the last item. Anything else, including absent, takes the first. | +#### FilterLimit — **Supported** + +`filter/limit.rb`. Limits how many items the whole pipeline passes downstream. +The limit is shared across feeds rather than applied once per feed. + +Recipe: + +```yaml + - module: FilterLimit + config: + max_items: 20 +``` + +| Key | Type | Meaning | +| --- | --- | --- | +| `max_items` | integer | Maximum items passed by the whole pipeline. Required; must be greater than zero. | + +Items are selected in feed order, then item order, up to `max_items`; the +grouping of the input feeds is kept in the output, and a feed that contributed +no item under the limit is left out of it. Once the limit is reached, later +feeds are not walked at all. Anything other than a positive integer — +including zero, a negative number, and a non-numeric or fractional string — is +a settings error, raised when the plugin is constructed; a numeric string such +as `"20"` is accepted. No network access, and no dependency on another plugin. + #### FilterRand — **Supported** `filter/rand.rb`. Shuffles each feed's items. Combined with `FilterOne`, picks @@ -914,7 +967,16 @@ this filter will therefore keep links it used to blank. `filter/image_source.rb`. Replaces each item with one item per image found: the images in the description, or, if there are none, the images on the page the -link points at. Fetching pages means network access. No settings. +link points at. Fetching pages means network access. + +| Key | Type | Meaning | +| --- | --- | --- | +| `interval` | integer | Seconds to wait after each page fetch attempt. Default `0`. | + +An item whose images come from the description is never fetched, and +`interval` does not apply to it. Where a page is fetched, the wait follows the +actual fetch attempt whether it succeeded or failed. A non-positive or +non-numeric `interval` — including it being absent — means no wait at all. Needs `nokogiri`: `gem install nokogiri`, or the `html` group in a checkout. @@ -974,9 +1036,14 @@ the body. | --- | --- | --- | | `clear_description` | `1` | Empty the description afterwards. Any other value leaves it. | | `get_title` | `1` | Fetch the new link and use its ``. Any other value skips it. | +| `interval` | integer | Seconds to wait after each title-page fetch attempt when `get_title` is `1`. Default `0`. | `get_title` makes one request per item; use `FilterOne` or a store plugin before -it on a large feed. +it on a large feed. When `get_title` is not `1`, no title page is fetched and +`interval` does not apply. A URL that is not fetchable is not read either, and +does not wait. Where a title page is read, the wait follows the actual fetch +attempt whether it succeeded or failed. A non-positive or non-numeric +`interval` — including it being absent — means no wait at all. **Both settings were being ignored in every real run.** The test that guarded them asked whether the settings mapping was a `Hash`, and the framework hands a @@ -999,6 +1066,12 @@ page. | Key | Type | Meaning | | --- | --- | --- | | `siteinfo` | string | File name under the assets directory. Required. | +| `interval` | integer | Seconds to wait after each article-page fetch attempt. Default `0`. | + +An item whose link matches no siteinfo record is never fetched, and `interval` +does not apply to it. Where a link does match, the wait follows the actual +fetch attempt whether it succeeded or failed. A non-positive or non-numeric +`interval` — including it being absent — means no wait at all. Needs `nokogiri`: `gem install nokogiri`, or the `html` group in a checkout. @@ -1045,6 +1118,35 @@ pipeline expects. Needed because GitHub publishes Atom, not RSS. No settings. A field that is already a string is taken as it stands, so a pipeline that has been through another filter first is no longer a `NoMethodError`. +#### FilterBatch — **Supported** + +`filter/batch.rb`. Groups all items in the pipeline into fixed-size batches. +Feed boundaries are intentionally discarded; each batch becomes one item in one +output feed. + +Recipe: + +```yaml + - module: FilterBatch + config: + batch_items: 5 +``` + +| Key | Type | Meaning | +| --- | --- | --- | +| `batch_items` | integer | Maximum source items in one batch. Required; must be greater than zero. | + +The whole pipeline is collected in feed order, then item order, and sliced +into batches of `batch_items` source items. Each batch becomes one item titled +`Batch N`; a batch item carries no link. Its description lists the source +items as `ARTICLE N`, `Title:`, `URL:` and the body, with the `ARTICLE` +numbering starting over at 1 in every batch. An empty pipeline produces an +empty pipeline. Anything other than a positive integer — including zero, a +negative number, and a non-numeric or fractional string — is a settings error, +raised when the plugin is constructed; a numeric string such as `"5"` is +accepted. It is independent of `FilterJoin` — it does not require it, call it, +or depend on it in any way — and reaches no network and no external service. + #### FilterJoin — **Supported** `filter/join.rb`. Joins every item in the pipeline into one item. Many items @@ -1832,21 +1934,19 @@ is a claim that the plugin works. --- -## 7. Summary - -| Status | Count | Plugins | -| --- | --- | --- | -| Supported | 26 | `SubscriptionFeed`, `SubscriptionLink`, `SubscriptionXml`, `SubscriptionText`, `CustomFeedWeb`, `FilterIgnore`, `FilterAccept`, `FilterSort`, `FilterOne`, `FilterRand`, `FilterClear`, `FilterImage`, `FilterImageSource`, `FilterAbsoluteURI`, `FilterSanitize`, `FilterTumblrResize`, `FilterDescriptionLink`, `FilterGithubFeed`, `FilterJoin`, `StorePermalink`, `StoreFullText`, `StoreDigest`, `StoreFile`, `PublishMarkdown`, `PublishConsole`, `PublishConsoleLink` | -| Supported (external) | 14 | `SubscriptionTumblr`, `CustomFeedSVNLog`, `FilterFullFeed`, `FilterOpenAI`, `FilterClaude`, `FilterGemini`, `FilterSakuraAI`, `ProvideFluentd`, `NotifyIkachan`, `PublishEject`, `PublishMemcached`, `PublishFluentd`, `PublishInstapaper`, `PublishAmazonS3` | -| Needs rework | 1 | `PublishHatenaBookmark` | +## 7. Catalogue maintenance -Forty-one plugins. Every one of them either runs, or names the one thing it -needs from the operator; the single exception says what is wrong with it and -what fixing it would take. +Section 6 is the single source of truth for the set of plugins that ship and +for each plugin's current status. Current plugin totals, per-status totals and +duplicate current plugin-name lists are not maintained here or in `README.md`; +adding or removing a plugin changes its implementation, its specification and +its Section 6 catalogue entry, not a second summary that has to be kept in +sync. -`spec/doc/plugins_catalogue_spec.rb` holds this table to the files in -`plugins/`: an entry with no file, a file with no entry, and a count that has -been left behind by an edit are all failures of the ordinary test suite. +`spec/doc/plugins_catalogue_spec.rb` verifies that every shipped plugin has +exactly one Section 6 entry, every Section 6 entry has a shipped plugin at the +loader-derived path, and every entry uses one of the statuses defined in +section 5. ## 8. Plugins that were removed diff --git a/doc/VERSIONS b/doc/VERSIONS index 80f8785..aeba796 100644 --- a/doc/VERSIONS +++ b/doc/VERSIONS @@ -4,6 +4,9 @@ automaticruby Repository Version History v26.09 (Release Date: TBD) -------------------------- - Refresh the bundled LDRFullFeed siteinfo database from the newer upstream snapshot. +- Add FilterLimit, FilterBatch, and FilterPresent for pipeline-wide limiting, fixed-size batching, and required-field filtering. +- Add per-fetch interval handling to FilterFullFeed, FilterImageSource, and FilterDescriptionLink. +- Make the Section 6 plugin catalogue the single source of truth instead of duplicating current plugin counts and lists. v26.08 (2026-08-22) ------------------- diff --git a/plugins/filter/batch.rb b/plugins/filter/batch.rb new file mode 100644 index 0000000..c68f559 --- /dev/null +++ b/plugins/filter/batch.rb @@ -0,0 +1,97 @@ +# -*- coding: utf-8 -*- +# Name:: Automatic::Plugin::Filter::Batch +# Description:: Group the whole pipeline into fixed-size item batches. +# Author: id774 (More info: http://id774.net) +# Source Code:: https://github.com/id774/automaticruby +# License:: The GPL version 3, or LGPL version 3 (Dual License). +# Contact:: idnanashi@gmail.com +# Created:: Aug 24, 2026 +# Updated:: Aug 24, 2026 +# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. + +module Automatic::Plugin + class FilterBatch + require 'rss' + + # Where one source item ends and the next begins inside a batch's + # description, in the manner of FilterJoin's own delimiter -- but built + # independently, since a batch item is not a joined item. + HEADING = 'ARTICLE'.freeze + + def initialize(config, pipeline = []) + @config = config || {} + @pipeline = pipeline + @batch_items = validated_batch_items + end + + # Collects the whole pipeline into one Array, discarding feed boundaries, + # and slices it into fixed-size batches. Each batch becomes one item in one + # output feed. + def run + items = collect + return [] if items.empty? + + feed(items.each_slice(@batch_items).to_a) + end + + private + + def validated_batch_items + value = begin + Integer(@config['batch_items'].to_s, 10) + rescue ArgumentError + nil + end + + if value.nil? || value < 1 + raise ArgumentError, 'FilterBatch needs batch_items to be a positive integer' + end + + value + end + + def collect + @pipeline.each_with_object([]) do |feeds, items| + next if feeds.nil? + + items.concat(feeds.items) + end + end + + def feed(batches) + [RSS::Maker.make('2.0') { |maker| + maker.channel.title = 'Automatic Ruby' + maker.channel.description = 'Automatic::Plugin::FilterBatch' + maker.channel.link = 'https://github.com/id774/automaticruby' + maker.items.do_sort = false + + batches.each_with_index do |batch, index| + item = maker.items.new_item + item.title = "Batch #{index + 1}" + item.description = description(batch) + item.date = Time.now + end + }] + end + + def description(batch) + batch.each_with_index.map { |item, index| section(index + 1, item) }.join("\n\n") + end + + def section(number, item) + ["#{HEADING} #{number}", + "Title: #{value(item, :title)}", + "URL: #{value(item, :link)}", + '', + value(item, :description)].join("\n") + end + + # A field an item does not carry is empty rather than absent, so that every + # ARTICLE has the same shape whatever the feed it came from left out. + def value(item, name) + return '' unless item.respond_to?(name) + + item.public_send(name).to_s.strip + end + end +end diff --git a/plugins/filter/description_link.rb b/plugins/filter/description_link.rb index 75cd339..73e7763 100644 --- a/plugins/filter/description_link.rb +++ b/plugins/filter/description_link.rb @@ -5,7 +5,7 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Oct 03, 2014 -# Updated:: Aug 15, 2026 +# Updated:: Aug 24, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. module Automatic::Plugin @@ -63,10 +63,27 @@ def retitle(item) def fetch_title(url) return nil unless Automatic::Http.fetchable?(url) - Nokogiri::HTML.parse(Automatic::Http.read(url)).xpath('//title').text + Nokogiri::HTML.parse(page(url)).xpath('//title').text rescue StandardError => e Automatic::Log.puts('warn', "Failed in get title for: #{url}, #{e.message}") nil end + + # The one place this plugin actually reaches the network. `wait` runs in + # the `ensure` so that a fetch attempt is waited out whether it succeeded + # or raised -- a URL that is not fetchable never gets here at all, and so + # never waits. + def page(url) + Automatic::Http.read(url) + ensure + wait + end + + # `interval` seconds after a real fetch attempt, positive values only. See + # doc/PLUGINS.md section 6.3. + def wait + seconds = @config['interval'].to_i + sleep(seconds) if seconds.positive? + end end end diff --git a/plugins/filter/full_feed.rb b/plugins/filter/full_feed.rb index 1235f68..aeec80c 100644 --- a/plugins/filter/full_feed.rb +++ b/plugins/filter/full_feed.rb @@ -5,7 +5,7 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Apr 29, 2012 -# Updated:: Aug 15, 2026 +# Updated:: Aug 24, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. module Automatic::Plugin @@ -154,13 +154,30 @@ def body(link, entry) # anywhere: it was recorded in 2013, and trusting it ahead of what the page # says would break every site that has changed encoding since. def document(link, entry) - page, declared = Automatic::Http.open(link) { |io| [io.read, declared_charset?(io)] } + page, declared = fetch_page(link) parsed = Nokogiri::HTML.parse(StringIO.new(page)) return parsed if declared || parsed.meta_encoding || entry.encoding.nil? Nokogiri::HTML.parse(StringIO.new(page), nil, entry.encoding) end + # The one place this plugin actually reaches the network. `wait` runs in + # the `ensure` so that a fetch attempt is waited out whether it succeeded + # or raised -- an item with no link and an item whose siteinfo did not + # match never get here at all, and so never wait. + def fetch_page(link) + Automatic::Http.open(link) { |io| [io.read, declared_charset?(io)] } + ensure + wait + end + + # `interval` seconds after a real fetch attempt, positive values only. See + # doc/PLUGINS.md section 6.3. + def wait + seconds = @config['interval'].to_i + sleep(seconds) if seconds.positive? + end + # Whether the response itself named a charset, as opposed to open-uri # having settled on one in the absence of an answer. def declared_charset?(io) diff --git a/plugins/filter/image_source.rb b/plugins/filter/image_source.rb index bffe7ed..8af3ea4 100644 --- a/plugins/filter/image_source.rb +++ b/plugins/filter/image_source.rb @@ -5,7 +5,7 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Feb 28, 2012 -# Updated:: Aug 15, 2026 +# Updated:: Aug 24, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. module Automatic::Plugin @@ -49,12 +49,29 @@ def images(item) end def page_images(link) - sources(Automatic::Http.read(link), link) + sources(page(link), link) rescue StandardError => e Automatic::Log.puts('warn', "Failed to read images from #{link}: #{e.message}") [] end + # The one place this plugin actually reaches the network. `wait` runs in + # the `ensure` so that a fetch attempt is waited out whether it succeeded + # or raised -- an item whose description already had images never gets + # here at all, and so never waits. + def page(link) + Automatic::Http.read(link) + ensure + wait + end + + # `interval` seconds after a real fetch attempt, positive values only. See + # doc/PLUGINS.md section 6.3. + def wait + seconds = @config['interval'].to_i + sleep(seconds) if seconds.positive? + end + # The images of an HTML fragment, as absolute URLs. This was a scan for # `<img src="` before, which found nothing in a document quoting its # attributes with apostrophes or writing src after another attribute; the diff --git a/plugins/filter/limit.rb b/plugins/filter/limit.rb new file mode 100644 index 0000000..91fc641 --- /dev/null +++ b/plugins/filter/limit.rb @@ -0,0 +1,57 @@ +# -*- coding: utf-8 -*- +# Name:: Automatic::Plugin::Filter::Limit +# Description:: Limit the number of items passed by the whole pipeline. +# Author: id774 (More info: http://id774.net) +# Source Code:: https://github.com/id774/automaticruby +# License:: The GPL version 3, or LGPL version 3 (Dual License). +# Contact:: idnanashi@gmail.com +# Created:: Aug 24, 2026 +# Updated:: Aug 24, 2026 +# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. + +module Automatic::Plugin + class FilterLimit + def initialize(config, pipeline = []) + @config = config || {} + @pipeline = pipeline + @max_items = validated_max_items + end + + # Selects items in feed order, item order, up to max_items shared across the + # whole pipeline rather than reset per feed. A feed that contributed no item + # is dropped from the output; the walk stops as soon as the limit is met. + def run + remaining = @max_items + returned = [] + + @pipeline.each do |feeds| + next if feeds.nil? + break if remaining <= 0 + + selected = feeds.items.first(remaining) + next if selected.empty? + + returned << Automatic::FeedMaker.create_pipeline(selected) + remaining -= selected.size + end + + returned + end + + private + + def validated_max_items + value = begin + Integer(@config['max_items'].to_s, 10) + rescue ArgumentError + nil + end + + if value.nil? || value < 1 + raise ArgumentError, 'FilterLimit needs max_items to be a positive integer' + end + + value + end + end +end diff --git a/plugins/filter/present.rb b/plugins/filter/present.rb new file mode 100644 index 0000000..5edf7d1 --- /dev/null +++ b/plugins/filter/present.rb @@ -0,0 +1,75 @@ +# -*- coding: utf-8 -*- +# Name:: Automatic::Plugin::Filter::Present +# Description:: Keep items whose configured fields are all present. +# Author: id774 (More info: http://id774.net) +# Source Code:: https://github.com/id774/automaticruby +# License:: The GPL version 3, or LGPL version 3 (Dual License). +# Contact:: idnanashi@gmail.com +# Created:: Aug 24, 2026 +# Updated:: Aug 24, 2026 +# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. + +module Automatic::Plugin + class FilterPresent + # The item fields presence may be checked over. `date` is not one, being + # absent from a plain summary as often as not, and `enclosure` is not + # either, being a structure rather than a value. + FIELDS = %w[title link description author comments source content_encoded].freeze + + def initialize(config, pipeline = []) + @config = config || {} + @pipeline = pipeline + @fields = validated_fields + end + + # Keeps only the items where every configured field is present. A feed + # that kept nothing is dropped from the output rather than passed on empty. + def run + @pipeline.each_with_object([]) do |feeds, returned| + next if feeds.nil? + + survivors = feeds.items.select { |item| present?(item) } + returned << Automatic::FeedMaker.create_pipeline(survivors) unless survivors.empty? + end + end + + private + + def validated_fields + given = @config['fields'] + raise ArgumentError, 'FilterPresent takes fields as a list' unless given.is_a?(Array) + + names = given.map(&:to_s) + raise ArgumentError, 'FilterPresent needs a non-empty fields list' if names.empty? + + unknown = names - FIELDS + unless unknown.empty? + raise ArgumentError, + "FilterPresent cannot inspect #{unknown.join(', ')}; " \ + "the fields are #{FIELDS.join(', ')}" + end + + duplicated = names.tally.select { |_name, count| count > 1 }.keys + raise ArgumentError, "FilterPresent was given #{duplicated.join(', ')} twice" unless duplicated.empty? + + names + end + + def present?(item) + @fields.all? { |field| field_present?(item, field) } + end + + # A field an item does not carry, or whose value is nil, is absent. A + # field whose value answers #content -- a parsed source or enclosure -- is + # judged on that content rather than on the element itself. + def field_present?(item, field) + return false unless item.respond_to?(field) + + value = item.public_send(field) + return false if value.nil? + + value = value.content if value.respond_to?(:content) + !value.to_s.strip.empty? + end + end +end diff --git a/spec/doc/plugins_catalogue_spec.rb b/spec/doc/plugins_catalogue_spec.rb index 9fbca41..f2a6086 100644 --- a/spec/doc/plugins_catalogue_spec.rb +++ b/spec/doc/plugins_catalogue_spec.rb @@ -5,14 +5,16 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Aug 15, 2026 -# Updated:: Aug 15, 2026 +# Updated:: Aug 24, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. # -# doc/PLUGINS.md section 6 is the catalogue of what ships, and README.md -# repeats its counts. Both are prose, and prose drifts from a directory. This -# holds them to it: a plugin with no entry, an entry with no plugin, and a -# count left behind by an edit are failures of the ordinary suite rather than -# something a reader discovers. +# doc/PLUGINS.md section 6 is the catalogue of what ships: the set of plugins +# and each one's current status. This spec compares that catalogue with the +# plugin files themselves -- a plugin with no entry, an entry with no plugin, +# and an entry duplicated within section 6 are all failures of the ordinary +# suite rather than something a reader discovers. No second count or plugin +# list is maintained anywhere as a target to keep this spec, or section 6, +# synchronized with. require File.expand_path(File.join(File.dirname(__FILE__), '../spec_helper')) @@ -42,18 +44,20 @@ def shipped_plugins # "#### SubscriptionFeed — **Supported**" ENTRY = /^\#\#\#\#\s+(\w+)\s+—\s+\*\*(.+?)\*\*/ - def catalogue + # Section 6's entries, in the order they appear, as [name, status] pairs. A + # name that appears twice is left duplicated here rather than collapsed by + # #to_h, so a spec can tell an accidental duplicate apart from a plugin with + # no entry at all. + def catalogue_entries document = File.read(File.join(APP_ROOT, 'doc', 'PLUGINS.md'), encoding: 'UTF-8') section = document[/^## 6\. The plugins$.*?^## 7\./m] raise 'doc/PLUGINS.md has no section 6' if section.nil? - section.scan(ENTRY).to_h + section.scan(ENTRY) end - # "| Supported (external) | 10 | `SubscriptionTumblr`, ... |" - def summary_rows - document = File.read(File.join(APP_ROOT, 'doc', 'PLUGINS.md'), encoding: 'UTF-8') - document.scan(/^\|\s(Supported|Supported \(external\)|Needs rework)\s\|\s(\d+)\s\|\s(.+?)\s\|$/) + def catalogue + catalogue_entries.to_h end let(:shipped) { shipped_plugins } @@ -88,40 +92,11 @@ def summary_rows end end - describe 'the summary table in section 7' do - it 'counts what section 6 lists' do - counted = entries.values.group_by { |status| status.sub(/,.*\z/, '') }. - transform_values(&:size) - - summary_rows.each do |status, count, _plugins| - count.to_i.should == counted.fetch(status, 0) - end - end - - it 'names what section 6 lists' do - summary_rows.each do |status, _count, plugins| - named = plugins.scan(/`(\w+)`/).flatten.sort - expected = entries.select { |_name, value| value.sub(/,.*\z/, '') == status }.keys.sort - named.should == expected - end - end - - it 'accounts for every plugin exactly once' do - summary_rows.sum { |_status, count, _plugins| count.to_i }.should == shipped.size - end - end - - describe 'README.md' do - let(:readme) { File.read(File.join(APP_ROOT, 'README.md'), encoding: 'UTF-8') } + it 'lists every plugin exactly once' do + names = catalogue_entries.map(&:first) + duplicates = names.tally.select { |_name, count| count > 1 }.keys.sort - it 'gives the same total' do - readme.should include("#{shipped.size} plugins") - end - - it 'gives the same count per status' do - summary_rows.each do |status, count, _plugins| - readme.should match(/\*\*#{Regexp.escape(status)}\*\*\s*\|\s*#{count}\s*\|/) - end - end + duplicates.should == [] + shipped.sort.should == names.sort end end diff --git a/spec/plugins/filter/batch_spec.rb b/spec/plugins/filter/batch_spec.rb new file mode 100644 index 0000000..8802d90 --- /dev/null +++ b/spec/plugins/filter/batch_spec.rb @@ -0,0 +1,101 @@ +# -*- coding: utf-8 -*- +# Name:: Automatic::Plugin::Filter::Batch +# Author: id774 (More info: http://id774.net) +# Source Code:: https://github.com/id774/automaticruby +# License:: The GPL version 3, or LGPL version 3 (Dual License). +# Contact:: idnanashi@gmail.com +# Created:: Aug 24, 2026 +# Updated:: Aug 24, 2026 +# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. + +require File.expand_path(File.dirname(__FILE__) + '../../../spec_helper') + +require 'filter/batch' + +describe Automatic::Plugin::FilterBatch do + def batch(config, pipeline) + Automatic::Plugin::FilterBatch.new(config, pipeline).run + end + + let(:pipeline) { + AutomaticSpec.generate_pipeline { + feed { + item 'https://example.com/a', 'A', 'the body of A' + item 'https://example.com/b', 'B', 'the body of B' + item 'https://example.com/c', 'C', 'the body of C' + } + feed { + item 'https://example.com/d', 'D', 'the body of D' + item 'https://example.com/e', 'E', 'the body of E' + item 'https://example.com/f', 'F', 'the body of F' + item 'https://example.com/g', 'G', 'the body of G' + } + } + } + + describe 'batching across feed boundaries' do + let(:returned) { batch({ 'batch_items' => 3 }, pipeline) } + let(:descriptions) { returned[0].items.map(&:description) } + + it 'puts every batch into one output feed' do + returned.should have(1).feed + returned[0].items.should have(3).items + end + + it 'titles each batch item in order' do + returned[0].items.map(&:title).should == ['Batch 1', 'Batch 2', 'Batch 3'] + end + + it 'sets no link on a batch item' do + returned[0].items.map(&:link).should == [nil, nil, nil] + end + + it 'groups A, B and C as ARTICLE 1 through 3 in the first batch' do + %w[A B C].each { |title| descriptions[0].should include("Title: #{title}") } + descriptions[0].should include('ARTICLE 1') + descriptions[0].should include('ARTICLE 2') + descriptions[0].should include('ARTICLE 3') + end + + it 'groups D, E and F as ARTICLE 1 through 3 in the second batch' do + %w[D E F].each { |title| descriptions[1].should include("Title: #{title}") } + descriptions[1].should include('ARTICLE 1') + descriptions[1].should include('ARTICLE 2') + descriptions[1].should include('ARTICLE 3') + end + + it 'puts G alone as ARTICLE 1 in the third batch' do + descriptions[2].should include('Title: G') + descriptions[2].should include('ARTICLE 1') + end + + it 'carries the original title, URL and description into each ARTICLE' do + descriptions[0].should include('Title: A') + descriptions[0].should include('URL: https://example.com/a') + descriptions[0].should include('the body of A') + end + + it 'restarts ARTICLE numbering at the second batch rather than continuing it' do + descriptions[1].index('ARTICLE 1').should < descriptions[1].index('ARTICLE 2') + descriptions[1].should_not include('ARTICLE 4') + end + end + + it 'accepts batch_items as a numeric string' do + returned = batch({ 'batch_items' => '2' }, pipeline) + + returned.should have(1).feed + returned[0].items.should have(4).items + end + + it 'returns an empty pipeline for empty input' do + batch({ 'batch_items' => 2 }, []).should == [] + end + + it 'rejects invalid batch_items' do + [nil, 0, -1, '', 'abc', '1.5'].each do |value| + lambda { Automatic::Plugin::FilterBatch.new({ 'batch_items' => value }, []) }. + should raise_error(ArgumentError, 'FilterBatch needs batch_items to be a positive integer') + end + end +end diff --git a/spec/plugins/filter/description_link_spec.rb b/spec/plugins/filter/description_link_spec.rb index e34cac1..f3376fa 100644 --- a/spec/plugins/filter/description_link_spec.rb +++ b/spec/plugins/filter/description_link_spec.rb @@ -5,7 +5,7 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Oct 03, 2014 -# Updated:: Aug 14, 2026 +# Updated:: Aug 24, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. require File.expand_path(File.dirname(__FILE__) + '../../../spec_helper') @@ -168,5 +168,81 @@ } end + describe "the interval setting" do + context "when interval is positive and every title page fetch succeeds" do + subject { + Automatic::Plugin::FilterDescriptionLink.new( + { 'get_title' => 1, 'interval' => 2 }, + AutomaticSpec.generate_pipeline { + feed { + item "http://test1.id774.net", "dummy title 1", + "aaa bbb ccc http://test2.id774.net ddd eee" + item "http://test3.id774.net", "dummy title 2", + "aaa bbb ccc http://test4.id774.net ddd eee" + } + } + ) + } + + before do + Automatic::Http.stub(:read). + and_return('<html><head><title>Fetched title') + end + + it "waits after every title page fetch" do + subject.should_receive(:sleep).with(2).twice + subject.run + subject.instance_variable_get(:@pipeline)[0].items.each do |item| + item.title.should == "Fetched title" + end + end + end + + context "when get_title is disabled" do + subject { + Automatic::Plugin::FilterDescriptionLink.new( + { 'interval' => 2 }, + AutomaticSpec.generate_pipeline { + feed { + item "http://test1.id774.net", "dummy title", + "aaa bbb ccc http://test2.id774.net ddd eee" + } + } + ) + } + + it "does not wait" do + Automatic::Http.should_not_receive(:read) + subject.should_not_receive(:sleep) + subject.run + subject.instance_variable_get(:@pipeline)[0].items[0].link. + should == "http://test2.id774.net" + end + end + + context "when the title page fetch fails" do + subject { + Automatic::Plugin::FilterDescriptionLink.new( + { 'get_title' => 1, 'interval' => 2 }, + AutomaticSpec.generate_pipeline { + feed { + item "http://test1.id774.net", "dummy title", + "aaa bbb ccc http://test2.id774.net ddd eee" + } + } + ) + } + + before { Automatic::Http.stub(:read).and_raise(StandardError, 'no such host') } + + it "waits once and keeps the existing title" do + subject.should_receive(:sleep).with(2).once + lambda { subject.run }.should_not raise_error + subject.instance_variable_get(:@pipeline)[0].items[0].title. + should == "dummy title" + end + end + end + end end diff --git a/spec/plugins/filter/full_feed_spec.rb b/spec/plugins/filter/full_feed_spec.rb index c4b05aa..089141b 100644 --- a/spec/plugins/filter/full_feed_spec.rb +++ b/spec/plugins/filter/full_feed_spec.rb @@ -5,7 +5,7 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Jan 24, 2013 -# Updated:: Aug 15, 2026 +# Updated:: Aug 24, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. require File.expand_path(File.dirname(__FILE__) + '../../../spec_helper') @@ -263,6 +263,63 @@ def response(body, content_type) should raise_error(ArgumentError, /siteinfo/) end end + + describe "the interval setting" do + context "when interval is positive and every fetch succeeds" do + subject { + Automatic::Plugin::FilterFullFeed.new( + { 'siteinfo' => 'test.json', 'interval' => 2 }, + AutomaticSpec.generate_pipeline { + feed { + item 'http://example.com/article-1', 'a title', 'the summary the feed gave' + item 'http://example.com/article-2', 'another title', 'another summary' + } + }) + } + + it "waits after each page fetch" do + subject.should_receive(:sleep).with(2).twice + lambda { subject.run }.should_not raise_error + end + end + + context "when the siteinfo does not match, so no page is fetched" do + let(:records) { [FullFeedSpec.record('^http://elsewhere\.example/', '//div')] } + + subject { + Automatic::Plugin::FilterFullFeed.new( + { 'siteinfo' => 'test.json', 'interval' => 2 }, + AutomaticSpec.generate_pipeline { + feed { item 'http://example.com/article', 'a title', 'the summary the feed gave' } + }) + } + + it "does not wait" do + Automatic::Http.should_not_receive(:open) + subject.should_not_receive(:sleep) + lambda { subject.run }.should_not raise_error + end + end + + context "when the fetch fails" do + before { Automatic::Http.stub(:open).and_raise(Errno::ECONNREFUSED) } + + subject { + Automatic::Plugin::FilterFullFeed.new( + { 'siteinfo' => 'test.json', 'interval' => 2 }, + AutomaticSpec.generate_pipeline { + feed { item 'http://example.com/article', 'a title', 'the summary the feed gave' } + }) + } + + it "waits once and keeps the existing summary" do + subject.should_receive(:sleep).with(2).once + lambda { subject.run }.should_not raise_error + subject.instance_variable_get(:@pipeline)[0].items[0].description. + should == 'the summary the feed gave' + end + end + end end describe Automatic::Plugin::FilterFullFeed do diff --git a/spec/plugins/filter/image_source_spec.rb b/spec/plugins/filter/image_source_spec.rb index 58fae8f..b2dc4fb 100644 --- a/spec/plugins/filter/image_source_spec.rb +++ b/spec/plugins/filter/image_source_spec.rb @@ -5,7 +5,7 @@ # License:: The GPL version 3, or LGPL version 3 (Dual License). # Contact:: idnanashi@gmail.com # Created:: Mar 1, 2012 -# Updated:: Aug 14, 2026 +# Updated:: Aug 24, 2026 # Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. require File.expand_path(File.dirname(__FILE__) + '../../../spec_helper') @@ -134,3 +134,59 @@ } end end + +describe Automatic::Plugin::FilterImageSource, "the interval setting" do + context "when interval is positive and every page fetch succeeds" do + subject { + Automatic::Plugin::FilterImageSource.new( + { 'interval' => 2 }, + AutomaticSpec.generate_pipeline { + feed { + item "http://example.com/a", "", "" + item "http://example.com/b", "", "" + }})} + + before do + Automatic::Http.stub(:read) { |url| "" } + end + + it "waits after every page fetch" do + subject.should_receive(:sleep).with(2).twice + lambda { subject.run }.should_not raise_error + end + end + + context "when the images come from the description" do + subject { + Automatic::Plugin::FilterImageSource.new( + { 'interval' => 2 }, + AutomaticSpec.generate_pipeline { + feed { + item "http://example.com/a", "", + "" + }})} + + it "does not wait" do + Automatic::Http.should_not_receive(:read) + subject.should_not_receive(:sleep) + subject.run[0].items.map(&:link).should == ['http://example.com/a.png'] + end + end + + context "when the page fetch fails" do + subject { + Automatic::Plugin::FilterImageSource.new( + { 'interval' => 2 }, + AutomaticSpec.generate_pipeline { + feed { item "http://example.com/a", "", "" } + })} + + before { Automatic::Http.stub(:read).and_raise(StandardError, 'no such host') } + + it "waits once and returns no images" do + subject.should_receive(:sleep).with(2).once + returned = subject.run + returned[0].items.should be_empty + end + end +end diff --git a/spec/plugins/filter/limit_spec.rb b/spec/plugins/filter/limit_spec.rb new file mode 100644 index 0000000..ffdedb2 --- /dev/null +++ b/spec/plugins/filter/limit_spec.rb @@ -0,0 +1,70 @@ +# -*- coding: utf-8 -*- +# Name:: Automatic::Plugin::Filter::Limit +# Author: id774 (More info: http://id774.net) +# Source Code:: https://github.com/id774/automaticruby +# License:: The GPL version 3, or LGPL version 3 (Dual License). +# Contact:: idnanashi@gmail.com +# Created:: Aug 24, 2026 +# Updated:: Aug 24, 2026 +# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. + +require File.expand_path(File.dirname(__FILE__) + '../../../spec_helper') + +require 'filter/limit' + +describe Automatic::Plugin::FilterLimit do + def limit(config, pipeline) + Automatic::Plugin::FilterLimit.new(config, pipeline).run + end + + # Automatic::FeedMaker.create_pipeline sorts each feed it builds by date, + # newest first; explicit, descending dates are what keep the output feeds' + # own item order predictable enough to assert on here. + let(:pipeline) { + AutomaticSpec.generate_pipeline { + feed { + item 'https://example.com/a', 'A', '', 'Mon, 07 Mar 2011 15:54:12 +0900' + item 'https://example.com/b', 'B', '', 'Mon, 07 Mar 2011 15:54:11 +0900' + } + feed { + item 'https://example.com/c', 'C', '', 'Mon, 07 Mar 2011 15:54:10 +0900' + item 'https://example.com/d', 'D', '', 'Mon, 07 Mar 2011 15:54:09 +0900' + } + } + } + + it 'limits the whole pipeline instead of each feed' do + returned = limit({ 'max_items' => 3 }, pipeline) + + returned.should have(2).feeds + returned[0].items.should have(2).items + returned[1].items.should have(1).item + + links = returned.flat_map { |feeds| feeds.items.map(&:link) } + links.should == %w[https://example.com/a https://example.com/b https://example.com/c] + links.should_not include('https://example.com/d') + end + + it 'keeps every item when max_items exceeds the pipeline size' do + returned = limit({ 'max_items' => 10 }, pipeline) + + returned.sum { |feeds| feeds.items.size }.should == 4 + end + + it 'accepts max_items as a numeric string' do + returned = limit({ 'max_items' => '2' }, pipeline) + + returned.sum { |feeds| feeds.items.size }.should == 2 + end + + it 'returns an empty pipeline for empty input' do + limit({ 'max_items' => 2 }, []).should == [] + end + + it 'rejects invalid max_items' do + [nil, 0, -1, '', 'abc', '1.5'].each do |value| + lambda { Automatic::Plugin::FilterLimit.new({ 'max_items' => value }, []) }. + should raise_error(ArgumentError, 'FilterLimit needs max_items to be a positive integer') + end + end +end diff --git a/spec/plugins/filter/present_spec.rb b/spec/plugins/filter/present_spec.rb new file mode 100644 index 0000000..d91cc44 --- /dev/null +++ b/spec/plugins/filter/present_spec.rb @@ -0,0 +1,86 @@ +# -*- coding: utf-8 -*- +# Name:: Automatic::Plugin::Filter::Present +# Author: id774 (More info: http://id774.net) +# Source Code:: https://github.com/id774/automaticruby +# License:: The GPL version 3, or LGPL version 3 (Dual License). +# Contact:: idnanashi@gmail.com +# Created:: Aug 24, 2026 +# Updated:: Aug 24, 2026 +# Copyright:: Copyright (c) 2012-2026 Automatic Ruby Developers. + +require File.expand_path(File.dirname(__FILE__) + '../../../spec_helper') + +require 'filter/present' + +describe Automatic::Plugin::FilterPresent do + def present(config, pipeline) + Automatic::Plugin::FilterPresent.new(config, pipeline).run + end + + it 'keeps only items where every configured field is present' do + returned = present({ 'fields' => %w[title description] }, + AutomaticSpec.generate_pipeline { + feed { + item 'https://example.com/a', 'A', 'body A' + item 'https://example.com/b', 'B', '' + item 'https://example.com/c', '', 'body C' + } + }) + + returned.should have(1).feed + returned[0].items.should have(1).item + returned[0].items[0].title.should == 'A' + end + + it 'treats whitespace-only text as absent' do + returned = present({ 'fields' => %w[description] }, + AutomaticSpec.generate_pipeline { + feed { item 'https://example.com/a', 'A', " \n\t " } + }) + + returned.should == [] + end + + it 'accepts a single present field' do + returned = present({ 'fields' => %w[description] }, + AutomaticSpec.generate_pipeline { + feed { + item 'https://example.com/a', 'A', 'body' + item 'https://example.com/b', 'B', '' + } + }) + + returned.should have(1).feed + returned[0].items.should have(1).item + returned[0].items[0].title.should == 'A' + end + + it 'rejects a non-list fields setting' do + lambda { Automatic::Plugin::FilterPresent.new({ 'fields' => 'description' }, []) }. + should raise_error(ArgumentError, 'FilterPresent takes fields as a list') + end + + it 'rejects an empty fields list' do + lambda { Automatic::Plugin::FilterPresent.new({ 'fields' => [] }, []) }. + should raise_error(ArgumentError, 'FilterPresent needs a non-empty fields list') + end + + it 'rejects unknown fields' do + lambda { Automatic::Plugin::FilterPresent.new({ 'fields' => %w[title date] }, []) }. + should raise_error( + ArgumentError, + 'FilterPresent cannot inspect date; the fields are title, link, description, ' \ + 'author, comments, source, content_encoded' + ) + end + + it 'rejects duplicated fields' do + lambda { Automatic::Plugin::FilterPresent.new({ 'fields' => %w[title title] }, []) }. + should raise_error(ArgumentError, 'FilterPresent was given title twice') + end + + it 'rejects a missing fields setting' do + lambda { Automatic::Plugin::FilterPresent.new({}, []) }. + should raise_error(ArgumentError, 'FilterPresent takes fields as a list') + end +end