Backport(v6): Reduce build time by disabling LTO flags on RHEL 10 / Ubuntu (#1101) by Watson1978 · Pull Request #1102 · fluent/fluent-package-builder

Watson1978 · 2026-08-06T00:58:18Z

Backport #1101

RPM

RPM 4.19, which AlmaLinux 10 ships, exports the distribution build flags into the build and install sections automatically. Previously (rpm 4.16 on AlmaLinux 9 and older) those flags were only applied through macros such as %configure or %make_build, which this spec does not use, so nothing was exported at all.

Since the whole build happens in the install section via "rake build:all", the flags now leak into jemalloc, our bundled Ruby and every native gem extension. -flto=auto -ffat-lto-objects is especially costly: mkmf links a small test program for every check in extconf.rb, and each of those links runs the full LTO machinery.

Measured on the AlmaLinux 10 x86_64 build:

configuring ext/* 90s -> 981s
Ruby core compilation 32s -> 81s
rdkafka native ext 62s -> 320s
whole job 9m34s -> 27m16s

We build Ruby with its own optflags anyway, so LTO buys us nothing here. Drop it, along with the annobin plugin which is loaded on every compile and link. The hardening flags are kept as they are.

This is a no-op on AlmaLinux 9 and older, where no flags were exported in the first place.

Ubuntu

Ubuntu enables optimize=+lto by default in dpkg-buildflags, and dpkg-buildpackage exports the resulting flags into the environment. Debian
does not enable it, which is why only the Ubuntu jobs are affected.

Since the whole build happens in override_dh_auto_install via "rake build:all", the flags leak into jemalloc, OpenSSL, our bundled Ruby
and every native gem extension. -flto=auto -ffat-lto-objects is especially
costly: mkmf links a small test program for every check in extconf.rb, and
each of those links runs the full LTO machinery.

Measured on the amd64 builds of a single run:

configuring ext/* Debian bookworm 21s vs Ubuntu Jammy 964s
whole job Debian bookworm 10m vs Ubuntu Jammy 33m,
Noble 36m, Resolute 39m

We build Ruby with its own optflags anyway, so LTO buys us nothing here. The hardening flags are kept as they are.

This is the deb counterpart of the previous commit, which did the same for
RHEL 10.

## RPM RPM 4.19, which AlmaLinux 10 ships, exports the distribution build flags into the build and install sections automatically. Previously (rpm 4.16 on AlmaLinux 9 and older) those flags were only applied through macros such as %configure or %make_build, which this spec does not use, so nothing was exported at all. Since the whole build happens in the install section via "rake build:all", the flags now leak into jemalloc, our bundled Ruby and every native gem extension. -flto=auto -ffat-lto-objects is especially costly: mkmf links a small test program for every check in extconf.rb, and each of those links runs the full LTO machinery. Measured on the AlmaLinux 10 x86_64 build: * configuring ext/* 90s -> 981s * Ruby core compilation 32s -> 81s * rdkafka native ext 62s -> 320s * whole job 9m34s -> 27m16s We build Ruby with its own optflags anyway, so LTO buys us nothing here. Drop it, along with the annobin plugin which is loaded on every compile and link. The hardening flags are kept as they are. This is a no-op on AlmaLinux 9 and older, where no flags were exported in the first place. ## Ubuntu Ubuntu enables optimize=+lto by default in dpkg-buildflags, and dpkg-buildpackage exports the resulting flags into the environment. Debian does not enable it, which is why only the Ubuntu jobs are affected. Since the whole build happens in override_dh_auto_install via "rake build:all", the flags leak into jemalloc, OpenSSL, our bundled Ruby and every native gem extension. -flto=auto -ffat-lto-objects is especially costly: mkmf links a small test program for every check in extconf.rb, and each of those links runs the full LTO machinery. Measured on the amd64 builds of a single run: * configuring ext/* Debian bookworm 21s vs Ubuntu Jammy 964s * whole job Debian bookworm 10m vs Ubuntu Jammy 33m, Noble 36m, Resolute 39m We build Ruby with its own optflags anyway, so LTO buys us nothing here. The hardening flags are kept as they are. This is the deb counterpart of the previous commit, which did the same for RHEL 10. --------- Signed-off-by: Shizuo Fujita <fujita@clear-code.com>

kenhys · 2026-08-06T01:37:46Z

For LTS:

The result of annocheck will be changed (could you record annocheck output (before/after) to explain in release note later)
Could you record ruby -e 'puts RbConfig::CONFIG["CFLAGS"], RbConfig::CONFIG["LDFLAGS"]' result (before/after) to explain in release note later

It is enough to record in this PR comment to look back later.

Watson1978 · 2026-08-06T02:06:24Z

Recorded for the release note: build flags and annocheck output (before / after)

Both packages are taken straight from CI artifacts. 2cdb88f is a direct child of 5ed7e1c, so this PR is the only difference between them.

	commit	workflow run	artifact
before	`5ed7e1c`	30968343393	`packages-almalinux-10`
after	`2cdb88f`	31061357464	`packages-almalinux-10`

Environment: almalinux:10.0 container, annocheck 13.02 (annobin-annocheck-13.02-2.el10). Both fluent-package and fluent-package-debuginfo were installed so that annocheck can resolve the separate debug files.

1. `RbConfig::CONFIG`

This is what fluent-gem install will use when a user builds a native extension later.

before

CFLAGS  : -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -march=x86-64-v3 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC
LDFLAGS : -L. -Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-hardened-ld-errors -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--no-as-needed

after

CFLAGS  : -O2  -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -march=x86-64-v3 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIC
LDFLAGS : -L. -Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-hardened-ld-errors   -Wl,--build-id=sha1 -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--no-as-needed

difference

	removed	added
CFLAGS	`-flto=auto`, `-ffat-lto-objects`, `-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1`	none
LDFLAGS	`-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1`	none

Every hardening flag is kept: -fstack-protector-strong, -D_FORTIFY_SOURCE=3, -D_GLIBCXX_ASSERTIONS, -specs=redhat-hardened-cc1, -fstack-clash-protection, -fcf-protection, -fasynchronous-unwind-tables, -mtls-dialect=gnu2, and on the link side -z relro, -z now, -z pack-relative-relocs, -specs=redhat-hardened-ld. -march=x86-64-v3 is also unchanged.

2. annocheck

Aggregate over all 161 shared objects under `/opt/fluent`

result	before	after
`FAIL: run-path`	157	157
`FAIL: property-note`	2	2
`FAIL: cf-protection`	2	2
`FAIL: bind-now`	1	1
`FAIL: optimization`	2	0
Overall: FAIL	158	158

No test changed from PASS to FAIL. The only difference is yajl.so: FAIL: optimization test because level too low (based upon annobin data), which disappears simply because annocheck no longer has the annobin data to judge it. The binary itself is not built any differently in that respect.

The remaining failures are pre-existing and unrelated to this PR: run-path fires on every object because we install under /opt/fluent rather than /usr, cf-protection and property-note come from zstdruby.so, and bind-now from protobuf_c.so.

`libruby.so.3.4.9`

before

PASS : bind-now test because BIND_NOW dynamic tag seen
PASS : dynamic-segment test
PASS : fast test because no negative result detected
PASS : glibcxx-assertions test
PASS : gnu-relro test
PASS : gnu-stack test because stack segment exists with the correct permissions
PASS : instrumentation test because no negative result detected
PASS : load-segments test because no negative result detected
PASS : lto test because detected in DW_AT_producer string
PASS : notes test because annobin notes found in the .annobin.notes section
PASS : openssl-engine test because no negative result detected
PASS : optimization test because option found in DW_AT_producer string
PASS : pic test because option found in DW_AT_producer string
PASS : pie test because the ELF file header has the correct type
PASS : production test because no negative result detected
PASS : run-path test because no negative result detected
PASS : short-enums test because no negative result detected
PASS : stack-clash test because option found in DW_AT_producer string
PASS : stack-prot test because option found in DW_AT_producer string
PASS : textrel test because no negative result detected
PASS : threads test because no negative result detected
PASS : unicode test because no negative result detected
PASS : writable-got test
skip : branch-protection test because not an AArch64 binary
skip : cf-protection test because mixed Rust and C code - control flow protection is needed but not yet supported by Rust
skip : dynamic-tags test because AArch64 specific
skip : entry test because shared libraries do not use entry points
skip : fips test because not a GO binary
skip : fortify test because LTO compilation discards preprocessor options
skip : gaps test because string notes imply full coverage
skip : go-revision test because no GO compiled code found
skip : implicit-values test because -Wimplicit-int setting is hidden by LTO
skip : property-note test because mixed Rust and C code - property notes are needed but not yet supported by Rust
skip : stack-realign test because not an i686 executable
skip : warnings test because LTO compilation discards preprocessor options

after

PASS : bind-now test because BIND_NOW dynamic tag seen
PASS : dynamic-segment test
PASS : fast test because no negative result detected
PASS : gaps test because no gaps found in .text section coverage
PASS : gnu-relro test
PASS : gnu-stack test because stack segment exists with the correct permissions
PASS : instrumentation test because no negative result detected
PASS : load-segments test because no negative result detected
PASS : notes test because no negative result detected
PASS : openssl-engine test because no negative result detected
PASS : optimization test because option found in DW_AT_producer string
PASS : pic test because option found in DW_AT_producer string
PASS : pie test because the ELF file header has the correct type
PASS : production test because no negative result detected
PASS : run-path test because no negative result detected
PASS : short-enums test because no negative result detected
PASS : stack-clash test because option found in DW_AT_producer string
PASS : stack-prot test because option found in DW_AT_producer string
PASS : textrel test because no negative result detected
PASS : threads test because no negative result detected
PASS : unicode test because no negative result detected
PASS : writable-got test
skip : branch-protection test because not an AArch64 binary
skip : cf-protection test because mixed Rust and C code - control flow protection is needed but not yet supported by Rust
skip : dynamic-tags test because AArch64 specific
skip : entry test because shared libraries do not use entry points
skip : fips test because not a GO binary
skip : fortify test because Rust compilation does not use the C preprocessor
skip : glibcxx-assertions test because source language not C++
skip : go-revision test because no GO compiled code found
skip : implicit-values test because  These tests are only relevent to C source code
skip : lto test because RUST sources are not compiled with LTO
skip : property-note test because mixed Rust and C code - property notes are needed but not yet supported by Rust
skip : stack-realign test because not an i686 executable
skip : warnings test because Rust compilation does not use the C preprocessor

`oj.so (a native gem extension)`

before

FAIL : run-path test because the DT_RPATH/DT_RUNPATH dynamic tag contains a path that does not start with /usr
PASS : bind-now test because BIND_NOW dynamic tag seen
PASS : cf-protection test because correct flags found in .note.gnu.property note
PASS : dynamic-segment test
PASS : fast test because no negative result detected
PASS : glibcxx-assertions test
PASS : gnu-relro test
PASS : gnu-stack test because stack segment exists with the correct permissions
PASS : instrumentation test because no negative result detected
PASS : load-segments test because no negative result detected
PASS : lto test because detected in DW_AT_producer string
PASS : notes test because annobin notes found in the .annobin.notes section
PASS : openssl-engine test because no negative result detected
PASS : optimization test because option found in DW_AT_producer string
PASS : pic test because option found in DW_AT_producer string
PASS : pie test because the ELF file header has the correct type
PASS : production test because no negative result detected
PASS : property-note test because CET enabled property note found
PASS : short-enums test because no negative result detected
PASS : stack-clash test because option found in DW_AT_producer string
PASS : stack-prot test because option found in DW_AT_producer string
PASS : textrel test because no negative result detected
PASS : threads test because no negative result detected
PASS : unicode test because no negative result detected
PASS : writable-got test
skip : branch-protection test because not an AArch64 binary
skip : dynamic-tags test because AArch64 specific
skip : entry test because shared libraries do not use entry points
skip : fips test because not a GO binary
skip : fortify test because LTO compilation discards preprocessor options
skip : gaps test because string notes imply full coverage
skip : go-revision test because no GO compiled code found
skip : implicit-values test because -Wimplicit-int setting is hidden by LTO
skip : stack-realign test because not an i686 executable
skip : warnings test because LTO compilation discards preprocessor options

after

FAIL : run-path test because the DT_RPATH/DT_RUNPATH dynamic tag contains a path that does not start with /usr
PASS : bind-now test because BIND_NOW dynamic tag seen
PASS : cf-protection test because correct flags found in .note.gnu.property note
PASS : dynamic-segment test
PASS : fast test because no negative result detected
PASS : gaps test because no gaps found in .text section coverage
PASS : gnu-relro test
PASS : gnu-stack test because stack segment exists with the correct permissions
PASS : instrumentation test because no negative result detected
PASS : load-segments test because no negative result detected
PASS : notes test because no negative result detected
PASS : openssl-engine test because no negative result detected
PASS : optimization test because option found in DW_AT_producer string
PASS : pic test because option found in DW_AT_producer string
PASS : pie test because the ELF file header has the correct type
PASS : production test because no negative result detected
PASS : property-note test because CET enabled property note found
PASS : short-enums test because no negative result detected
PASS : stack-clash test because option found in DW_AT_producer string
PASS : stack-prot test because option found in DW_AT_producer string
PASS : textrel test because no negative result detected
PASS : threads test because no negative result detected
PASS : unicode test because no negative result detected
PASS : writable-got test
skip : branch-protection test because not an AArch64 binary
skip : dynamic-tags test because AArch64 specific
skip : entry test because shared libraries do not use entry points
skip : fips test because not a GO binary
skip : fortify test because C sources compiled without notes are not checked by this test
skip : glibcxx-assertions test because source language not C++
skip : go-revision test because no GO compiled code found
skip : implicit-values test because  These tests are only relevent to C source code
skip : lto test because C sources compiled without notes are not checked by this test
skip : stack-realign test because not an i686 executable
skip : warnings test because C sources compiled without notes are not checked by this test

What actually changes in the per-test results

Unchanged, because annocheck derives these either from the ELF structure or from the DW_AT_producer string (we still pass -grecord-gcc-switches, so the compiler options stay recorded in DWARF):

pie, bind-now, gnu-relro, gnu-stack, writable-got, dynamic-segment, cf-protection, property-note, textrel, entry, run-path, and notably optimization, pic, stack-clash and stack-prot, which keep reporting option found in DW_AT_producer string.

Changed, because the annobin notes are gone:

test	before	after
`notes`	`PASS` — annobin notes found in the `.annobin.notes` section	`PASS` — no negative result detected
`glibcxx-assertions`	`PASS`	`skip` — source language not C++
`lto`	`PASS` — detected in `DW_AT_producer` string	`skip`
`gaps`	`skip` — string notes imply full coverage	`PASS` — no gaps found in `.text` section coverage

glibcxx-assertions is the only hardening property that can no longer be confirmed from the binary. -D_GLIBCXX_ASSERTIONS is still passed to the compiler, annocheck simply has no note left to verify it with.

It is worth noting that fortify, warnings and implicit-values are skipped both before and after. In the before build they were skipped because of LTO (LTO compilation discards preprocessor options), so LTO was itself preventing those checks from running.

Summary for the release note

The hardening flags of the shipped binaries are unchanged. Only LTO is removed.
annocheck's overall verdict is unchanged: 158 objects report Overall: FAIL, all of which are pre-existing and caused by installing under /opt/fluent instead of /usr, plus three third party gem extensions.
Users who run annocheck themselves will see three differences: the lto test no longer reports LTO, the notes test no longer reports annobin notes, and glibcxx-assertions is skipped instead of passing.
Users who build native extensions with fluent-gem install will get binaries compiled without -flto=auto -ffat-lto-objects and without the annobin plugin. All hardening flags are still applied.

Watson1978 requested a review from kenhys August 6, 2026 00:58

kenhys approved these changes Aug 6, 2026

View reviewed changes

kenhys merged commit bc871bd into fluent-package-v6 Aug 6, 2026
358 of 362 checks passed

kenhys deleted the backport-pr1101 branch August 6, 2026 02:19

Navigation Menu

Uh oh!

Backport(v6): Reduce build time by disabling LTO flags on RHEL 10 / Ubuntu (#1101) - #1102

Backport(v6): Reduce build time by disabling LTO flags on RHEL 10 / Ubuntu (#1101)#1102
kenhys merged 1 commit into
fluent-package-v6from
backport-pr1101

Watson1978 commented Aug 6, 2026

Uh oh!

kenhys commented Aug 6, 2026

Uh oh!

Watson1978 commented Aug 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Watson1978 commented Aug 6, 2026

RPM

Ubuntu

Uh oh!

kenhys commented Aug 6, 2026

Uh oh!

Watson1978 commented Aug 6, 2026

Recorded for the release note: build flags and annocheck output (before / after)

1. RbConfig::CONFIG

before

after

difference

2. annocheck

Aggregate over all 161 shared objects under /opt/fluent

libruby.so.3.4.9

oj.so (a native gem extension)

What actually changes in the per-test results

Summary for the release note

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

1. `RbConfig::CONFIG`

Aggregate over all 161 shared objects under `/opt/fluent`

`libruby.so.3.4.9`

`oj.so (a native gem extension)`