[FLINK-35324][formats] Avro format can not perform projection pushdown for specific fields

[johntomcat7408-cmyk]

Supersedes #28746 (targeted release-1.17), which targeted release-1.17. Moved to master per @davidradl's review; can be backported later.

What is the purpose of the change

The avro format returns a ProjectableDecodingFormat, but projection pushdown only works when the projected fields start at index 0; a projection such as [[1], [2]] fails with ArrayIndexOutOfBoundsException. The deserializer used the projected schema as both the writer and reader schema, so the positional Avro decoding reads the wrong fields. This passes the full physical schema as the writer schema and the projected schema as the reader schema, so Avro schema resolution reads only the projected fields.

Brief change log

AvroDeserializationSchema: support an optional writer schema and resolve the input with setSchema(writer) + setExpected(reader); add a forGeneric(Schema reader, Schema writer) overload.
AvroFormatFactory#createRuntimeDecoder: use the projected schema as the reader schema and the full physical schema as the writer schema.

Verifying this change

AvroRowDataDeSerializationSchemaTest

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): yes
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• 7a04a22 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build