I have not done this, but from looking at the Fastify JWT library, it looks pretty straight forward:
https://github.com/fastify/fastify-jwt#options
Since there's no Fastify FusionAuth integration, you need to pass the public and private key if you want to verify and sign the JWTs, respectively. Since FusionAuth is creating the JWT, in theory you could just pass the public key of an RSA keypair, since you'll never need to sign the JWT (haven't tested that though).
To add the keys to FusionAuth, you'll want to use Keymaster: https://fusionauth.io/docs/v1/tech/core-concepts/key-master
Hope that helps.