There are a few things to consider.
how long tokens live for
what happens if permisssion are modified in FusionAuth but the protected resource still allows access?
any performance worries due to a large number of accessToken validation calls being made by the protected resource.
With the first approach (validating the access token without communicating with FusionAuth) the holder of the token will be able to access your API as long as the token is valid (unless the API server communicates periodically with FusionAuth to check the validity). In addition, changes to user privileges won't take place until the JWT expires and the client retrieves a new access token using the refresh token.
With the second approach, if a token is revoked in FusionAuth (if for instance the user is disabled) the access is cut off immediately. The cost is that you're making an additional network call every time, which has a performance impact. Note that if you could use the userinfo endpoint instead of the token if you want updated user claims. The token endpoint isn't going to give you that information, just a yes/no depending on if the token is valid.
So it's hard to make a recommendation without knowing what the consequences of unauthorized access to your API or protected resource would be. It also would be helpful to know the expected traffic; if it is expected to be low, the performance impact of the second approach will be minimal.