I haven't used pytorch3d in the past, but I wouldn't be surprised if this happens because of different conventions for the camera coordinate system between pyrender and pytorch3d. If you are able to ...

When I wanted to calculate the distance from points to SMPL mesh, I found that the result is almost wrong, far from the result calculated by igl.signed_distance dists, f_id = ...