Reduce run time for GitHub Actions

Created by: asogaard

Currently build tests run for 20 mins. or more, meaning that workflows with two dependent tasks may take 45 mins. to run. As these workflows run for each push and pull request, waiting for this long potentially introduces development friction. Therefore, we should try to reduce the run time, which is currently dominated by the installation of torch and related packages.