Is the change in time step necessary for the grid independent study? As the CFL is based on the relation between dt and dx. In mesh independent study, only change should be mesh i.e, dx isn't it so?
You've almost certainly got to reduce the timestep to maintain stability due to the CFL condition imposed by your explicit timestepping method choice. That being said, for benchmarking purposes, I'd be inclined to pose the problem as computing to a particular final time and want to compare that to the quality of the answers and times to solution on other finer and coarser grids. Maybe you solve them all with the smallest timestep imposed by the finest grid so that the ultimate comparison is all about the grid. Don't throw away the run times, but don't put all your focus on that either, especially if you're submitting to a venue that requires a mesh-refinement study as part of your submission.
As pointed out in the comments, the CFL condition dictates how big timestep can be. Therefore as we progressively refine in space, we would be required to take smaller time steps if an explicit temporal discretization is employed. Regardless of the smaller time steps, it would still be acceptable to get a time-averaged quantity to compare across the various cases with different spatial resolutions.