The reconstructed image of digital holography is affected by the speckle noise which is random distribution of spatial intensity and granular. The traditional Total Variation (TV) model algorithm has achieved widely attention and great success for solving noise reduction. However, overly smooth on smooth regional of image is easily produced as the smooth area spread along with the edge direction, which even results in the false phenomenon and step effect. We propose a method based on total variation (TV) model algorithm by using gradient descent of the neighborhood mean value for suppressing speckle noise. Firstly, we construct the integral functional of the image by setting the image smooth and fidelity to enable the image pixel values to change with the solutions of this integral functional equation. Secondly, the gradient descent method is used to minimize the function to solve the equation as the pixel values of the noise image are significantly larger than that of the noise free image. Thirdly, the solutions are replaced by the pixels of eight neighborhood average for overcoming the problems of overly smooth and step effect. The effect of denoising by the proposed method is compared with other methods. The experimental results show that speckle reduction is improved, and the algorithm has the best advantage of image edge protection and eliminates the false edge to some degree.