Wearable device usage is burgeoning, with representative applications ranging from patient/athelete monitoring to stress/fatigue identification to the so-called quantified self movement. Typically, cardiac information is monitored via electrocardiograms (ECG) and information such as heart rate (HR) and heart rate variability (HRV) are used as key health related metrics. With many wearable devices, however, lower quality sensors are used, thus resulting in devices that are highly sensible to artifacts due to e.g., user's movement. The introduced artifacts hamper HR/HRV analyses, thus ECG enhancement has been the focus of recent research. Existing enhancement algorithms, however, do not perform well in very noisy conditions, as well as add additional computational processing to already battery-hungry wearable applications. Here, we propose to overcome these limitations by describing a new ECG signal representation called the modulation spectrum. By quantifying the rate-of-change of ECG spectral components, signal and artifactual components become separable, thus allowing for accurate HR and HRV measurement from the noisy signal, even in very extreme conditions typically seen in athletic performance training. The proposed MD-HRV (modulation domain HRV) metric is tested with noise-corrupted synthetic ECG signals and is compared to "true" HRV values obtained from the clean signals. Experimental results show the proposed metric significantly outperforming conventional HRV indices computed on both the noisy, as well as enhanced ECG signals processed by a state-of-the-art wavelet-based algorithm. The obtained findings suggest that the proposed metric is well suited for wearable applications, particularly those involved with intense movement (e.g., in elite athletic training).