Should writers really worry about adverbs in their fiction?

Should writers really worry about adverbs in their fiction?