This a tougher question than it sounds. The quick simple answer is: The National Fire Protection Association standard 211 says, Chimneys, fireplaces, and vents shall be inspected at least once a year for soundness, freedom from deposits, and correct clearances. Cleaning, maintenance, and repairs shall be done if necessary.” This is the national safety standard and is the correct way to approach the problem. It takes into account the fact that even if you don’t use your chimney much, animals may build nests in the flue or there may be other types of deterioration that could make the chimney unsafe to use.
The Chimney Safety Institute of America recommends that open masonry fireplaces should be cleaned at 1/4″ of sooty buildup, and sooner if there is any glaze present in the system. Factory-built fireplaces should be cleaned when any appreciable buildup occurs. This is considered to be enough fuel buildup to cause a chimney fire capable of damaging the chimney or spreading to the home.