As long as the damper ring and rubber are doing their job, a damper with no or very minimal interference fit will not be a problem for awhile assuming the damper bolt is tight.  As the damper becomes aged with either the ring becoming loose and/or the rubber getting hard and having less flexibility, then the damper will have an increased tendency to move at the crankshaft snout regardless of how tight the damper bolt is.  The higher the compression ratio, the more the issue with this.
 
If the damper is of inadequate size or weight for the engine such as in the case of a modified engine using a stock oem style damper, then the lack of an interference fit will see some movement between the damper and the snout regardless of damper condition.  This will eventually create wear at the crankshaft key that holds it in place.  As the key becomes worn, so does the slot in the crankshaft snout that retains the key.  As James mentions, there may be some galling or metal transfer taking place with any unwanted movement between the crankshaft snout and the damper.
 
The higher the horsepower output, the more sensitive the engine is to needing a damper that is ‘tuned’ to the needs of the engine.  An interference fit insures that the damper does its job in counteracting those harmonics being created by the crankshaft.

Lorena, Texas (South of Waco)

I had my damper rebuilt by Winslow mfg.  The tools from CASCO did the trick, taking it off and re-installing it.  Especially with the engine still in the car.

My experience, tho limited in running time, always “exchanged” metal when they were not a tight fit on both surfaces. The end where mine butted up to the crank and the sliding surface fit.
By run time I mean these were on racing engines not street driven and saw rpm in the 5500 to 6000 range.
On our current race 422” engine where the damper is tight fit and needs a breaker bar to press on, there has never been any exchanged metal and it sees 7200 at the end of the straights..

56 Vic, B'Ville 200 MPH Club Member, So Cal.

I bought a NOS dampener to install on my spare motor, and when I realized it wouldn’t go on without honing, I took it to a local machine shop that had the proper tool.  I THOUGHT my measuring tools were accurate, and gave the shop specific specs.  When I finally installed it, I found it went on just a bit too easy (I had the correct installation tool).  Lesson learned; Better to underestimate how much material to remove, since you can always take it back again if it’s still too tight.
P.S. Wondering the consequences of not enough installation interference, even if the dampener is well bolted down? (Ted?)

6 VOLTS/POS. GRD. NW INDIANA

It’s why I buy longer cap screws and flat washers to install dampers. Generously lube the threads. The first engines I ever over hauled had no threads. I was taught to install the damper with the pan off and use wedges to protect the thrust bearing.

56 Vic, B'Ville 200 MPH Club Member, So Cal.

Beating on the damper affects the thrust bearing surface, regardless of the material used.  It shocks all the bearings in the rods, crankshaft, and can create stress cracks.  Don't do it.  Joe-JDC

JDC

I bought the damper from Summit Racing and the box had no labels.  It did have timing marks. 

I would like to report success.  With a lot of emery cloth, some anti seize lubricant and a mallet (I hadn't read Ted's comments) it is now in place.  I used a large round piece of wood so I didn't hit directly on the damper.  Also, as soon as I could get the bolt started I used the bolt to take it the final distance.

You don’t mention what brand damper or the application.  High pressure lubricant or anti-seize is always recommended when installing the damper and that’s assuming the amount of interference fit is correct.  This should not be said but I’ll mention it anyway, “Always use a damper installing tool and not a mallet to install the damper”.
 
I do deal with a large number of ATI dampers that require honing before they will press fit on the crankshaft snouts and those do require some honing to get the correct fit on the snout.  When it comes to honing the damper holes, a special hone is required to resize a hole with a keyway slot.  Trying to enlarge a hole with a keyway slot using the incorrect hone will egg shape the hole.

Lorena, Texas (South of Waco)

Beside a brake hone on the damper, look closely at the snout of the crank to see if there are any spots that cold be causing the problem.  Specifically inspect the key way in the crank and pulley for any dirt or other material that can cause the problem.  Also check the key closely for any thing amiss.


54 Victoria 312;  48 Ford Conv 302, 56 Bird 312
Forever Ford
Midland Park, NJ

I find that it helps to have the right tools. I just bought a damper puller from CASCO. It arrived within 3 days and works well. They also sell a damper installer. Tools aren't cheap, but not having the right tools can be costly.