They were limited produced in the Davidsons backyard till 1906 when they moved into a full fledged factory

They moved into their first factory at Chestnut Street which has remained their factory since

Boy do you have the wrong information.
The first engine was machined in the Milwaukee garage of Henry Melk. Then their "First Factory" was built in the Davidson's back yard. From there there were at least five factories before the Juneau Ave facility was built. Now H-D has several factories all over the country where parts are made and shipped to York, PA and Kansas City where the bikes are assembled.