The true differences between a go fast bike and a touring bike are much exaggerated. Do you want to have a one unit bike collection and still go fast? Buy a touring bike. Put skinny tires on it for your go-fast days. Your misinformed riding companions will marvel at your moose-like muscles, but you and I will know that you really aren't overcoming a huge mechanical disadvantage.
Here's the weight difference: a go-fast bike's frame weighs two to four pounds. A touring bike's frame weighs four to five pounds. There's another half pound weight difference in the fork, and a sprinkling of ounces here and there in other components. The most important of these differences is in the wheels and tires, and you can minimize that by using the lightest tires your touring rims will allow.
Your touring bike's rims will be a bit wider than a go-fast bike's rims. The touring bikes I've tested in the past few years have rim widths of 22 to 24mm, and a typical go-fast rim is 18mm. This means you can't fit the narrowest tires on your touring bike. Depending on your bike, the narrowest tire that works well may be in the 23C to 26C range.
But is this a disadvantage? I don't think so. A 20C tire may look cool, but a 26C tire handles better.
The Surly Long Haul Trucker I recently road tested weighed 24.16 pounds with its fat tires. I used to ride USCF races on 23-pound bikes. SO I think Long Haul Trucker is pretty good. Sure, today's carbon wonders routinely trounce the 20-pound barrier, but so what? You can go plenty fast on a 24-pound bike. The "additional" weight hampers your acceleration, but you won't notice that unless you're doing exhaustive sprint workouts. The weight doesn't hamper your top speed. It hampers your hill climbing, but only as much as the weight of a few more water bottles.
Finally, an anecdote: decades ago, I held a USCF racing license and I trained daily on exotic racing bikes. Each morning, I would time my commute to work. This commute was six miles long, it had 68o vertical feet of climbing, and it had 1,030 feet of descending. It had hard cornering, washboard pavement, and one memorable switchback climb. My stopwatch and I rode many a world championship racing machine over those hills, and yet my fastest time on that commute was on my Cannondale touring bike with fenders, a rear rack, and 27 x 1 1/8 inch touring tires.
So if you want to go fast, it ain't the bike that's holding you back.
Source: John Schubert, "What Kind of Bike Should You Buy?", Adventure Cyclist Magazine, Feb 2007.