Weigh your options.
In many situations there are great solutions available on AppExchange that can be adopted easily and don't require a complicated integration process. Also, there might be third-party solutions that will be far easier to integrate with the CRM than your legacy software system. To minimize the risks associated with starting a costly integration project, you need to come up with all options, calculate the costs of each, and think about timelines in each case.
Get your team involved.
After all, it's your team who'll have to deal with the newly integrated software. Ask for opinions and suggestions, and they could give you hints about what solution is the best.
Plan in detail.
You should have a clear goal behind the integration and also an understanding of what business processes it will facilitate. This is too demanding an undertaking to do it just because "others do it". Then think about whether data flow between the CRM and the app should be one-way or two-way and whether it will be one-time migration or ongoing synchronization. Select the exact types of data that should be extracted from one system and transferred to another. Remember to include data backup in the strategy.
Consult with an experienced Salesforce integration specialist.
Even if you have a qualified admin on board who has great coding skills, most surely he or she has little to no experience integrating legacy systems with Salesforce. A lot goes into such project, and doing it right from the very beginning will save you time and money. The very least you should do is consulting with a Salesforce developer who has expertise in third-party integration. Otherwise, consider hiring an expert for the job.