There's not actually a way to stop some of these caught exceptions. Some occur within DevForce and some occur within other .NET components. The URI format exception occurs in a call to MEF (Managed Extensibility Framework). We changed the MEF logic (unrelated to this exception) in 6.1.0 and this particular exception is no longer thrown (and caught) by MEF.
As for optimizing load time, the biggest bang for the buck is usually to set the SearchPatterns and/or IgnorePatterns on the CompositionHost. These settings control which assemblies are probed at startup, and reducing the number of assemblies probed usually results in a performance improvement.
Breaking out a Silverlight application into modules and using application library caching is another good way to improve startup performance.