Construct a section of railroad by having an engineer in one of the tiles you wish to join and clicking on the target tile when the railroad icon appears. If you have the technology required to build into the target terrain and out of the tile your engineer is on, you should see an icon of railroad track above the target tile before clicking.
Swamps, hills, and mountains are not available at the start of a standard game. Swamps become available soonest, in about 1821 with the purchase of the Iron Railroad Bridge technology. About 50 turns later, you can research Compound Steam Engine to get the ability to use hills. After another 200 turns or so, you can get Dynamite for the mountains.
If the $100 or more cost would take you over your credit limit, you will get a polite message from your Interior Minister advising that the treasury cannot meet the cost; wait till next turn or reverse another expenditure you have made in the current turn. (Maybe less training for more train, ha ha.)
Linking only to a portEdit
Linking to a port should be as good as linking to the capital, and AI nations seem to do it, but it seems not to work for the human player with the free download version of the game.
In a conquered minor nation, depots need to be linked to the former capital, not just to a port, but in a conquered ex-GP province a link to any port seems to work.
The subsidiary value of your transport system, in addition to its regular job of carrying items to your capital, is the rapid movement of Army units to non-adjacent provinces (even overseas). An attempt to move them will show a black railroad engine icon in any available target province. However, the icon is misleading, because the caoacity for that is based on your number of freight cars, not on the length or location of railroad: five cars are needed for every "gun" that went into the assembly of each regiment involved.