Some of the algorithms are available from the source code for the different packages. PyMol is one such, and VMD's source is also accessible.
I implemented VMD's ribbon algorithm in the 1990s. The first step is the structure determination - where are the amino acids? which are connected into a chain? where are the C-alpha atoms?
Next, as Kyle said, is the spline. VMD uses a Catmull–Rom spline, with the C-alphas as control points. This is a 3rd order spline, and the splines goes through the C-alphas. If you work out the math, there's a single free parameter, which corresponds to how stiff the spline is around the control point. I tried a few values until I found the one which was esthetically pleasing.
There's also some trickiness about how to handle the end, which don't have enough C-alphas. I extrapolated to get the other points.
That gives the path. A circular extrusion along the path gives a tube. You can vary the cross-section radii to give an ellipse, and with a bit more work define the ribbon.
The problem is to find the correct norm, so that the ribbons are aligned with an alpha helix. I tried various things then gave up, looked at the Raster3D implementation, got permission to use it, and added that to VMD. It's a cumulative sum of the previous vector norm and the current norm defined by the C-alpha trace. I would have to look at the source for how it works again. Interestingly, Ethan Merritt, the Raster3D author, pointed out that he got that bit of code from FRODO, so it has a long history.
VMD now has "NewRibbons", which was implemented after my time. I don't know how it works.
The easiest way to do an alpha helix is to draw a line from the first to the last residue; extrude a circle along the line and you have a cylinder. You can also do a linear best-fit to the helix, but I think that caused problems for short helices. There's likely more clever ways to do it, including ways like Kyle suggested, which allows for soft bends.
Beta strands are easy. There are two control paths, one for each side. Those define the strand path, and the normal. You do have to be a bit careful with twists, so that your strand doesn't twist 290 degrees when it should twist 70 degrees, but that wasn't hard to handle.
A hard part, which you didn't mention, is how to detect where the alpha-helix and beta-strands are located. Some PDB records contain that, but not all. I punted and used a 3rd-party tool, STRIDE, for that. Warren implemented his own algorithm. Roger Sayle implemented his own version of DSSP for Raster3D.