Coin-op tables make money by storing pocketed balls in a little pen that opens only after being fed more quarters. But they also have to follow the rules of the game and return a pocketed cue ball.

How do they do it? Usually with magnets. Sometimes with size.

When a ball is pocketed on a coin-op table, it goes down a chute and onto a metal track. The track consists of three parallel rails, making two separate tracks. Any pocketed ball at first runs on one set of tracks, which leads to the storage pen. Various mechanisms are used to force the cue ball over onto the other set of tracks, which leads to the return slot.
The preferred way is to load the cue ball with steel and use a magnet to pull it over onto the second track. The favorite type is the “cat’s eye,” which has a steel bearing inserted into a drilled hole, which is then plugged and ground smooth.

Another kind, the “mudball,” has steel particles mixed into the rosin that is used to form the cue ball. Mudballs can have a slightly uneven surface and balance, making them unpopular with serious players.
About 25 years ago, the Valley company got a patent on magnetic systems, and other companies had to innovate. One method is using an oversize cue ball (2 3/8 inches as opposed to the standard 2 ¼ inches) just big enough to catch on a spring that flings it over onto the second track.
Another is using a heavier cue ball that tips a tiny scale and dumps the ball onto the second track.

For those not familiar with coin-operated pool tables, all the balls are held inside the table until the coins are inserted, after which they all fall out. As each ball is hit into a pocket, it rolls down a simple system of chutes that return it to a storage area, where it's retained until the game is over and more money is inserted. However, since it's common to scratch by sinking the cue ball, there has to be a system for returning just that ball. Otherwise, the game would be over the first time anybody scratched. Some coin-operated pool tables solve this problem by using cue balls that are a different size from the colored balls. The standard diameter for pocket billiard balls, as defined by the Billiard Congress of America, is 2-1/4 inches plus or minus .005 inch (ref 1). In the case of tables using oversized cue balls, the cue ball is typically 2-3/8 inches in diameter, or about 1/8 inch larger than the other balls. This system works fairly well, although some players allege that having a larger diameter cue ball throws off their shots and changes the dynamics of the game. Magnetic cue ball return systems rely on a magnet embedded in the cue ball that triggers a sensor as the ball rolls down the return chute. This sensor causes a switch to flip that moves a gate, sending the cue ball into a separate path that returns it to the players. This system apparently works very well (based on calls to a couple local coin-operated pool table supply stores), although tables using magnetic balls seem to be far less common than ones using oversized balls. Magnetic cue balls are a bit more expensive than oversized ones, running around $10 to $15 each as opposed to $4 to $7 for oversized. I couldn't find a significant difference in the cost of the tables themselves, but due to differences in features and design it was difficult to make a broad comparison. Magnetic cue balls are regulation size and weight and thus unlikely to affect play, although anecdotally speaking some say magnetic balls roll and respond differently.