The simple answer is "history". A longer answer is that the oilfield is evolutionary in product development, and that there are a lot of twists and turns in the evolutionary bunny trail.
To explain the differences a bit more, in the early days of drilling and production, neither rig design nor the time to run tubulars were as 'engineered' as they are now, so there was not as tremendous an amount of importance placed on product development or standardization as there is now. Further, there were also sound marketing tactics involved. For example, going back to the mid- '70's, one of the tactics / philosophies adopted by several of the Japanese OCTG manufacturers when they were breaking into the US market, was to provide casing and tubing in lengths that varied by no more than a few inches ... it looked impressive and made product utilization much simpler, so was perceived as a higher quality product.
Further complicating the issue is that there were and still are some fairly cockamamie engineering / marketing concepts mixed in the industry as well (I could give you half-a-dozen or so clear, crystal clear examples, but don't need the lawsuits).
And, that's without getting into the various personalities of the personnel running the API Committees that set the specifications for API standard products.
Hope that helps a bit.