Previous responses are well founded and I think Ian covered most of them. In a former life I worked for Gearhart Geodata and, on occasions, drill cutting samples were collected from diesel based muds. The fumes were very unpleasant and significant headaches were the norm. In those days ROPs for mud logging units were determined via a kelly bottle linked to a transducer and there was a constant requirement to re-tape the lines to the kelly hose as a result of any diesel mud on the outside of the hose. This is very old school system and no longer in use by mud logging companies. Rig crew were always having to re-tape their lines.
As a mud engineer from 1990 I have not come across any diesel based systems but that is because I have worked exclusively in all the North Sea sectors where regulations have stopped the use of diesel for the oil phase. Mineral oils are the norm and although they still give off fumes with elevated flowline temperatures they are nothing like the fumes experienced in the past from diesel.
If diesel can be avoided then it is best to do so with respect to HSE for the rig and service crews.
One point of note (which may not apply to you but will apply to many readers) is that as part of the OSPAR agreement in Copenhagen in 2000, it was agreed that the use of diesel oil-based drilling fluids is prohibited. This was based on protection of the marine environment in the NE Atlantic but if your company has similar policies as mine then this is followed globally.