This work proposes an efficient fourth-order accurate Cartesian grid-based method for the two- dimensional Helmholtz scattering and transmission problems around irregular obstacles. The pro- posed method solves problems in the framework of boundary integral method but does not need to compute (nearly) singular integrals. This is, the integral is evaluated indirectly by solving an equivalent but simple interface problem together with a procedure of polynomial interpolation. The interface problem can be efficiently calculated by a modified fourth-order compact finite difference scheme together with an FFT-based solver. The advantage of using the modified fourth-order com- pact finite difference scheme is that the leading truncation errors of this scheme have less relevance to the wave number of the Helmholtz equation. In contrast with the classical truncated method, this method deals with irregular scatterers naturally and efficiently. Numerical experiments are provided to demonstrate the accuracy and efficiency of the proposed method.