What are main differences between FEM and XFEM? When should we (not) use XFEM intead of FEM and vice versa? In other words, when I meet a new problem, how I can know to use which one of them?
The Finite Element Method (FEM) is the parent method which has inspired many, many other methods and methods which are actually FEM but pretend not to be.
In the finite element method, "shape functions" are used to provide an approximation space so that the solution can be represented by a vector. In the classical FEM, these shape functions are polynomials.
In the Extended Finite Element Method (XFEM), additional "enrichment" functions are used to approximate the solution in addition to the polynomial shape functions. These enrichment functions are chosen to have properties that the solution is known to follow.
The most obvious XFEM enrichment functions are power functions introduced at cracks sharp corners to represent the singularities in the solution gradient (i.e., the singularity in the stress for solid mechanics problems). The XFEM can be used for other enrichment functions and other solution domains (notably heat transfer), but the name synonymous with fracture analysis.
The distinction between various methods -- is this XFEM or not?, etc. -- is tricky and subtle and unimportant.
As for which to use, XFEM sees very little practical use. There are a handful of applications in real finite element codes, most notably Abaqus, but they have not seen widespread acceptance.
For almost all practical problems, the classical FEM would be used. For most fracture analysis problems, classical FEM might still be used with suitable mesh refinement and/or p-refinement in the area of the crack tip. Other, less rigorous, fracture models may also be used.
Both Mike's answer and Jed's one describe well the XFEM/FEM dichotomy and correctly point out that the most important area of application is 3D Fracture Mechanics, where you have a crack, i.e. a displacement discontinuity across a surface inside your domain.
Cracks are hard to model in classical FEM for two reasons:
The mesh has to be congruent across the crack: more precisely the crack has to be at the boundary of a subdomain of FE's. The crack cannot lay inside (pass though) a finite element.
The singular stress field at crack tip requires special elements and/or meshing techniques (quarter point elements, focused mesh) to be modelled with good accuracy.
From an engineering point of view in fracture mechanics you have two main types of problems:
Stress intensity factor computation,
crack propagation analysis, e.g. in fatigue or damage tolerance analysis.
For the first type of problem classical FEM is more than adequate and is the standard engineering tool. (This is because, fortunately, there are energy methods to evaluate the stress intensity factors that are not sensitive to numerical errors near the crack tip.)
Crack propagation analysis is a completely different story: in most cases you do not know beforehand the crack path, and therefore frequent remeshing is necessary. The major promise of XFEM is to allow for crack propagation inside a fixed FEM mesh, the crack cutting his way not only at the boundary between subdomains, but inside the FE's themselves.
XFEM is a relatively new technique, still far from being a standard engineering tool. My answer to the the OP question, at least in solid mechanics and engineering analysis, is that XFEM has a very narrow and specialised application field in crack and damage propagation analysis, for complex 3D geometries, when the crack path cannot be estimated a priori.
Nevertheless let me stress that fracture mechanics is a very important field in engineering: e.g. today's aiplanes are safe also because it is possible to numerically predict damage and crack propagation between maintenance intervals. XFEM, or similar new techniques, are to become important tools in the near future.
FEM is a subset of XFEM. XFEM is a methodology for enriching finite-element spaces to handle problems with discontinuities (such as fracture). With classical FEM, attaining similar accuracy typically requires complicated conformal meshing and adaptive refinement, where as XFEM does it with a single mesh, moving that geometric complexity into the elements (XFEM is very complicated to implement, especially in 3D). Meanwhile, XFEM results in extremely ill-conditioned matrices that require either direct solvers or very specialized multigrid methods (e.g., Gerstenberger and Tuminaro (2012)).