Refining Protein Envelopes with a Transition Region for Enhanced Direct Phasing in Protein Crystallography

In protein crystallography, the determination of an accurate protein envelope is of paramount importance for ab initio phasing of diffraction data. In our previous work, we introduced an approach to ascertain the protein envelope by seeking an optimal cutoff value on a weighted-average density map. In this paper, we present a significant advancement in our approach by focusing on identifying a transition region that demarcates the boundary between the protein and solvent regions, rather than relying solely on a single cutoff value. Within this transition region, we conducted a meticulous search for the protein envelope using a finer map and our proposed transition hybrid input–output (THIO) algorithm. Through this improvement, we achieved a refined protein envelope even when starting from random phases, enabling us to determine protein structures with irregular envelopes and successfully phase crystals with reduced solvent contents. To validate the efficacy of our method, we conducted tests using real diffraction data from five protein crystals, each containing solvent contents ranging from 60% to 65%. Solving these structures through conventional direct methods proved difficult due to the limited solvent content. The mean phase error obtained through our proposed method was about 30°. The reconstructed model matched with the structure in the protein data bank with a root mean square deviation (r.m.s.d.) of about 1 Å. These results serve as compelling evidence that the utilization of the proposed transition region in conjunction with the THIO algorithm contributes significantly to the construction of a reliable protein envelope. This, in turn, becomes indispensable for the direct phasing of protein crystals with lower solvent contents.