Finding the largest separating rectangle among two point sets

Given a set R of n red points and a set B of m blue points, we study the problem of finding a rectangle that contains all the red points, the minimum number of blue points and has the maximum area. We call such a rectangle a maximum-area separating rectangle (MSR). We first address the planar, axis-aligned (2D) version, and present an $O(m\log m+n)$ time, $O(m+n)$ space algorithm. The running time reduces to $O(m+n)$ if the points are pre-sorted by one of the coordinates. We also consider the planar arbitrary orientation version, in which the MSR is allowed to have arbitrary orientation. For this arbitrary orientation version, our algorithm takes $O(m^3+n\log n)$ time and $O(m+n)$ space. Finally, we address the 3D axis-aligned version, which asks for the maximum-volume separating box (MSB), i.e., the maximum-volume axis-aligned box containing all the red points and the fewest blue points. For this version, we give an algorithm that runs in $O\left(m^2\right(m+n\left)\right)$ time and $O(m+n)$ space.