Filter operators

A more robust way to filter when comparing numbers.

In any real-world online shop, customers would not search for products with an exact price, but below a certain price or within a price range, for example. This involves a certain complexity, so we should better explore and discuss this situation and hypothetical scenarios before proceeding further.

When finding products that cost up to $50, we should receive:

• The value itself (50)

• The filter operator (lte, short for Lower Than or Equal)

• Ex: price=lte:50

We may then follow the aforementioned design with other unary filter operators, they are:

• lt - Lower Than

• lte - Lower Than or Equal

• gt - Greater Than

• gte - Greater Than or Equal

• eq - Equal

We can notice an extra layer of complexity here, as both the value and the operator must be sent together, and they should be extracted correctly. But there is more: when finding products that cost between two prices, a binary filter operator is needed, as it receives two operands.

So, when finding products that cost between $100 and $200, we would need:

• The first value (100)

• The second value (200)

• The filter operator (btw, short for Between)

• Ex: price=btw:100,200

What we can notice there:

• This data needs to be correctly "sliced", as they will all arrive concatenated as a single query param

• The price may assume different forms, being either one or two values

• The operator may operate with a varying amount of operands

It seems this step of the journey won't be quite a walk in the park... But don't worry, we'll get through this together. So, let's start with a positive mood and good spirits.

The first step is to create a DTO to represent a filter operator and its operands. Let's then create it in querying -> dto -> filter-operation.dto.

const Operator = ['gt', 'gte', 'lt', 'lte', 'eq', 'btw'] as const;
type Operator = (typeof Operator)[number];

export class FilterOperationDto {
  @IsIn(Operator)
  readonly operator: Operator;

  @IsNumber({}, { each: true })
  readonly operands: number[];
}

Well, so far there's nothing here that deviates from what is already known. As we want this DTO to be reusable, we just enforce the operands to be numbers, even though there are no negative prices and the validation then becomes a bit more lenient. This could be improved at another moment.

However, note that the operator should determine the length of the operands array. Due to this, we'll need to create a new kind of decorator: one whose validation depends on many fields, and not just on the one upon which it is placed. Let's then create it in querying -> decorators -> validate-filter-operands-length.decorator.

You'll also notice that some of this content is nothing new, we created decorators from scratch when improving the password validation. The difference is in the function that contains the validation logic:

• Through the args parameter, of type ValidationArguments, we can obtain a reference to the object being validaded, and then perform a type assertion because this will only be used in the FilterOperationDto anyway

• Extract the operator and operands, which we assume to exist due to the assertion

• Then, according to the operator, we check if the amount of operands is correct

const VALIDATE_FILTER_OPERANDS_LENGTH_KEY = 'validateFilterOperandsLength';

const validateFilterOperandsLength = (args: ValidationArguments) => {
  const filterOperationDto = args.object as FilterOperationDto;
  const { operator, operands } = filterOperationDto;

  switch (operator) {
    case 'lt':
    case 'lte':
    case 'gt':
    case 'gte':
    case 'eq':
      return operands.length === 1;

    case 'btw':
      return operands.length === 2;

    default:
      const exhaustiveCheck: never = operator;
      return exhaustiveCheck;
  }
};

export const ValidateFilterOperandsLength = (): PropertyDecorator =>
  ValidateBy({
    name: VALIDATE_FILTER_OPERANDS_LENGTH_KEY,
    validator: {
      validate: (value, args): boolean => validateFilterOperandsLength(args),
      defaultMessage: () => 'Operands length is not according to filter operator',
    },
  });

The exhaustiveCheck at the end of the switch ensures that no case is overlooked.

And remember that the validation of this decorator does not depend on the field upon which it is placed. Due to this, we may create a field just for this purpose. As it will simply receive the decorator and won't be directly used, we may make it private and set its type to any.

@ValidateFilterOperandsLength()
private readonly manyFieldValidation: any;

The DTO validation is done. The next step is to create a decorator for a custom Transform, in order to correctly separate and obtain the data that will arrive as the price query param. Only after this, the validation we just implemented should take place. Let's then create it in querying -> decorators -> to-filter-operation-dto.decorator.

The steps performed here are:

• The value is split() on the : and what is before the colon will be the operator, and everything after it is the concOperands, that is, the operands still concatenated

If there is no colon, the operator will be the entire value, and concOperands will be undefined. If there are many colons, the operator will be what's before the first colon, and the concOperands will be what's between the first and second colons (the rest will simply be ignored).

• It is checked if there was any colon, if not then the operandsStr receives an empty array, otherwise it receives the value of concOperands split on the ,

• operands receives the values of the operandsStr array, cast to number

• The plainDto object receives the values of operator and operands

• An instance of FilterOperationDto, generated from the plainDto, is returned

const toFilterOperationDto = (value: string) => {
  const [operator, concOperands] = value.split(':');
  const operandsStr = concOperands ? concOperands.split(',') : [];
  const operands = operandsStr.map((operand) => +operand);

  const plainDto = { operator, operands };
  return plainToInstance(FilterOperationDto, plainDto);
};

export const ToFilterOperationDto = () =>
  Transform(({ value }) => toFilterOperationDto(value));

If you wish, you may visit the TypeScript playground to test this logic yourself.

We may then, after all of this, go back to the ProductsFilterDto and alter the price field. It will be transformed into an instance of FilterOperationDto and, after that, receive its validation.

@IsOptional()
@ValidateNested()
@ToFilterOperationDto()
readonly price?: FilterOperationDto;

Remember that transformation always takes place before validation.

We're almost done. The DTO is correctly transformed and validated. The last step is to actually use it. Luckily, this is the easiest step. Back in the FilteringService, let's create a method that receives this DTO and returns the corresponding TypeORM filter operation. Nothing out of this world here.

compare(filterOperationDto: FilterOperationDto) {
  if (!filterOperationDto) return;

  const { operator, operands } = filterOperationDto;
  const [operand, secondOperand] = operands;

  switch (operator) {
    case 'lt':
      return LessThan(operand);
    case 'lte':
      return LessThanOrEqual(operand);
    case 'gt':
      return MoreThan(operand);
    case 'gte':
      return MoreThanOrEqual(operand);
    case 'eq':
      return Equal(operand);
    case 'btw':
      return Between(operand, secondOperand);

    default:
      const exhaustiveCheck: never = operator;
      return exhaustiveCheck;
  }
}

Then, we just need to use it in the findAll() method in the ProductsService, and we're finished!

price: this.filteringService.compare(price),

Commit - Filtering by price with filter operators