Storage service

A dedicated service for managing files.

A service for the purpose of managing files may prove quite useful. What we'll first do then is to generate a files module and a storage service.

nest g mo files
nest g s files/storage

The overall structure was inspired by this article from Prateek Kathal (blog).

Here, we'll adopt an ideia similar to the HashingService. That is, to have an abstract class with the methods' signatures and a concrete class with their respective implementations. So, in the StorageService, we should have signatures to:

• Save a file

• Create a directory

• Get a file

• Get the names of files inside a directory

• Calculate the amount of files inside a directory

• Delete a file or directory

• Validate a path

@Injectable()
export abstract class StorageService {
  abstract saveFile(path: string, file: File): Promise<void>;
  abstract createDir(path: string): Promise<void>;
  abstract getFile(path: string): StreamableFile;
  abstract getDirFilenames(path: string): Promise<string[]>;
  abstract getDirFilecount(path: string): Promise<number>;
  abstract delete(path: string): Promise<void>;
  abstract validatePath(path: string): Promise<void>;
}

Each method will be better detailed during its implementation. Therefore, let's proceed to implement the concrete class.

Node.js, by default, has the fs module for managing files. However, the fs-extra package has the same functionalities and more features/improvements. It's already installed, so let's just add its typing.

yarn add -D @types/fs-extra

Now, we can create a concrete service that implements the methods of the StorageService using fs-extra. After creating it, we should make it implement the StorageService.

nest g s files/storage/fse --flat

Let's also already adjust the providers in the FilesModule. Remember to add the StorageService to the exports.

{
  provide: StorageService,
  useClass: FseService,
},

In file.constants, let's define a constant for the base directory where all the files will be stored, which will be the upload folder at the root of the project.

export const BASE_PATH = 'upload';

Also add this folder in the .gitignore file, as we won't publish files to the code repository.

# Upload
/upload

Let's then, finally, implement each one of the methods in the FseService.

• saveFile() - Receives the path and the file, and then saves it in this path prepended with the BASE_PATH, with the file's originalname, from its buffer

async saveFile(path: string, file: File) {
  const fullPath = join(BASE_PATH, path, file.originalname);
  await writeFile(fullPath, file.buffer);
}

The join() function combines many paths into one safely.

• createDir() - The function mkdirp() allows to create folders and even nested ones

async createDir(path: string) {
  const fullPath = join(BASE_PATH, path);
  await mkdirp(fullPath);
}

If the path already exists, it simply doesn't do anything.

• getFile() - This function returns the file as a StreamableFile, which allows for more flexibility

getFile(path: string) {
  const fullPath = join(BASE_PATH, path);
  const stream = createReadStream(fullPath);
  return new StreamableFile(stream);
}

This NestJS doc further explains the class mentioned above.

• getDirFilenames() - Obtains the names of files inside a folder

getDirFilenames(path: string) {
  const fullPath = join(BASE_PATH, path);
  return readdir(fullPath);
}

• getDirFilecount() - Counts how many files are inside a folder, using the previous method

async getDirFilecount(path: string) {
  const dirFilenames = await this.getDirFilenames(path);
  return dirFilenames.length;
}

Notice that the fullPath is not obtained here; it is in the previous method.

• delete() - Deletes a file or folder

async delete(path: string) {
  const fullPath = join(BASE_PATH, path);
  await remove(fullPath);
}

Some notes:

• The remove() function also allows for removing folders with contents

• If the path does not exist, it simply doesn't do anything

• validatePath() - Checks if a path exists, throwing an exception if it does not

async validatePath(path: string) {
  const fullPath = join(BASE_PATH, path);
  if (!(await pathExists(fullPath))) {
    throw new NotFoundException('Path not found');
  }
}

And we're done! We now have a quite interesting set of util functions to aid us when managing files.

Commit - Implementing file storage functions

Before finishing the StorageService, let's just add two more methods there. We may then write methods' signatures to:

• Validate the amount of files to be saved into a directory

• Generate a unique filename from the original one

abstract validateFilecount(count: number, max: number): void;
abstract genUniqueFilename(filename: string): string;

And now, in the FseService, implement them.

• validateFilecount() - Checks if the file count exceeds a maximum

validateFilecount(count: number, max: number) {
  if (count > max) {
    throw new ConflictException('File count exceeds max limit');
  }
}

• genUniqueFilename() - Prepends a filename with a unique prefix

genUniqueFilename(filename: string) {
  const uniquePrefix = `${Date.now()}-${Math.round(Math.random() * 1e9)}`;
  return `${uniquePrefix}-${filename}`;
}

The unique prefix was obtained from the Multer docs.

It's a good practice to save files with unique filenames.

Lastly, adjust the saveFile() method to use the unique filename.

async saveFile(path: string, file: File) {
  const { originalname, buffer } = file;

  const uniqueFilename = this.genUniqueFilename(originalname);
  const fullPath = join(BASE_PATH, path, uniqueFilename);
  await writeFile(fullPath, buffer);
}

Commit - Some more features in file storage