简化 NestJS 中的文件上传：无需磁盘存储即可高效内存解析 CSV 和 XLSX-js教程-PHP中文网

首页

web前端

js教程

简化 NestJS 中的文件上传：无需磁盘存储即可高效内存解析 CSV 和 XLSX

DDD

Oct 01, 2024 am 06:26 AM

NestJS 中的轻松文件解析：管理内存中的 CSV 和 XLSX 上传，以提高速度、安全性和可扩展性

介绍

在 Web 应用程序中处理文件上传是一项常见任务，但处理不同的文件类型并确保正确处理它们可能具有挑战性。通常，开发人员需要解析上传的文件而不将其保存到服务器，这对于降低服务器存储成本并确保敏感数据不会被不必要地保留尤为重要。在本文中，我们将逐步介绍创建自定义 NestJS 模块来处理专门针对 CSV 和 XLS/XLSX 文件的文件上传的过程，并且我们将使用 Node.js 流在内存中解析这些文件，因此不会有静态文件在服务器上创建。

为什么选择 NestJS？

NestJS 是一个渐进式 Node.js 框架，它利用 TypeScript 并提供开箱即用的应用程序架构，使您能够构建高度可测试、可扩展、松散耦合且易于维护的应用程序。通过使用 NestJS，我们可以利用其模块化结构、强大的依赖注入系统和广泛的生态系统。

第 1 步：设置项目

在深入研究代码之前，让我们先建立一个新的 NestJS 项目。如果您还没有安装 NestJS CLI：

npm install -g @nestjs/cli

登录后复制

创建一个新的 NestJS 项目：

nest new your-super-name

登录后复制

导航到项目目录：

cd your-super-name

登录后复制

第2步：安装所需的软件包

我们需要安装一些额外的软件包来处理文件上传和解析：

npm install @nestjs/platform-express multer exceljsfile-type

登录后复制

Multer：处理multipart/form-data的中间件，主要用于上传文件。
Exlesjs：用于解析 CSV/XLS/XLSX 文件的强大库。
File-Type：用于检测流或缓冲区的文件类型的库。

步骤 3：创建 Multer 存储引擎而不保存文件

为了自定义文件上传过程，我们将创建一个自定义 Multer 存储引擎。该引擎将确保仅接受 CSV 和 XLS/XLSX 文件，使用 Node.js 流在内存中解析它们，并返回解析后的数据，而不将任何文件保存到磁盘。

为我们的引擎创建一个新文件：

import { PassThrough } from 'stream';
import * as fileType from 'file-type';
import { BadRequestException } from '@nestjs/common';
import { Request } from 'express';
import { Workbook } from 'exceljs';
import { createParserCsvOrXlsx } from './parser-factory.js';

const ALLOWED_MIME_TYPES = [
  'text/csv',
  'application/vnd.ms-excel',
  'text/comma-separated-values',
  'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
  'application/vnd.ms-excel',
] as const;

export class CsvOrXlsxMulterEngine {
  private destKey: string;
  private maxFileSize: number;
  constructor(opts: { destKey: string; maxFileSize: number }) {
    this.destKey = opts.destKey;
    this.maxFileSize = opts.maxFileSize;
  }
  async _handleFile(req: Request, file: any, cb: any) {
    try {
      const contentLength = Number(req.headers['content-length']);
      if (
        typeof contentLength === 'number' &&
        contentLength > this.maxFileSize
      ) {
        throw new Error(`Max file size is ${this.maxFileSize} bytes.`);
      }
      const fileStream = await fileType.fileTypeStream(file.stream);
      const mime = fileStream.fileType?.mime ?? file.mimetype;
      if (!ALLOWED_MIME_TYPES.includes(mime)) {
        throw new BadRequestException('File must be *.csv or *.xlsx');
      }
      const replacementStream = new PassThrough();
      fileStream.pipe(replacementStream);
      const parser = createParserCsvOrXlsx(mime);
      const data = await parser.read(replacementStream);
      cb(null, {
        [this.destKey]:
          mime === 'text/csv' ? data : (data as Workbook).getWorksheet(),
      });
    } catch (error) {
      cb(error);
    }
  }
  _removeFile(req: Request, file: any, cb: any) {
    cb(null);
  }
}

登录后复制

此自定义存储引擎检查文件的 MIME 类型并确保它是 CSV 或 XLS/XLSX 文件。然后，它使用 Node.js 流完全在内存中处理该文件，因此不会在服务器上创建临时文件。这种方法既高效又安全，尤其是在处理敏感数据时。

第 4 步：创建解析器工厂

解析器工厂负责根据文件类型确定合适的解析器。

为我们的解析器创建一个新文件：

import excel from 'exceljs';

export function createParserCsvOrXlsx(mime: string) {
  const workbook = new excel.Workbook();
  return [
    'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
    'application/vnd.ms-excel',
  ].includes(mime)
    ? workbook.xlsx
    : workbook.csv;
}

登录后复制

此工厂函数检查 MIME 类型并返回适当的解析器（xlsx 或 csv）。

第5步：在NestJS控制器中配置Multer

接下来，让我们创建一个控制器来使用我们的自定义存储引擎处理文件上传。

生成一个新的控制器：

nest g controller files

登录后复制

在files.controller.ts中，使用Multer和自定义存储引擎配置文件上传：

import {
  Controller,
  Post,
  UploadedFile,
  UseInterceptors,
} from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import { Worksheet } from 'exceljs';
import { CsvOrXlsxMulterEngine } from '../../shared/multer-engines/csv-xlsx/engine.js';
import { FilesService } from './files.service.js';

const MAX_FILE_SIZE_IN_MiB = 1000000000; // Only for test

@Controller('files')
export class FilesController {
  constructor(private readonly filesService: FilesService) {}
  @UseInterceptors(
    FileInterceptor('file', {
      storage: new CsvOrXlsxMulterEngine({
        maxFileSize: MAX_FILE_SIZE_IN_MiB,
        destKey: 'worksheet',
      }),
    }),
  )
  @Post()
  create(@UploadedFile() data: { worksheet: Worksheet }) {
    return this.filesService.format(data.worksheet);
  }
}

登录后复制

该控制器设置一个端点来处理文件上传。上传的文件由 CsvOrXlsxMulterEngine 处理，解析后的数据在响应中返回，而不会保存到磁盘。

第 6 步：设置模块

最后，我们需要设置一个模块来包含我们的控制器。

生成一个新模块：

nest g module files

登录后复制

在files.module.ts中，导入控制器：

import { Module } from '@nestjs/common';
import { FilesController } from './files.controller.js';
import { FilesService } from './files.service.js';

@Module({
  providers: [FilesService],
  controllers: [FilesController],
})
export class FilesModule {}

登录后复制

确保将此模块导入到您的 AppModule 中：

第 7 步：使用 HTML 测试文件上传

为了测试文件上传功能，我们可以创建一个简单的 HTML 页面，允许用户上传 CSV 或 XLS/XLSX 文件。此页面会将文件发送到我们的 /api/files 端点，该端点将在内存中进行解析和处理。

这是用于测试文件上传的基本 HTML 文件：

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>File Upload</title>
</head>
<body>
    <h1>Upload a File (CSV or XLSX)</h1>
    <form action="/api/files" method="post" enctype="multipart/form-data">
        <label for="file">Choose file:</label>
        <input type="file" id="file" name="file" accept=".csv, .xlsx" required>
        <br><br>
        <button type="submit">Upload</button>
    </form>
</body>
</html>

登录后复制

为了渲染文件上传的 HTML 页面，我们首先需要安装一个名为 @nestjs/serve-static 的附加 NestJS 模块。您可以通过运行以下命令来完成此操作：

npm install @nestjs/serve-static

登录后复制

安装后，我们需要在AppModule中配置该模块：

import { Module } from '@nestjs/common';
import { join } from 'path';
import { ServeStaticModule } from '@nestjs/serve-static';
import { FilesModule } from './modules/files/files.module.js';

@Module({
  imports: [
    FilesModule,
    ServeStaticModule.forRoot({
      rootPath: join(new URL('..', import.meta.url).pathname, 'public'),
      serveRoot: '/',
    }),
  ],
})
export class AppModule {}

登录后复制

此设置将允许我们从公共目录提供静态文件。现在，我们可以通过在浏览器中导航到http://localhost:3000来打开文件上传页面。

Streamline File Uploads in NestJS: Efficient In-Memory Parsing for CSV & XLSX Without Disk Storage

上传您的文件

To upload a file, follow these steps:

Choose a file by clicking on the ‘Choose file’ button.
Click on the ‘Upload’ button to start the upload process.

Once the file is uploaded successfully, you should see a confirmation that the file has been uploaded and formatted.

Streamline File Uploads in NestJS: Efficient In-Memory Parsing for CSV & XLSX Without Disk Storage

Note: I haven’t included code for formatting the uploaded file, as this depends on the library you choose for processing CSV or XLS/XLSX files. You can view the complete implementation on GitHub.
Comparing Pros and Cons of In-Memory File Processing
When deciding whether to use in-memory file processing or saving files to disk, it’s important to understand the trade-offs.

Pros of In-Memory Processing:

No Temporary Files on Disk:

Security: Sensitive data isn’t left on the server’s disk, reducing the risk of data leaks.
Resource Efficiency: The server doesn’t need to allocate disk space for temporary files, which can be particularly useful in environments with limited storage.

Faster Processing:

Performance: Parsing files in memory can be faster since it eliminates the overhead of writing and reading files from disk.
Reduced I/O Operations: Fewer disk I/O operations means lower latency and potent ially higher throughput for file processing.

Simplified Cleanup:

No Cleanup Required: Since files aren’t saved to disk, there’s no need to manage or clean up temporary files, simplifying the codebase.

Cons of In-Memory Processing:

Memory Usage:

High Memory Consumption: Large files can consume significant amounts of memory, which might lead to out-of-memory errors if the server doesn’t have enough resources.
Scalability: Handling large files or multiple file uploads simultaneously may require careful memory management and scaling strategies.

File Size Limitations:

Limited by Memory: The maximum file size that can be processed is limited by the available memory on the server. This can be a signific ant drawback for applications dealing with very large files.

Complexity in Error Handling:

Error Management: Managing errors in streaming data can be more complex than handling files on disk, especially in cases where partial data might need to be recovered or analyzed.

When to Use In-Memory Processing:

Small to Medium Files: If your application deals with relatively small files, in-memory processing can offer speed and simplicity.

Security-Sensitive Applications: When handling sensitive data that shouldn’t be stored on disk, in-memory processing can reduce the risk of data breaches.

High-Performance Scenarios: Applications that require high throughput and minimal latency may benefit from the reduced overhead of in-memory processing.

When to Consider Disk-Based Processing:

Large Files: If your application needs to process very large files, disk-based processing may be necessary to avoid running out of memory.

Resource-Constrained Environments: In cases where server memory is limited, processing files on disk can prevent memory exhaustion and allow for better resource management.

Persistent Storage Needs: If you need to retain a copy of the uploaded file for auditing, backup, or later retrieval, saving files to disk is necessary.

Integration with External Storage Services: For large files, consider uploading them to external storage services like AWS S3, Google Cloud

Storage, or Azure Blob Storage. These services allow you to offload storage from your server, and you can process the files in the cloud or retrieve them for in-memory processing as needed.

Scalability: Cloud storage solutions can handle massive files and provide redundancy, ensuring that your data is safe and easily accessible from multiple geographic locations.

Cost Efficiency: Using cloud storage can be more cost-effective for handling large files, as it reduces the need for local server resources and provides pay-as-you-go pricing.