linux单目录文件不要过多,多少是过多?

我们都知道一个文件夹文件不宜过多,文件过多,会导致性能下降,那到底怎样算过多呢?

今天板子决定蹂躏下自己的新mac

因本文旨在讨论相同状态下,性能随单目录文件数量变化而发生的变化,所以未做其他特别设置

准备:
Mac Pro (ssd)一台 (安装ubuntu 12.04 虚拟机 LNMP 环境)
5个文件夹,各分别包含1w、5w、10w、50w、100w个同样内容小文件
压测命令:ab -n 10000 -c 100 ‘http://s1.localhost.com/files.php?num=xxx’

files.php:

$num = empty($_GET['num']) ? 10000 : $_GET['num'];
echo file_get_contents("/..filepath../$num/file_".rand(1,$num));

以下是测试结果:


1.1w个文件


Document Path:          /files.php?num=10000
Document Length:        2 bytes

Concurrency Level:      100
Time taken for tests:   4.385 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1500000 bytes
HTML transferred:       20000 bytes
Requests per second:    2280.35 [#/sec] (mean)
Time per request:       43.853 [ms] (mean)
Time per request:       0.439 [ms] (mean, across all concurrent requests)
Transfer rate:          334.04 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0       6
Processing:     3   44   4.5     43      69
Waiting:        3   43   4.5     43      69
Total:          9   44   4.3     43      69

Percentage of the requests served within a certain time (ms)
  50%     43
  66%     44
  75%     45
  80%     46
  90%     49
  95%     52
  98%     54
  99%     56
 100%     69 (longest request)

2. 5w个小文件

Document Path:          /files.php?num=50000
Document Length:        2 bytes

Concurrency Level:      100
Time taken for tests:   5.088 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1500000 bytes
HTML transferred:       20000 bytes
Requests per second:    1965.44 [#/sec] (mean)
Time per request:       50.879 [ms] (mean)
Time per request:       0.509 [ms] (mean, across all concurrent requests)
Transfer rate:          287.91 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0       4
Processing:     5   51   8.2     50     568
Waiting:        5   50   8.2     50     568
Total:          9   51   8.2     50     569

Percentage of the requests served within a certain time (ms)
  50%     50
  66%     51
  75%     52
  80%     53
  90%     56
  95%     58
  98%     61
  99%     62
 100%    569 (longest request)

3. 10w个小文件

Document Path:          /files.php?num=100000
Document Length:        2 bytes

Concurrency Level:      100
Time taken for tests:   6.541 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1500000 bytes
HTML transferred:       20000 bytes
Requests per second:    1528.79 [#/sec] (mean)
Time per request:       65.411 [ms] (mean)
Time per request:       0.654 [ms] (mean, across all concurrent requests)
Transfer rate:          223.94 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       6
Processing:    11   65  35.7     56    1076
Waiting:       11   65  35.7     56    1076
Total:         17   65  35.9     56    1076

Percentage of the requests served within a certain time (ms)
  50%     56
  66%     58
  75%     60
  80%     62
  90%     69
  95%    119
  98%    212
  99%    268
 100%   1076 (longest request)

4. 50w个小文件

Document Path:          /files.php?num=500000
Document Length:        2 bytes

Concurrency Level:      100
Time taken for tests:   19.078 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1500000 bytes
HTML transferred:       20000 bytes
Requests per second:    524.17 [#/sec] (mean)
Time per request:       190.776 [ms] (mean)
Time per request:       1.908 [ms] (mean, across all concurrent requests)
Transfer rate:          76.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0       5
Processing:     9  190  72.0    180     672
Waiting:        9  190  72.0    180     672
Total:         13  190  72.0    180     672

Percentage of the requests served within a certain time (ms)
  50%    180
  66%    216
  75%    236
  80%    248
  90%    280
  95%    304
  98%    334
  99%    508
 100%    672 (longest request)

5.100w个小文件

Document Path:          /files.php?num=1000000
Document Length:        2 bytes

Concurrency Level:      100
Time taken for tests:   40.261 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1500000 bytes
HTML transferred:       20000 bytes
Requests per second:    248.38 [#/sec] (mean)
Time per request:       402.614 [ms] (mean)
Time per request:       4.026 [ms] (mean, across all concurrent requests)
Transfer rate:          36.38 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.7      0       9
Processing:    24  401  91.3    407     726
Waiting:       23  401  91.3    407     726
Total:         29  401  91.3    407     726

Percentage of the requests served within a certain time (ms)
  50%    407
  66%    440
  75%    461
  80%    475
  90%    512
  95%    543
  98%    592
  99%    620
 100%    726 (longest request)

结论:自行参考结果吧,要我说的话,以1w为限吧

什么?你没法在一个目录创建100w个文件?那是另外一个话题了,请自行google: linux inode

linux单目录文件不要过多,多少是过多?》上有 5 条评论

  1. 感觉不太准确。如果不用rand()函数,直接指定文件名的话,几乎没有差别。

  2. 就喜欢最后一句”自豪地采用 WordPress”,正好遇到个业务需求,服务器载了1Tb的SSD.根目录下有10个目录,每个都含有10万个子目录.其中内含1万个2Kb的’.json’文件.感觉过于水平.目前存放了实际100G的东西却占了十倍的空间.所以过于碎片还真不是什么好事.

发表评论

您的电子邮箱地址不会被公开。