2

My company has a few servers that we dedicate to running Sphinx Search. All are CentOS 5 with about 48GB of memory and our version of searchd is 1.11. Recently I've been trying to understand why Sphinx uses so little of the available memory for its regular indexes (we don't use RT indexes at the moment). According to the official Sphinx website, all files related to an index except for the .spd and .spp files are stored in RAM. At present this constitutes about 14GG for all our indexes, which can easily fit in RAM. However, when I consult htop for memory usage, it's showing a little less than 1.5GB being used! Interestingly enough, it reports that swap is being used as well, even with vm.swappiness set to 0. If I stop searchd, swap goes down to 0. (It should be noted that the amount of swap used is no greater that the amount of memory being used.) We have it configured so that all our indexes are preopened on startup (preopen_indexes = 1 in our sphinx conf).

I have two questions, given the above information:

  1. What is Sphinx doing with the data in those indexes, if it isn't keeping it in memory?
  2. Is it possible to force Sphinx to keep more data in memory?
altendo
  • 21

2 Answers2

1

try mlock = 1 and reindex all indexes.

index index1
{
    source      = main1
    path        = /ssd/sphinx/index.main
    charset_type    = utf-8
    docinfo     = extern
    mlock       = 1
}
jim
  • 11
0

What is Sphinx doing with the data in those indexes, if it isn't keeping it in memory?

It probably does put them into memory. But the host OS, notices they are not being used much, and so swaps them out.

Would suggest looking at making sure your swappiness is really being applied.

Is it possible to force Sphinx to keep more data in memory?

If you have the memory to spare. Put the indexes on a RAM-Disk :)

And if you really have plenty of memory to spare, turn OFF swap altogether instead. You risk a crash if run out of memory, but it sounds unlikly to happen on that server.

barryhunter
  • 101