Replies: 4 comments 17 replies
|
I'm not 100% sure this is your slow replication issue, but, the defaults have rebalancing queues set at a relatively low level compared to endangered. These rates are controlled (and well documented) in the mfsmaster.cfg file by the following configurations: I've found that the defaults are quite conservative, requiring a very long time to balance drives. In the perspective of petabyte storage arrays, conservative defaults totally make sense. I've increased these limits in my two smaller arrays. To find an acceptable level, I basically doubled the values until I saw the performance I wanted. |
|
Thanks for the reply! I have these parameters set (CHUNKS_WRITE_REP_LIMIT=32, CHUNKS_READ_REP_LIMIT=24), but as I understand it, they only affect the rebalancing speed between CSs, not between HDDs within a single CS. As I understand it, the master itself has no relation to internal rebalancing (despite the fact that rebalancing is active, there is practically no traffic between the CS and the master, as seen both in the graphs and via netdump). I also tried changing HDD_TEST_SPEED on the CS, but that had no effect either. |
|
I think I found the settings I needed. |
|
Yes, these settings are the only ones that control the speed of so-called "internal rebalance", that is, internal rebalance of chunks between disks in one chunkserver. This is a separate process, not in any way connected with the "cluster-wide rebalance", which is the process overseen by master, in which the system tries to even out the capacity of all chunkservers. There are 2 modes of internal rebalance:
High speed rebalance disables the regular rebalance while it's active. It's recommended only in situations like yours, when you put in a new, blank drive in a chunkserver with other drives filled with data. It's NOT recommended to keep it on at all times, especially in busy instances, as it sends regular "jobs" to the common pool of "worker threads", thus occupying resources normally available only for tasks sent over by the master. |
|
@chogata Thank you very much for the clarification! Am I understanding correctly that high-speed rebalancing "excludes" this node from all write operations? |
|
High speed rebalance doesn't exclude a chunkserver from i/o operations, but the chunk server is presented to the master as "busy", so the master tries not to burden it with i/o operations, unless necessary. So it may seem like it's "excluded", it's not, it's just "much less used" :) |
Got it, thanks for the clarification! |
|
BTW, there's probably an error in the statistics table. As you can see in the screenshot, the write and read data don't match. |
|
Hmm, I wonder if we display Mb/sec, incorrectly marked as MiB/sec... I will have it checked :) |
|
OK, I consulted extensively on this matter and now I can write more :) Short answer: everything is correct and those 2 views show different kind of information. Long answer: the charts show the average i/o speed that happened in a period of time, scaled to that period of time. So, if, during one minute 600 MiB of data was read, the average read speed for that minute will be 600MiB/60sec = 10MiB/sec. The difference between the 2 speeds: 10MiB/sec on the chart and 30MiB/sec in the disk tab is because in the chart, we show how "busy" the chunkserver is with data read operations. In disk tab, we show the actual i/o average speeds. Now, I wrote "as reported by kernel". This is an important distinction, because we don't really know how much time it actually takes to physically read or write data to disk. We know how much time the system call "read" or "write" took to complete. But if you have caches, both in kernel and in your (very smart) disk controller, then the "read" or "write" time may be much shorter than actual reading or writing would take. Especially writing, because nowadays the data is usually stored in some kind of cache and the physical write operation is done in the background, with a slight delay. The more busy your instance is and the more often a chunkserver must write data to disks, the less of a difference there will be between the speed on the chart and the speed in the disks tab, because at some point any cache, be it kernel or controller, will be exhausted and physical write operations must be completed for new operations to take place. So, in your particular case it looks like the recording of data is done with the average speed of 80MiB/sec, but the "write" operations are done in short bursts, so that the kernel reports them done very fast and the i/o speed calculation reports 800MiB/sec, due to cache utilisation. This probably means that you still have some "breathing space" and could up your replication limits a bit more. |
|
Thanks for the detailed answer! So, in essence, on the "Disks" tab, the write speed is roughly equal to the write speed to the cache (dirty pages, HDD cache)? If not, then that's odd. Even with all the caches (let's say 50GB dirty pages + HDD cache), such high speeds should be exhausted in a few minutes. This would be logical for a single write, e.g. VM image, but it rebalancing for several days now, and data is mostly written to this drive non-stop. As for limits, I currently have HDD_HIGH_SPEED_REBALANCE_LIMIT=10 (why is this maximum hardcoded?), and with this value, my iowait is ~40% and I have almost 100% utilization of all the drives MFS reading data from. No need to overload it any further :) |
|
Basically, yes. That is, I cannot answer with 100% certainty what your os/kernel does, when chunkserver performs a "write" call, but my educated guess would be it goes to some kind of cache, considering the speed displayed in your GUI. Again - we use posix "write" function, and what it does, exactly, with the data passed to it, is the kernel's decision. It should be written to disk, but any use of kernel cache, controller cache etc. is not controlled by us. |
|
Okay, thank you very much! |
|
I think that his answer, like the entire discussion, will be useful to people who face the same misunderstanding of what is happening. |
|
Actually – her answer :) |
|
Oops, sorry for the typo :) |
Uh oh!
There was an error while loading. Please reload this page.
Hello everyone!
I have an MFS cluster with 3 nodes, each running a CS on top of 8 HDDs of 4TB. I also have 2 chunk servers on SSDs, but that’s not related to the issue.
~85 TB of raw space, ~60 million objects (2CP replication). Interconnect via IPoIB.
Yesterday, I replaced one failed 4TB drive on the third node with a 22TB HDD (eventually all drives will be replaced).
What I want:
What I get:
At the second stage, I changed ACCEPTABLE_PERCENTAGE_DIFFERENCE to 10%, because the third node now had 43 TB instead of 28 TB, and its 4TB disks were filling up, while the new disk was practically unused. After the rebalancing between CSs was completed, rebalancing within the CS began, but for some reason it is proceeding too slowly.
After almost two days, the new disk is less than 1% full.
The graphs show all the stages.
First, write operations appeared on the first and second nodes -- this was replication of missing chunks.
Then, at ~02:00, writes on the first and second nodes disappeared, but increased on the third -- this was rebalancing between CS.
Then, at ~21:00, writes on the third node decreased, and rebalancing within the third node began.
As you can see, during all three stages, st3:/data/sdg/ was not used as intensively as the other disks. It is also clear that if I had left ACCEPTABLE_PERCENTAGE_DIFFERENCE at the default 1%, then most likely, upon reaching full balancing between CSs, on the third node, all disks except the new one would have run out of space.
All reactions