| Time |
Nick |
Message |
| 00:21 |
|
Ark joined #gluster |
| 00:28 |
|
georgeh|workstat joined #gluster |
| 00:29 |
|
dtrainor joined #gluster |
| 01:08 |
glusterbot |
New news from resolvedglusterbugs: [Bug 987555] Glusterfs ports conflict with qemu live migration <https://bugzilla.redhat.com/show_bug.cgi?id=987555> |
| 01:11 |
|
georgeh|workstat joined #gluster |
| 01:13 |
|
siel joined #gluster |
| 01:33 |
|
georgeh|workstat joined #gluster |
| 01:38 |
glusterbot |
New news from resolvedglusterbugs: [Bug 764655] NetBSD port <https://bugzilla.redhat.com/show_bug.cgi?id=764655> |
| 02:08 |
|
harish joined #gluster |
| 02:10 |
|
RioS2 joined #gluster |
| 02:14 |
|
theron joined #gluster |
| 02:30 |
|
gmcwhistler joined #gluster |
| 02:33 |
|
baojg joined #gluster |
| 02:36 |
|
rameez joined #gluster |
| 02:37 |
rameez |
hello team |
| 02:37 |
rameez |
requesting your help on a geo replication issue. |
| 02:37 |
rameez |
i have two datacenters where the geo-replication is implemented. |
| 02:38 |
rameez |
from A to B it is working as expected. |
| 02:38 |
rameez |
but from B to A to (different set of volumes), it is spawning additional seessions. |
| 02:39 |
rameez |
there are required Active sessions, But equal number of sessions goes in to defunct status. |
| 02:40 |
|
Bullardo joined #gluster |
| 02:40 |
|
theron joined #gluster |
| 02:53 |
|
churnd joined #gluster |
| 02:55 |
|
bharata-rao joined #gluster |
| 02:56 |
|
haomaiw__ joined #gluster |
| 02:56 |
|
baojg joined #gluster |
| 03:07 |
|
sjm joined #gluster |
| 03:14 |
|
vpshastry joined #gluster |
| 03:15 |
|
theron_ joined #gluster |
| 03:27 |
|
rejy joined #gluster |
| 03:29 |
|
kdhananjay joined #gluster |
| 03:30 |
|
gmcwhistler joined #gluster |
| 03:31 |
|
RioS2 joined #gluster |
| 03:47 |
|
shubhendu__ joined #gluster |
| 03:49 |
|
atinm joined #gluster |
| 03:55 |
|
sauce joined #gluster |
| 03:57 |
|
kanagaraj joined #gluster |
| 03:57 |
|
prasanthp joined #gluster |
| 03:58 |
|
davinder15 joined #gluster |
| 04:05 |
|
gmcwhist_ joined #gluster |
| 04:13 |
|
itisravi joined #gluster |
| 04:15 |
|
Pupeno joined #gluster |
| 04:15 |
|
gildub joined #gluster |
| 04:16 |
|
nishanth joined #gluster |
| 04:22 |
|
baojg joined #gluster |
| 04:22 |
|
ndarshan joined #gluster |
| 04:27 |
|
RameshN joined #gluster |
| 04:31 |
|
spandit joined #gluster |
| 04:36 |
|
bala joined #gluster |
| 04:38 |
|
vpshastry joined #gluster |
| 04:40 |
|
ppai joined #gluster |
| 04:46 |
|
rjoseph joined #gluster |
| 04:48 |
|
vimal joined #gluster |
| 04:56 |
|
lalatenduM joined #gluster |
| 05:01 |
|
kshlm joined #gluster |
| 05:02 |
|
baojg joined #gluster |
| 05:03 |
|
Pupeno joined #gluster |
| 05:04 |
|
nbalachandran joined #gluster |
| 05:04 |
|
baojg joined #gluster |
| 05:14 |
|
hagarth joined #gluster |
| 05:16 |
|
psharma joined #gluster |
| 05:18 |
|
aravindavk joined #gluster |
| 05:23 |
|
kdhananjay joined #gluster |
| 05:24 |
|
ppai joined #gluster |
| 05:28 |
|
saurabh joined #gluster |
| 05:32 |
|
Ark joined #gluster |
| 05:45 |
|
Pupeno joined #gluster |
| 05:49 |
|
aravindavk joined #gluster |
| 05:51 |
|
bala1 joined #gluster |
| 05:52 |
|
vkoppad joined #gluster |
| 05:55 |
|
ws2k33 joined #gluster |
| 05:55 |
|
hagarth joined #gluster |
| 05:56 |
|
Pupeno joined #gluster |
| 05:58 |
|
nhayashi joined #gluster |
| 05:59 |
|
prasanthp joined #gluster |
| 05:59 |
|
Philambdo joined #gluster |
| 06:00 |
|
mbukatov joined #gluster |
| 06:03 |
|
Pupeno joined #gluster |
| 06:10 |
|
Pupeno joined #gluster |
| 06:11 |
|
raghu` joined #gluster |
| 06:19 |
|
pureflex joined #gluster |
| 06:27 |
|
Pupeno joined #gluster |
| 06:30 |
|
_polto_ joined #gluster |
| 06:33 |
|
karnan joined #gluster |
| 06:33 |
|
firemanxbr joined #gluster |
| 06:35 |
|
rastar joined #gluster |
| 06:35 |
|
ctria joined #gluster |
| 06:38 |
|
shylesh__ joined #gluster |
| 06:42 |
|
psharma_ joined #gluster |
| 06:45 |
|
psharma__ joined #gluster |
| 06:52 |
|
psharma_ joined #gluster |
| 06:53 |
|
pureflex joined #gluster |
| 06:57 |
|
aravindavk joined #gluster |
| 06:58 |
|
hagarth joined #gluster |
| 07:03 |
|
prasanthp joined #gluster |
| 07:03 |
|
overclk joined #gluster |
| 07:04 |
|
hflai joined #gluster |
| 07:04 |
|
eseyman joined #gluster |
| 07:04 |
|
ndarshan joined #gluster |
| 07:07 |
|
vpshastry joined #gluster |
| 07:08 |
|
kevein joined #gluster |
| 07:12 |
|
ppai joined #gluster |
| 07:17 |
|
keytab joined #gluster |
| 07:18 |
|
hflai joined #gluster |
| 07:20 |
|
hagarth joined #gluster |
| 07:25 |
|
Swaroop joined #gluster |
| 07:26 |
Swaroop |
Hello All, I need some help in setting up glusterfs with peer replication |
| 07:26 |
Swaroop |
I would like to have the peers replicate each other instead of clients replicating data to peers |
| 07:26 |
Swaroop |
Had anyone has this kind of setup? |
| 07:29 |
|
ktosiek joined #gluster |
| 07:32 |
|
fsimonce joined #gluster |
| 07:37 |
glusterbot |
New news from newglusterbugs: [Bug 957917] gluster create volume doesn't cleanup after its self if the create fails. <https://bugzilla.redhat.com/show_bug.cgi?id=957917> |
| 07:47 |
|
ppai joined #gluster |
| 07:49 |
|
baojg joined #gluster |
| 07:49 |
|
hflai joined #gluster |
| 07:49 |
|
baojg joined #gluster |
| 07:54 |
|
vpshastry joined #gluster |
| 07:58 |
|
vpshastry joined #gluster |
| 08:00 |
|
spandit joined #gluster |
| 08:04 |
|
wgao joined #gluster |
| 08:04 |
pasqd |
hello, how can i check if the heal process is done? without checking each brick's size? |
| 08:05 |
|
pureflex joined #gluster |
| 08:05 |
|
_polto_ joined #gluster |
| 08:06 |
|
ricky-ti1 joined #gluster |
| 08:07 |
glusterbot |
New news from newglusterbugs: [Bug 1112980] NFS subdir authentication doesn't correctly handle multi-(homed,protocol,etc) network addresses <https://bugzilla.redhat.com/show_bug.cgi?id=1112980> |
| 08:14 |
|
baojg joined #gluster |
| 08:19 |
|
Sunghost joined #gluster |
| 08:21 |
|
pureflex joined #gluster |
| 08:23 |
Sunghost |
Hello, i have a question for debian wheezy <-> jessie and glusterfs. I actually run wheezy but i need the last xfsprogs which are only since jessie in repo. |
| 08:25 |
Sunghost |
I installed gluster via own source list file, like it is described in the howto on the downloadfolder of gluster for debian. i have installed version 3.5.1beta2 |
| 08:26 |
Sunghost |
My question is can i upgrade from wheezy to jessie without problems or have i change the source file? |
| 08:27 |
Sunghost |
Version in Jessie is 3.5.0-1 for amd64. When will latest 3.5.1 available? |
| 08:27 |
fraggeln |
Sunghost: when its done ;) |
| 08:27 |
fraggeln |
best anwser ever.. :) |
| 08:28 |
Sunghost |
Yes quiet good answer ;) |
| 08:30 |
Sunghost |
My question was more on timing of both packages - so came the package on the gluster downloadserver much earlier as on debian repo or nearly at the same time. |
| 08:31 |
Sunghost |
So if it came at the same time, i could use the debian repos, but if the time between the releases differntiate much, than i whould use the extra source file. |
| 08:34 |
|
dusmant joined #gluster |
| 08:34 |
|
hagarth joined #gluster |
| 08:35 |
|
meghanam joined #gluster |
| 08:37 |
|
Slashman joined #gluster |
| 08:37 |
|
meghanam_ joined #gluster |
| 08:48 |
Sunghost |
Any infos to this timing of debian repo and upload on glusterfs fileserver? |
| 08:52 |
|
ndarshan joined #gluster |
| 08:55 |
|
overclk joined #gluster |
| 08:59 |
|
liquidat joined #gluster |
| 09:01 |
|
overclk joined #gluster |
| 09:02 |
|
nishanth joined #gluster |
| 09:03 |
|
spandit joined #gluster |
| 09:03 |
|
karnan joined #gluster |
| 09:07 |
|
overclk joined #gluster |
| 09:15 |
|
tomased joined #gluster |
| 09:17 |
|
ndarshan joined #gluster |
| 09:22 |
|
nishanth joined #gluster |
| 09:25 |
|
dusmant joined #gluster |
| 09:26 |
|
nthomas joined #gluster |
| 09:33 |
|
davinder15 joined #gluster |
| 09:41 |
|
nthomas_ joined #gluster |
| 09:48 |
|
Ark joined #gluster |
| 09:53 |
|
nishanth joined #gluster |
| 09:57 |
|
swebb joined #gluster |
| 10:10 |
|
hagarth joined #gluster |
| 10:14 |
|
gildub joined #gluster |
| 10:20 |
|
aravindavk joined #gluster |
| 10:22 |
|
pureflex joined #gluster |
| 10:27 |
|
_polto_ joined #gluster |
| 10:43 |
|
Nightshader2 joined #gluster |
| 10:54 |
|
tdasilva joined #gluster |
| 10:55 |
|
RameshN joined #gluster |
| 11:05 |
|
tdasilva_ joined #gluster |
| 11:07 |
|
coreping joined #gluster |
| 11:09 |
|
77CAAJHTE joined #gluster |
| 11:09 |
|
6JTAAI72K joined #gluster |
| 11:10 |
|
edward1 joined #gluster |
| 11:11 |
|
baojg joined #gluster |
| 11:12 |
|
baojg joined #gluster |
| 11:17 |
|
ppai joined #gluster |
| 11:20 |
|
nshaikh joined #gluster |
| 11:22 |
|
aravindavk joined #gluster |
| 11:25 |
|
LebedevRI joined #gluster |
| 11:27 |
|
keds joined #gluster |
| 11:31 |
partner |
that topic was slightly discussed last night |
| 11:31 |
partner |
as a user i of course assume all the releases to be built, otherwise i'm not sure what's the point of building anything |
| 11:32 |
partner |
but i also know how the stuff works, volunteers doing the packaging and stuff so can't ask too much from them either |
| 11:38 |
glusterbot |
New news from newglusterbugs: [Bug 1002907] changelog binary parser not working <https://bugzilla.redhat.com/show_bug.cgi?id=1002907> || [Bug 1113066] DHT: Log new layout of directory generated during directory self healing <https://bugzilla.redhat.com/show_bug.cgi?id=1113066> |
| 11:41 |
|
simulx joined #gluster |
| 11:43 |
fraggeln |
partner: the buildserver can build the packages as well? ;) |
| 11:43 |
fraggeln |
I know both bamboo and jenkins has good support for building both .rpm's and .deb's |
| 11:44 |
partner |
i would assume the project does not have such but individuals build packages wherever |
| 11:46 |
|
diegows joined #gluster |
| 11:52 |
kkeithley |
Our jenkins in running on CentOS 6. It'll build rpms in mock for el5 and el6. It has problems building Fedora and el7 rpms. I haven't seen any sign that it'll build .debs, but I may not have looked in the right place. Or maybe it'll do that if it's running on debian or ubuntu. |
| 11:53 |
kkeithley |
That aside, we build gluster for Fedora and epel in the Fedora koji build system. That works pretty good, and the Fedora rpms go straight into Fedora. |
| 11:55 |
kkeithley |
We'd like to build .debs for Debian and Ubuntu ourselves, and I believe we're getting closer to doing that. Until then we're still relying on volunteers who do that on their own time. |
| 11:56 |
fraggeln |
kkeithley: I think you need a jenkins-slave running debian to build packages, but im not sure. |
| 11:57 |
kkeithley |
Our jenkins is pretty outdated too. Once we get it updated to the latest-and-greatest it may do some of those things, or do them better. |
| 11:59 |
fraggeln |
just wget the war-file and let tomcats autodeploy do the rest ;) |
| 11:59 |
fraggeln |
jenkins is a pain in the ass some times, I have been running one for almost 2 years now, every now and then something breaks. |
| 11:59 |
fraggeln |
jobs not showing up and stuff like that |
| 12:00 |
fraggeln |
oh, you are running 1.515 |
| 12:00 |
fraggeln |
yea, it could use some upgrading. |
| 12:04 |
kkeithley |
yep, it's in the works |
| 12:05 |
fraggeln |
btw, green balls ftw ;) |
| 12:05 |
fraggeln |
thats the default plugin you always install ;) |
| 12:08 |
|
tdasilva joined #gluster |
| 12:11 |
|
calum_ joined #gluster |
| 12:12 |
fraggeln |
kkeithley: how much volontary work is done in relation to the work that redhat employees do? |
| 12:17 |
kkeithley |
hard to say. |
| 12:17 |
kkeithley |
because I don't know, and it's difficult to track |
| 12:18 |
|
itisravi joined #gluster |
| 12:20 |
fraggeln |
anyway, I like the product, its really nice, keep up the good work. |
| 12:20 |
kkeithley |
thanks |
| 12:21 |
fraggeln |
we get awesome performance on large files, now we just need to optimize it for smaller files. |
| 12:21 |
kkeithley |
yeah, we hear that a lot. ;-) |
| 12:22 |
fraggeln |
but if we manage to get good enough performance we will switch to gluster for our main website. |
| 12:22 |
|
pureflex joined #gluster |
| 12:23 |
kkeithley |
are you using PHP by any chance? |
| 12:23 |
fraggeln |
and imho, that is really cool. |
| 12:23 |
|
Nightshader joined #gluster |
| 12:23 |
fraggeln |
well, we use php for some sites, but the main page runs escenic |
| 12:23 |
kkeithley |
okay |
| 12:23 |
|
deepakcs joined #gluster |
| 12:24 |
fraggeln |
but, the webfronts will have local ssd-disks for scaled images, so it will not be that much load on the gluster I hope |
| 12:27 |
|
pureflex joined #gluster |
| 12:31 |
fraggeln |
kkeithley: is it possible to fix the "issues" with performance when dealing with small files, or is it impossible dueto the design? |
| 12:35 |
fraggeln |
or is the solution to switch to infiniband between the servers? :) |
| 12:35 |
|
kanagaraj joined #gluster |
| 12:44 |
|
velladecin joined #gluster |
| 12:46 |
|
coredump joined #gluster |
| 12:48 |
|
sputnik13 joined #gluster |
| 12:49 |
|
sputnik13 joined #gluster |
| 12:49 |
|
aravindavk joined #gluster |
| 12:51 |
|
sroy joined #gluster |
| 12:53 |
kkeithley |
IIRC, jdarcy got a big increase in small file performance by only sending lookups to one server instead of all of them. Maybe we'll see that in 3.6. |
| 12:55 |
|
deeville joined #gluster |
| 12:55 |
fraggeln |
last question for the day, is it possible to use RoCE with gluster? |
| 12:56 |
fraggeln |
regarding lookups, is that the client that does that or the servers itself? (in version 3.5) |
| 12:57 |
|
theron joined #gluster |
| 12:58 |
|
theron joined #gluster |
| 12:59 |
|
bennyturns joined #gluster |
| 12:59 |
kkeithley |
clients do lookups, generally |
| 13:00 |
kkeithley |
gluster will use IBoIB. I haven't used RoCE myself, so I'm not sure. I'd think if RoCE looks like an ethernet/TCP-IP then GlusterFS would "just work", but can't say that with any authority |
| 13:00 |
|
chirino joined #gluster |
| 13:01 |
|
japuzzo joined #gluster |
| 13:01 |
kkeithley |
IPoIB |
| 13:03 |
|
lpabon joined #gluster |
| 13:03 |
|
julim joined #gluster |
| 13:07 |
fraggeln |
kkeithley: thank you very much for your time, have a very good day/morning/night o what best suits your timezone :) |
| 13:08 |
glusterbot |
New news from newglusterbugs: [Bug 1101111] [RFE] Add regression tests for the component geo-replication <https://bugzilla.redhat.com/show_bug.cgi?id=1101111> |
| 13:09 |
tty00 |
btw, have anyone of you tried to decrease the NIC queue/buffers so the NIC is flushing packets to the wire faster? |
| 13:19 |
|
Ark joined #gluster |
| 13:28 |
|
mjsmith2 joined #gluster |
| 13:31 |
|
baojg joined #gluster |
| 13:32 |
|
primechuck joined #gluster |
| 13:44 |
|
fsimonce` joined #gluster |
| 13:44 |
|
gmcwhistler joined #gluster |
| 13:45 |
|
burnalot joined #gluster |
| 13:52 |
|
nshaikh joined #gluster |
| 13:53 |
|
fsimonce joined #gluster |
| 13:53 |
|
Kedsta joined #gluster |
| 14:02 |
|
marbu joined #gluster |
| 14:03 |
|
theron joined #gluster |
| 14:09 |
|
mortuar joined #gluster |
| 14:10 |
|
jag3773 joined #gluster |
| 14:16 |
|
hagarth joined #gluster |
| 14:17 |
|
sjm joined #gluster |
| 14:18 |
|
wushudoin joined #gluster |
| 14:20 |
jezier_ |
hi.. we have couple of messages in logs like [server-rpc-fops.c:1739:server_stat_cbk] 0-storage-server: 2415179: STAT (null) (08521fc9-1824-415c-9972-eae65ac19c43) ==> (No such file or directory)... it looks like race condition.. file creation happens on one node and open/read on another node... when I verify after a couple of minutes, this file exists... how can we fix this? (ubuntu 12.04, glusterfs3.4.3) |
| 14:21 |
|
hybrid512 joined #gluster |
| 14:24 |
|
mjsmith2 joined #gluster |
| 14:27 |
|
ultrabizweb joined #gluster |
| 14:30 |
|
doc|holliday joined #gluster |
| 14:31 |
doc|holliday |
is it safe to connect to cluster running 3.4.2 with clients running 3.4.4? |
| 14:32 |
doc|holliday |
my clients (mount via fuse) keep leaking memory with 3.4.2 and echo 2 > .../drop_caches didn't help |
| 14:33 |
doc|holliday |
want to try 3.4.4 |
| 14:34 |
|
lmickh joined #gluster |
| 14:36 |
|
ndk joined #gluster |
| 14:40 |
|
_polto_ joined #gluster |
| 14:48 |
|
Guest41106 joined #gluster |
| 14:49 |
Guest41106 |
Hi guys, I'm getting memory leak on glusterfs 3.5.0beta3 built on Feb 11 2014 20:45:24 |
| 14:49 |
Guest41106 |
and the kernel kills the process after it eats 7GB |
| 14:50 |
|
fsimonce joined #gluster |
| 14:52 |
Guest41106 |
quit |
| 14:52 |
Guest41106 |
|
| 14:58 |
|
harish joined #gluster |
| 15:02 |
|
kdhananjay joined #gluster |
| 15:04 |
|
Ark joined #gluster |
| 15:04 |
|
zaitcev joined #gluster |
| 15:05 |
|
jdarcy joined #gluster |
| 15:06 |
|
ndk` joined #gluster |
| 15:10 |
|
mbukatov joined #gluster |
| 15:11 |
|
gts joined #gluster |
| 15:11 |
JustinClift |
*** Gluster Community Meeting time is now in #gluster-meeting on irc.freenode.net *** |
| 15:16 |
|
lpabon joined #gluster |
| 15:28 |
|
sage__ joined #gluster |
| 15:28 |
|
_polto_ joined #gluster |
| 15:30 |
|
dberry joined #gluster |
| 15:30 |
|
dberry joined #gluster |
| 15:39 |
|
shubhendu__ joined #gluster |
| 16:04 |
|
lalatenduM joined #gluster |
| 16:05 |
|
rwheeler joined #gluster |
| 16:25 |
|
mjsmith2 joined #gluster |
| 16:25 |
|
Mo_ joined #gluster |
| 16:27 |
|
fsimonce joined #gluster |
| 16:32 |
|
bala joined #gluster |
| 16:34 |
|
jobewan joined #gluster |
| 16:34 |
|
wushudoin joined #gluster |
| 16:42 |
|
lmickh joined #gluster |
| 16:43 |
|
sjm joined #gluster |
| 16:43 |
|
zerick joined #gluster |
| 16:46 |
|
coredump joined #gluster |
| 16:47 |
|
sputnik13 joined #gluster |
| 16:48 |
|
jag3773 joined #gluster |
| 17:09 |
|
rwheeler joined #gluster |
| 17:11 |
|
dusmant joined #gluster |
| 17:15 |
|
dusmant joined #gluster |
| 17:18 |
|
Matthaeus joined #gluster |
| 17:19 |
|
daMaestro joined #gluster |
| 17:22 |
|
sjm joined #gluster |
| 17:31 |
|
lmickh joined #gluster |
| 17:34 |
|
coredump joined #gluster |
| 17:37 |
|
qdk joined #gluster |
| 17:51 |
|
sjm joined #gluster |
| 17:52 |
|
MacWinner joined #gluster |
| 17:58 |
|
davinder15 joined #gluster |
| 17:59 |
|
vpshastry joined #gluster |
| 18:03 |
|
ramteid joined #gluster |
| 18:12 |
|
_dist joined #gluster |
| 18:16 |
|
mjrosenb joined #gluster |
| 18:21 |
|
sjm joined #gluster |
| 18:51 |
|
sjm joined #gluster |
| 18:53 |
|
lmickh joined #gluster |
| 18:57 |
|
dtrainor joined #gluster |
| 18:59 |
|
dtrainor joined #gluster |
| 19:02 |
n0de |
In Gluster 3.2.4, was there no .glusterfs/ directoy ? |
| 19:06 |
semiosis |
sounds right |
| 19:06 |
semiosis |
simpler times |
| 19:10 |
|
rotbeard joined #gluster |
| 19:12 |
|
sjm joined #gluster |
| 19:20 |
|
calum_ joined #gluster |
| 19:20 |
n0de |
Hi semiosis, this is the gluster in which I have a failed raid6 on for one of my bricks |
| 19:21 |
semiosis |
do you have a question? |
| 19:22 |
n0de |
yes, so the brick in question consists of software 4 x raid6 in a raid0 |
| 19:23 |
n0de |
one of those raid6s has failed and I will need to wipe it |
| 19:23 |
n0de |
everything is mounted still and works, just the files on that failed raid6 will spit out i/o errors |
| 19:23 |
n0de |
prior to wiping the failed raid6, I want to make sure I have the correct plan in place in order to avoid loosing any data. |
| 19:25 |
n0de |
Volume type is Distributed-Replicate, it feels like if I simply fix the broken raid6, gluster will heal the files on it's own? |
| 19:26 |
|
Paul-C joined #gluster |
| 19:28 |
|
lpabon_ joined #gluster |
| 19:36 |
|
bene2 joined #gluster |
| 19:41 |
|
dtrainor joined #gluster |
| 19:41 |
fraggeln |
is there any good way to optimize the clients? like sysctl-stuff or tcp-offloading stuff? |
| 19:43 |
|
sjm joined #gluster |
| 19:44 |
|
theron joined #gluster |
| 19:50 |
|
tdasilva joined #gluster |
| 19:51 |
semiosis |
fraggeln: is that really your bottleneck? |
| 19:52 |
semiosis |
n0de: well what i would do is: kill the brick daemon for the badd brick. firewall the port for the bad brick so nothing can connnect to it. replace the drive/raid/whatever with a new one, formatted & mounted to the same place. restart glusterd to respwn the brick daemon. unfirewall the port |
| 19:53 |
semiosis |
n0de: now you'll have an empty brick, and as files are accessed they will be healed on demand, in the background. |
| 19:53 |
semiosis |
since this is pre-3.3 you'll need to stat each file to make it heal. i recommend using the ,,(targeted self heal) strategy which I wrote just for this situation (though it's not needed since 3.3) |
| 19:53 |
glusterbot |
https://web.archive.org/web/20130314122636/http://community.gluster.org/a/howto-targeted-self-heal-repairing-less-than-the-whole-volume/ |
| 19:54 |
|
dtrainor joined #gluster |
| 19:54 |
semiosis |
summary of that is: use find on the good brick to stat files through a local mount |
| 19:54 |
semiosis |
see article for details |
| 19:54 |
semiosis |
s/local mount/local client mount/ |
| 19:54 |
glusterbot |
What semiosis meant to say was: summary of that is: use find on the good brick to stat files through a local client mount |
| 19:55 |
n0de |
semiosis: I will do that, thank you for your help |
| 19:55 |
semiosis |
yw |
| 19:56 |
n0de |
Can I skip the firewalling step if I just stop gluster, fix raid, and then restart it? |
| 19:56 |
n0de |
it == gluster |
| 19:56 |
semiosis |
you need to stop the volume, gluster volume stop $volname |
| 19:56 |
semiosis |
if you want to do that |
| 19:56 |
semiosis |
it will stop all brick daemons on all servers |
| 19:56 |
semiosis |
cutting off clients |
| 19:56 |
n0de |
I rather not stop the volume |
| 19:56 |
semiosis |
right |
| 19:57 |
semiosis |
so the other way is to kill the brick daemon process & firewall the port (in case it gets restarted prematurely) |
| 19:59 |
n0de |
got it |
| 19:59 |
|
dtrainor joined #gluster |
| 19:59 |
n0de |
looks like its port 9001 for glusterfs and port 8036 for glusterd |
| 20:02 |
semiosis |
that doesnt sound right |
| 20:03 |
semiosis |
should be something around 24009 or a bit above |
| 20:03 |
semiosis |
,,(ports) |
| 20:03 |
glusterbot |
glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4. |
| 20:04 |
n0de |
hrm, that is not what I am seeing |
| 20:05 |
n0de |
on my gluster 3.4.2 setup, I see glusterd is on 28166 and glusterfs is on various 282xx ports |
| 20:05 |
|
chirino joined #gluster |
| 20:05 |
semiosis |
possible you're looking at process ids? |
| 20:06 |
semiosis |
please pastie.org the output of 'sudo netstat -anp | grep gluster' |
| 20:06 |
n0de |
crap, I am my apologies |
| 20:06 |
semiosis |
lol |
| 20:06 |
n0de |
did that thing again of being human :) |
| 20:07 |
n0de |
glusterd == 24007 and glusterfs == 38465, 38466,38467 |
| 20:08 |
semiosis |
see ,,(processes) |
| 20:08 |
glusterbot |
The GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). |
| 20:08 |
semiosis |
you want to kill the 'glusterfsd' (brick daemon) for the bad brick, and firewall it's port, which will be 24009 or higher |
| 20:09 |
semiosis |
when it's running its command line will clearly identify the brick path & the port number, use 'ps axwww | grep glusterfsd' |
| 20:10 |
semiosis |
to see the full command line for all the brick daemons |
| 20:14 |
|
sjm joined #gluster |
| 20:15 |
n0de |
# ps axwww | grep glusterfsd |
| 20:15 |
n0de |
10892 pts/0 S+ 0:00 grep --colour=auto glusterfsd |
| 20:15 |
n0de |
got nothing it seems |
| 20:16 |
n0de |
I see glusterfsd running on other bricks though |
| 20:16 |
n0de |
the brick with the failed raid6 does not have glusterfsd running |
| 20:18 |
|
bennyturns joined #gluster |
| 20:19 |
n0de |
I assume the firewalling is done so that any new writes will be recorded as not successful on the brick I am working on? |
| 20:20 |
semiosis |
without a brick daemon there's nothing to write to. the firewall is a safeguard in case that brick daemon get started before you're ready for it, for example if you restart the system |
| 20:23 |
|
sroy joined #gluster |
| 20:23 |
fraggeln |
semiosis: yes, I think it is, since we have a shitload of small files :) |
| 20:24 |
semiosis |
fraggeln: i'm still not convinced |
| 20:24 |
fraggeln |
I am. |
| 20:24 |
|
dtrainor joined #gluster |
| 20:24 |
fraggeln |
3 servers, gbit network, if I copy 1 4gb file, I get wirespeed. |
| 20:25 |
semiosis |
whats the latency from client to the servers |
| 20:25 |
semiosis |
? |
| 20:25 |
semiosis |
average rtt? |
| 20:25 |
fraggeln |
they are in the same switch, so I guess its pretty low, cant check atm, im not connected to the vpn |
| 20:25 |
_dist |
JoeJulian: when you're around just wondering about that vm healing issue I think you said it was on your list to look into :) |
| 20:26 |
semiosis |
fraggeln: ok same switch should be <1ms |
| 20:27 |
fraggeln |
3 days of testing. |
| 20:27 |
fraggeln |
:) |
| 20:27 |
semiosis |
that's still likely to be the bottleneck though, context switches are ~1us |
| 20:28 |
semiosis |
roughly speaking |
| 20:29 |
fraggeln |
the problem is the workload, its a shitload of small files |
| 20:29 |
semiosis |
how small? |
| 20:29 |
fraggeln |
between 10k and 5mb |
| 20:29 |
fraggeln |
around 90.000 of them ;) |
| 20:29 |
JoeJulian |
_dist: Tried to duplicate it again yesterday... |
| 20:31 |
fraggeln |
semiosis: we are a newssite, so we have a shitload of pictures ;) |
| 20:31 |
semiosis |
JoeJulian: got any quick small-file optimization tips for fraggeln? |
| 20:31 |
semiosis |
istr something about a scheculer recommendation? |
| 20:31 |
fraggeln |
semiosis: we talked yesterday about it |
| 20:31 |
semiosis |
we did? |
| 20:32 |
fraggeln |
naa, me and JoeJulian |
| 20:32 |
* semiosis |
goes to logs |
| 20:32 |
semiosis |
ohhh |
| 20:32 |
semiosis |
ok, still want to read that log tho |
| 20:32 |
|
zaitcev left #gluster |
| 20:32 |
fraggeln |
I did get a big performance-boost wen adding more bricks to the volume. |
| 20:32 |
fraggeln |
when.. |
| 20:32 |
fraggeln |
version is 3.5, on debian wheezy. |
| 20:33 |
semiosis |
i'll have 3.5.1 for wheezy available tonight |
| 20:33 |
fraggeln |
oh! |
| 20:33 |
semiosis |
dont ask whats new in 3.5.1, i dont know :) |
| 20:33 |
fraggeln |
ill read the changelog later on, no worries. |
| 20:34 |
semiosis |
let me know! |
| 20:34 |
fraggeln |
Im gonna go and fetch my laptop, so I can hook the vpn up, and get the current config for the volume. |
| 20:34 |
fraggeln |
brb |
| 20:38 |
|
primechuck joined #gluster |
| 20:38 |
|
primechuck joined #gluster |
| 20:39 |
fraggeln |
semiosis: this is my config for the volume atm, http://fpaste.org/113242/72876214/ |
| 20:39 |
glusterbot |
Title: #113242 Fedora Project Pastebin (at fpaste.org) |
| 20:40 |
fraggeln |
servers 1 and 3 have local disks, 6 sas-disks in raid10 |
| 20:40 |
fraggeln |
server 2 has a 8gbit FC-lun attached. |
| 20:40 |
semiosis |
wow lots of options reconfigured |
| 20:40 |
semiosis |
do those do anything for you? |
| 20:40 |
fraggeln |
some dell compellent-thingy |
| 20:40 |
fraggeln |
semiosis: like I said, 3 days of testing :) |
| 20:41 |
semiosis |
neat |
| 20:41 |
fraggeln |
I have tried a shitload of different options |
| 20:41 |
fraggeln |
boxes themselvs are dual quadcores with 32gb ram |
| 20:41 |
fraggeln |
so, it should not be HW-related. |
| 20:46 |
SpeeR |
I'm having some problems cleaning up a split-brain problem, I have this file on 2 brick, I've remove the file, and the link-file, then stat the file. |
| 20:46 |
SpeeR |
on my next heal info, it's back in the list |
| 20:48 |
n0de |
semiosis: I will not be writing to the volume while the raid is getting fixed, nor will any of my clients. |
| 20:48 |
n0de |
But will firewall regardless as a precaution |
| 20:49 |
JoeJulian |
SpeeR: is it "back" in the list, or is it just "still" in the list? The list is a log with timestamps. As long at it's not newly in the list you're probably good. |
| 20:50 |
SpeeR |
it disappears from the list every few seconds |
| 20:50 |
SpeeR |
then comes back |
| 20:50 |
JoeJulian |
If it disappears, that's a bug. |
| 20:50 |
JoeJulian |
wait |
| 20:50 |
|
theron joined #gluster |
| 20:50 |
JoeJulian |
the "info split-brain" list or just the "info" list? |
| 20:51 |
SpeeR |
just info |
| 20:51 |
JoeJulian |
Oh. That probably just means it's being actively changed. |
| 20:51 |
SpeeR |
ahh ok, so I should rely more on the info split-brain? |
| 20:52 |
SpeeR |
ohh yeh, I see it's timestamped |
| 20:53 |
SpeeR |
ok, that makes a lot more sense... so a file that is active can show up there and disappear from there |
| 20:53 |
_dist |
JoeJulian: were you not able to duplicate it? |
| 20:56 |
fraggeln |
semiosis: http://fpaste.org/113251/40372975/ <-- some "performance" messures |
| 20:56 |
glusterbot |
Title: #113251 Fedora Project Pastebin (at fpaste.org) |
| 20:56 |
_dist |
If you find that's the case, I don't mind setup a new cluster that has the issue and giving you VPN access to |
| 21:02 |
semiosis |
fraggeln: can you ditch the raid altogether and try just plain old disks for bricks? |
| 21:03 |
semiosis |
you already have gluster replica 3, why bother with raid?! |
| 21:03 |
semiosis |
seems like it's just going to slow you down |
| 21:03 |
fraggeln |
semiosis: its hw-raid, should not be a problem. |
| 21:03 |
fraggeln |
and, there is no io-wait on the servers |
| 21:03 |
semiosis |
worth a try |
| 21:04 |
semiosis |
ok ok |
| 21:04 |
semiosis |
suit yourself |
| 21:04 |
* semiosis |
is biased against raid |
| 21:04 |
fraggeln |
sorry, I was wrong, 0.2 wa, |
| 21:04 |
semiosis |
a ha! |
| 21:05 |
semiosis |
i mean, how could it *not* get in the way? :P |
| 21:05 |
fraggeln |
bite me ;) |
| 21:06 |
fraggeln |
not sure if that raidcontroller support jbod tho. |
| 21:07 |
_dist |
semiosis: implemented correctly raid could actually speed things up. Personally I don't use HW raid but SW, however, I'd also rather replace a disk in my raid set than a brick in gluster as well. |
| 21:07 |
fraggeln |
real6m45.909s <-- that was the time that it took to do a copy of the testworkload to local disks on one of the clients |
| 21:07 |
_dist |
but, truthfully I never considered using multiple bricks on the same server as a type of sw raid |
| 21:07 |
fraggeln |
_dist: i use raid10 atm, 6 disks |
| 21:07 |
semiosis |
i like knowing if i have to get data off a disk it's as easy as mounting it |
| 21:08 |
_dist |
yeap that's fair, especially with HW raid you can run into scenarios where md can't mount it |
| 21:08 |
fraggeln |
_dist: thats why you have replicas, and a good backup-strategy |
| 21:08 |
semiosis |
+1 |
| 21:08 |
fraggeln |
imho raid != backup. |
| 21:09 |
fraggeln |
ahh, fuck, look at the time, I need to get my beautysleep. |
| 21:09 |
_dist |
fraggeln: I totally agree, raid and gluster replica is just HA, not a backup. If you're using raid you just need to understand that your entire replica volume will be as slow as your slowest brick (raid set) |
| 21:10 |
_dist |
(assuming there is no network bottleneck of course) |
| 21:10 |
fraggeln |
_dist: yea, but disk-speed is not really the issue here, I think. |
| 21:10 |
|
sjm joined #gluster |
| 21:12 |
_dist |
I'd expect a raid10 of 6 would give you the read speed of 3 disks and the write speed of 2 disks. Your volume would never be faster than that, 1gbe would be a bottleneck, 2gbe probably not |
| 21:12 |
_dist |
(depends on your disks of course) |
| 21:12 |
fraggeln |
_dist: I max out 1gbit when I do a read or write of 1 large file |
| 21:12 |
fraggeln |
my issue is when I try to handle a shitload of small files :) |
| 21:13 |
semiosis |
fraggeln: you dont get any iowait when you have your small file activity storm? |
| 21:13 |
_dist |
I'd test locally first, what is a shitload ? :) |
| 21:14 |
fraggeln |
_dist: 40k small files packed into 8gb |
| 21:14 |
fraggeln |
in test-folder |
| 21:14 |
fraggeln |
but, in real world, when/if we go live, it will be like 900k files |
| 21:15 |
_dist |
Well, replica bricks will help that, what kind of test are you doing, how are you mounting the volume? |
| 21:15 |
fraggeln |
usingthe glusterfs client |
| 21:15 |
semiosis |
@options |
| 21:15 |
glusterbot |
semiosis: See config options and their defaults with 'gluster volume set help' and see also this page about undocumented options: http://goo.gl/mIAe4E |
| 21:17 |
semiosis |
idk how but there might be some way to disable replication checking on the client |
| 21:17 |
semiosis |
leave it up to the self heal daemon, hopefully |
| 21:17 |
semiosis |
that should help quite a bit |
| 21:17 |
fraggeln |
sounds risky ;) |
| 21:17 |
fraggeln |
but, its worth a try. |
| 21:17 |
semiosis |
for your read-heavy workload, shouldnt be too risky |
| 21:17 |
fraggeln |
since I will not have that many writes |
| 21:17 |
semiosis |
oh btw, did you use noatime,nodiratime on your brick mounts? |
| 21:17 |
semiosis |
that's important too |
| 21:18 |
fraggeln |
semiosis: I thought that noatime didnt really have any affect on xfs |
| 21:18 |
_dist |
fraggeln: do your tests represent what's going to be going on in prod? |
| 21:18 |
semiosis |
hmm, might not |
| 21:18 |
semiosis |
idk |
| 21:18 |
fraggeln |
_dist: yes, mostly reads. |
| 21:19 |
fraggeln |
when someone is crawling our site, it goes through all links, and then it needs to resize files that are not in our cache, and that hurts :/ |
| 21:19 |
fraggeln |
and for some reason, people tend to do that alot :( |
| 21:20 |
|
julim joined #gluster |
| 21:21 |
fraggeln |
and the cms creates a metadatafile next to each picture, with worthless info, and that metadata-file needs to be read eveytime the picture needs to be scaled. |
| 21:21 |
* fraggeln |
dont like Escenic..... |
| 21:23 |
_dist |
fraggeln: This is older, but it may help |
| 21:23 |
_dist |
http://joejulian.name/blog/optimizing-web-performance-with-glusterfs/ |
| 21:23 |
glusterbot |
Title: Optimizing web performance with GlusterFS (at joejulian.name) |
| 21:24 |
fraggeln |
_dist: I did read that one yesterday. |
| 21:24 |
fraggeln |
doesnt really tell how to optimize glusterfs, just how to prevent using it, with the help of varnish,apc,memcached and stuff like that. |
| 21:24 |
_dist |
I think the reverse proxy is a good idea in your case? |
| 21:25 |
JoeJulian |
fraggeln: did you see the "dht misses are expensive" page? |
| 21:25 |
JoeJulian |
Might have useful info too. |
| 21:25 |
fraggeln |
JoeJulian: didnt read that one, ill have a look. |
| 21:25 |
JoeJulian |
@lucky dht misses are expensive |
| 21:25 |
glusterbot |
JoeJulian: http://joejulian.name/blog/dht-misses-are-expensive/ |
| 21:26 |
fraggeln |
_dist: we have enough caching ;) |
| 21:26 |
fraggeln |
we are swedens biggest newspaper online ;) |
| 21:26 |
JoeJulian |
nice |
| 21:27 |
JoeJulian |
I've been wondering. |
| 21:27 |
fraggeln |
Wondering what? |
| 21:28 |
_dist |
JoeJulian: would it help if I gave you a test cluster with the vm heal issue to work with? I could probably set that up in an hour or so, (not today though) |
| 21:28 |
JoeJulian |
fraggeln: wondering who you were. |
| 21:29 |
fraggeln |
oh. |
| 21:29 |
JoeJulian |
_dist: sure |
| 21:29 |
fraggeln |
operationsmanager at Aftonbladet |
| 21:29 |
fraggeln |
thats me |
| 21:29 |
_dist |
ok, I'll see if I can have that setup before the end of the week and get you creds to get in :) |
| 21:31 |
semiosis |
"The king stopped Aftonbladet from being printed and banned it. This was answered by starting the new newspaper "Det andra Aftonbladet" (The second Aftonbladet), which was subsequently banned, followed by new versions named in similar fashion until the newspaper had been renamed 26 times, after which it was allowed by the king." |
| 21:31 |
semiosis |
from wikipedia |
| 21:31 |
semiosis |
that's awesome |
| 21:32 |
fraggeln |
;) |
| 21:32 |
JoeJulian |
lol |
| 21:33 |
_dist |
alright, I'm outta here :) ttyl |
| 21:33 |
JoeJulian |
o/ |
| 21:33 |
JoeJulian |
fraggeln: Ok, so what is the goal that's not being met? Let's try phrasing it into something quantifiable. |
| 21:34 |
JoeJulian |
because, imho, "small files" and "too slow" are not. |
| 21:37 |
|
_polto_ joined #gluster |
| 21:37 |
|
_polto_ joined #gluster |
| 21:53 |
|
_polto_ joined #gluster |
| 21:53 |
|
_polto_ joined #gluster |
| 22:03 |
|
RicardoSSP joined #gluster |
| 22:03 |
|
RicardoSSP joined #gluster |
| 22:12 |
JoeJulian |
fraggeln: I hope you didn't read that as adversarial. I just ask so I have some idea of a target to help you shoot for. |
| 22:20 |
|
jcsp1 joined #gluster |
| 22:37 |
|
sjm left #gluster |
| 22:57 |
|
tdasilva joined #gluster |
| 23:03 |
|
theron joined #gluster |
| 23:06 |
|
primechuck joined #gluster |
| 23:17 |
|
lpabon joined #gluster |
| 23:19 |
|
lpabon joined #gluster |
| 23:22 |
|
_polto_ joined #gluster |
| 23:22 |
|
_polto_ joined #gluster |
| 23:27 |
|
gildub joined #gluster |
| 23:32 |
|
Ark joined #gluster |
| 23:37 |
|
sjm joined #gluster |
| 23:54 |
|
theron joined #gluster |
